The Linux Journal
(From https://sematext.com/blog/journald-logging-tutorial/)
journald is the part of systemd that deals with logging. systemd, at its core, is in charge of managing services: it starts them up and keeps them alive.
All services and systemd itself need to log: “ssh started” or “user root logged in”, they might say. That's where journald comes in: to capture these logs, record them, make them easy to find, and remove them when they pass a certain
Most Linux distros use journald for system logging by default. Most applications running as a service will also log to the journal.
So how do you make use of these logs to:
- find the error or debug message that you're looking for?
- make sure logs don't fill your disk?
- centralize journals so you don't have to ssh to each box?
Managing journald
Learn about the current status of journald by typing:
systemctl stop systemd-journald.service
You can stop and start journald:
sudo systemctl stop systemd-journald.service
and
sudo systemctl stop systemd-journald.service
Still, when you try to stop it you get the following warning:
Stopping 'systemd-journald.service', but its triggering units are still active: systemd-journald.socket, systemd-journald-dev-log.socket
Configuring journald
To tweak how journald behaves, you'll edit /etc/systemd/journald.conf and then reload the journal service:
systemctl reload systemd-journald.service
Storage*
The Storage option controls whether the journal is stored in memory (under /run/log/journal) or on disk (under /var/log/journal). Setting Storage=volatile will store the journal in memory, while Storage=persistent will store it on disk. Most distributions have it set to auto, which means it will store the journal on disk if /var/log/journal exists, otherwise it will be stored in memory.
Once you've decided where to store the journal, you may want to set some limits. For example, SystemMaxUse=4G will limit /var/log/journal to about 4GB. Similarly, SystemKeepFree=10G will try to keep 10GB of disk space free. If you choose to keep the journal in memory, the equivalent options are RuntimeMaxUse and RuntimeKeepFree.
You can check the current disk usage of the journal with journalctl by typing journalctl --disk-usage. If you need to, you can clean it up on demand by typing journalctl --vacuum-size=4GB (i.e. to reduce it to 4GB).
Bursts
or Rate
Also by default, journald will drop all log messages from a service if it passes certain limits. These limits can be configured via RateLimitBurst and RateLimitIntervalSec, which default to 10000 and 30s respectively. Actual values will depend on the available free space. For example, if you have more than 64GB of free disk space, the multiplier will be 6. Meaning it will drop logs from a service after 60K messages sent in 30 seconds.
The rate limit defaults are sensible, unless you have a specific service that's generating lots of logs (e.g. a web server). In that case, it might be better to tweak or reduce LogRateLimitBurst and LogRateLimitIntervalSec in that application's service definition.
The journald.conf Configuration File(s)*
These files configure various parameters of the systemd journal service, systemd-journald.service(8). See systemd.syntax (7) (man 7 systemd.syntax) for a general description of the syntax.
The systemd-journald instance managing the default namespace is configured by /etc/systemd/journald.conf and associated drop-ins. Instances managing other namespaces read /etc/systemd/journald@NAMESPACE.conf and associated drop-ins with the namespace identifier filled in. This allows each namespace to carry a distinct configuration. See systemd-journald.service(8) (man 8 systemd-journald.service) for details about journal namespaces.