Random thoughts

Yeah, well, that's just, like, your opinion, man.

Jeff Lebowski

Journald all your logs

Let’s write about logs today. All those files that are scattered everywhere on your systems, by processes that try to dump their actions in files.

Before

From what I’ve seen and experienced, on a GNU/Linux or BSD system, there is a common path where the system logs should go: /var/log. Sometimes you’ll read a recommendation to have a dedicated mount point for this path, or its parent /var. It can be useful to avoid a denial of service if it gets filled up. Spoiler: you might not need it after all…

There often is a daemon that is supposed to be the main entry point to write logs. This is the component that is reading all the data sent to him through /dev/log or the syslog system call. You could also use the logger user command.

Depending on the facility, priority level or other rules, the messages are written in a specific log file in /var/log. This is why you’ll often find a messages, kern.log, daemon.log and many other files in you log directory.

When another daemon is not configured to use the main syslog, it can write directly in this directory or its own subdirectory. That’s what gdm is doing on a Buster Debian GNU/Linux distribution.

As you can see by yourself, after a few days, a lot of log files get new little friends with a number at the end of their name or even compressed with gzip. This is managed by the log rotation. It can be a cron job calling another dedicated utility, it can be managed by the syslog daemon or directly by the process writing its log files.

The problem

As I’ve tried to show you in the previous section, this can be complicated and there are a lot of actors in all this log files mess. As often said, this can be interpreted as the “UNIX way”: each program does one thing and does it well; furthermore, everything is managed as text files and it can be really handy when you just have to view or grep those files.

When your system is getting old or when your applications handle real traffic, this can get out of control. That’s why the log messages journey should be evaluated long before your file system is full.

In my humble sysadmin life, I often have to find the reasons why it’s getting full. It’s a kind of cat and mouse game where you have to start from the log file and find the journey of the log messages through your system. It’s fun a few times, then it gets really annoying and you don’t want to do this anymore. Here are a list of commands that can help you:

ls -altrh /var/log
du -x -h --max-depth=1 /var/log
lsof +D /var/log
lnav /var/log/<your big log file>

With those first three commands you should know who is writing files, what are the latest ones and the biggest space consumers. Then, if there is a level of indirection, you could be searching in the syslog daemon or log rotation configuration… Your goal is to fix this once for all, you just can’t remove the files and leave, it will bite you once more in the future.

A few words about lnav before going on. I just discovered this command line utility (thanks Sébastien by the way). If you just look at log files once in a month or less frequently, a pager like less is enough. But if you have to dig in log files every week, take a look at this great tool that can save you a lot of time: http://lnav.org/

Journald to the rescue

Yes, we’re coming to a potential friend: journald, which is part of systemd (I love systemd ;-). For all the applications that know how to write their logs to a pipe or syslog it can be used as the only log daemon.

Let’s get straight to a specific thing: instead of simple text files, journald writes logs in a binary format that might be cleared on reboot. You need to use journalctl to read the logs and it can slow you down at first. On Debian, you have to create a directory to get persistent log data retention; as specified in the man page, you have to run this as root:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

So, what do you get if you lose the text file log format? A lot. Let’s dive in with some of the configuration options.

SystemMaxUse
SystemKeepFree

With those settings you can tame the file space usage of all the logs. It can let you specify how much space you want it to use at most and how much it should leave free. Even the default values are great!

ForwardToSyslog

If you want, you can still forward logs to a traditional syslog daemon with this setting. Otherwise, the only way is to use a program aware of the format used by journald. It might be your only option if you want to forward logs to another host that’s using the RFC 5424 Syslog Protocol.

You also get compression (by default), forward secure sealing and a lot of features. For details dive in the man pages: journald.conf(5) , systemd-journald.service(8) and journalctl(1).

Next, let’s see a few commands that can save you a lot of time.

journalctl --disk-usage

You’ll know how much you’re using with this one.

journalctl --vacuum-size=1G

It lets you specify that it should remove your oldest log messages until the disk space they use falls below 1GB. You can also use a time constraint with --vacuum-time=.

What I use

Apache HTTP Server

By default, Apache HTTP server uses its own logfile management. As I like to have a minimal number of actors writing logs, here is my recipe for journald usage with Apache.

Because I use different virtual hosts, I don’t want to repeat myself and only use one setting in the main configuration file: /etc/apache2/apache2.conf:

ErrorLog syslog:daemon

And in the /etc/apache2/conf-available/, I add a file named other-vhosts-access-log.conf with this content:

# Define an access log for VirtualHosts that don't define their own logfile
CustomLog "|/usr/bin/logger --tag apache2 --priority daemon.info" vhost_combined

Then you can enable this configuration file and restart you server:

a2enconf other-vhosts-access-log
apache2ctl -t && apache2ctl graceful

Next thing you know, no apache2 process should be using any file descriptors in /var/log.

Docker and containers

I did not write about this topic yet on purpose because your containers should not write log files. They should use their standard and error outputs and let the container management daemon forward everything with all the good tags.

For Docker, here is a link to the documentation on the specific log driver: https://docs.docker.com/config/containers/logging/journald/

TL;DR here is a part of the content of my /etc/docker/daemon.json file:

"log-driver": "journald",
"log-opts": {
  "tag": "{{.DaemonName}}/{{.Name}}"
},
"userns-remap": "default"

(I also let the user namespace isolation setting, as you already should have this enabled.)

PHP-FPM or any PHP usage

If you use PHP-FPM (I guess you should switch to it if you’re using mod_php) the option to define in /etc/php/7.3/fpm/php-fpm.conf is:

error_log = syslog

For the PHP CLI or any other PHP entry point, the documentation for the error logging is in the manual. Use syslog as value for the error_log directive and you’re good to go.

Nextcloud

Nextcloud is a PHP app but it also has its own log management by default. Fortunately, there is a setting that can help you forward the log messages to journald. Here is what I use:

 'log_type' => 'syslog',
 'loglevel' => 2,

The full logging documentation is here in the admin manual.

By default, it uses the Nextcloud tag. If you want to see the logs: journalctl -t Nextcloud.

That’s all for today, but maybe I’ll revisit this post to add more apps.