Random thoughts

Home row mod with kmonad and systemd

Fri, 08 Mar 2024 00:00:00 +0000

Here are a few notes and explanations about how I use kmonad under linux to apply home row modifier keys. For a fantastic introduction and configuration guide about home row modifiiers, read this first: A guide to home row mods

I’ll base my kmonad configuration files on what’s written in this aforementioned guide.

If you first want to read about kmonad and what it can do (spoiler: a lot of things I thought only possible with firmware), the official project is hosted there: https://github.com/kmonad/kmonad

Making it work with hotplug

The thing that I find missing, is how to have this working for devices that you hotplug, may it be USB or Bluetooth. This is where both systemd and udev come in handy.

udev is used for device identification and reacting to changes. systemd and udev can work together to manage the associated kmonad process and its arguments, as a service. Then it’s the kmonad process that will do the work of making the home row modifiers magic.

Device and udev rule creation

Let’s start with the device and its file. We’ll need a way to identify it with the udevadm tool and let kmonad know which device file to use. If you know the input event device file, you can use something like that to help you build a udev rule:

udevadm info --attribute-walk /dev/input/by-id/usb-SONiX_USB_DEVICE-event-kbd

For a bluetooth device, you can identify the device by its Bluetooth address as displayed by bluetoothctl. But you need to convert it to lowercase if you want to find it in udev’s tree.

BT_ADDRESS="00:1F:20:76:41:30"
BT_ADDRESS_LC=$(echo ${BT_ADDRESS,,}) # bash lowercase
udevadm info --tree | grep -C 20 $BT_ADDRESS_LC

You should try to find a device from the input subsystem and its event sub-device. Once you have it, use the same kind of udevadm command as for the USB keyboard. For my USB keyboard, I have a device alias (a symbolic link) that is stable in /dev/input/by-id/, so I’ll use that in the kmonad config. For the Bluetooth device, I’ll use the udev rule SYMLINK field to create a stable device name in /dev.

For udev rules, here is an exemple file to put in /etc/udev/rules.d/60-keyboard-homerowmod.rules:

SUBSYSTEMS=="input", TAG="systemd", ATTRS{name}=="SONiX USB DEVICE Keyboard", ENV{SYSTEMD_WANTS}+="kmonad@rk61.service", ENV{ID_MODEL}="Royal Kludge RK61 USB keyboard"

Or, for my bluetooth variant, also creating an alias to find it easily later:

SUBSYSTEM=="input", TAG="systemd", ATTRS{name}=="RK-Bluetooth keyboard", ENV{SYSTEMD_WANTS}+="kmonad@rk61bt.service", SYMLINK+="rk61bt"

As you might have seen in the rules, there is a way to make udev and systemd work together.

systemd to the rescue

For the systemd service unit, here is a possible solution using templating, defined as a /etc/systemd/system/kmonad@.service file:

[Unit]
Description=kmonad advanced keyboard daemon
StopWhenUnneeded=true

[Service]
Restart=on-abort
RestartSec=3
ExecStart=-/usr/local/bin/kmonad %E/kmonad/%i.kbd
Nice=-20

[Install]
WantedBy=default.target

The %i part is the name of your configuration file and the instance name of this template.

kmonad configuration

For each instance of the service, as defined in your udev rules, you’ll need a kmonad configuration file. They should be at the path defined in the systemd service. For instance: /etc/kmonad/rk61bt.kbd

Inside your configuration files, you need to use the same stable device files as the ones created by udev.

For my case, here are the two stable devices files:

/dev/rk61bt for the Bluetooth device, as requested in the udev rule as a SYMLINK
/dev/input/by-id/usb-SONiX_USB_DEVICE-event-kbd for the USB one

And here is a snippet of the lines for the two inputs in my two kbd files:

input  (device-file "/dev/rk61bt")
input  (device-file "/dev/input/by-id/usb-SONiX_USB_DEVICE-event-kbd")

With all this, the kmonad process is started when the device is plugged and stopped when it disappears.

Music everywhere

Wed, 03 Jan 2024 00:00:00 +0000

Here is an article about how I play music around my different computers or devices.

Collection

I still have a music collection coming from various vinyl and compact discs that I started converting to MP3s starting around 1999 when I was offered a Rio PMP300. This device had so few memory space (32MB) that I needed to convert with bitrates as low as 64kbps. In those days it was really bad quality-wise. With the LAME Project open-source encoding quality made good progress. And I then often used the VBR standard quality of this encoder.

Then Ogg Vorbis appeared and I heavily used this format. It had good quality but hardware support was hard to find on portable devices.

These days I tend to use FLAC when I want to keep quality at its premium but still often use LAME with VBR or 320kbps CBR when I don’t want to use too much space.

One thing is for sure, my dad was pretty wrong when he told me in 1999: “MP3 will be replaced in a few years”. MP3 is still here in the 2020s and the quality is still good enough for me.

All my music is in a shared folder available to any family user of my Nextcloud instance. It makes it really easy to share with anyone or access it from anywhere. I can also sync a few albums easily on my various devices to listen later offline.

To keep it clean and well tagged, I use beets. On the Nextcloud server, I use the CLI with a user that can have read and write access to the actual files on the filesystem managed by Nextcloud. So, if anything change, an occ groupfolders:scan -- <group ID> is required to keep both apps (NC and beets) in sync.

Let’s start with one main use case.

Living room

I have a pair of studio monitors in my living room where I want to listen to my music collection. Those monitors are connected to a USB soundcard connected to my Raspberry Pi 4 (2GB) that is the Nextcloud server (and a lot more, it may be the server sending you this page).

As a long time MPD user, this was the obvious choice to play from the music collection. It’s very old and stable, it’s using very little resources, it has an API and can be accessed over the network or a local socket. Another strong point: it has an ecosystem of various clients that range from CLI tools to dedicated apps or even web front ends.

As we still use Android devices (I’d like to avoid that, Pinephone, I’m looking at you for the future) the obvious app is: MALP.

https://gitlab.com/gateship-one/malp

While on the local home Wi-Fi, we can use this as a remote that is really light and fast. Almost no latency.

If I’m using a laptop in the room, I tend to use ncmpcpp over Wi-Fi to control MPD. It’s simple, light and when you learn (or set) the keyboard bindings, it’s easy to use. I was a Sonata user before, but I try to use keyboard for everything it made sense to switch to ncmpcpp.

Streaming or remote listening

When out of range of my Wi-Fi or when I didn’t think about syncing my music before leaving, Nextcloud is still there to the rescue. I use the Music app for Nextcloud. On a desktop computer it does the job to browse and listen to anything in the library. But the less known feature is that it provides APIs to serve the music over the net: Ampache and Subsonic compatible.

On my Android devices, I use the fantastic subtracks app that is Subsonic-compatible. It’s simple and have, to my eyes, a great UI.

What’s missing

All my music needs are fulfilled with all those tools and services in place. All? Really? No, there’s still a Podcast centralized sync service missing. I tested various things and did not find one that fulfilled the way I want it to work. At this time, I use my RSS aggregator to store which episodes I listened to. But it still requires a lot of manual downloading, app switching or web browsing.

If you have any ideas, send me a mail or contact me on Mastodon.

Adding an alternative name to a Let's Encrypt certificate

Wed, 23 Aug 2023 00:00:00 +0000

Here is a short guide to add a new subject alternative name to an existing certificate managed by certbot.

From time to time, I need to add or remove a domain name to my reverse-proxy. After updating the DNS zone of my domain, I need to let the certbot utility update its certificates.

First, to view the certbot managed certificates, type this (with sudo or as root if needed):

certbot certificates

It will tell you about the domains managed, the first one being used as the certificate name.

Then a dry-run of the update with the actual update:

certbot certonly --cert-name <certificate name> --expand --dry-run \
    --domains <main domain>,<other domain>,<you new domain>
certbot certonly --cert-name <certificate name> --expand \
    --domains <main domain>,<other domain>,<you new domain>

You’re new certificate should be ready. I tend to check it manually with:

openssl x509 -noout -text -in <path to cert>

And if it’s all OK and depending on your software using your certificates, reload your app so that it uses it.

How I use my IEMs under Linux

Sun, 05 Mar 2023 00:00:00 +0000

Here is an overview of my software audio setup. The main goal here is to have an equalizer than can compensate for the tuning of my in-ear monitors. I bought cheap Chinese IEMs (for In-Ear Monitors) for around 15€. I knew that the tuning was bass heavy with what is called a V-shaped frequency response graph. I like neutral frequency response. Here are a few notes on what I do to fix that.

First, let’s start with the basics. I don’t use any fancy sound cards, only what’s on the device I use. It is all well supported and working with Alsa. Then, depending on my workstation or distribution, I use either PulseAudio or PipeWire.

Then, to have my preferred audio frequency response, I use the fantastic AutoEq project. Go there for more information or documentation on usage. To put it simply, I get the settings for my device, use them in Easy Effects or its former application name: PulseEffects.

Now enjoy your music!

Join the dark side… of Gnome 42

Wed, 30 Mar 2022 00:00:00 +0000

As of today, my main computer has a few pieces of software from Gnome 42. I like to use a dark theme these days. I used to set my preference in the Gnome Tweaks application. Now, I believe the setting can be changed in the Settings.

But, as the package is not there yet, I tried to do it with the command line. Here we go:

gsettings get org.gnome.desktop.interface color-scheme
gsettings set org.gnome.desktop.interface color-scheme prefer-dark

You can find the definition of this setting here on Github.

The allowed values are:

default
prefer-dark
prefer-light

Raspberry PI OS 64-bit announced

Wed, 02 Feb 2022 00:00:00 +0000

Good news everyone! At last, the official 64-bit operating system for Raspberry Pi is out (kernel and user-space). You can check the announcement here:

https://www.raspberrypi.com/news/raspberry-pi-os-64-bit/

As an happy user and administrator of a Raspberry Pi 4 Model B, I was waiting for this since I started encountering errors because of pieces of software dropping support for ARM 32-bit architectures.

I’ll wait for things to settle a bit and enjoy a more stable OS. Then, I’ll switch and at the same time try to ditch the Micro SD card and go fully on a SATA/USB3 disk. See you next time for details.

Let's backup

Mon, 27 Dec 2021 00:00:00 +0000

It’s been a long time without an article, let’s try to fix that with the subject of today: backups.

What could be a good backup strategy?

One of the rules I try to follow and kind of makes sense to me is the 3 2 1 rule :

3 copies
2 different medias
1 copy off site

Let’s not be too rigid with those expectations. For me, the major difficulty when you start to have a lot of content and don’t want to spend too much money, is the off site copy. Three years ago, my off site copy was a small hard disk that was synced almost every month and was stored at a place of a family relative. Clearly, it worked one or two times and then was not done on time and there, let’s face it, it was a big failure: I did not have an actual copy, off site, with the data of the last month.

Off site copy solution

In order to avoid this pain of bringing back and forth the disk, I needed a kind of replication software that just worked. Here comes Syncthing, a wonderful small portable peace of free software. You need to find a colleague, friend or relative that has a fiber or xDSL connection at his home, a relatively always on host with disk space (a NAS is often the best thing). You help them install Syncthing on it, you pair this and it’s done.

Depending on the confidence you have on the security of the off site location, you might want to configure Syncthing folders on local and remote as, Send only and Receive only, respectively. Further details in the documentation there: Folder Types. You can even go further and ask Syncthing to encrypt the data on remote copies for you (but it’s still in beta as of the date of this article).

Inside the backups

When I hear about backups, sometimes, people mean copies. To me, a backup solution should be more than just a copy. I want to prevent the deletion incident and be able to go back in time. As soon as this expectation comes into place, it gets messy. I’ve been using a few technical solutions to keep historical snapshots of my data: rsnapshot, manual rsync with complicated rules, duplicity (and its DejaDup cousin) or even Bacula.

But, for simplicity and my usage, restic is the one I chose. I won’t dig too much in the features of this tool, I know it’s not the best performing solution in town, but I like it and it fills my requirements. I’ve never met (yet) a situation where I couldn’t find an old copy of the files I was looking for.

Tying it all together

I use only one Restic repository in a directory on an NFS share that is also a source Syncthing share. For this NFS share to be accessible on every hosts, I use two Systemd units.

Here is the backup.mount unit:

[Unit]
Description=Backup shared file system mount point

[Mount]
Where=/mnt/local-dir
What=192.168.0.5:/my-syncthing-folder
Type=nfs

Here is the associated backup.automount unit:

[Unit]
Description=Automounting for backups NFS

[Install]
WantedBy=multi-user.target

[Automount]
TimeoutIdleSec=10min

These 2 units are located in the /etc/systemd/system directory. It allows the restic repository to be available on every hosts and be unmounted automatically when not needed.

System backup job

To backup a standard host, here is the systemd unit (backup.service) to launch the task:

[Unit]
Description=Backup with restic
Wants=backup.mount

[Service]
Environment="RESTIC_REPOSITORY=<path to your restic repo>" \
            "RESTIC_PASSWORD_FILE=/etc/restic-pass"
ExecStart=/usr/bin/restic backup \
  --cleanup-cache --cache-dir="%C/restic" \
  --exclude='/home/**/.cache' \
  --exclude='/home/**/.local/share/Trash' \
  --exclude='/home/**/.vagrant.d' \
  --exclude='/home/**/cache' \
  --exclude='/home/*/Nextcloud' \
  --exclude='/home/*/Videos' \
  --exclude='/home/*/VirtualBox*' \
  /etc /home
SuccessExitStatus=3

And then a timer (backup.timer) to schedule this weekly:

[Unit]
Description=Launch a backup weekly

[Install]
WantedBy=timers.target

[Timer]
OnCalendar=Fri *-*-1..31 23:34:12
Persistent=true
RandomizedDelaySec=180

Node backup elements

I’ve got a main backup job scheduled on a host that takes care of backuping the most complicated system. I use a personal script that includes snapshots or dumps (BTRFS or MariaDB) and that is also in charge of putting the services in maintenance mode if needed. It also do maintenance tasks on the Restic repository: checks and enforcing my retention policy.

Here is the way I enforce my retention policy:

restic forget --prune --quiet \
  --keep-hourly=0 \
  --keep-daily=4 \
  --keep-weekly=3 \
  --keep-monthly=3 \
  --keep-yearly=1

Wrap-up

Here are a few things I use to have backups I can rely on. This is a bit too complicated, it doesn’t follow a backup tip I like: make it very simple so that restore is as painless as possible. To be quick in case of emergency, I like to have a simple text file with the required commands to be able to restore as fast as possible.

Next step: try to restore the data and all components with Ansible. But that’s a story for another day…

Journald all your logs

Fri, 05 Mar 2021 14:41:00 +0000

Let’s write about logs today. All those files that are scattered everywhere on your systems, by processes that try to dump their actions in files.

Before

From what I’ve seen and experienced, on a GNU/Linux or BSD system, there is a common path where the system logs should go: /var/log. Sometimes you’ll read a recommendation to have a dedicated mount point for this path, or its parent /var. It can be useful to avoid a denial of service if it gets filled up. Spoiler: you might not need it after all…

There often is a daemon that is supposed to be the main entry point to write logs. This is the component that is reading all the data sent to him through /dev/log or the syslog system call. You could also use the logger user command.

Depending on the facility, priority level or other rules, the messages are written in a specific log file in /var/log. This is why you’ll often find a messages, kern.log, daemon.log and many other files in you log directory.

When another daemon is not configured to use the main syslog, it can write directly in this directory or its own subdirectory. That’s what gdm is doing on a Buster Debian GNU/Linux distribution.

As you can see by yourself, after a few days, a lot of log files get new little friends with a number at the end of their name or even compressed with gzip. This is managed by the log rotation. It can be a cron job calling another dedicated utility, it can be managed by the syslog daemon or directly by the process writing its log files.

The problem

As I’ve tried to show you in the previous section, this can be complicated and there are a lot of actors in all this log files mess. As often said, this can be interpreted as the “UNIX way”: each program does one thing and does it well; furthermore, everything is managed as text files and it can be really handy when you just have to view or grep those files.

When your system is getting old or when your applications handle real traffic, this can get out of control. That’s why the log messages journey should be evaluated long before your file system is full.

In my humble sysadmin life, I often have to find the reasons why it’s getting full. It’s a kind of cat and mouse game where you have to start from the log file and find the journey of the log messages through your system. It’s fun a few times, then it gets really annoying and you don’t want to do this anymore. Here are a list of commands that can help you:

ls -altrh /var/log
du -x -h --max-depth=1 /var/log
lsof +D /var/log
lnav /var/log/<your big log file>

With those first three commands you should know who is writing files, what are the latest ones and the biggest space consumers. Then, if there is a level of indirection, you could be searching in the syslog daemon or log rotation configuration… Your goal is to fix this once for all, you just can’t remove the files and leave, it will bite you once more in the future.

A few words about lnav before going on. I just discovered this command line utility (thanks Sébastien by the way). If you just look at log files once in a month or less frequently, a pager like less is enough. But if you have to dig in log files every week, take a look at this great tool that can save you a lot of time: http://lnav.org/

`Journald` to the rescue

Yes, we’re coming to a potential friend: journald, which is part of systemd (I love systemd ;-). For all the applications that know how to write their logs to a pipe or syslog it can be used as the only log daemon.

Let’s get straight to a specific thing: instead of simple text files, journald writes logs in a binary format that might be cleared on reboot. You need to use journalctl to read the logs and it can slow you down at first. On Debian, you have to create a directory to get persistent log data retention; as specified in the man page, you have to run this as root:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

So, what do you get if you lose the text file log format? A lot. Let’s dive in with some of the configuration options.

SystemMaxUse
SystemKeepFree

With those settings you can tame the file space usage of all the logs. It can let you specify how much space you want it to use at most and how much it should leave free. Even the default values are great!

ForwardToSyslog

If you want, you can still forward logs to a traditional syslog daemon with this setting. Otherwise, the only way is to use a program aware of the format used by journald. It might be your only option if you want to forward logs to another host that’s using the RFC 5424 Syslog Protocol.

You also get compression (by default), forward secure sealing and a lot of features. For details dive in the man pages: journald.conf(5) , systemd-journald.service(8) and journalctl(1).

Next, let’s see a few commands that can save you a lot of time.

journalctl --disk-usage

You’ll know how much you’re using with this one.

journalctl --vacuum-size=1G

It lets you specify that it should remove your oldest log messages until the disk space they use falls below 1GB. You can also use a time constraint with --vacuum-time=.

What I use

Apache HTTP Server

By default, Apache HTTP server uses its own logfile management. As I like to have a minimal number of actors writing logs, here is my recipe for journald usage with Apache.

Because I use different virtual hosts, I don’t want to repeat myself and only use one setting in the main configuration file: /etc/apache2/apache2.conf:

ErrorLog syslog:daemon

And in the /etc/apache2/conf-available/, I add a file named other-vhosts-access-log.conf with this content:

# Define an access log for VirtualHosts that don't define their own logfile
CustomLog "|/usr/bin/logger --tag apache2 --priority daemon.info" vhost_combined

Then you can enable this configuration file and restart you server:

a2enconf other-vhosts-access-log
apache2ctl -t && apache2ctl graceful

Next thing you know, no apache2 process should be using any file descriptors in /var/log.

Docker and containers

I did not write about this topic yet on purpose because your containers should not write log files. They should use their standard and error outputs and let the container management daemon forward everything with all the good tags.

For Docker, here is a link to the documentation on the specific log driver: https://docs.docker.com/config/containers/logging/journald/

TL;DR here is a part of the content of my /etc/docker/daemon.json file:

"log-driver": "journald",
"log-opts": {
  "tag": "{{.DaemonName}}/{{.Name}}"
},
"userns-remap": "default"

(I also let the user namespace isolation setting, as you already should have this enabled.)

PHP-FPM or any PHP usage

If you use PHP-FPM (I guess you should switch to it if you’re using mod_php) the option to define in /etc/php/7.3/fpm/php-fpm.conf is:

error_log = syslog

For the PHP CLI or any other PHP entry point, the documentation for the error logging is in the manual. Use syslog as value for the error_log directive and you’re good to go.

Nextcloud

Nextcloud is a PHP app but it also has its own log management by default. Fortunately, there is a setting that can help you forward the log messages to journald. Here is what I use:

 'log_type' => 'syslog',
 'loglevel' => 2,

The full logging documentation is here in the admin manual.

By default, it uses the Nextcloud tag. If you want to see the logs: journalctl -t Nextcloud.

That’s all for today, but maybe I’ll revisit this post to add more apps.

Why I'm using vim

Sat, 23 Jan 2021 15:30:00 +0100

Sorry, this is going to be more technical than the previous post: feel free to skip this post if you’re not into geeky details.

What’s vim?

Let’s get in the context: a program to edit files. But a program that have really old roots and working well when you don’t want to touch your mouse. It even works in the console as the interface is all text.

The project home page is here: https://www.vim.org/

There are a lot of text editors, I’ve used a few: Emacs, VSCode, neovim, IntelliJ, Eclipse, nano or even Notepad++. But vim is the one that’s working great for me.

Why I use and love it

When I discovered Linux, somewhere around 1997, I didn’t manage to have a graphical environement, X11 was so hard to configure. The only text editor that my mother found in the default console was vim. I was really annoyed when she explained to me what I actually had to type to edit my X11 configuration file. And I couldn’t event use the mouse. Don’t even think about quitting vi!

Fast forward a few years and I enjoyed using other GNU/Linux distributions. This time I tried to understand a few commands and it saved me a bit of time while trying to fix things. When I started working, every Unix I worked on had this editor. So I had to learn to use it. The documentation is good, you can start with vimtutor and there is the :help command inside.

In 2015, I tried to improve my typing and I switched my keyboard layout. It was a great time to revisit my usage. I read a lot of articles and documentation to improve my skills in this editor. I discovered the blockwise visual mode and insert mode completion. I never supposed this kind of features before.

Settings

Default settings can change, it can vary depending on your Linux distribution. In my case, in almost all the places I use this editor, it’s a Debian GNU/Linux context. Even between versions you can have surprises such as the 'mouse' setting.

Here are a few settings that I come to appreciate on my local machine, in the ~/.vim/vimrc file:

set list listchars=nbsp:⎵,eol:↲
set number
set smartindent
set textwidth=80

:filetype plugin indent on

As I like to use unbreakable spaces, very handy in French before punctuation, I want to see them. And nothing fancy, that’s almost all.

For format specific options, I use the ftplugin-overrule capability. As a mutt user, I like to set a few options, but just for this usage. That’s why I have a ~/.vim/after/ftplugin/mail.vim file with this content:

set linebreak
set formatoptions+=w
set number
set spell
set spelllang=fr,en

I don’t want to go into details with all those settings, if you want more info, read the manual: :help in vim.

I don’t use any additional plugins and I try to learn and, at least, know what features it can do. As a production/DevOps engineer, under any conditions, in the middle of the night on an emergency for example, I know I’ll have to use it. I don’t want to rely on a personal macro or setting that I won’t find on one of the hundreds of environments (I have to share with my colleagues) that I couldn’t tweak or prepare.

Beyond the editor

As I’m a long time (almost 20 years) vim user, a lot of the key bindings and logic feel natural to me. That’s why sometimes I’m looking for relatives in the free software world. Just to list a few of them:

newsboat as an RSS reader
mutt as an email reader where vim is a first time citizen to compose your messages
tridactyl to browse the web with your keyboard in Firefox with vim-style key bindings

And to finish this post, I’ll give a huge thanks to Bram and all the project contributors. Also thank you everyone sharing your tips and tricks anywhere. Sometimes I like to visit the VIM Community on Reddit and find really interesting points of view and new commands.

Best Book of 2020

Sat, 02 Jan 2021 00:00:00 +0000

Since I was offered an Amazon gift card when quitting my previous job, I have a Kindle e-book reader (I would still not recommend this device anyway). And since then I read more books than ever. I love the way you focus on only reading on this kind of device. It’s even easier for me (french speaking) to have a dictionary included when I’m reading in English.

But let’s focus on what I read in 2020 and my best book: The Abominable by Dan Simmons. Why? The story of this book is fascinating. No, not only the story in this book, the story of how this book came to be. I’ve never been a fan of prefaces. But this one is the best I’ve ever read.

The plot is full of details on mountain climbing expeditions in places where no human life should be. As it’s often said, men or women are not adapted to this kind of extreme altitude.

My piece of advice: read this book. I loved a lot of Dan Simmons books, it might not be the best but it’s still an easy recommendation.

What I want with this site

Sun, 27 Dec 2020 00:00:00 +0000

These are my random thoughts about what I want with this site:

be able to write in english and french
write on anything, no need to think about sections or categories
people can subscribe with their favorite feed reader
no bullshit!