Random thoughts

Yeah, well, that's just, like, your opinion, man.

Jeff Lebowski

Let's backup

It’s been a long time without an article, let’s try to fix that with the subject of today: backups.

What could be a good backup strategy?

One of the rules I try to follow and kind of makes sense to me is the 3 2 1 rule :

Let’s not be too rigid with those expectations. For me, the major difficulty when you start to have a lot of content and don’t want to spend too much money, is the off site copy. Three years ago, my off site copy was a small hard disk that was synced almost every month and was stored at a place of a family relative. Clearly, it worked one or two times and then was not done on time and there, let’s face it, it was a big failure: I did not have an actual copy, off site, with the data of the last month.

Off site copy solution

In order to avoid this pain of bringing back and forth the disk, I needed a kind of replication software that just worked. Here comes Syncthing, a wonderful small portable peace of free software. You need to find a colleague, friend or relative that has a fiber or xDSL connection at his home, a relatively always on host with disk space (a NAS is often the best thing). You help them install Syncthing on it, you pair this and it’s done.

Depending on the confidence you have on the security of the off site location, you might want to configure Syncthing folders on local and remote as, Send only and Receive only, respectively. Further details in the documentation there: Folder Types. You can even go further and ask Syncthing to encrypt the data on remote copies for you (but it’s still in beta as of the date of this article).

Inside the backups

When I hear about backups, sometimes, people mean copies. To me, a backup solution should be more than just a copy. I want to prevent the deletion incident and be able to go back in time. As soon as this expectation comes into place, it gets messy. I’ve been using a few technical solutions to keep historical snapshots of my data: rsnapshot, manual rsync with complicated rules, duplicity (and its DejaDup cousin) or even Bacula.

But, for simplicity and my usage, restic is the one I chose. I won’t dig too much in the features of this tool, I know it’s not the best performing solution in town, but I like it and it fills my requirements. I’ve never met (yet) a situation where I couldn’t find an old copy of the files I was looking for.

Tying it all together

I use only one Restic repository in a directory on an NFS share that is also a source Syncthing share. For this NFS share to be accessible on every hosts, I use two Systemd units.

Here is the backup.mount unit:

[Unit]
Description=Backup shared file system mount point

[Mount]
Where=/mnt/local-dir
What=192.168.0.5:/my-syncthing-folder
Type=nfs

Here is the associated backup.automount unit:

[Unit]
Description=Automounting for backups NFS

[Install]
WantedBy=multi-user.target

[Automount]
TimeoutIdleSec=10min

These 2 units are located in the /etc/systemd/system directory. It allows the restic repository to be available on every hosts and be unmounted automatically when not needed.

System backup job

To backup a standard host, here is the systemd unit (backup.service) to launch the task:

[Unit]
Description=Backup with restic
Wants=backup.mount

[Service]
Environment="RESTIC_REPOSITORY=<path to your restic repo>" \
            "RESTIC_PASSWORD_FILE=/etc/restic-pass"
ExecStart=/usr/bin/restic backup \
  --cleanup-cache --cache-dir="%C/restic" \
  --exclude='/home/**/.cache' \
  --exclude='/home/**/.local/share/Trash' \
  --exclude='/home/**/.vagrant.d' \
  --exclude='/home/**/cache' \
  --exclude='/home/*/Nextcloud' \
  --exclude='/home/*/Videos' \
  --exclude='/home/*/VirtualBox*' \
  /etc /home
SuccessExitStatus=3

And then a timer (backup.timer) to schedule this weekly:

[Unit]
Description=Launch a backup weekly

[Install]
WantedBy=timers.target

[Timer]
OnCalendar=Fri *-*-1..31 23:34:12
Persistent=true
RandomizedDelaySec=180

Node backup elements

I’ve got a main backup job scheduled on a host that takes care of backuping the most complicated system. I use a personal script that includes snapshots or dumps (BTRFS or MariaDB) and that is also in charge of putting the services in maintenance mode if needed. It also do maintenance tasks on the Restic repository: checks and enforcing my retention policy.

Here is the way I enforce my retention policy:

restic forget --prune --quiet \
  --keep-hourly=0 \
  --keep-daily=4 \
  --keep-weekly=3 \
  --keep-monthly=3 \
  --keep-yearly=1

Wrap-up

Here are a few things I use to have backups I can rely on. This is a bit too complicated, it doesn’t follow a backup tip I like: make it very simple so that restore is as painless as possible. To be quick in case of emergency, I like to have a simple text file with the required commands to be able to restore as fast as possible.

Next step: try to restore the data and all components with Ansible. But that’s a story for another day…