Set up backups

A backup on the same server does not help if the server is gone, or if you deleted something by mistake and only notice days later. This guide backs up your data daily to another target and shows how to check that the backup can really be restored.

Covers Debian 12 and 13, Ubuntu 22.04 and 24.04, AlmaLinux 9 and Rocky Linux 9. All commands run as root.

What CitexNode already backs up

We back up Xeon VPS every night as an image of the whole disk and keep 7 days; you can get a point in time back in the Backups tab (see Backups). That is a convenience without a guarantee and covers neither single files nor databases in a consistent state.

The guide below shows how to back up files and databases regularly to a target outside the server as well.

What to back up

You do not need to back up the system itself (/usr, /var/lib/apt, …): a reinstall gives you that back.

Back up databases as a dump, not as files from /var/lib/mysql. Database files copied from a running server are often broken. How to dump is in Set up MariaDB.

The target

You need space outside this VPS, for example:

• a second server or a storage box you can reach via SFTP (SSH),
• S3-compatible storage at a provider of your choice.

The examples below use SFTP with the target backup@backup.example.com. Create an SSH key without a passphrase on the VPS so the timer runs without you, and add it to the target:

ssh-keygen -t ed25519 -f /root/.ssh/backup -N ''
ssh-copy-id -i /root/.ssh/backup.pub backup@backup.example.com

And in /root/.ssh/config:

Host backup.example.com
    User backup
    IdentityFile /root/.ssh/backup

Option A: restic (recommended)

restic encrypts, only stores changes and keeps many versions without using much space.

Install

apt install -y restic                     # Debian and Ubuntu
dnf install -y epel-release && dnf install -y restic   # AlmaLinux and Rocky Linux

Create the repository

Create a password. Without this password the backups cannot be read. Also keep it outside the server, e.g. in your password manager.

openssl rand -base64 32 > /root/.restic-password
chmod 600 /root/.restic-password

Settings in /root/.restic.env:

RESTIC_REPOSITORY=sftp:backup.example.com:/backups/vserver
RESTIC_PASSWORD_FILE=/root/.restic-password

Create the repository once:

set -a; . /root/.restic.env; set +a
restic init

The backup script

/usr/local/bin/backup.sh:

#!/bin/bash
set -euo pipefail
set -a; . /root/.restic.env; set +a

# Dump databases first (remove this line if you have no MariaDB)
mkdir -p /var/backups/mariadb
mariadb-dump --single-transaction --routines --triggers --all-databases | gzip > /var/backups/mariadb/all.sql.gz

# Only list folders that exist on your server, otherwise the run ends with an error
restic backup /etc /var/www /var/backups /root /opt --exclude-caches
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

chmod 700 /usr/local/bin/backup.sh
/usr/local/bin/backup.sh

The first run takes longest; later runs take seconds to minutes.

Option B: rsync

rsync mirrors folders to the target. Simple, but without encryption and without older versions: if you delete something and the backup runs afterwards, it is gone on the target too (with --delete), or old files pile up (without it).

apt install -y rsync   # Debian and Ubuntu
dnf install -y rsync   # AlmaLinux and Rocky Linux

/usr/local/bin/backup.sh with rsync:

#!/bin/bash
set -euo pipefail
mkdir -p /var/backups/mariadb
mariadb-dump --single-transaction --routines --triggers --all-databases | gzip > /var/backups/mariadb/all.sql.gz
# Only list folders that exist on your server, otherwise the run ends with an error
rsync -aR --delete /etc /var/www /var/backups /root /opt backup.example.com:/backups/vserver/

rsync must be installed on the target too.

Daily and automatic: systemd timer

/etc/systemd/system/backup.service:

[Unit]
Description=Daily backup
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

/etc/systemd/system/backup.timer:

[Unit]
Description=Daily backup at 3 am

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

Enable it:

systemctl daemon-reload
systemctl enable --now backup.timer
systemctl list-timers backup.timer

Persistent=true catches up on a missed run if the server was off at 3 am. Prefer cron? See Cron jobs and timers.

Check that it worked

Did the last run finish without errors?

systemctl status backup.service
journalctl -u backup.service -n 50 --no-pager

With restic, list the snapshots and check the repository:

set -a; . /root/.restic.env; set +a
restic snapshots
restic check

Test restore

A backup is only worth something if you can restore it. Do it once now and then now and again, e.g. every month.

With restic, restore the latest snapshot into a test folder:

set -a; . /root/.restic.env; set +a
restic restore latest --target /tmp/restore-test --include /etc/hostname --include /var/backups
ls -l /tmp/restore-test/etc/hostname /tmp/restore-test/var/backups/mariadb/

Check that the database dump is complete. The last line of a finished dump starts with -- Dump completed:

gunzip -t /tmp/restore-test/var/backups/mariadb/all.sql.gz
gunzip -c /tmp/restore-test/var/backups/mariadb/all.sql.gz | tail -n 1
rm -rf /tmp/restore-test

A dump made with --all-databases recreates the databases under their old names and overwrites them. For a real test, import it on a second server or test machine, not on the live one.

With rsync, fetch files back from the target, e.g. rsync -a backup.example.com:/backups/vserver/etc/hostname /tmp/.

Common errors

• Fatal: unable to open config file with restic. The repository has not been created with restic init yet, or the path in RESTIC_REPOSITORY is wrong.
• Fatal: wrong password or no key found. The password in /root/.restic-password does not match the repository. Without the right password you cannot get at the data.
• The timer runs, but SSH asks for a password. The key is not on the target or /root/.ssh/config is missing. Test with ssh backup.example.com.
• unable to create lock. An earlier run was interrupted. Make sure no restic is still running, then restic unlock.
• The VPS disk fills up. Old dumps pile up in /var/backups. The script above overwrites the dump each time; older versions live in restic.

What next

• After a few days, look at restic snapshots. If the number and times are right, everything works.
• You only see failures in the journal. Check it regularly or have a monitoring tool notify you.

If you get stuck, open a ticket.