How to do encrypted incremental backups and snapshots with GNU Tar and GnuPG

Published on 2021-07-24. Modified on 2021-07-29.

There exist a bunch of really cool open source backup tools such as Borg and Restic, but did you know that you can use GNU Tar and GnuPG to create encrypted incremental snapshots?

While I think that both Borg and Restic are great tools I generally prefer solutions that I can easily take apart if I need to. Borg and Restic offers more advanced features but there is a great benefit to the simplicity of combining GNU Tar and GnuPG.

I have ZFS deeply integrated into my daily workflow and I don't trust anything else with my important files, but occasional I need to do a file level incremental backup with encryption on Linux and GNU Tar and GnuPG are really great for that. It's simple and you can write the data unto whatever storage solution you prefer or pipe it into SSH.

When you do incremental backups with GNU Tar you use a meta data file, or a snapshot file, that you need to store somewhere. The meta data file keeps track of added or deleted files.

Let's dive into it by a simple example where I backup a directory called important-data in my home directory.

You can do this in a bunch of different ways, but I like to create a small encryption script, in this exampled called tarcryptor. You can put this script anywhere you like, but I suggets you keep in a bin directory in your home directory.

gpg -c --cipher-algo AES256 --compress-algo none --yes -o - --trust-model=always -c

You can add compression if you want, in this case I'll add gzip just as an example.

gzip -9 | gpg -c --cipher-algo AES256 --compress-algo none --yes -o - --trust-model=always -c

Then make the script executable.

$ chmod +x bin/tarcryptor

Now you can use that with the --use-compress-program option for GNU Tar. It doesn't matter whether you actually use compression, it will still work.

I am going to backup files from the important-data directory.

$ mkdir backup
$ cd backup
$ tar -g important-data.snar --use-compress-program=/home/foo/bin/tarcryptor -cvf backup-1.tar.gz.gpg /home/foo/important-data/

GnuPG will prompt you for a password.

The -g option is for the new GNU-format incremental backups and the important-data.snar file contains the relevant meta data. The rest of the options should be familiar. In this example I have just numbered the backup file backup-1.tar.gz.gpg, but you should consider using a date and time format instead, like backup-20210724-1855.tar.gz.gpg.

The next time you need to do an incremental backup you simply increase the number in the file name or set a new date and time.

$ tar -g important-data.snar --use-compress-program=/home/foo/bin/tarcryptor -cvf backup-2.tar.gz.gpg /home/foo/important-data/

GNU Tar will now create a new snapshot of the changes made to the important-data directory.

When you reach a point where you need to restore some files, you can extract the relevant files from the relevant snapshot.

Because the archive is encrypted GNU Tar cannot simply list the files in the snapshot as it normally would, you need to decrypt the snapshot first. Let's say we want to list the files in the first snapshot.

$ gpg backup-1.tar.gz.gpg

Then we can use GNU Tar to list the files.

$ tar --list --incremental -vvf backup-1.tar.gz
Y foo.txt
Y bar.txt
Y baz.txt

-rw-r--r-- foo/wheel     4051 2021-07-01 15:03 home/foo/important-data/foo.txt
-rw------- foo/wheel  5902741 2021-07-19 09:53 home/foo/important-data/bar.txt
-rw------- foo/wheel    41090 2021-07-19 08:52 home/foo/important-data/baz.txt

Should you ever need to do a full restoration from the latest snapshot, simply decrypt the lastest snapshot and extract the files.

$ cd important-data
$ gpg -d ../backup/backup-2.tar.gz.gpg | tar -xvz --listed-incremental=/dev/null -f -

GNU Tar will then extract the files from the snapshot.

If you don't need encryption you can take GnuPG out of the equation and simply use GNU Tar alone. I recommend you also don't use compression as that will enable tar to verify the integrity of the archive.

$ cd backup
$ tar -g important-data.snar -PWcvf backup-1.tar /home/foo/important-data/

Final notes

GNU Tar with gzip and GnuPG are excellent tools for "do it yourself" encrypted backup and they performed very well. However, searching backups are far more difficult when using incremental tar compared to something like Borg, Restic or duplicity. Yet, these tools adds multiple layers of complexity.

You have to weigh all the pros and cons of each of these tools and then choose the solution that best fit your requirements.

Relevant reading