Incremental Backups with Borg Backup

When I redid my personal folder structure and moved to iCloud Drive in 2023, I also needed a new backup solution.

Previously, I stored my personal files on my Synology NAS and its data was – and still is – regularly backed-up to a cloud provider and to local hard drives.

By changing my primary storage location to iCloud Drive, creating a backup got a bit more complicated:

• For one, iCloud Drive is only available on Apple devices or on Windows. As far as I know, it's impossible to install an iCloud Drive client on a Synology NAS or a Linux server.
• When "Advanced Data Protection" is enabled, accessing my iCloud data is even more restricted, as all data is end-to-end encrypted.
• The MacBook Pro I own doesn't have the storage capacity to hold my entire iCloud Drive content locally. The 400GB I currently store in Apple's cloud doesn't fit on my 256GB MacBook Pro disk.
• There is no SDK or web API available to interact with iCloud Drive. (Probably wouldn't have helped me, as I have Advanced Data Protection enabled)

As it's impossible for me to keep an offline snapshot of my iCloud Drive on my MacBook Pro, it became quickly clear, that I won't be able to create regular backup of all my iCloud Drive data.^[1]
I decided that I therefore only regularly back up my most important files. In my private note on this project I wrote:

The definition of "most important files" is quite hard. For me I consider all the files in the folders 10-29 as most important. The folders contain all personal and financial information from the last 2 decades. If I would lose them, life would become complicated quickly.^[2]

Given iCloud Drive's limitations, I set these goals for my backup software search:

• end-to-end encryption: only I should be able to see the contents of my files in the backup.
• fast: creating a backup should not take hours
• cheap: a one time purchase; not a monthly or yearly subscription
• incremental backups: to preserve disk space, the software shouldn't back up all files whenever a new backup is created. Only the changed files should be copied.
• files need to be able to be restored without relying on third party applications on third party hardware^[3]

With this out of the way, the search began.

The beginnings with rsync #

At first I've used rsync to create incremental backups.

I used the following script to create incremental backups of my "Personal Documents" folder.

#!/bin/sh

src="~/iCloud/Documents/10-19 Personal-Documents/"

# Path to folder for backups
# Note: ULDUAR is the name the USB SSD where I would store my backups.
dest='/Volumes/ULDUAR/Backups/';

# Set the retention period for incremental backups in days
retention=30

# Start the backup process
rsync --archive \
--delete \
--backup \
--inplace \
--recursive \
--verbose \
--exclude=.DS_Store \
--backup-dir=${dest}/increment/`date +%Y-%m-%d-%H%M`/ \
"${src}" \
${dest}/full/

# Clean up incremental archives older than the specified retention period
find ${dest}/increment/ -mindepth 1 -maxdepth 2 -type d -mtime +${retention} -exec rm -rf {} \;

This system met a few of my requirements: cheap, fast, incremental backups and restoring files does not require third party software on third-party hardware.

But it was finicky. My prior knowledge of rsync was limited to using it as a deploy-tool and debugging the script was a bit of a nightmare.

My attempt to automate the running of the script using launchd was also a failure. Somehow macOS doesn't allow a scheduled script to copy files from the internal hard disk to an external drive. The script always failed with rsync: [sender] opendir "/path/to/destination" failed: Operation not permitted (1).

End-to-end encryption was also not yet solved. I've used this system, while I continued my search for a perfect match, so that I at least got a backup of my files.

Borg Backup #

After many hours of research and reading too many Reddit threads I've landed on my current solution: Borg Backup or Borg for short.^[4]

After creating several proof of concept script to test different backup strategies, I created several scripts in a ~/bin/borg-backup/ -older that now do all the heavy lifting of backing up my files.

config.sh #

At the core sits the config.sh file which contains helper functions and config values. Here is an abbreviated version of the file. I use the 1Password CLI to get the passphrase for the backups.

#!/bin/sh

# ...

# Secrets
# ---------------------------------------------------------------

# Setting this, so the repo does not need to be given on the commandline:
# export BORG_REPO=ssh://[email protected]:2022/~/backup/main
export BORG_REPO='/Volumes/ULDUAR/Backups/Borg Repositories/iCloud Drive'

# Extract Passphrase from 1Password item
export BORG_PASSPHRASE=$(op item get <1password-id> --fields label=password);

# Utilities
# ---------------------------------------------------------------
colorOutput() {
    local color="$1"
    local content="$2"

    # Color mappings
    local default="\033[0m"
    local red="\033[31m"
    local green="\033[0;32m"
    local yellow="\033[0;33m"
    local cyan="\033[0;35m"
    local magenta="\033[0;36m"

    printf "${!color}$content$default"
}

printDefault () {
    colorOutput "default" "$1\r\n"
}

printInfo() {
    colorOutput "cyan" "$1\r\n"
}

# ...

backup.sh #

The other core script is obviously backup.sh which creates the backup. I didn't write all of this on my own. I think the basic structure is available in the Borg docs.

#!/bin/sh

# source config.sh
source "$(dirname "$0")/config.sh"

printSection "Starting backup"

# Backup the most important directories into an archive named after
# the machine this script is currently running on:

borg create                                \
    --verbose                              \
    --filter AME                           \
    --list                                 \
    --stats                                \
    --show-rc                              \
    --compression lz4                      \
    --exclude-caches                       \
    --exclude '/Users/stefan/.cache/*'     \
                                           \
    ::'{hostname}-{now}'                   \
    '/iCloud/Documents/00-00 INBOX' \
    '/iCloud/Documents/10-19 Personal-Documents' \
    '/iCloud/Documents/20-29 Finances' \
    '/iCloud/Notes' \

backup_exit=$?

# If available, copy the Borg Repositories from the ULDUAR drive (SSD)
# to the NAS. The "Backups" directory on the NAS is regularly
# being backed up to different locations.

if [ -d "/Volumes/NAS/Backups/" ]; then

    printInfo "🟢 Copy Repository to NAS"

    rsync --archive \
    --protect-args \
    --verbose \
    --recursive \
    --exclude=.DS_Store \
    --delete \
    '/Volumes/ULDUAR/Backups/Borg Repositories/' \
    '/Volumes/NAS/Backups/Borg Repositories/'

else
  printCaution "⚠️ NAS not available. Backup not mirrored to NAS."
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-*' matching is very important to
# limit prune's operation to this machine's archives and not apply to
# other machines' archives also:

printInfo "🧹 Pruning repository"

borg prune                          \
    --list                          \
    --glob-archives '{hostname}-*'  \
    --show-rc                       \
    --keep-daily    7               \
    --keep-weekly   4               \
    --keep-monthly  6

prune_exit=$?

# actually free repo disk space by compacting segments

printInfo "🗜️ Compacting repository"

borg compact

compact_exit=$?

# use highest exit code as global exit code
global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))
global_exit=$(( compact_exit > global_exit ? compact_exit : global_exit ))

if [ ${global_exit} -eq 0 ]; then
    printSuccess "🟢 Backup, Prune, and Compact finished successfully"
elif [ ${global_exit} -eq 1 ]; then
    printCaution "⚠️ Backup, Prune, and/or Compact finished with warnings"
else
    printError "🚨 Backup, Prune, and/or Compact finished with errors"
fi

exit ${global_exit}

check.sh, list.sh and export.sh #

There are three other files in my script directory. check.sh runs borg check and checks the repository consistency. list.sh runs borg list to get a list of backups and export.sh runs borg export-tar to extract files from a backup into my ~/Downloads folder.

Running the Backup Script #

Running the script is as easy as typing sh ./backup.sh in a terminal of your choice. In my case, 1Password will also prompt me to authenticate the request to access the encryption passphrase.

I've added a task to my weekly review project in Things, to remind me to create a backup.
As I don't want to open a terminal, navigate to the right directory and run the script (or add an alias for all of this) I've added a custom workflow to Alfred to run the script for me.
All I have to do now is open Alfred and type "borg:backup".^[5]

The Bigger Picture #

The Borg script and the storage on the USB-SSD is – however – only a small piece in a bigger backup strategy system.

As you might have noticed in the backup.sh-script above, the repository Borg creates is also copied to my NAS, if the drive is available. This way, the backups for "Important Docs" is located in 2 locations.

And my NAS is then also independently backed up to 2 different location. This illustrations explains my current backup strategy quite well.

Another important piece in my backup strategy are the yearly photo archives. At the end of each year, I export all photos taken during that year, zip them up and upload them into an AWS S3 bucket.

I've been using this system for over a year now and I'm quite happy with it. It has already happend that I needed to restore a specific file once or twice and the system held up.

For now, I'm quite happy with this system and I will use it for the foreseeable future.