Incremental Backups with Borg Backup

• 9 min read

When I redid my personal folder structure and moved to iCloud Drive in 2023, I also needed a new backup solution.

Previously, I stored my personal files on my Synology NAS and its data was – and still is – regularly backed-up to a cloud provider and to local hard drives.

By changing my primary storage location to iCloud Drive, creating a backup got a bit more complicated:

  • For one, iCloud Drive is only available on Apple devices or on Windows. As far as I know, it's impossible to install an iCloud Drive client on a Synology NAS or a Linux server.
  • When "Advanced Data Protection" is enabled, accessing my iCloud data is even more restricted, as all data is end-to-end encrypted.
  • The MacBook Pro I own doesn't have the storage capacity to hold my entire iCloud Drive content locally. The 400GB I currently store in Apple's cloud doesn't fit on my 256GB MacBook Pro disk.
  • There is no SDK or web API available to interact with iCloud Drive. (Probably wouldn't have helped me, as I have Advanced Data Protection enabled)

As it's impossible for me to keep an offline snapshot of my iCloud Drive on my MacBook Pro, it became quickly clear, that I won't be able to create regular backup of all my iCloud Drive data.[1]
I decided that I therefore only regularly back up my most important files. In my private note on this project I wrote:

The definition of "most important files" is quite hard. For me I consider all the files in the folders 10-29 as most important. The folders contain all personal and financial information from the last 2 decades. If I would lose them, life would become complicated quickly.[2]

Given iCloud Drive's limitations, I set these goals for my backup software search:

  • end-to-end encryption: only I should be able to see the contents of my files in the backup.
  • fast: creating a backup should not take hours
  • cheap: a one time purchase; not a monthly or yearly subscription
  • incremental backups: to preserve disk space, the software shouldn't back up all files whenever a new backup is created. Only the changed files should be copied.
  • files need to be able to be restored without relying on third party applications on third party hardware[3]

With this out of the way, the search began.

The beginnings with rsync #

At first I've used rsync to create incremental backups.

I used the following script to create incremental backups of my "Personal Documents" folder.


src="~/iCloud/Documents/10-19 Personal-Documents/"

# Path to folder for backups
# Note: ULDUAR is the name the USB SSD where I would store my backups.

# Set the retention period for incremental backups in days

# Start the backup process
rsync --archive \
--delete \
--backup \
--inplace \
--recursive \
--verbose \
--exclude=.DS_Store \
--backup-dir=${dest}/increment/`date +%Y-%m-%d-%H%M`/ \
"${src}" \

# Clean up incremental archives older than the specified retention period
find ${dest}/increment/ -mindepth 1 -maxdepth 2 -type d -mtime +${retention} -exec rm -rf {} \;

This system met a few of my requirements: cheap, fast, incremental backups and restoring files does not require third party software on third-party hardware.

But it was finicky. My prior knowledge of rsync was limited to using it as a deploy-tool and debugging the script was a bit of a nightmare.

My attempt to automate the running of the script using launchd was also a failure. Somehow macOS doesn't allow a scheduled script to copy files from the internal hard disk to an external drive. The script always failed with rsync: [sender] opendir "/path/to/destination" failed: Operation not permitted (1).

End-to-end encryption was also not yet solved. I've used this system, while I continued my search for a perfect match, so that I at least got a backup of my files.

Borg Backup #

After many hours of research and reading too many Reddit threads I've landed on my current solution: Borg Backup or Borg for short.[4]

After creating several proof of concept script to test different backup strategies, I created several scripts in a ~/bin/borg-backup/ -older that now do all the heavy lifting of backing up my files. #

At the core sits the file which contains helper functions and config values. Here is an abbreviated version of the file. I use the 1Password CLI to get the passphrase for the backups.


# ...

# Secrets
# ---------------------------------------------------------------

# Setting this, so the repo does not need to be given on the commandline:
# export BORG_REPO=ssh://[email protected]:2022/~/backup/main
export BORG_REPO='/Volumes/ULDUAR/Backups/Borg Repositories/iCloud Drive'

# Extract Passphrase from 1Password item
export BORG_PASSPHRASE=$(op item get <1password-id> --fields label=password);

# Utilities
# ---------------------------------------------------------------
colorOutput() {
    local color="$1"
    local content="$2"

    # Color mappings
    local default="\033[0m"
    local red="\033[31m"
    local green="\033[0;32m"
    local yellow="\033[0;33m"
    local cyan="\033[0;35m"
    local magenta="\033[0;36m"

    printf "${!color}$content$default"

printDefault () {
    colorOutput "default" "$1\r\n"

printInfo() {
    colorOutput "cyan" "$1\r\n"

# ... #

The other core script is obviously which creates the backup. I didn't write all of this on my own. I think the basic structure is available in the Borg docs.


# source
source "$(dirname "$0")/"

printSection "Starting backup"

# Backup the most important directories into an archive named after
# the machine this script is currently running on:

borg create                                \
    --verbose                              \
    --filter AME                           \
    --list                                 \
    --stats                                \
    --show-rc                              \
    --compression lz4                      \
    --exclude-caches                       \
    --exclude '/Users/stefan/.cache/*'     \
    ::'{hostname}-{now}'                   \
    '/iCloud/Documents/00-00 INBOX' \
    '/iCloud/Documents/10-19 Personal-Documents' \
    '/iCloud/Documents/20-29 Finances' \
    '/iCloud/Notes' \


# If available, copy the Borg Repositories from the ULDUAR drive (SSD)
# to the NAS. The "Backups" directory on the NAS is regularly
# being backed up to different locations.

if [ -d "/Volumes/NAS/Backups/" ]; then

    printInfo "🟢 Copy Repository to NAS"

    rsync --archive \
    --protect-args \
    --verbose \
    --recursive \
    --exclude=.DS_Store \
    --delete \
    '/Volumes/ULDUAR/Backups/Borg Repositories/' \
    '/Volumes/NAS/Backups/Borg Repositories/'

  printCaution "⚠️ NAS not available. Backup not mirrored to NAS."

# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-*' matching is very important to
# limit prune's operation to this machine's archives and not apply to
# other machines' archives also:

printInfo "🧹 Pruning repository"

borg prune                          \
    --list                          \
    --glob-archives '{hostname}-*'  \
    --show-rc                       \
    --keep-daily    7               \
    --keep-weekly   4               \
    --keep-monthly  6


# actually free repo disk space by compacting segments

printInfo "🗜️ Compacting repository"

borg compact


# use highest exit code as global exit code
global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))
global_exit=$(( compact_exit > global_exit ? compact_exit : global_exit ))

if [ ${global_exit} -eq 0 ]; then
    printSuccess "🟢 Backup, Prune, and Compact finished successfully"
elif [ ${global_exit} -eq 1 ]; then
    printCaution "⚠️ Backup, Prune, and/or Compact finished with warnings"
    printError "🚨 Backup, Prune, and/or Compact finished with errors"

exit ${global_exit}, and #

There are three other files in my script directory. runs borg check and checks the repository consistency. runs borg list to get a list of backups and runs borg export-tar to extract files from a backup into my ~/Downloads folder.

Running the Backup Script #

Running the script is as easy as typing sh ./ in a terminal of your choice. In my case, 1Password will also prompt me to authenticate the request to access the encryption passphrase.

I've added a task to my weekly review project in Things, to remind me to create a backup.
As I don't want to open a terminal, navigate to the right directory and run the script (or add an alias for all of this) I've added a custom workflow to Alfred to run the script for me.
All I have to do now is open Alfred and type "borg:backup".[5]

The Bigger Picture #

The Borg script and the storage on the USB-SSD is – however – only a small piece in a bigger backup strategy system.

As you might have noticed in the above, the repository Borg creates is also copied to my NAS, if the drive is available. This way, the backups for "Important Docs" is located in 2 locations.

And my NAS is then also independently backed up to 2 different location. This illustrations explains my current backup strategy quite well.

Diagram showing my backup strategy. It shows rectangles labeled MacBook Pro, iCloud, Google Drive Synology NAS, AWS S3 and USB-SDD. Arrows point to the different rectangles signifining which devices is backed up to which platform.
As you can see, different data is stored on different devices and platforms and is backed up in different ways to different locations.

Another important piece in my backup strategy are the yearly photo archives. At the end of each year, I export all photos taken during that year, zip them up and upload them into an AWS S3 bucket.

I've been using this system for over a year now and I'm quite happy with it. It has already happend that I needed to restore a specific file once or twice and the system held up.

For now, I'm quite happy with this system and I will use it for the foreseeable future.

  1. You might ask yourself: "Stefan, why do you have 400 GB of data in iCloud Drive? What is all that stuff?". Fair question. The biggest files are in itself backups/archives. Be it yearly archives of my photo library dating back to 1992, backups of old software I would like to keep or my music library. ↩︎

  2. "folder 10-29" refer to my Johnny Decimal folder structure. ↩︎

  3. For example Synology Hyper Backup is a backup software I use on my NAS. In case of an emergency (say my flat burns down and the NAS is destroyed), I would need to purchase another NAS in order to restore my files. In such a situation, purchasing a NAS and setting it up is not the first thing I want to do, just to access my files. ↩︎

  4. Discovering Borg was quite the challenge as its SEO isn't great. It's headline isn't "backup software" but rather the nerdy term "deduplicating archiver with compression and encryption". ↩︎

  5. I want to give Raycast another try soon. Wonder if I could create a neat extension that would work with my script. ↩︎