Author: Eiko

Tags: rsync, backup, incremental, script, cron

Time: 2024-09-20 01:20:33 - 2024-11-01 09:43:16 (UTC)

Create Incremental Backups using rsync

rsync is an extremely powerful utility that can do copy and backups blazingly fast.

One very useful way to use it is to create incremental backups running periodically. The key magic in use is the --link-dest option, which specifies a path to provide hard links for old files from.

Hard link is a feature in UNIX file systems that allows multiple files to point to the same actual file on the disk, so you can have multiple same files on your file system that does not take as much space. This allows the update to be incremental.

The following script demonstrates creating robust backups to a remote backup machine. You can easily modify it to do anything you want, like backup locally to another partition or disk etc.

#!/bin/bash
# This script performs an incremental backup using rsync with --link-dest option.

# Variables
SOURCE="/path/to/source_directory/"
DEST_BASE="/path/to/backup/directory/"
REMOTE_USER="user"
REMOTE_HOST="remote_host"
LAST_BACKUP_FILE="/path/to/last_backup.txt" # File to store the last successful backup date

DATE=$(date +"%Y-%m-%d_%H-%M")
DEST="$REMOTE_USER@$REMOTE_HOST:$DEST_BASE$DATE/"

# Include/Exclude options (modify as needed)
INCLUDE_EXCLUDE_OPTIONS=(
    --include='dir1/***'
    --include='dir3/***'
    --exclude='*'
)

# Retrieve the previous backup date from the local file, if it exists
if [ -f "$LAST_BACKUP_FILE" ]; then
    PREV_DATE=$(cat "$LAST_BACKUP_FILE")
    LINK_DEST="../$PREV_DATE/"
else
    # If no previous backup, do a full backup without --link-dest
    LINK_DEST=""
fi

# Perform rsync with --link-dest if available
if [ -n "$LINK_DEST" ]; then
    RSYNC_COMMAND="rsync -avz --delete --link-dest=$LINK_DEST"
else
    RSYNC_COMMAND="rsync -avz --delete"
fi

# Execute the rsync command and check if it succeeds
if $RSYNC_COMMAND "${INCLUDE_EXCLUDE_OPTIONS[@]}" "$SOURCE" "$DEST"; then
    # If rsync succeeds, save the current date to the last_backup.txt file
    echo "$DATE" > "$LAST_BACKUP_FILE"
else
    echo "Backup failed at $(date)."
    exit 1
fi

To automate it, you can put the path of the script in crontabs,

crontab -e

use something like

0 2 * * *  /path/to/backup.sh >> /path/to/backup_logs.log
# This performs backup at 2AM each day.