Home Lab Backups

The why behind my backups!

Recently the hardware I’d purchased for a hardware raid failed on me! Restoring everything was a pain and I wasn’t able to get another of the same raid hardware so I was unable to recover any of the data. I lost most of my servers and had to do a lot of rebuilding. Fortunately I did have a lot of the configurations and docker compose yaml’s backed up to an external HDD as well as my laptop.

The Backup Plan

Proxmox VM and CT Backups

Each service is now run on their own Proxmox Container (CT) the CTs are set to backup each Sunday and retain 3 backups.

Tier 1: Software Raid

Through my desktop I’m using as my server I’ve installed the 4 HDD from the previous raid in a virtual raid using Proxmox ZFS feature. This is my first line of defense. The raid has a pool set within Proxmox for the Tank storage. It contains a couple of important items.

1
2
3
4


/tank/backups/dump # Proxmox CT and VM backups.
/tank/backups/nextcloud # Nextcloud backups
/tank/storage/container-points/immich/ # immich backups
/tank/storage/container-points/media/Plex # Plex media server backups

Tier 2: Cloud backup

I have an account with Backblaze and a script setup for backups. This is set in my Proxmox servers Cron job to run each morning at 6:00am. The script will will sync the backup and notify me when it’s complete or if there are errors. At $6 per TB it seems well worth the price. Though I am not backing up my Plex media to the cloud as that would get costly. I am backing up my photo server data, Nextcloud server data and all the Proxmox CT and CM backups.

1
2


# Cron Job for backup
0 6 * * * /usr/local/sbin/cloudbackup.sh

cloudbackup.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


#!/bin/bash

WEBHOOK_URL_SUCCESS="https://discord.com/api/webhooks/{webhook_id}/{webhook_token}"
WEBHOOK_URL_FAILURE="https://discord.com/api/webhooks/{webhook_id}/{webhook_token}"

# Function to send a success notification
send_success_notification() {
    currentTime=$(date +"%Y-%m-%d %H:%M:%S")
    JSON_PAYLOAD=$(jq -n \
        --arg title "Backblaze Daily Complete" \
        --arg message "Successfully synced data at $currentTime" \
        '{embeds: [{title: $title, description: $message, color: 5763719}]}') # Green color
    curl -H "Content-Type: application/json" -X POST -d "$JSON_PAYLOAD" "$WEBHOOK_URL_SUCCESS"
}

# Function to send an error notification
send_error_notification() {
    ERROR_LOG="$1"
    JSON_PAYLOAD=$(jq -n \
        --arg title "Backblaze Daily Backup Sync Error" \
        --arg message "Error during data sync:\n\`\`\`\n$ERROR_LOG\n\`\`\`" \
        '{embeds: [{title: $title, description: $message, color: 15548997}]}') # Red color
    curl -H "Content-Type: application/json" -X POST -d "$JSON_PAYLOAD" "$WEBHOOK_URL_FAILURE"
}

# Run the sync commands and capture output/errors
{
    b2 sync /tank/backups/dump b2://{bucket-name}/proxmox
    b2 sync /tank/backups/nextcloud b2://{bucket-name}/nextcloud
    b2 sync /tank/storage/container-points/immich/ b2://{bucket-name}/immich
} &> /tmp/b2_sync_output.log # Redirect all output (stdout and stderr) to a temporary log file

# Check the exit code of the last command
if [ $? -eq 0 ]; then
    send_success_notification
else
    ERROR_CONTENT=$(cat /tmp/b2_sync_output.log)
    send_error_notification "$ERROR_CONTENT"
fi

# Clean up the temporary log file
rm /tmp/b2_sync_output.log

Tier 3: External HDD

I have an external HDD that I can use for archive storage. This is used for a manual backup by running rsync on the directories I need backed up. Here I backup the entire tank drive including all my Plex Media.

1
2
3


mount -t ntfs-3g  -o rw /dev/{drive_id} /mnt/backup

rsync -avz --delete /tank/ /mnt/backup/ProxmoxServer/