---
id: linux-administration-data-backup-and-recovery
slug: linux-administration-data-backup-and-recovery
title: "Linux Administration #6: Data Backup and Recovery"
excerpt: "Learn the fundamentals of Linux data backup: From backup strategies and automation to recovery. With practical examples for administrators."
date: "2024-11-03T09:00:00+01:00"
updated: "2024-11-03T12:52:00+01:00"
author:
name: "Sebastian Palencsar"
handle: "AdminDocs"
category: "linux-administration"
tags: ["backup", "restore", "tar", "rsync", "compression", "disaster-recovery", "linux-administration"]
toc: true
reading_time: 45
---
**Welcome to the sixth part of our technical wiki series on Linux administration!**
After we covered the [fundamentals](/en/linux-administration/linux-administration-fundamentals-of-linux-administration){.badge-link-text}, [process management](/en/linux-administration/linux-administration-advanced-user-management){.badge-link-text}, and [shell scripting](/en/linux-administration/linux-administration-network-configuration-and-management){.badge-link-text} in previous articles, we now turn to a critical aspect of system administration: data backup and recovery.
⚠️ **Note:** In this article, we use Ubuntu/Debian as the example distribution. The basic concepts are the same on all Linux systems, but package installation and some configuration paths may vary depending on the distribution. If you use a different distribution, please consult the relevant documentation for the specific installation commands and paths.
**As a Linux administrator, backing up and recovering data is one of your most important tasks.**
Imagine your system is like a valuable book – and backups are like copies of this book that you store at various secure locations. If the original is damaged or lost, you can restore it at any time.
## Backup Strategies
As a Linux administrator, data backup is one of your most important tasks. Think of backups like different insurance policies – each has its own advantages and disadvantages.
**Backup types:**
```markdown
┌─────────── Full Backup ─────────────────────────────────────┐
│ • Complete data backup │
│ • High storage consumption │
│ • Long backup time │
├─────────── Differential ────────────────────────────────────┤
│ • Changes since last full │
│ • Medium storage consumption │
│ • Faster recovery │
├─────────── Incremental ─────────────────────────────────────┤
│ • Only new changes │
│ • Low storage consumption │
│ • More complex recovery │
└─────────────────────────────────────────────────────────────┘
```
### 1. Full Backup
A full backup is like a complete moving box with all its contents. It saves all selected data, regardless of whether they have changed or not.
**Advantages:**
* Simple recovery
* All data in one backup
* Independent of other backups
**Disadvantages:**
* Requires a lot of storage space
* Time-consuming
* High network load for remote backups
### 2. Differential Backup
A differential backup saves all changes since the last full backup. This is like a diary that records all changes since a specific date.
**Backup timeline (differential):**
```markdown
┌─── Day 1 ───┬─── Day 2 ───┬─── Day 3 ───────────────────────┐
│ Full backup │ Diff (20GB) │ Diff (25GB) │
│ (100GB) │ since Day 1 │ since Day 1 │
└─────────────┴─────────────┴─────────────────────────────────┘
```
**Advantages:**
* Faster than full backup
* Less storage than full backup
* Simpler recovery than incremental backup
**Disadvantages:**
* More storage than incremental backup
* Longer backup time than incremental backup
* More complex recovery than incremental backup
### 3. Incremental Backup
An incremental backup saves only changes since the last backup (regardless of type). This is like a daily entry in a diary.
**Backup timeline:**
```markdown
┌─── Day 1 ───┬─── Day 2 ───┬─── Day 3 ───────────────────────┐
│ Full backup │ Increment 1 │ Increment 2 │
│ (100 GB) │ (5 GB) │ (3 GB) │
└─────────────┴─────────────┴─────────────────────────────────┘
```
**Advantages:**
* Very fast backup time
* Minimal storage consumption
* Ideal for daily backups
**Disadvantages:**
* More complex recovery
* Requires all previous backups
* Higher risk with corrupted backups
### 4. Snapshot Backups
A snapshot is a point-in-time capture of the system state at a specific moment. Unlike traditional backups, snapshots use copy-on-write technology.
**How snapshots work:**
```markdown
┌─────────── Original Data ───────────────────────────────────┐
│ Block A │ Block B │ Block C │
├─────────── Snapshot 1 ──────────────────────────────────────┤
│ Only changed blocks are copied │
│ A' │ B │ C │
├─────────── Snapshot 2 ──────────────────────────────────────┤
│ Further changes │
│ A' │ B' │ C │
└─────────────────────────────────────────────────────────────┘
```
**Advantages:**
* Very fast creation (seconds)
* Space-efficient (only changes)
* Consistent backup
**Disadvantages:**
* File system dependent (e.g., ZFS, Btrfs)
* All snapshots affected on system failure
* Not a replacement for external backups
**Use cases:**
* Before system updates
* Development environments
* Virtual machines
## Backup Tools and Technologies
As a Linux administrator, you have access to various powerful backup tools. Each tool has its specific strengths and use cases.
**Backup tools:**
```markdown
┌─────────── Traditional ─────────────────────────────────────┐
│ tar - Archiving │
│ dd - Bit-exact copies │
│ rsync - Synchronization │
└─────────────────────────────────────────────────────────────┘
```
### Traditional
rsync
A powerful tool for incremental backups and synchronization. **Key features:**
* Only transfers changed files
* Supports remote backups
* Bandwidth-efficient
```bash
# Local backup
rsync -av /source/ /target/
# Remote backup
rsync -avz -e ssh /local/data/ user@server:/backup/
# With exclusions
rsync -av --exclude='*.tmp' /source/ /target/
```
tar
The classic tool for archiving and compression.
**Key features:**
* Creates single archive files
* Various compression methods
* Preserves metadata and permissions
```bash
# Create backup with compression
tar -czf backup.tar.gz /data/to/backup/
# Backup with date
tar -czf backup_$(date +%Y%m%d).tar.gz /data/
# Extract archive
tar -xzf backup.tar.gz
```
dd
A powerful tool for bit-exact copies of storage media.
**Key features:**
* Creates exact copies (disk images)
* Can back up entire hard disks
* Operates at the lowest level
```bash
# Back up disk to image
dd if=/dev/sda of=/backup/disk.img bs=4M status=progress
# Write image back to new disk
dd if=/backup/disk.img of=/dev/sdb bs=4M status=progress
# Back up partition
dd if=/dev/sda1 of=/backup/partition.img bs=4M
```
### Modern
**Backup tools:**
```markdown
┌─────────── Modern ──────────────────────────────────────────┐
│ borg - Deduplication │
│ restic - Encryption │
└─────────────────────────────────────────────────────────────┘
```
borg
A modern backup tool with deduplication and encryption.
**Key features:**
* Intelligent deduplication
* Built-in encryption
* Fast recovery
```bash
# Initialize repository
borg init --encryption=repokey /path/to/backup
# Create backup
borg create --stats --progress \
/path/to/backup::backup-{now} \
/home/user/data
# List backups
borg list /path/to/backup
```
### restic
A secure backup program with cloud support.
**Key features:**
* Encryption
* Cloud storage support
* Fast incremental backups
```bash
# Initialize repository
restic init --repo /backup/restic-repo
# Create backup
restic -r /backup/restic-repo backup /home/user/data
# Display snapshots
restic -r /backup/restic-repo snapshots
```
## Deep Dive: The 3-2-1 Backup Rule in Practice
In professional data backup, the **3-2-1 rule** is the recognized industry standard for preventing total loss:
```markdown
┌─────────────────────────────────────────────────────────────┐
│ THE 3-2-1 BACKUP ARCHITECTURE │
├─────────────────────────────────────────────────────────────┤
│ │
│ [ 3 ] COPIES OF DATA │
│ ├── 1x Primary data on production server │
│ ├── 1x Local backup (fast access) │
│ └── 1x Secondary offsite backup │
│ │
│ [ 2 ] USE DIFFERENT STORAGE MEDIA │
│ ├── NVMe/SSD storage local │
│ └── NAS / SAN / Object storage (S3) / Tape │
│ │
│ [ 1 ] COPY AT AN EXTERNAL LOCATION (OFFSITE) │
│ └── Protection against fire, theft, ransomware │
│ │
└─────────────────────────────────────────────────────────────┘
```
## Deep Dive: Database Backups During Live Operation (Hot Backups)
File-based backups of active database directories (like `/var/lib/mysql` or `/var/lib/postgresql`) inevitably lead to inconsistent or corrupted data states, as write operations occur during copying. Therefore, administrators use native dump tools:
### 1. Consistent MySQL and MariaDB Backup (`mysqldump`)
```bash
# Transaction-safe dump of all databases during live operation:
sudo mysqldump --all-databases --single-transaction --quick \
--lock-tables=false | gzip > /backup/db/all_databases_$(date +%Y%m%d).sql.gz
# Restore individual database:
gunzip < /backup/db/appdb_20260220.sql.gz | sudo mysql appdb
```
### 2. Consistent PostgreSQL Backup (`pg_dump` and `pg_dumpall`)
```bash
# Back up all PostgreSQL cluster databases:
sudo -u postgres pg_dumpall | gzip > /backup/db/postgres_all_$(date +%Y%m%d).sql.gz
# Back up individual database in custom format (fast, parallelizable):
sudo -u postgres pg_dump -Fc mydb -f /backup/db/mydb_$(date +%Y%m%d).dump
# Restore with pg_restore:
sudo -u postgres pg_restore -d mydb /backup/db/mydb_20260220.dump
```
## Deep Dive: Hardlink Snapshots with `rsync --link-dest`
With the `--link-dest` switch, `rsync` creates incremental snapshots that behave like full backups but only occupy new disk space for changed files:
```bash
#!/usr/bin/env bash
set -euo pipefail
BACKUP_PATH="/backup/snapshots"
TODAY=$(date +%Y-%m-%d)
YESTERDAY=$(ls -1d $BACKUP_PATH/20* 2>/dev/null | tail -n 1 || true)
mkdir -p "$BACKUP_PATH/$TODAY"
if [ -n "$YESTERDAY" ] && [ -d "$YESTERDAY" ]; then
echo "Using $YESTERDAY as hardlink base..."
rsync -a --delete --link-dest="$YESTERDAY" /var/www/ "$BACKUP_PATH/$TODAY/"
else
echo "Initial full backup..."
rsync -a /var/www/ "$BACKUP_PATH/$TODAY/"
fi
```
## Backup Automation
After the backup tools, we now turn to the automation of backup processes. A good backup strategy is only effective if it is executed reliably and regularly.
### Creating Backup Scripts
**a) Simple backup script with logging**
```bash
#!/bin/bash
# simple_backup.sh - Basic backup with logging
# Configuration
SOURCE_DIR="/home/user/data"
BACKUP_DIR="/backup"
LOG_FILE="/var/log/backup.log"
MAX_BACKUPS=5
# Logging function
log_message() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
echo "$1"
}
```
**b) Incremental backup script**
```bash
#!/bin/bash
# incremental_backup.sh - Incremental backup system
# Configuration
SOURCE="/home/user/data"
BACKUP_BASE="/backup/incremental"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
LATEST_LINK="$BACKUP_BASE/latest"
# Incremental backup with rsync
if [ -d "$LATEST_LINK" ]; then
rsync -av --link-dest="$LATEST_LINK" "$SOURCE/" "$BACKUP_BASE/backup_$TIMESTAMP/"
else
rsync -av "$SOURCE/" "$BACKUP_BASE/backup_$TIMESTAMP/"
fi
# Update "latest" link
ln -snf "$BACKUP_BASE/backup_$TIMESTAMP" "$LATEST_LINK"
```
**c) Full backup script**
```bash
#!/bin/bash
# full_backup.sh - Complete system backup
# Configuration
SOURCE_DIRS=("/home" "/etc" "/var/www")
BACKUP_DIR="/backup"
DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=30
# Logging function
log_message() {
local message="$1"
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $message" >> "$LOG_FILE"
echo "$message"
}
# Check/create backup directory
check_backup_dir() {
if [ ! -d "$BACKUP_DIR" ]; then
mkdir -p "$BACKUP_DIR"
log_message "Backup directory created: $BACKUP_DIR"
fi
}
# Check disk space
check_disk_space() {
local required_space=$(du -sc "${SOURCE_DIRS[@]}" | tail -n1 | cut -f1)
local available_space=$(df "$BACKUP_DIR" | tail -n1 | awk '{print $4}')
if [ $available_space -lt $required_space ]; then
log_message "ERROR: Not enough disk space!"
exit 1
fi
}
# Create backup
create_backup() {
local backup_file="${BACKUP_DIR}/full_backup_${DATE}.tar.gz"
log_message "Starting full backup..."
tar -czf "$backup_file" "${SOURCE_DIRS[@]}" 2>> "$LOG_FILE"
if [ $? -eq 0 ]; then
log_message "Backup successful: $backup_file"
return 0
else
log_message "ERROR: Backup failed!"
return 1
fi
}
# Clean up old backups
cleanup_old_backups() {
log_message "Deleting backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -name "full_backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete
}
# Main program
main() {
log_message "=== Starting backup process ==="
check_backup_dir
check_disk_space
create_backup
cleanup_old_backups
log_message "=== Backup process finished ==="
}
# Execute script
main
```
### Setting Up Cron Jobs
Regular execution of backups is crucial for data security. Cron is the perfect tool for this:
**Cron time format:**
```markdown
┌─────────────────────────────────────────────────────────────┐
│ CRON SYNTAX TIME FORMAT │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────── Minute (0 - 59) │
│ │ ┌───────────── Hour (0 - 23) │
│ │ │ ┌───────────── Day of month (1 - 31) │
│ │ │ │ ┌───────────── Month (1 - 12) │
│ │ │ │ │ ┌───────────── Weekday (0 - 7, 0/7 = Sunday) │
│ │ │ │ │ │ │
│ * * * * * /path/to/script.sh │
│ │
└─────────────────────────────────────────────────────────────┘
```
Setting up backup cron:
```bash
# Prepare backup script for cron
chmod +x /usr/local/sbin/backup_system.sh
# Edit crontab
crontab -e
# Daily backup at 2 AM
0 2 * * * /usr/local/sbin/backup_system.sh
# Weekly full backup on Sunday at 3 AM
0 3 * * 0 /usr/local/sbin/full_backup.sh
```
## Monitoring and Notifications
An effective backup system requires not only the backup itself but also reliable monitoring. Only in this way can we ensure that our backups are performed successfully and available in an emergency.
### Components of Backup Monitoring
**Monitoring components:**
```markdown
┌─────────── Status Check ────────────────────────────────────┐
│ • Backup success │
│ • Backup size │
│ • Storage space │
├─────────── Notification ────────────────────────────────────┤
│ • Email alerts │
│ • Log entries │
│ • System messages │
└─────────────────────────────────────────────────────────────┘
```
### Status Monitoring
Status monitoring continuously checks various aspects of our backups:
* Successful execution
* Currentness of backups
* Integrity of backup files
**Here is an example of a monitoring script:**
```bash
#!/bin/bash
# backup_monitor.sh
# Configuration
BACKUP_DIR="/backup"
LOG_FILE="/var/log/backup_monitor.log"
MAIL_TO="admin@domain.com"
MIN_SIZE=1048576 # 1MB in bytes
# Monitoring functions
check_backup_status() {
# Check last backup
local latest_backup=$(find "$BACKUP_DIR" -type f -name "backup_*.tar.gz" -mtime -1)
if [ -z "$latest_backup" ]; then
send_alert "WARNING: No current backup found!"
return 1
fi
# Check backup size
local size=$(stat -f %z "$latest_backup")
if [ "$size" -lt "$MIN_SIZE" ]; then
send_alert "WARNING: Backup size too small: $size bytes"
return 1
fi
log_message "Backup status OK: $(basename "$latest_backup")"
return 0
}
# Alert function
send_alert() {
local message="$1"
echo "$message" | mail -s "Backup Alert" "$MAIL_TO"
log_message "ALERT: $message"
}
```
## Backup Verification and Tests
The best backup strategy is useless if the backups don't work. Therefore, regular verification and testing of backups is essential.
**Verification process:**
```markdown
┌─────────── Integrity ───────────────────────────────────────┐
│ • Check checksums │
│ • Verify backup size │
│ • Perform archive test │
├─────────── Tests ───────────────────────────────────────────┤
│ • Test recovery │
│ • Perform spot checks │
│ • Validate recovery plan │
└─────────────────────────────────────────────────────────────┘
```
### 1. Check Integrity
The integrity of your backups should be checked regularly:
**a) Create and verify checksums**
```bash
# Create MD5 checksum
md5sum backup.tar.gz > backup.md5
# Create SHA256 checksum (more secure)
sha256sum backup.tar.gz > backup.sha256
# Verify checksum
sha256sum -c backup.sha256
```
**b) Test archive integrity**
```bash
# Test tar archive
tar -tvf backup.tar.gz
# Test zip archive
unzip -t backup.zip
```
### 2. Test Recoveries
Regular test recoveries are crucial for the reliability of your backup system.
**Verification process:**
```markdown
┌─────────── Integrity ───────────────────────────────────────┐
│ • Check checksums │
│ • Verify backup size │
│ • Perform archive test │
├─────────── Tests ───────────────────────────────────────────┤
│ • Test recovery │
│ • Perform spot checks │
│ • Validate recovery plan │
└─────────────────────────────────────────────────────────────┘
```
**a) Prepare test environment**
```bash
# Create test directory
mkdir /tmp/backup_test
cd /tmp/backup_test
# Check backup archive
tar -tvf /backup/full_backup_20240101.tar.gz
# Perform test recovery
tar -xzf /backup/full_backup_20240101.tar.gz
```
**b) Check data integrity**
```bash
# Check directory structure
find . -type d | sort > structure.txt
# Check file permissions
ls -lR > permissions.txt
# Spot-check files
file important_file.txt
md5sum critical_data.db
```
### 3. Documentation
Thorough documentation is crucial for a successful backup system. It helps you and your team to understand processes and respond quickly in an emergency.
**Documentation structure:**
```markdown
┌─────────── Backup Plan ─────────────────────────────────────┐
│ • Backup strategies │
│ • Schedules │
│ • Responsibilities │
├─────────── Processes ───────────────────────────────────────┤
│ • Backup procedures │
│ • Recovery │
│ • Emergency plan │
└─────────────────────────────────────────────────────────────┘
```
**Important documentation elements:**
1. **Backup configuration**
* Backup types and schedules
* Storage locations and retention periods
* Tools and scripts used
2. **Recovery processes**
* Step-by-step instructions
* Contact persons and responsibilities
* Emergency procedures
3. **Test protocols**
* Tests performed
* Test results
* Identified problems and solutions
## Disaster Recovery
Disaster recovery is about restoring your system after a major failure. A well-thought-out plan is crucial here.
**Recovery process:**
```markdown
┌─────────── Preparation ─────────────────────────────────────┐
│ 1. Analyze situation │
│ 2. Activate recovery plan │
│ 3. Inform team │
├─────────── Recovery ────────────────────────────────────────┤
│ 4. Restore system │
│ 5. Restore data │
│ 6. Restart services │
├─────────── Validation ──────────────────────────────────────┤
│ 7. Perform tests │
│ 8. Activate monitoring │
│ 9. Update documentation │
└─────────────────────────────────────────────────────────────┘
```
1. Recovery Strategy
**a) Define recovery goals:**
* RPO (Recovery Point Objective): Maximum acceptable data loss
* RTO (Recovery Time Objective): Maximum acceptable downtime
* Identify critical systems and services
**b) Set recovery level:**
**Recovery levels:**
```markdown
┌─────────── Level 1 ─────────────────────────────────────────┐
│ Critical business processes │
├─────────── Level 2 ─────────────────────────────────────────┤
│ Important support systems │
├─────────── Level 3 ─────────────────────────────────────────┤
│ Non-critical systems │
└─────────────────────────────────────────────────────────────┘
```
### Documentation
Complete and current documentation is crucial:
**a) System documentation:**
* Hardware configuration
* Software versions
* Network settings
* Backup configuration
* Dependencies between systems
**b) Recovery procedures:**
* Step-by-step instructions
* Recovery order
* Validation processes
* Rollback plans
### Responsibilities
Clear responsibilities are essential for quick action:
**a) Recovery team:**
* Team leader (decision maker)
* System administrators
* Network specialists
* Database administrators
* Support staff
**b) Escalation processes:**
**Escalation hierarchy:**
```markdown
┌─────────── Level 1 ─────────────────────────────────────────┐
│ First response & assessment │
├─────────── Level 2 ─────────────────────────────────────────┤
│ Technical escalation │
├─────────── Level 3 ─────────────────────────────────────────┤
│ Management escalation │
└─────────────────────────────────────────────────────────────┘
```
2. System Recovery
Recovering a system after a failure requires a systematic approach. As a Linux administrator, you must master various recovery scenarios.
**Recovery process:**
```markdown
┌─────────── Preparation ─────────────────────────────────────┐
│ • Create boot medium │
│ • Identify backups │
│ • Check hardware │
├─────────── System ──────────────────────────────────────────┤
│ • Install base system │
│ • Restore configuration │
│ • Set up services │
├─────────── Data ────────────────────────────────────────────┤
│ • Restore backup │
│ • Check permissions │
│ • Test integrity │
└─────────────────────────────────────────────────────────────┘
```
Preparatory Measures
* Prepare boot medium with rescue system
* Identify backup media and recovery points
* Check hardware for errors
* Ensure network connection
System Recovery
**a) Install base system:**
* Restore partitioning
* Install base system
* Configure bootloader
**b) System configuration:**
* Restore `/etc` directory
* Adjust network settings
* Set up users and groups
**c) Services:**
* Configure system services
* Check dependencies
* Start services in correct order
3. Data Recovery
Data recovery requires a systematic approach, depending on the type of backup and data to be recovered.
**Recovery process:**
```markdown
┌─────────── Preparation ─────────────────────────────────────┐
│ • Identify backup │
│ • Check target directory │
│ • Verify storage space │
├─────────── Execution ───────────────────────────────────────┤
│ • Restore data │
│ • Check permissions │
│ • Test integrity │
└─────────────────────────────────────────────────────────────┘
```
Restore Data from Full Backup
```bash
# Extract backup
tar -xzf /backup/full_backup_20240101.tar.gz -C /recovery/
# Check and correct permissions
chown -R user:group /recovery/data/
chmod -R 755 /recovery/data/
```
Incremental Recovery
```bash
# Restore full backup
tar -xzf /backup/full_20240101.tar.gz -C /recovery/
# Incremental backups in chronological order
tar -xzf /backup/incr_20240102.tar.gz -C /recovery/
tar -xzf /backup/incr_20240103.tar.gz -C /recovery/
```
4. Emergency Plans
A well-thought-out emergency plan is crucial for quick recovery in a crisis.
**Emergency plan structure:**
```markdown
┌─────────── Preparation ─────────────────────────────────────┐
│ • Contact list │
│ • Backup inventory │
│ • Recovery instructions │
├─────────── Measures ────────────────────────────────────────┤
│ • Immediate actions │
│ • Recovery steps │
│ • Validation │
└─────────────────────────────────────────────────────────────┘
```
Preparation and Documentation
* Current contact list of all responsible persons
* Locations of all backup media
* Access data and passwords (stored securely)
* Hardware requirements
* Network configurations
Immediate Actions
* Damage limitation
* Identify affected systems
* Inform team
* Provide backup media
Recovery Steps
* Prioritized recovery list
* Observe dependencies
* Test procedures
* Document steps taken
## Best Practices
Adherence to proven practices is crucial for a reliable backup system. Here are the most important best practices for your backup strategy.
**Best practices overview:**
```markdown
┌─────────── 3-2-1 Rule ──────────────────────────────────────┐
│ • 3 backup copies │
│ • 2 different media │
│ • 1 offsite backup │
├─────────── Security ────────────────────────────────────────┤
│ • Encryption │
│ • Access controls │
│ • Verification │
└─────────────────────────────────────────────────────────────┘
```
### The 3-2-1 Backup Rule
This proven rule helps you optimally protect your data:
Backup copies:
* Original data
* First backup copy (e.g., `external hard drive`)
* Second backup copy (e.g., `cloud storage`)
Different media:
* Different storage types (e.g., `SSD` and `HDD`)
* Different manufacturers
* Different technologies
1 Offsite Backup:
* Physically separated from main location
* Cloud storage or external data center
* Protection against local disasters
### Encryption
Encrypting your backups is an important security aspect, especially when backing up sensitive data.
**Encryption options:**
```markdown
┌─────────── Symmetric ───────────────────────────────────────┐
│ • One key │
│ • Fast │
│ • Simple │
├─────────── Asymmetric ──────────────────────────────────────┤
│ • Key pair │
│ • More secure │
│ • More complex │
└─────────────────────────────────────────────────────────────┘
```
**a) Backup encryption with GPG:**
```bash
# Create and encrypt backup
tar -czf - /important/data | \
gpg -e -r "admin@domain.com" > backup.tar.gz.gpg
# Decrypt backup
gpg -d backup.tar.gz.gpg | tar -xzf -
```
**b) Encrypted backups with borg:**
```bash
# Initialize repository with encryption
borg init --encryption=repokey /path/to/backup
# Create encrypted backup
borg create /path/to/backup::backup-{now} /data
```
### Retention Periods
The correct retention period for backups is crucial for an effective backup system.
**Retention strategies:**
```markdown
┌─────────── Short-term ──────────────────────────────────────┐
│ • Daily backups: 7-14 days │
│ • Weekly: 4-8 weeks │
├─────────── Long-term ───────────────────────────────────────┤
│ • Monthly: 12-24 months │
│ • Annual: 5-7 years │
└─────────────────────────────────────────────────────────────┘
```
**Retention policies:**
1. **Daily backups**
* For quick recovery
* Keep 7-14 days
* Automatic rotation
2. **Weekly backups**
* For medium-term security
* Keep 4-8 weeks
* Create every Sunday
3. **Monthly backups**
* For long-term archiving
* Keep 12-24 months
* Create at month end
**Implementation:**
```bash
# Set up backup rotation
find /backup/daily -type f -mtime +14 -delete
find /backup/weekly -type f -mtime +60 -delete
find /backup/monthly -type f -mtime +730 -delete
```
### Documentation
Thorough documentation is the key to a maintainable backup system.
**Documentation structure:**
```markdown
┌─────────── System ──────────────────────────────────────────┐
│ • Hardware configuration │
│ • Software versions │
│ • Network setup │
├─────────── Backup ──────────────────────────────────────────┤
│ • Backup strategies │
│ • Storage locations │
│ • Recovery plans │
└─────────────────────────────────────────────────────────────┘
```
**1. System documentation**
* Complete hardware inventory
* Installed software and versions
* Network configuration
* Users and permissions
**2. Backup documentation**
* Backup tools used
* Backup schedules
* Storage locations and access data
* Retention policies
**3. Recovery documentation**
* Step-by-step instructions
* Emergency contacts
* Test protocols
* Error handling
## Exercise
### Setting Up a Backup System
**Scenario:** As a Linux administrator, you need to set up a complete backup system for a small company. Data from different departments must be backed up and quickly recoverable in an emergency.
**Requirements:**
```markdown
┌─────────── Backup ──────────────────────────────────────────┐
│ • Daily backup │
│ • Weekly full backup │
│ • Encryption │
├─────────── Monitoring ──────────────────────────────────────┤
│ • Status check │
│ • Email notification │
│ • Log rotation │
└─────────────────────────────────────────────────────────────┘
```
### Tasks
**1. Create backup structure:**
* Create backup directories
* Set permissions
* Set up encryption
**2. Develop backup scripts:**
* Daily incremental backup
* Weekly full backup
* Automatic rotation of old backups
**3. Set up monitoring:**
* Monitor backup status
* Configure email notifications
* Implement logging
### Possible Solution
1. Create backup structure
```bash
# Create directory structure
sudo mkdir -p /backup/{daily,weekly,logs}
sudo mkdir -p /backup/keys
# Set permissions
sudo chmod 700 /backup/keys
sudo chmod 755 /backup/{daily,weekly,logs}
# Create GPG key for encryption
gpg --gen-key
gpg --export-secret-keys --armor backup@domain.com > /backup/keys/backup.key
```
2. Develop backup script
```bash
#!/bin/bash
# backup_system.sh
# Configuration
SOURCE_DIRS=("/home" "/etc" "/var/www")
BACKUP_DIR="/backup"
LOG_FILE="/backup/logs/backup.log"
DATE=$(date +%Y%m%d_%H%M)
RETENTION_DAILY=7
RETENTION_WEEKLY=4
# Logging function
log_message() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
echo "$1"
}
# Create backup
create_backup() {
local type=$1
local backup_file="${BACKUP_DIR}/${type}/backup_${DATE}.tar.gz"
log_message "Starting ${type} backup..."
tar -czf - "${SOURCE_DIRS[@]}" | \
gpg -e -r backup@domain.com > "$backup_file"
if [ $? -eq 0 ]; then
log_message "Backup successful: $backup_file"
return 0
else
log_message "ERROR: Backup failed"
return 1
fi
}
```
3. Set up monitoring
```bash
#!/bin/bash
# backup_monitor.sh
# Configuration
BACKUP_DIR="/backup"
LOG_FILE="/backup/logs/backup.log"
MAIL_TO="admin@domain.com"
MIN_SIZE=1048576 # 1MB in bytes
# Monitoring functions
check_backup() {
local latest_backup=$(find "$BACKUP_DIR" -type f -name "backup_*.tar.gz" -mtime -1)
# Check if backup exists
if [ -z "$latest_backup" ]; then
send_alert "ERROR: No current backup found!"
return 1
fi
# Check backup size
local size=$(stat -f %z "$latest_backup")
if [ "$size" -lt "$MIN_SIZE" ]; then
send_alert "WARNING: Backup size too small: $size bytes"
return 1
fi
# Check backup integrity
if ! tar -tzf "$latest_backup" >/dev/null 2>&1; then
send_alert "ERROR: Backup archive corrupted!"
return 1
fi
log_message "Backup status OK: $(basename "$latest_backup")"
return 0
}
# Cron job for daily check:
0 7 * * * /backup/scripts/backup_monitor.sh
```
## Command Reference (Cheatsheet)
For quick access during planning and implementation of backup and recovery processes, the following reference table summarizes the most important Linux commands and tools:
| Command / Syntax | Tool / Area | Function & Description |
|---|---|---|
| `tar -czf archive.tar.gz /path` | tar | Creates a gzip-compressed archive. |
| `tar -xzf archive.tar.gz -C /target` | tar | Extracts a gzip archive to a defined target directory. |
| `tar -tvf archive.tar.gz` | tar | Lists archive contents without extracting. |
| `rsync -avz /source/ user@host:/target/` | rsync | Synchronizes directories over SSH with archive mode and compression. |
| `rsync -a --delete /source/ /target/` | rsync | Synchronizes data and deletes files no longer present in target. |
| `rsync -a --link-dest=REF /source/ /target/` | rsync | Creates space-saving hardlink snapshots based on reference state. |
| `dd if=/dev/sda of=/backup/disk.img bs=4M status=progress` | dd | Creates a bit-exact 1:1 image of a storage medium. |
| `borg init --encryption=repokey /repo` | BorgBackup | Initializes an encrypted, deduplicated backup repository. |
| `borg create /repo::backup-{now} /data` | BorgBackup | Creates a new deduplicated and encrypted backup archive. |
| `restic backup /path/to/data` | Restic | Backs up files encrypted and incrementally to a Restic repository. |
| `sha256sum file.tar.gz > file.sha256` | Checksum | Calculates cryptographic SHA-256 checksum for integrity verification. |
| `sha256sum -c file.sha256` | Checksum | Validates checksums against stored reference data. |
| `mysqldump --all-databases --single-transaction` | Database | Consistent hot dump of MySQL/MariaDB without locking read tables. |
| `pg_dumpall \| gzip > all_db.sql.gz` | Database | Complete PostgreSQL dump of all databases and roles. |
| `pg_restore -d dbname dump.dump` | Database | Restores PostgreSQL database from binary custom dump. |
## Further Resources
The following guides, manuals, and internal course modules deepen the backup strategies and system administration covered:
| Resource | Description |
|---|---|
| [BorgBackup Documentation](https://borgbackup.readthedocs.io/){.badge-link-text} | Official documentation for deduplicating archiving with Borg. |
| [Restic Documentation](https://restic.readthedocs.io/){.badge-link-text} | Official guide for secure cloud and on-premises backups with Restic. |
| [GNU Tar Manual](https://www.gnu.org/software/tar/manual/){.badge-link-text} | Complete reference manual of the GNU Tar archiving tool. |
| [Linux Administration #5: Shell Scripting](/en/linux-administration/linux-administration-shell-scripting-and-automation){.badge-link-text} | The previous module: shell scripting, traps, cron and systemd timers. |
| [Linux Administration #7: System Security](/en/linux-administration/linux-administration-system-security-and-hardening){.badge-link-text} | The next module: SSH hardening, firewalls, Fail2ban and LUKS encryption. |
| [Command Line Processor in Linux](/en/linux-beginners/command-line-processor-in-linux){.badge-link-text} | Fundamental knowledge about shells, I/O streams, pipes, and process redirections. |
## Conclusion
In this sixth part of our Linux administration series, you learned how to build and manage a reliable backup system. You now understand the various backup strategies, from full backups to incremental and differential backups to hardlink snapshots and deduplication, and know their respective strengths and challenges.
Backing up and recovering data is one of the most responsible tasks in the daily work of a Linux administrator. With the knowledge you have acquired, you can independently develop backup concepts, set up automated backup pipelines, and reliably recover data in an emergency. Particularly important is the understanding of the various tools and their applications, as well as the ability to continuously monitor backups and perform integrity checks.
The practical exercises showed you how to build a complete backup system, create encrypted archives, and secure them with monitoring scripts. Careful documentation and regular tests ensure that you can respond quickly and calmly in an emergency.
💡 **Practical Tip:** An untested backup is not a backup! Set up a quarterly emergency test (*Disaster Recovery Drill*) where you restore a backup to a fresh, isolated test VM and verify that all services and databases start without errors.
In the next module of our course, we turn to system hardening and defending against cyber attacks:
👉 **Next up:** [Linux Administration #7: System Security and Hardening](/en/linux-administration/linux-administration-system-security-and-hardening){.badge-link-text}
👉 **Course Overview:** [All Linux Administration Articles & Modules](/en/category/linux-administration){.badge-link-text}