Linux Administration #5: Shell Scripting and Automation

Welcome to the fifth part of our technical wiki series on Linux administration!

After we covered the fundamentals, process management, and network configuration in previous articles, we now turn to shell scripting and task automation.

As a Linux administrator, you spend a lot of time in the shell. The ability to automate tasks is an important key to efficient system administration. Think of shell scripting as a personal assistant who handles recurring tasks for you.

⚠️ Note: In this article, we use Ubuntu/Debian as the example distribution. The basic concepts are the same on all Linux systems, but package installation and some configuration paths may vary depending on the distribution. If you use a different distribution, please consult the relevant documentation for the specific installation commands and paths.

Understanding Shell Fundamentals

The shell is your main interface to the Linux system. Let's go through the most important concepts:

Shell types:


┌─────────── Bash (Standard) ─────────────────────────────────┐
│  • Bourne Again Shell                                       │
│  • Widely used                                              │
│  • Many features                                            │
├─────────── Zsh ─────────────────────────────────────────────┤
│  • Modern alternative                                       │
│  • Better auto-completion                                   │
│  • Extended features                                        │
└─────────────────────────────────────────────────────────────┘

Shell Environment


# Display shell version
echo $SHELL
echo $BASH_VERSION

# Important environment variables
echo $PATH        # Search paths for programs

echo $HOME        # User home directory

echo $USER        # Current user

Shell Configuration


# Configuration files
~/.bashrc        # Personal Bash configuration

~/.bash_profile  # Login shell configuration

~/.bash_history  # Command history
# Define aliases
alias ll='ls -la'
alias update='sudo apt update && sudo apt upgrade'

Shell Functions

Basic Shell Functions

echo # Output text

cd # Change directory

ls # List files

pwd # Current directory

cp # Copy files

mv # Move files

rm # Delete files

mkdir # Create directory

Shell functions:


┌─────────── Input/Output ────────────────────────────────────┐
│  • echo, cat, less                                          │
│  • read, printf                                             │
├─────────── Navigation ──────────────────────────────────────┤
│  • cd, pwd, ls                                              │
│  • find, locate                                             │
├─────────── File Operations ─────────────────────────────────┤
│  • cp, mv, rm                                               │
│  • mkdir, rmdir                                             │
└─────────────────────────────────────────────────────────────┘

Shell Scripting

After the shell fundamentals, we now turn to the details of shell scripting. Here we learn how to write effective scripts:

Basic Syntax


#!/bin/bash              # Shebang - defines the interpreter

# This is a comment
# Define variables
name="Anna"              # String

alter=25                 # Integer

datum=$(date +%Y-%m-%d)  # Command substitution
# Output
echo "Hello $name"       # Variable expansion

echo 'Hello $name'       # No variable expansion

Variables and Data Types


# String operations
text="Hello World"
echo ${#text}              # Length: 10

echo ${text:0:5}           # Substring: "Hello"

echo ${text/Welt/Linux}    # Replacement: "Hello Linux"
# Arrays
declare -a fruits=("apple" "pear" "orange")
echo ${fruits[0]}          # First element

echo ${fruits[@]}          # All elements

echo ${#fruits[@]}         # Array length

fruits+=("banana")         # Add element
# Integer operations
declare -i number=42
((number+=5))              # Arithmetic operation

let "result = number * 2"  # Alternative calculation

echo $((16 / 4))           # Direct calculation

Control Structures

Control structures are fundamental for programming logic. Let's examine the most important structures in detail:

1. If Conditions


# Basic syntax
if [ condition ]; then

# Actions
elif [ condition ]; then

# Alternative actions
else

# Fallback actions
fi

# Practical examples
# File check
if [ -f "/etc/hosts" ]; then
	echo "Hosts file exists"
	cat /etc/hosts
elif [ -f "/etc/hostname" ]; then
	echo "Only hostname file found"
else
	echo "No system files found"
fi

# Numeric comparisons
age=25
if [ $age -gt 18 ]; then
	echo "Of legal age"
	if [ $age -lt 67 ]; then
		echo "Working age"
	fi
fi

2. Loops


# For loop with list
for name in Anna Bob Charlie; do
	echo "Hello $name"
	mkdir -p /home/$name/documents
done

# For loop with number range
for i in {1..5}; do
	echo "Iteration $i"
	sleep 1
done

# While loop with condition
count=1
while [ $count -le 3 ]; do
	echo "Counter: $count"
	((count++))
	if [ $count -eq 2 ]; then
		echo "Halfway there!"
	fi
done

3. Case Statements


# Basic syntax
case $1 in
	start)
		echo "Starting service"
		systemctl start apache2
		;;
	stop)
		echo "Stopping service"
		systemctl stop apache2
		;;
	restart)
		echo "Restarting service"
		systemctl restart apache2
		;;
	*)
		echo "Unknown option"
		echo "Usage: $0 {start|stop|restart}"
		exit 1
		;;
esac

Functions

Functions are reusable code blocks that make your scripts more organized and maintainable:


# Basic function syntax
function name() {

# Code block
	echo "Execute function"
}

# Alternative syntax
name() {

# Code block
	echo "Execute function"
}

1. Functions with Parameters


# Use parameters in functions
check_user() {
	local username=$1    # First parameter

	local uid=$2         # Second parameter
# Check if user exists
	if id "$username" &>/dev/null; then
		echo "User $username exists"
		return 0
	else
		echo "User $username does not exist"
		return 1
	fi
}

# Call function
check_user "anna" 1001

2. Return Values and Exit Codes


# Function with return value
get_disk_usage() {
	local directory=$1
	local usage=$(df -h "$directory" | tail -n 1 | awk '{print $5}')

# Remove percent sign and return as number
	echo "${usage%\%}"

# Set exit code
	if [ "${usage%\%}" -gt 90 ]; then
		return 1    # Error: Over 90% full

	fi
	return 0       # Success

}

# Use function
if ! disk_usage=$(get_disk_usage "/"); then
	echo "Warning: Disk is too full ($disk_usage%)"
fi

Parameter Processing

Parameter processing is an important component for flexible and reusable scripts:

Parameter types:


┌─────────────────────────────────────────────────────────────┐
│  SHELL PARAMETER OVERVIEW                                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  $0           → Name of the executed script                 │
│  $1, $2, ...  → Positional parameters (arguments)           │
│  $#           → Number of passed arguments                  │
│  $@           → All arguments as separate list              │
│  $*           → All arguments as continuous string           │
│  $?           → Exit status of the last executed command     │
│  $$           → Process ID (PID) of the current shell       │
│                                                             │
└─────────────────────────────────────────────────────────────┘

1. Basic Parameter Processing


#!/bin/bash
# backup.sh - A backup script
# Check if parameters were passed
if [ $# -eq 0 ]; then
	echo "Usage: $0 <source-directory> <target-directory>"
	exit 1
fi

# Use parameters
source="$1"
target="$2"

# Check parameters
if [ ! -d "$source" ]; then
	echo "Error: Source directory does not exist"
	exit 1
fi

2. Advanced Parameter Processing


#!/bin/bash
# Parameters with default values
target=${2:-"/backup"}        # Default if $2 is empty

verbose=${VERBOSE:-false}   # Environment variable or false
# Iterate over all parameters
for param in "$@"; do
	echo "Processing: $param"
done

# Count parameters
echo "Number of parameters: $#"

# Exit status of last command
if [ $? -eq 0 ]; then
	echo "Last command successful"
fi

Error Handling

Error handling is essential for robust shell scripts. Here are the most important concepts and techniques:

Error handling:


┌─────────── Exit Codes ──────────────────────────────────────┐
│  0     = Success                                             │
│  1-255 = Various errors                                      │
├─────────── Checking ────────────────────────────────────────┤
│  $?    = Last exit code                                      │
│  &&    = AND connection                                      │
│  ||    = OR connection                                       │
└─────────────────────────────────────────────────────────────┘

1. Exit Codes and Error Checking


#!/bin/bash
# Basic error handling
# Function with error checking
check_file() {
	local file="$1"

	if [ ! -f "$file" ]; then
		echo "Error: File '$file' not found!" >&2
		return 1
	fi
	return 0
}

# Example with exit code checking
if ! check_file "config.txt"; then
	echo "Configuration file missing!" >&2
	exit 1
fi

2. Advanced Error Handling


#!/bin/bash
# Catch and log errors
# Set up error logging
log_error() {
	local message="$1"
	echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') - $message" >&2
	echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') - $message" >> error.log
}

# set options for better error handling
set -e    # Exit script on error

set -u    # Error on undefined variables

set -o pipefail  # Catch errors in pipes
# Trap for cleanup
trap 'echo "Script is being terminated..."; rm -f /tmp/tempfile' EXIT

💡 Tip: For a deeper introduction to shell scripting, we recommend our Bash Basics Course. There you will learn step by step how to create and manage effective shell scripts.

Practical Examples

Error handling is crucial for robust shell scripts. Here are practical examples:

1. Error Handling in Functions


#!/bin/bash
# Function with error handling
backup_files() {
	local source="$1"
	local target="$2"

# Check if source directory exists
	if [ ! -d "$source" ]; then
		echo "Error: Source directory '$source' not found!" >&2
		return 1
	fi

# Check write permissions in target directory
	if [ ! -w "$target" ]; then
		echo "Error: No write permissions in '$target'!" >&2
		return 2
	fi

# Perform backup
	cp -r "$source"/* "$target"/ || {
		echo "Error copying files!" >&2
		return 3
	}

	return 0
}

2. Error Logs and Debugging


#!/bin/bash
# Enable debugging
set -x    # Show executed commands

trap 'echo "Line $LINENO: Command failed!"' ERR
# Error logging
log_error() {
	local message="$1"
	echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: $message" >> error.log
	return 1
}

# Example for error handling
if ! mkdir -p /path/to/directory; then
	log_error "Could not create directory"
	exit 1
fi

3. Debugging Techniques


# Enable debug mode
set -x          # Show every command before execution

set -e          # Exit script on errors

set -u          # Error on undefined variables

set -o pipefail # Show errors in pipes
# Debug output
echo "DEBUG: Variable=$variable" >&2
logger "DEBUG: Script reached point A"

# Step-by-step execution
bash -x ./script.sh

4. Error Logs and Logging


# Custom logging function
log() {
	local level="$1"
	shift
	echo "[$(date +'%Y-%m-%d %H:%M:%S')] [$level] $*" >> script.log
}

# Usage
log "INFO" "Script started"
log "ERROR" "File not found"

Automation

After the shell scripting fundamentals, we now turn to the practical automation of tasks. Here we learn how to efficiently automate recurring tasks:

Automation concepts:


┌─────────── Backup Scripts ──────────────────────────────────┐
│  • Back up files                                             │
│  • Rotate logs                                               │
│  • Clean up                                                  │
├─────────── Monitoring ──────────────────────────────────────┤
│  • Monitor system                                            │
│  • Check resources                                           │
│  • Send alerts                                               │
└─────────────────────────────────────────────────────────────┘

Automating System Maintenance


#!/bin/bash
# system_maintenance.sh
# Define variables
LOG_DIR="/var/log"
BACKUP_DIR="/backup"
MAX_LOG_DAYS=30

# Clean up old logs
find "$LOG_DIR" -type f -name "*.log" -mtime +$MAX_LOG_DAYS -delete

# Perform system updates
apt update && apt upgrade -y

# Create backup
tar -czf "$BACKUP_DIR/backup_$(date +%Y%m%d).tar.gz" /home/user/data/

More Automation Examples

Here are practical examples for typical automation tasks:

1. Log Rotation and Cleanup


#!/bin/bash
# cleanup_logs.sh - Clean up and compress old logs
# Variables
LOG_DIR="/var/log"
MAX_DAYS=30
ARCHIVE_DIR="/backup/logs"

# Create directory for archives
mkdir -p "$ARCHIVE_DIR"

# Find and archive old logs
find "$LOG_DIR" -name "*.log" -type f -mtime +$MAX_DAYS -exec gzip -c {} \; \
	-exec mv {} "$ARCHIVE_DIR/" \;

# Delete very old archives (older than 90 days)
find "$ARCHIVE_DIR" -type f -mtime +90 -delete

2. System Monitoring


#!/bin/bash
# system_monitor.sh - Monitor important system resources
# Thresholds
DISK_THRESHOLD=90
MEM_THRESHOLD=80
CPU_THRESHOLD=90

# Monitoring function
check_resources() {

# Disk usage
	disk_usage=$(df -h / | tail -n1 | awk '{print $5}' | tr -d '%')

# RAM usage
	memory_usage=$(free | grep Mem | awk '{print $3/$2 * 100}' | cut -d. -f1)

# CPU load
	cpu_load=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d. -f1)

# Send warnings
	if [ "$disk_usage" -gt "$DISK_THRESHOLD" ]; then
		echo "WARNING: Disk is ${disk_usage}% full!" | mail -s "Disk Alert" admin@domain.com
	fi
}

3. Resource Monitoring and Alerts


#!/bin/bash
# monitor_and_alert.sh - Monitor system resources and send warnings
# Thresholds
DISK_THRESHOLD=90
MEM_THRESHOLD=80
CPU_THRESHOLD=90
LOAD_THRESHOLD=4

# Alert function
send_alert() {
	local message="$1"
	local subject="$2"

# Via email
	echo "$message" | mail -s "$subject" admin@domain.com

# Via log
	logger -p user.warning "$subject: $message"

# Optional: Via Slack/Discord/Teams
# curl -X POST -H 'Content-type: application/json' \
#    --data "{\"text\":\"$message\"}" \
#    $WEBHOOK_URL
}

# Check resources
check_resources() {

# Disk usage
	local disk_usage=$(df -h / | tail -n1 | awk '{print $5}' | tr -d '%')
	if [ "$disk_usage" -gt "$DISK_THRESHOLD" ]; then
		send_alert "Disk is ${disk_usage}% full!" "Disk Alert"
	fi

# RAM usage
	local memory_usage=$(free | grep Mem | awk '{print $3/$2 * 100}' | cut -d. -f1)
	if [ "$memory_usage" -gt "$MEM_THRESHOLD" ]; then
		send_alert "RAM usage at ${memory_usage}%!" "Memory Alert"
	fi

# CPU load
	local load_average=$(uptime | awk '{print $(NF-2)}' | tr -d ',')
	if [ "$(echo "$load_average > $LOAD_THRESHOLD" | bc)" -eq 1 ]; then
		send_alert "Load Average too high: ${load_average}!" "CPU Alert"
	fi
}

# Main program
while true; do
	check_resources
	sleep 300  # Check every 5 minutes

done

4. Process Automation


#!/bin/bash
# process_automation.sh
# Monitor and restart processes
check_and_restart_process() {
	local process_name="$1"
	if ! pgrep "$process_name" >/dev/null; then
		systemctl restart "$process_name"
		logger "Process $process_name was restarted"
	fi
}

# Monitor important services
check_and_restart_process "nginx"
check_and_restart_process "mysql"

5. Network Automation


#!/bin/bash
# network_automation.sh
# Check network connection
check_network() {
	if ! ping -c 1 8.8.8.8 >/dev/null; then
		systemctl restart networking
		logger "Network was restarted"
	fi
}

# Monitor VPN connection
check_vpn() {
	if ! ip link show tun0 >/dev/null 2>&1; then
		systemctl restart openvpn
		logger "VPN was restarted"
	fi
}

Cron Jobs and Time-based Tasks

Time-based execution of tasks is an important part of automation. Cron provides a flexible way to execute scripts and commands at specific times:

Cron time format:


┌─────────────────────────────────────────────────────────────┐
│  CRON SYNTAX TIME FORMAT                                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌───────────── Minute (0 - 59)                             │
│  │ ┌───────────── Hour (0 - 23)                             │
│  │ │ ┌───────────── Day of month (1 - 31)                   │
│  │ │ │ ┌───────────── Month (1 - 12)                        │
│  │ │ │ │ ┌───────────── Weekday (0 - 7, 0/7 = Sunday)       │
│  │ │ │ │ │                                                  │
│  * * * * *  /path/to/script.sh                               │
│                                                             │
└─────────────────────────────────────────────────────────────┘

1. Cron Configuration


# Edit crontab
crontab -e

# Example entries:
# Every hour
0 * * * * /path/to/hourly.sh

# Every day at 3:30 AM
30 3 * * * /path/to/daily.sh

# Every Monday at 8 AM
0 8 * * 1 /path/to/weekly.sh

Advanced Cron Functions

Advanced cron functions offer even more possibilities for automation:

1. Special Cron Expressions


# Special time specifications
@yearly    # Once per year (0 0 1 1 *)

@monthly   # Once per month (0 0 1 * *)

@weekly    # Once per week (0 0 * * 0)

@daily     # Once per day (0 0 * * *)

@hourly    # Once per hour (0 * * * *)

@reboot    # After every reboot

2. Environment Variables in Cron


# In crontab -e:
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=admin@domain.com

# Backup with log
0 2 * * * /scripts/backup.sh >> /var/log/backup.log 2>&1

3. Error Handling and Logging


# Redirect errors to separate file
30 1 * * * /scripts/cleanup.sh > /var/log/cleanup.log 2> /var/log/cleanup.error

# Suppress output
0 * * * * /scripts/check.sh >/dev/null 2>&1

# Log with timestamp
15 * * * * echo "$(date '+\%Y-\%m-\%d \%H:\%M:\%S') - Start" >> /var/log/cron.log && /scripts/task.sh

Automating System Tasks

After shell scripting and cron configuration, we come to the practical automation of system tasks. These tasks are essential for smooth system operation:

System tasks:


┌─────────── Maintenance ─────────────────────────────────────┐
│  • Log rotation                                              │
│  • Backup management                                         │
│  • Cleanup work                                              │
├─────────── Updates ─────────────────────────────────────────┤
│  • Package updates                                           │
│  • Security updates                                          │
│  • System upgrades                                           │
└─────────────────────────────────────────────────────────────┘

Log Rotation


#!/bin/bash
# logrotate.sh - Automatic log management
# Variables
LOG_DIR="/var/log"
MAX_DAYS=30
ARCHIVE_DIR="/backup/logs"

# Archive old logs
find "$LOG_DIR" -name "*.log" -type f -mtime +$MAX_DAYS -exec gzip {} \;
find "$LOG_DIR" -name "*.gz" -type f -mtime +90 -delete

# Log status
logger "Log rotation performed: $(date)"

Update Automation

Automatic system updates are important for security and stability. Here is a detailed example:

1. Automating System Updates


#!/bin/bash
# auto_update.sh - Automatic system updates
# Variables
LOG_FILE="/var/log/system_updates.log"
MAIL_TO="admin@domain.com"
UPDATE_TIMEOUT=1800  # 30 minutes timeout

# Logging function
log_message() {
	echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Perform updates
perform_updates() {
	log_message "Starting system update"

# APT update cache
	if ! apt-get update; then
		log_message "ERROR: apt-get update failed"
		return 1
	fi

# Security updates first
	if ! DEBIAN_FRONTEND=noninteractive apt-get -y --with-new-pkgs upgrade; then
		log_message "ERROR: Security updates failed"
		return 2
	fi

# Cleanup
	apt-get clean
	apt-get autoremove -y

	log_message "System update completed successfully"
	return 0
}

2. Reboot Management


# Check if reboot is required
check_reboot() {
	if [ -f /var/run/reboot-required ]; then
		log_message "System reboot required"

# Only reboot at night
		if [ $(date +%H) -ge 2 ] && [ $(date +%H) -le 4 ]; then
			log_message "Performing scheduled reboot"
			/sbin/shutdown -r +5 "System reboot after updates"
		fi
	fi
}

Exercise

In this exercise, we will create a practical automation script that monitors and logs various system tasks.

Scenario: You are a Linux administrator in a small company and need to set up a monitoring system that monitors important system resources and automatically responds to problems.

The script should:

Monitor system resources (CPU, RAM, disk)
Send warnings when thresholds are exceeded
Create and rotate logs
Automatically run as a cron job

Requirements

Monitoring requirements:


┌─────────── Monitoring ──────────────────────────────────────┐
│  • CPU load > 80%                                            │
│  • RAM usage > 90%                                           │
│  • Disk > 85%                                                │
├─────────── Actions ─────────────────────────────────────────┤
│  • Email notification                                        │
│  • Logging of all events                                     │
│  • Automatic log rotation                                    │
└─────────────────────────────────────────────────────────────┘

Possible solution


#!/bin/bash
# monitor.sh - System monitoring script
# Thresholds
CPU_THRESHOLD=80
MEM_THRESHOLD=90
DISK_THRESHOLD=85

# Monitoring function
check_resources() {

# Check CPU load
	cpu_load=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d. -f1)

# Check memory usage
	mem_used=$(free | grep Mem | awk '{print $3/$2 * 100}' | cut -d. -f1)

# Check disk usage
	disk_used=$(df / | tail -n1 | awk '{print $5}' | tr -d '%')

# Logging and alerts
	if [ "$cpu_load" -gt "$CPU_THRESHOLD" ]; then
		echo "WARNING: CPU load at ${cpu_load}%" | mail -s "CPU Alert" admin@company.de
	fi
}

Checklist

Implementation checklist:


┌─────────── Basic Functions ─────────────────────────────────┐
│  □ Monitoring functions                                      │
│  □ Alert system                                              │
│  □ Logging mechanism                                         │
├─────────── Automation ──────────────────────────────────────┤
│  □ Set up cron job                                           │
│  □ Activate log rotation                                     │
│  □ Error handling                                            │
└─────────────────────────────────────────────────────────────┘

Command Reference (Cheatsheet)

For quick access during script development and automation, the following reference table summarizes the most important constructs and commands:

Command / Syntax	Category	Function & Description
`#!/usr/bin/env bash`	Shebang	Portable standard interpreter call for Bash scripts.
`set -euo pipefail`	Robustness	Aborts on errors, unset variables, and pipe errors.
`trap '<command>' EXIT ERR`	Signal Handling	Executes cleanup functions on script end or error case.
`crontab -e`	Cron	Opens the personal crontab in the configured default editor.
`crontab -l`	Cron	Lists all active cron jobs of the current user.
`0 2 * <command>`	Cron	Cron time format: Executes command daily at 2:00 AM.
`systemctl list-timers`	systemd	Shows active systemd timers with status and next execution time.
`logger -t <TAG> "<msg>"`	Logging	Writes messages directly to the system journal (`syslog` / `journald`).
`getopts ":vd:h" opt`	CLI Parsing	Processes flags and parameters in shell scripts.
`local var="value"`	Functions	Restricts variable scope strictly to the function.
`bash -x <script.sh>`	Debugging	Executes script in tracing mode with display of every step.
`${VAR:-fallback}`	Parameter Expansion	Uses default value if `VAR` is unset or empty.
`date +'%Y-%m-%d_%H-%M'`	Timestamp	Generates sortable timestamps for backups and log files.

Further Resources

The following guides, specifications, and internal course modules deepen shell scripting and system automation:

Resource	Description
Bash Reference Manual	Official GNU Bash reference documentation of the Free Software Foundation.
systemd.timer Documentation	Official documentation for systemd timers and calendar events.
Bash Basics #1: First Script	Fundamental introduction to Bash script programming.
Linux Administration #4: Networking	The previous module: IP routing, netplan, nmcli, DNS & UFW.
Linux Administration #6: Data Backup	The next module: backup strategies, rsync, Borg & recovery.
Command Line Processor in Linux	Architecture of shell, TTY, pipes, and I/O streams.

Conclusion

Shell scripting and automation form the backbone of modern Linux administration. Through the consistent use of best practices like set -euo pipefail, modular functions with local variables, reliable error handling with trap, flexible parameter processing, and periodic execution via cron or systemd timers, you transform error-prone manual routine tasks into reproducible, fault-resistant background processes.

💡 Practical Tip: Start every production Bash script without exception with set -euo pipefail and a trap cleanup function for temporary files under /tmp/. This way, you prevent unnoticed errors or unset variables from causing devastating collateral damage on your production server.

In the next module of our administration course, we turn to securing your server data: 👉 Next up: Linux Administration #6: Data Backup and Recovery

👉 Course Overview: All Linux Administration Articles & Modules