Arch Linux Advanced Security Features and Maintenance

Security: Mandatory Access Control with AppArmor

Why AppArmor?

Discretionary Access Control (DAC) – the standard Linux permission model with rwx for user, group, others – has a fundamental weakness: once a process runs as root, it can access everything. A compromised web server with CAP_NET_BIND_SERVICE and root access can read /etc/shadow, mount filesystems, or load kernel modules. DAC offers no barrier.

Mandatory Access Control (MAC) closes this gap. The kernel enforces a policy that applies to all processes – including root. Even a fully compromised service can only access resources explicitly allowed in its profile.


┌─────────────────────────────────────────────────────────────┐
│ DAC vs. MAC: ACCESS CONTROL COMPARISON                      │
├─────────────────────────────────────────────────────────────┤
│ DAC (Discretionary Access Control):                         │
│ • Owner decides who gets access                             │
│ • root bypasses all restrictions                            │
│ • Weakness: One compromised process = full system access    │
│                                                             │
│ MAC (Mandatory Access Control):                             │
│ • Kernel enforces policy for ALL processes                  │
│ • root is ALSO subject to MAC rules                         │
│ • Strength: root compromise -> limited damage               │
└─────────────────────────────────────────────────────────────┘

Arch Linux supports two MAC frameworks: AppArmor (profile-based, Ubuntu/Debian standard) and SELinux (label-based, Red Hat/Fedora standard). AppArmor is the better choice for Arch because:

Simpler configuration: text-based profiles instead of complex type enforcement
Faster learning curve: aa-genprof generates profiles from running processes
Arch Wiki has extensive AppArmor documentation
Kernel linux-hardened includes AppArmor support out of the box

AppArmor in Practice

Installation and activation:


# Install AppArmor
sudo pacman -S apparmor

# Enable the service
sudo systemctl enable --now apparmor

# Verify status
sudo aa-status
# apparmor module is loaded.
# 37 profiles are loaded.
# 37 profiles are in enforce mode.

First steps: Learn mode for existing services

Before blocking anything, AppArmor must learn what files and capabilities a service actually needs:


# Start nginx in learn mode
sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx
# Change to complain mode (log violations, don't block)
sudo aa-complain /etc/apparmor.d/usr.sbin.nginx

# Restart the service and exercise normal operations
sudo systemctl restart nginx
curl -s https://localhost/ > /dev/null
# Access all vhosts, static files, PHP applications

# Check learn mode logs
sudo journalctl -u apparmor --since "1 hour ago" | grep "profile"

Create a profile from learn mode output:


# Generate profile from learn mode
sudo aa-genprof /usr/sbin/nginx
# Scanning system logs for AppArmor events...
#
# The following local profile file was generated:
#   /etc/apparmor.d/usr.sbin.nginx

Verify and activate:


# Verify profile syntax
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx

# Switch to enforce mode
sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx

# Verify
sudo aa-status | grep nginx
# 1 profiles are in enforce mode.
#   /usr/sbin/nginx

Custom profile for a web application:


sudo nano /etc/apparmor.d/usr.sbin.mywebapp


#include <tunables/global>

/usr/sbin/mywebapp {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/openssl>

  # Binary
  /usr/sbin/mywebapp mr,

  # Configuration
  /etc/mywebapp/** r,
  /etc/mywebapp/config.yml r,

  # Data directory
  /var/lib/mywebapp/** rwk,
  /var/lib/mywebapp/tmp/ rw,

  # Logs
  /var/log/mywebapp/** w,
  /run/mywebapp.pid rw,

  # Network
  network inet stream,
  network inet6 stream,

  # Deny dangerous capabilities
  deny capability sys_admin,
  deny capability sys_rawio,
  deny capability sys_module,

  # Deny write to sensitive paths
  deny /etc/shadow w,
  deny /etc/passwd w,
  deny /etc/sudoers w,
  deny /proc/sys/** w,
}


# Load and activate profile
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mywebapp
sudo aa-enforce /etc/apparmor.d/usr.sbin.mywebapp

# Verify
sudo aa-status | grep mywebapp

⚠️ Restart required: AppArmor profiles are only loaded when the service starts. After profile changes, always restart the affected service: sudo systemctl restart nginx.

AppArmor vs. SELinux

Both frameworks solve the same problem, but with different approaches:

Feature	AppArmor	SELinux
Configuration	Text-based profiles	Type enforcement (labels)
Learning curve	Low	High
Granularity	Per-binary	Per-process + per-file context
Default on Arch	Optional (install required)	Optional (install required)
`linux-hardened` support	✅	✅
Real-world deployment	Ubuntu, SUSE, Debian	RHEL, Fedora, CentOS
Debugging tools	`aa-status`, `aa-logprof`	`audit2allow`, `sealert`

💡 Recommendation for Arch: AppArmor. The simpler configuration and faster debugging make it the better choice for small to medium Arch deployments. SELinux is better suited for enterprise environments with dedicated security teams.

Kernel Hardening with linux-hardened

Why a hardened kernel?

The standard Linux kernel is optimized for compatibility and performance – not for maximum security. Many security-relevant features are either disabled or configured conservatively by default. The linux-hardened kernel patch set enables these features and adds additional protections.


┌─────────────────────────────────────────────────────────────┐
│ linux-hardened vs. standard kernel                          │
├─────────────────────────────────────────────────────────────┤
│ STANDARD KERNEL:                                            │
│ • Defaults for compatibility (weaker security settings)     │
│ • Many hardening options disabled or configurable only      │
│ • No compile-time restrictions                              │
│                                                             │
│ linux-hardened:                                             │
│ • Security features enabled by default                      │
│ • Compile-time restrictions (e.g., no module loading)       │
│ • Additional patches (e.g., stronger ASLR, restricted      │
│   /proc visibility)                                         │
└─────────────────────────────────────────────────────────────┘

Installation:


# Install linux-hardened
sudo pacman -S linux-hardened linux-hardened-headers

# GRUB: Set as default kernel
sudo nano /etc/default/grub
# GRUB_DEFAULT=saved
# GRUB_SAVEDEFAULT=true

# Set linux-hardened as default
sudo grub-set-default "Arch Linux, with linux-hardened linux"

# Update GRUB
sudo grub-mkconfig -o /boot/grub/grub.cfg

# Reboot into hardened kernel
sudo reboot

# Verify
uname -r
# 7.1.9-hardened1-1-hardened

Kernel Parameters for Maximum Security

sysctl configuration:


sudo nano /etc/sysctl.d/99-hardening.conf


# Kernel Hardening Parameters

# 1. ASLR (Address Space Layout Randomization)

# Full randomization (0=off, 1=half, 2=full)
kernel.randomize_va_space = 2

# 2. Process hiding

# Hide kernel pointers from unprivileged users
kernel.kptr_restrict = 2

# Restrict dmesg access
kernel.dmesg_restrict = 1

# Restrict /proc visibility
kernel.yama.ptrace_scope = 1

# 3. Network hardening

# Disable IP forwarding (unless router)
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0

# Disable source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0

# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# SYN flood protection
net.ipv4.tcp_syncookies = 1

# 4. File system hardening

# Restrict symlinks
fs.protected_symlinks = 1
fs.protected_hardlinks = 1

# Restrict FIFO and regular file in world-writable dirs
fs.protected_fifos = 2
fs.protected_regular = 2


# Apply sysctl changes
sudo sysctl --system

# Verify key settings
cat /proc/sys/kernel/randomize_va_space
# 2

cat /proc/sys/kernel/kptr_restrict
# 2

Boot parameters for additional security:


sudo nano /etc/default/grub


# Add to GRUB_CMDLINE_LINUX_DEFAULT:
# quiet loglevel=3 slab_nomerge init_on_alloc=1 init_on_free=1
# page_alloc.shuffle=1 randomize_kstack_offset=on
# vsyscall=none debugfs=off


# Update GRUB
sudo grub-mkconfig -o /boot/grub/grub.cfg

# Reboot
sudo reboot

Parameter	Effect
`slab_nomerge`	Prevents slab merging (hardens heap exploitation)
`init_on_alloc=1`	Zero memory on allocation (prevents info leaks)
`init_on_free=1`	Zero memory on free (prevents use-after-free)
`page_alloc.shuffle=1`	Randomizes page allocator (hardens heap layout)
`randomize_kstack_offset=on`	Randomizes kernel stack offset per syscall
`vsyscall=none`	Disables legacy vsyscall page (attack surface reduction)
`debugfs=off`	Disables debugfs (attack surface reduction)

⚠️ Test first: Not all sysctl and boot parameters work with every hardware combination. Test on a non-production system first, especially init_on_alloc=1 and init_on_free=1, which can impact performance.

Container Security with Podman

Rootless Containers

Docker's daemon runs as root – a single container escape gives the attacker full host access. Podman eliminates this architecture with rootless containers: each user runs their own Podman process, with no central daemon and no root requirement.


┌─────────────────────────────────────────────────────────────┐
│ DOCKER vs. PODMAN: ARCHITECTURE                             │
├─────────────────────────────────────────────────────────────┤
│ DOCKER:                                                     │
│ • dockerd runs as root (PID 1 or systemd-managed)          │
│ • Container processes run under root's namespace            │
│ • Single point of failure: compromised daemon = full host  │
│                                                             │
│ PODMAN (rootless):                                          │
│ • No central daemon                                         │
│ • Each user runs their own containers                       │
│ • User namespaces: container root != host root             │
│ • No privilege escalation possible (in theory)             │
└─────────────────────────────────────────────────────────────┘

Rootless setup:


# Install Podman
sudo pacman -S podman

# Create subuid/subgid mappings for your user
sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER

# Verify
cat /etc/subuid
# sebastian:100000:65536

cat /etc/subgid
# sebastian:100000:65536

# Run a rootless container
podman run --rm -it alpine sh
# You are now "root" inside the container, but a regular user on the host
id
# uid=0(root) gid=0(root)

# On the host:
ps aux | grep container
# sebastian  12345  ... podman run --rm -it alpine sh
# (running as regular user, NOT root)

SELinux Integration

If SELinux is available, Podman supports --selinux-enabled for additional isolation:


# Check SELinux status
getenforce
# Enforcing (if configured)

# Run with SELinux labels
podman run --rm --security-opt label=type:container_t alpine sh

# Verify label
ps -eZ | grep container
# system_u:system_r:container_t:s0:c123,c456  12345  ... podman run ...

Seccomp and AppArmor for Containers

Seccomp: Restrict system calls


# Default seccomp profile (already applied by Podman)
podman run --rm alpine cat /proc/self/status | grep Seccomp
# Seccomp:	2
# Seccomp_filters:	1

AppArmor for containers:


# Run with AppArmor profile
podman run --rm --security-opt apparmor=my-custom-profile alpine sh

# Check which AppArmor profiles are loaded
podman info | grep -i apparmor
#   apparmorEnabled: true

Container Networking with nftables

Docker creates default bridge networks with automatic NAT – convenient but dangerous. Containers can reach each other, the host, and external networks without restrictions. Podman's rootless networking is more restrictive by default, but explicit rules are still necessary for production.

nftables rules for container isolation:


sudo nano /etc/nftables.d/containers.conf


#!/usr/sbin/nft -f
# Container Network Segmentation

flush ruleset

# 1. Define container networks

# Docker/Podman default bridge
define PODMAN_BRIDGE = 10.89.0.0/24

# Custom container networks
define WEB_CONTAINERS = 10.89.1.0/24
define DB_CONTAINERS = 10.89.2.0/24
define MONITOR_CONTAINERS = 10.89.3.0/24

# 2. Default policy: deny all

table inet container-filter {
    chain input {
        type filter hook input priority 0; policy drop;

        # Allow established connections
        ct state established,related accept

        # Allow loopback
        iif "lo" accept

        # Allow ICMP (ping)
        ip protocol icmp accept
        ip6 nexthdr icmpv6 accept
    }

    chain forward {
        type filter hook forward priority 0; policy drop;

        # Allow established connections
        ct state established,related accept

        # 3. Inter-container rules

        # Web -> DB (only specific ports)
        ip saddr $WEB_CONTAINERS ip daddr $DB_CONTAINERS \
            tcp dport { 5432, 6379 } accept comment "web to db"

        # Web -> Internet (HTTP/HTTPS only)
        ip saddr $WEB_CONTAINERS \
            tcp dport { 80, 443 } accept comment "web to internet"

        # DB -> NO internet
        # (default drop covers this)

        # Monitor -> all containers (read-only)
        ip saddr $MONITOR_CONTAINERS \
            tcp dport { 9100, 9090, 3000 } accept comment "monitor scrape"

        # 4. Deny everything else
    }
}


# Load rules
sudo nft -f /etc/nftables.d/containers.conf

# Verify
sudo nft list ruleset | grep "container-filter"


┌─────────────────────────────────────────────────────────────┐
│ CONTAINER NETWORK SEGMENTATION                              │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│  │ Web         │    │ DB          │    │ Monitor     │     │
│  │ 10.89.1.0/24│    │ 10.89.2.0/24│    │ 10.89.3.0/24│     │
│  └──────┬──────┘    └──────┬──────┘    └──────┬──────┘     │
│         │                  │                  │            │
│         │   ┌──────────────┼──────────────────┘            │
│         │   │              │                               │
│         ▼   ▼              ▼                               │
│  ┌─────────────────────────────────────────────┐           │
│  │ nftables (inet container-filter)            │           │
│  │ • Web -> DB: 5432, 6379 only                │           │
│  │ • Web -> Internet: 80, 443 only             │           │
│  │ • DB -> Internet: BLOCKED                   │           │
│  │ • Monitor -> all: 9100, 9090, 3000          │           │
│  │ • Everything else: DROP                    │           │
│  └─────────────────────────────────────────────┘           │
└─────────────────────────────────────────────────────────────┘

Logging, Debugging and Troubleshooting

Audit logs for dropped container packets:


# /etc/rsyslog.d/10-container.conf
:msg, contains, "[CONTAINER-" /var/log/container-firewall.log
& stop

Logrotate configuration:


# /etc/logrotate.d/container-firewall
/var/log/container-firewall.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 root root
}

💡 Debugging with tcpdump: When containers have connectivity issues, capture traffic directly on the interface: sudo tcpdump -i docker0 -n.

Multi-Host Logging: Central Logs with TLS Forwarding

Why Central Logs?

An attacker who compromises a server will delete local log files first. ~/.bash_history, /var/log/auth.log, /var/log/syslog – everything gets overwritten or deleted. Without central logs, you have no forensic foundation left.

Central logs solve three problems:

Prevent anti-forensics: Logs on the log server are unreachable by the attacker
Correlation: Detect warnings across multiple hosts (e.g., simultaneous SSH logins on web and DB servers)
Compliance: GDPR and the German IT Security Act require auditable access logs with retention periods


┌─────────────────────────────────────────────────────────────┐
│ MULTI-HOST LOGGING: TLS FORWARDING TOPOLOGY                 │
├─────────────────────────────────────────────────────────────┤
│ HOST A (Web)         HOST B (DB)          HOST C (Monitor)  │
│ ┌─────────────┐     ┌─────────────┐      ┌─────────────┐    │
│ │ rsyslog     │     │ rsyslog     │      │ rsyslog     │    │
│ │ (Sender)    │     │ (Sender)    │      │ (Sender)    │    │
│ └──────┬──────┘     └──────┬──────┘      └──────┬──────┘    │
│        │                   │                    │           │
│        │   mTLS (Port 6514)│   mTLS (Port 6514) │           │
│        └───────────────────┼────────────────────┘           │
│                            │                                │
│                 ┌──────────┴──────────┐                     │
│                 │ LOG-SERVER (Host D) │                     │
│                 │ rsyslog Receiver    │                     │
│                 │ /logs/{web,db,...}  │                     │
│                 └─────────────────────┘                     │
│                                                             │
│ ADVANTAGE: Host compromise keeps logs on D untouchable.     │
└─────────────────────────────────────────────────────────────┘

Certificates for TLS Logging

TLS forwarding protects logs from being read on the network. Each host needs a client certificate, the log server needs a server certificate. We create our own CA (Certificate Authority) for this.

Step 1: Create CA


# Create directory for certificates
sudo mkdir -p /etc/rsyslog/ca
cd /etc/rsyslog/ca

# Generate CA private key (4096 bit, RSA)
sudo openssl genrsa -out ca.key 4096

# Generate CA certificate (valid for 10 years)
sudo openssl req -x509 -new -nodes -key ca.key \
    -sha256 -days 3650 \
    -out ca.crt \
    -subj "/C=DE/ST=Bayern/L=Muenchen/O=Homelab/CN=Log-CA"

# Verification
sudo openssl x509 -in ca.crt -text -noout | grep -A2 "Issuer"
# Issuer: C = DE, ST = Bayern, L = Muenchen, O = Homelab, CN = Log-CA

Step 2: Server certificate for log receiver


cd /etc/rsyslog/ca

# Generate server key
sudo openssl genrsa -out server.key 4096

# Generate CSR (Certificate Signing Request)
sudo openssl req -new -key server.key \
    -out server.csr \
    -subj "/C=DE/ST=Bayern/L=Muenchen/O=Homelab/CN=logserver.local"

# Sign server certificate (valid for 5 years)
sudo openssl x509 -req -in server.csr \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -out server.crt -days 1825 -sha256

# Add SAN (Subject Alternative Name)
# WARNING: Without SAN, rsyslog refuses the connection!
sudo openssl x509 -in server.crt -out server-san.crt \
    -extfile <(printf "subjectAltName=DNS:logserver.local,DNS:logserver,IP:192.168.1.12")
sudo mv server-san.crt server.crt

Step 3: Client certificates for each host


# Generate a certificate for each host
for HOST in webserver dbserver monitor; do
    sudo openssl genrsa -out ${HOST}.key 4096
    sudo openssl req -new -key ${HOST}.key \
        -out ${HOST}.csr \
        -subj "/C=DE/ST=Bayern/L=Muenchen/O=Homelab/CN=${HOST}.local"
    sudo openssl x509 -req -in ${HOST}.csr \
        -CA ca.crt -CAkey ca.key -CAcreateserial \
        -out ${HOST}.crt -days 1825 -sha256
done

Step 4: Distribute certificates to hosts


# On each host:
sudo mkdir -p /etc/rsyslog/certs

# CA certificate (on all hosts)
sudo cp ca.crt /etc/rsyslog/certs/

# Server certificate (only on log server)
sudo cp server.key server.crt /etc/rsyslog/certs/

# Client certificate (on the respective host)
sudo cp webserver.key webserver.crt /etc/rsyslog/certs/

Step 5: Verification


# Check if certificate is signed by CA
sudo openssl verify -CAfile ca.crt server.crt
# server.crt: OK

sudo openssl verify -CAfile ca.crt webserver.crt
# webserver.crt: OK

# Display certificate details
sudo openssl x509 -in server.crt -text -noout | grep -A2 "Subject:"
# Subject: C = DE, ST = Bayern, L = Muenchen, O = Homelab, CN = logserver.local

⚠️ SAN is mandatory: Without subjectAltName in the server certificate, rsyslog refuses the TLS connection with SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed. The SAN must contain the hostname AND the IP address of the log server.

Configure rsyslog Sender

On each host that sends logs to the log server:


sudo nano /etc/rsyslog.d/50-forward.conf


# rsyslog TLS Forwarding (Sender)

# Load global modules
module(load="imuxsock")    # Local logs
module(load="imklog")      # Kernel logs
module(load="imfile")      # File monitoring
module(load="omfwd")       # Forwarding

# TLS transport definition

# TLS certificates
$DefaultNetstreamDriverCAFile /etc/rsyslog/certs/ca.crt
$DefaultNetstreamDriverCertFile /etc/rsyslog/certs/webserver.crt
$DefaultNetstreamDriverKeyFile /etc/rsyslog/certs/webserver.key

# Enable TLS transport
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/certvalid

# Forwarding rules

# Send all local logs to log server
*.* @@(omfwd)logserver.local:6514;RSYSLOG_SyslogProtocol23Format

# Queueing for reliability

# Queue type: Disk-assisted (overflow to disk)
$ActionQueueType LinkedList
$ActionQueueFileName fwd_queue
$ActionQueueMaxDiskSpace 256m
$ActionQueueSaveOnShutdown on
$ActionQueueTimeoutEnqueue 100
$ActionResumeRetryCount -1
$ActionResumeInterval 30

# Rate limiting against log flooding

# Maximum 1000 messages per second
$NetstatsRateLimit 1000


# Verification: Check syntax
sudo rsyslogd -N1
# rsyslogd: version ..., config validation run took 0.001 seconds

# Restart rsyslog
sudo systemctl restart rsyslog

# Check connection status
sudo systemctl status rsyslog
# active (running)

What happens during network failure:


┌─────────────────────────────────────────────────────────────┐
│ QUEUEING BEHAVIOR DURING NETWORK FAILURE                     │
├─────────────────────────────────────────────────────────────┤
│ 1. NETWORK FAILURE:                                         │
│ • rsyslog buffers logs in /var/spool/rsyslog/fwd_queue      │
│ • Max 256 MB disk-assisted spooling active                  │
│                                                             │
│ 2. RECONNECTION:                                            │
│ • TCP connection to port 6514 automatically reactivated     │
│ • Local queue transmitted chronologically to log server     │
│                                                             │
│ 3. REBOOT SAFE:                                             │
│ • Unsent logs survive a server restart                      │
│ • No data loss on connection drops                          │
└─────────────────────────────────────────────────────────────┘

Configure rsyslog Receiver

On the log server:


sudo nano /etc/rsyslog.d/00-receiver.conf


# rsyslog TLS Receiver (Log Server)

# Load module for TCP input
module(load="imtcp"
       StreamDriver="gtls"
       StreamDriver.AuthMode="x509/certvalid"
       StreamDriver.Mode="1")

# TLS certificates
$DefaultNetstreamDriverCAFile /etc/rsyslog/certs/ca.crt
$DefaultNetstreamDriverCertFile /etc/rsyslog/certs/server.crt
$DefaultNetstreamDriverKeyFile /etc/rsyslog/certs/server.key

# Open port
input(type="imtcp" port="6514")

# Log storage by host/tag

# Template: Storage path with hostname
template(name="HostLog" type="string"
    string="/logs/%programname%.log")

# Template: Path with hostname + date
template(name="HostLogRotated" type="string"
    string="/logs/%HOSTNAME%/%$year%-%$month%-%$day%.log")

# Store received logs by host
if $hostname != 'logserver' then {
    # Store logs of the respective host
    action(type="omfile"
           dynaFile="HostLogRotated"
           FileCreateMode="0640"
           DirCreateMode="0750")

    # Store auth logs separately (for audit)
    if $programname == "sshd" or $programname == "sudo" then {
        action(type="omfile"
               file="/logs/auth/audit.log"
               FileCreateMode="0640")
    }
}

Create log directory structure:


# Create directories
sudo mkdir -p /logs/{auth,web,db,monitor}
sudo mkdir -p /logs/{webserver,dbserver,monitor}

# Set permissions
sudo chown -R root:adm /logs
sudo chmod -R 750 /logs

# Configure log rotation
sudo nano /etc/logrotate.d/remote-logs


/logs/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
    dateext
    dateformat -%Y-%m-%d
}

journald Remote Forwarding

If you use journald instead of rsyslog as the primary logger (or additionally):


# On the sender (Host A/B/C):
sudo nano /etc/systemd/journald.conf


[Journal]
# Remote forwarding to rsyslog
ForwardToSyslog=yes
ForwardToSyslogSocket=yes
SyslogSocket=192.168.1.12:6514

# Keep local logs
Storage=persistent
SystemMaxUse=1G
SystemMaxFileSize=100M
MaxRetentionSec=30day


# On the receiver (log server):
sudo nano /etc/rsyslog.d/10-journald.conf


# Accept journald logs (if in addition to rsyslog)
module(load="imudp")
input(type="imudp" port="514")


# Restart journald
sudo systemctl restart systemd-journald

💡 rsyslog vs. journald for multi-host: rsyslog is better suited for multi-host logging. It supports TLS natively, has queueing, and can split logs by host/tag/program. journald is primarily designed for local logs. Recommendation: rsyslog as sender, journald only locally.

Log Integrity

Logs must be protected against manipulation. Two methods: signing and append-only mount.

Method 1: Log signing with rsyslog


# On the log server:
sudo nano /etc/rsyslog.d/20-signing.conf


# Log signing

# Generate signing key (one-time)
# openssl genrsa -out /etc/rsyslog/signing.key 4096

module(load="omfile")

# Sign logs after writing
template(name="SignedLog" type="string"
    string="/logs/%HOSTNAME%/%programname%.log")

action(type="omfile"
       dynaFile="SignedLog"
       FileCreateMode="0640"
       DirCreateMode="0750"
       cmd="/usr/bin/rsyslog-log-signer -s /etc/rsyslog/signing.key")

Method 2: Append-only mount (simpler and more effective)


# Mount log partition (append-only)
# WARNING: After this, logs can only be added, not deleted!
sudo mount -o remount,append-only /logs

# Verification
mount | grep /logs
# /dev/sdb1 on /logs type ext4 (rw,appendonly)


┌─────────────────────────────────────────────────────────────┐
│ LOG INTEGRITY: PROTECTION METHOD COMPARISON                  │
├─────────────────────────────────────────────────────────────┤
│ APPEND-ONLY FILESYSTEM (mount -o append-only / chattr +a):  │
│ • Prevents: Deleting and overwriting existing lines         │
│ • Allows: Only continuously appending new log data          │
│ • Practice: Low overhead, ideal for Linux log servers       │
│                                                             │
│ CRYPTOGRAPHIC SIGNING (rsyslog-log-signer / RFC 5424):      │
│ • Prevents: Forgery and manipulation by attackers           │
│ • Protection: Integrity check of every individual log block │
│                                                             │
│ RECOMMENDATION: mTLS forwarding combined with append-only!  │
└─────────────────────────────────────────────────────────────┘

Practice: 3-Host Setup

Host overview:

Host	IP	Role	rsyslog Role
webserver	192.168.1.10	nginx, Docker	Sender
dbserver	192.168.1.11	PostgreSQL	Sender
logserver	192.168.1.12	Log aggregation	Receiver

Complete configuration for each host:


# ─────────────────────────────────────────────────────
# Log Server (192.168.1.12)
# ─────────────────────────────────────────────────────

# 1. Install certificates
sudo cp ca.crt server.key server.crt /etc/rsyslog/certs/

# 2. Enable receiver configuration
# /etc/rsyslog.d/00-receiver.conf

# 3. Open firewall
sudo nft add rule inet filter input \
    tcp dport 6514 accept comment "rsyslog TLS"

# 4. Create log directories
sudo mkdir -p /logs/{auth,web,db,monitor}
sudo chown -R root:adm /logs

# 5. Start rsyslog
sudo systemctl enable --now rsyslog

# 6. Verification
sudo ss -tlnp | grep 6514
# LISTEN  0  128  0.0.0.0:6514  0.0.0.0:*  users:(("rsyslogd",pid=1234,fd=6))


# ─────────────────────────────────────────────────────
# Web Server (192.168.1.10)
# ─────────────────────────────────────────────────────

# 1. Install certificates
sudo cp ca.crt webserver.key webserver.crt /etc/rsyslog/certs/

# 2. Store sender configuration
# /etc/rsyslog.d/50-forward.conf

# 3. Start rsyslog
sudo systemctl enable --now rsyslog

# 4. Verification
sudo rsyslogd -N1
# config validation run took 0.001 seconds


# ─────────────────────────────────────────────────────
# DB Server (192.168.1.11)
# ─────────────────────────────────────────────────────

# 1. Install certificates
sudo cp ca.crt dbserver.key dbserver.crt /etc/rsyslog/certs/

# 2. Store sender configuration
# /etc/rsyslog.d/50-forward.conf

# 3. Additionally capture PostgreSQL logs
# /etc/rsyslog.d/51-postgres.conf:
# $AddUnixListenSocket /var/run/postgresql/.s.PGSQL.5432
# :programname, isequal, "postgres" /var/log/postgres.log
# & stop

# 4. Start rsyslog
sudo systemctl enable --now rsyslog

End-to-end verification:


# 1. On web server: Generate test log
logger -t test "Multi-Host-Logging Test from webserver"

# 2. On log server: Check
sudo tail -f /logs/webserver/test.log
# Aug 27 12:34:56 webserver test: Multi-Host-Logging Test from webserver

# 3. Check auth logs
sudo tail -f /logs/auth/audit.log
# Aug 27 12:34:56 webserver sshd[1234]: Accepted publickey for user

# 4. Check TLS connection
openssl s_client -connect logserver:6514 \
    -cert /etc/rsyslog/certs/webserver.crt \
    -key /etc/rsyslog/certs/webserver.key \
    -CAfile /etc/rsyslog/certs/ca.crt
# Verify return code: 0 (ok)

Retention and Compliance

GDPR requirements:

Access logs: 10 days (minimum retention)
Complete logs: 6 months (in case of data breach)
Deletion concept: Automatic deletion after expiry

German IT Security Act (IT-SiG):

Technical log data: 12 months
Audit logs: 12 months
For critical infrastructure (KRITIS): 24 months

Log rotation with automatic deletion:


# /etc/logrotate.d/remote-logs
/logs/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
    dateext
    dateformat -%Y-%m-%d
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service > /dev/null 2>&1 || true
    endscript
}


# /etc/cron.d/log-cleanup (Daily at 03:00: Delete logs older than 365 days)
0 3 * * * root find /logs -name "*.log.*" -mtime +365 -delete

💡 Audit before deletion: An admin should review the affected logs before automatic deletion.

Verification


# 1. On ALL hosts: Check rsyslog syntax
sudo rsyslogd -N1
# Expectation: "config validation run took 0.001 seconds"

# 2. Log server: Port 6514 open?
sudo ss -tlnp | grep 6514
# LISTEN  0  128  0.0.0.0:6514  0.0.0.0:*  users:(("rsyslogd",...))

# 3. Sender -> Receiver: TLS connection
sudo openssl s_client -connect logserver:6514 \
    -cert /etc/rsyslog/certs/webserver.crt \
    -key /etc/rsyslog/certs/webserver.key \
    -CAfile /etc/rsyslog/certs/ca.crt \
    </dev/null 2>/dev/null | grep "Verify return code"
# Verify return code: 0 (ok)

# 4. Log arrival on log server
logger -t verification "Test from $(hostname)"
sleep 1
sudo tail -5 /logs/$(hostname)/$(date +%Y-%m-%d).log
# Aug 27 12:34:56 webserver verification: Test from webserver

# 5. Queue status (during network issues)
sudo find /var/spool/rsyslog/ -name "fwd_queue*" -ls

# 6. Log integrity check
mount | grep /logs
# /dev/sdb1 on /logs type ext4 (rw,appendonly)

⚠️ Do not forget the firewall: The log server must allow Port 6514 (TCP) from each host: nft add rule inet filter input tcp dport 6514 accept. Without this rule, no logs can be received.

Backup Security: Encryption, Integrity, and the 3-2-1 Rule

Why Backup Security?

An attacker who gains root access on a server has two priorities: first, cover tracks (delete logs), second, take data hostage. This means: encrypt backups, delete local snapshots, and issue a ransom demand. Without secured, encrypted offsite backups, you face a choice: data lost or pay ransom.

GDPR Art. 34 requires notification to the supervisory authority in case of data breaches. If you can prove that backups exist and are restorable, the damage is considerably reduced. If not, a fine looms.


┌─────────────────────────────────────────────────────────────┐
│ ATTACK PLAYBOOK: RANSOMWARE ON LINUX SERVERS                 │
├─────────────────────────────────────────────────────────────┤
│ 1. GAIN ROOT ACCESS:                                        │
│ • SSH compromise, kernel exploit, or container escape       │
│                                                             │
│ 2. IDENTIFY BACKUPS & SNAPSHOTS:                            │
│ • Local BTRFS snapshots: `btrfs subvolume list /`           │
│ • Local archives under /backup/ and /var/backup/            │
│                                                             │
│ 3. DESTROY LOCAL BACKUPS & ENCRYPT FILES:                   │
│ • Delete snapshots: `btrfs subvolume delete /snapshots/*`  │
│ • Encrypt production data & leave ransom note               │
│                                                             │
│ PROTECTION: Untouchable, immutable offsite backups!         │
└─────────────────────────────────────────────────────────────┘

LUKS Header Backup (Cross-Reference)

The backup of the LUKS header was covered in detail in the section LUKS2 Header Management and Offline Backups.

Summary:

Header backup on USB stick (store offline, not on the same server)
Regular test: sudo cryptsetup luksHeaderRestore from the USB stick
Why critical: Without the header, all data is inaccessible, even with a known password

💡 LUKS header + offsite backup = complete protection. The LUKS header protects the boot process, offsite backups protect the data from ransomware. Together they form a complete security concept.

BTRFS Snapshot Security

BTRFS snapshots are a powerful tool for quick rollbacks, but they are not a backup. A snapshot is a consistent state at a specific point in time – but it offers no protection against deletion by an attacker or hardware failure.

What happens during compromise:


# Attacker finds snapshots
sudo btrfs subvolume list /
# ID 257 gen 5 path @
# ID 258 gen 100 path @home
# ID 300 gen 200 path @snapshots/2026-08-01
# ID 301 gen 210 path @snapshots/2026-08-15
# ID 302 gen 220 path @snapshots/2026-08-27

# Attacker deletes ALL snapshots
sudo btrfs subvolume delete /@snapshots/2026-08-01
sudo btrfs subvolume delete /@snapshots/2026-08-15
sudo btrfs subvolume delete /@snapshots/2026-08-27
# -> All snapshots gone. No rollback possible.

Protection: Read-only snapshots


# Create read-only snapshot (cannot be deleted, only via *)
sudo btrfs subvolume snapshot -r / /@snapshots/2026-08-27-readonly

# WARNING: read-only snapshots CANNOT be deleted
# * Unless: attacker mounts the volume and makes it read-write
# -> Therefore: secure snapshots on EXTERNAL volume!

Secure snapshots to external volume:


# ─────────────────────────────────────────────────────
# Snapper + Push-to-NAS (automated)
# ─────────────────────────────────────────────────────

# Mount NAS (e.g., via NFS)
sudo mount -t nfs 192.168.1.200:/backup /mnt/nas

# Adjust Snapper configuration
sudo nano /etc/snapper/configs/root


# Snapper configuration
SUBVOLUME="/"
FSTYPE="btrfs"

# Snapshot directory on NAS
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="5"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="4"
TIMELINE_LIMIT_MONTHLY="6"

# Post-snapshot script: push snapshot to NAS
RUN_FREQUENCY=1


# Create post-snapshot script
sudo nano /etc/snapper/post-snapshot.d/10-push-to-nas.sh


#!/bin/bash
# Synchronize snapshot to NAS (after each new snapshot)

SNAPSHOT_DIR="/@snapshots"
NAS_TARGET="/mnt/nas/snapshots/$(hostname)"

# Create NAS target directory
mkdir -p "$NAS_TARGET"

# Push newest snapshot to NAS
NEWEST=$(ls -td ${SNAPSHOT_DIR}/*/  | head -1)
rsync -a --delete "${NEWEST}" "${NAS_TARGET}/"

# Verification
echo "Snapshot pushed: $(basename $NEWEST) -> $NAS_TARGET"


# Make executable
sudo chmod +x /etc/snapper/post-snapshot.d/10-push-to-nas.sh

BTRFS Scrub (integrity check):


# Regular integrity check of all snapshots
sudo btrfs scrub start /

# Check status
sudo btrfs scrub status /
# UUID:             xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# Scrub started:    Mon Aug 27 12:00:00 2026
# Status:           finished
# Duration:         0:05:23
# Total to scrub:   128.00GB
# Rate:             400.00MB/s
# Error summary:    no errors found

# Cron: Monthly automatic scrub
echo "0 3 1 * * root btrfs scrub start /" | \
    sudo tee /etc/cron.d/btrfs-scrub

⚠️ Snapshots are not a backup. A snapshot protects against accidental file deletion and enables quick rollbacks. But it does NOT protect against: hardware failure, ransomware (which deletes snapshots), server theft. For these scenarios, you need offsite backups.

Cross-Reference: Snapper Basics

The installation and configuration of Snapper with BTRFS was covered in detail in the previous article. There you will find the setup for snapper, snap-pac, and grub-btrfs.

Backup Encryption

There are several tools for encrypted backups. Each has its own strengths:

Comparison of methods:

Feature	Restic	BorgBackup	rsync + GPG	LUKS Container
Encryption	AES-256	AES-256	GPG (varies)	AES-256 (LUKS)
Deduplication	✅	✅	❌	❌
Incremental	✅	✅	✅ (rsync)	❌
Anonymization	❌	❌	❌	❌
Complexity	Medium	Medium	Low	Low
Performance	Good	Very good	Good	Very good
Multi-backend	✅ (S3, B2, SFTP)	✅ (SSH)	✅ (SSH)	❌ (local volume)

Method 1: Restic (recommended for homelab)


# ─────────────────────────────────────────────────────
# Restic: Encrypted backup with deduplication
# ─────────────────────────────────────────────────────

# Install
sudo pacman -S restic

# Create repository (encrypted)
# Password is requested interactively
restic init --repo /mnt/nas/backups/$(hostname)

# Or: Use password file
echo "MySecureBackupPassword123!" | \
    restic init --repo /mnt/nas/backups/$(hostname) --password-file /dev/stdin

# Create backup
restic backup /home /etc /var/log \
    --repo /mnt/nas/backups/$(hostname) \
    --password-file /mnt/usb/backup-passwd.txt \
    --verbose

# Incremental backup (only changed data)
restic backup /home /etc /var/log \
    --repo /mnt/nas/backups/$(hostname) \
    --password-file /mnt/usb/backup-passwd.txt

# Display snapshots
restic snapshots --repo /mnt/nas/backups/$(hostname)

# Restore backup
restic restore latest \
    --repo /mnt/nas/backups/$(hostname) \
    --target /restore/

Restic cron for automatic backups:


sudo nano /etc/cron.d/restic-backup


# Daily at 02:00: Create backup
0 2 * * * root restic backup /home /etc /var/log \
    --repo /mnt/nas/backups/$(hostname) \
    --password-file /mnt/usb/backup-passwd.txt \
    --verbose >> /var/log/restic-backup.log 2>&1

# Weekly (Sunday 04:00): Delete old snapshots
# (keeps: 7 daily, 4 weekly, 6 monthly)
0 4 * * 0 root restic forget --repo /mnt/nas/backups/$(hostname) \
    --password-file /mnt/usb/backup-passwd.txt \
    --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
    --prune --verbose >> /var/log/restic-forget.log 2>&1

Method 2: BorgBackup (Pull mode)


# ─────────────────────────────────────────────────────
# BorgBackup: Deduplication + Encryption
# ─────────────────────────────────────────────────────

# On the backup server (NAS):
sudo pacman -S borg

# Create Borg repository
borg init --encryption=repokey /mnt/nas/backups/$(hostname)

# On the source server (pull mode):
# The backup server pulls the data (no SSH key needed on source!)

# Borg pull configuration on backup server
sudo nano /etc/borg.d/pull-webserver.conf


# Borg Pull configuration
SOURCE_HOST="webserver.local"
SOURCE_USER="backup"
SOURCE_PATH="/home /etc"
REMOTE_BORG_REPO="/mnt/nas/backups/webserver"

# SSH connection to source server
borg create --stats --progress \
    ssh://${SOURCE_USER}@${SOURCE_HOST}${REMOTE_BORG_REPO}::'{hostname}-{now}' \
    ${SOURCE_PATH}

Method 3: rsync + GPG (simple and flexible)


# ─────────────────────────────────────────────────────
# rsync + GPG: Encrypted transfer
# ─────────────────────────────────────────────────────

# Generate GPG key (one-time)
gpg --gen-key
# Note the fingerprint!

Security Audits and Auditd

Why Security Audits?

A security audit is not a one-time action, but a recurring process. The question is not whether a system will be compromised, but when. Regular audits find vulnerabilities before an attacker exploits them.

When is an audit performed?

Occasion	Frequency	Focus
Routine	Semi-annually	Complete check
After changes	As needed	Affected component
Incident response	After incident	Damage limitation
Compliance	Annually	GDPR, IT-SiG, BSI IT-Grundschutz
Before deploy	As needed	Configuration, dependencies

Audit Checklist

The following checklist serves as a working template for every audit. Each item is documented with status (OK/missing/critical) and justification.

System:

#	Check	Command
1.1	Kernel version (hardened?)	`uname -r`
1.2	Installed packages up to date?	`checkupdates`
1.3	Users without password?	`awk -F: '($2 == "") {print $1}' /etc/shadow`
1.4	Root without password?	`passwd -S root`
1.5	Unused users deactivated?	`lastlog \| grep -v "Never logged in"`
1.6	SUID binaries reduced?	`find / -perm -4000 -type f 2>/dev/null`

Network:

#	Check	Command
2.1	Open ports only necessary?	`ss -tlnp`
2.2	Firewall active?	`nft list ruleset`
2.3	SSH: Root login disabled?	`grep PermitRootLogin /etc/ssh/sshd_config`
2.4	SSH: Password auth disabled?	`grep PasswordAuthentication /etc/ssh/sshd_config`
2.5	TLS on all services?	`openssl s_client -connect ...`

Encryption:

#	Check	Command
3.1	LUKS active?	`lsblk -f`
3.2	LUKS header backup present?	Physically checked
3.3	SSH key encryption?	`ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key`
3.4	TLS certificates valid?	`openssl x509 -in ... -checkend 0`

Logging:

#	Check	Command
4.1	auditd active?	`systemctl status auditd`
4.2	rsyslog active?	`systemctl status rsyslog`
4.3	Central logs present?	`ls /logs/`
4.4	Log rotation active?	`cat /etc/logrotate.d/remote-logs`

Backups:

#	Check	Command
5.1	Backups present?	`restic snapshots --repo ...`
5.2	Integrity checked?	`restic check --read-data ...`
5.3	Restore test performed?	Documentation present
5.4	Offsite backup present?	Physically checked

auditd: The Linux Audit Framework

auditd is the kernel-level auditing for Linux. It logs system calls, file accesses, and user actions at kernel level – meaning even a root-compromised account cannot delete the audit logs without stopping the audit daemon (which is normally not possible).

Installation and activation:


# Install
sudo pacman -S audit

# Enable and start
sudo systemctl enable --now auditd

# Verify
sudo systemctl status auditd
# active (running)

Write audit rules:


sudo nano /etc/audit/rules.d/10-security.rules


# Audit rules for security monitoring

# 1. Monitor file access

# /etc/shadow: Every access is logged
-w /etc/shadow -p rwa -k shadow_access

# /etc/sudoers: Changes are logged
-w /etc/sudoers -p wa -k sudoers_mod
-w /etc/sudoers.d/ -p wa -k sudoers_mod

# /etc/ssh/sshd_config: Changes are logged
-w /etc/ssh/sshd_config -p wa -k sshd_config

# /etc/passwd, /etc/group: Changes are logged
-w /etc/passwd -p wa -k user_mod
-w /etc/group -p wa -k group_mod
-w /etc/gshadow -p wa -k group_mod

# 2. Monitor system calls

# execve: Which programs are executed?
-a always,exit -F arch=b64 -S execve -k program_exec

# connect: Which network connections are established?
-a always,exit -F arch=b64 -S connect -k network_connect

# mount: Filesystems are mounted
-a always,exit -F arch=b64 -S mount -k mount_op

# umount: Filesystems are unmounted
-a always,exit -F arch=b64 -S umount2 -k umount_op

# 3. Monitor user actions

# Login attempts (successful and failed)
-w /var/log/lastlog -p wa -k login_events
-w /var/run/faillock/ -p wa -k login_events

# SSH logins
-w /var/log/auth.log -p wa -k auth_log

# sudo usage
-w /var/log/sudo.log -p wa -k sudo_usage

# 4. Monitor kernel modules

# Load/unload modules
-a always,exit -F arch=b64 -S init_module -k module_load
-a always,exit -F arch=b64 -S delete_module -k module_unload

# 5. Performance: Log only relevant events

# Increase buffer size for audit events
-b 8192

# Prevent loss of audit events
-f 1

Load audit rules:


# Load rules
sudo auditctl -R /etc/audit/rules.d/10-security.rules

# Display loaded rules
sudo auditctl -l
# -w /etc/shadow -p rwa -k shadow_access
# -w /etc/sudoers -p wa -k sudoers_mod
# -a always,exit -F arch=b64 -S execve -k program_exec
# ...

Evaluate audit logs:


# ─────────────────────────────────────────────────
# ausearch: Search for specific events
# ─────────────────────────────────────────────────

# All accesses to /etc/shadow
sudo ausearch -k shadow_access --interpret

# All execve events (program calls)
sudo ausearch -k program_exec --interpret

# All network connections
sudo ausearch -k network_connect --interpret

# Events from the last hour
sudo ausearch -ts recent -k auth_log

# Events of a specific user
sudo ausearch -ua 1000 --interpret

# ─────────────────────────────────────────────────
# aureport: Create summaries
# ─────────────────────────────────────────────────

# Overall overview
sudo aureport

# Auth report (login attempts)
sudo aureport --auth

# Failed login report
sudo aureport --failed

# Login report by user
sudo aureport --login --summary

# File access report
sudo aureport -f

# Error report
sudo aureport -e

Example: Detecting suspicious activity


# All accesses to /etc/shadow in the last 24 hours
sudo ausearch -k shadow_access -ts today --interpret

# Expected output:
# type=PROCTITLE msg=audit(2026-08-27 12:34:56.789:1234) : proctitle=cat /etc/shadow
# type=SYSCALL msg=audit(2026-08-27 12:34:56.789:1234) : arch=c000003e syscall=257 success=yes ...
#   uid=0 root (expected: only root should read this)
#   exe=/usr/bin/cat

# Or: Access by unknown user (CRITICAL!)
# type=SYSCALL msg=audit(...) : uid=1001 webserver
#   exe=/usr/bin/cat -> WARNING: webserver reads /etc/shadow!

⚠️ Audit performance: Too many audit rules can impact performance. Every execve event creates an audit entry. On an active webserver with many HTTP requests, this can cause significant overhead. Restrict execve auditing to suspicious programs, not everything.

Optimize audit rules:


# ─────────────────────────────────────────────────
# Performance-optimized audit rules
# ─────────────────────────────────────────────────

# ONLY log execve for specific programs
# (not for everything, only for suspicious ones)
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/sudo -k sudo_exec
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/su -k su_exec
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/passwd -k passwd_exec

# Only log connect for specific UIDs
-a always,exit -F arch=b64 -S connect -F uid=0 -k root_network

# Excluded programs (performance)
# These programs are NOT logged
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/bash
-a never,exit -F arch=b64 -S execve -F exe=/usr/bin/zsh

LinPEAS: Automated Vulnerability Scanning

LinPEAS (Linux Privilege Escalation Awesome Scripts) is an automated tool that checks a system for known vulnerabilities and misconfigurations that could lead to privilege escalation.

Installation and execution:


# Download LinPEAS
curl -L https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh -o /tmp/linpeas.sh
chmod +x /tmp/linpeas.sh

# Run as root (with colored output)
sudo /tmp/linpeas.sh

# Run as root (without colors, for log file)
sudo /tmp/linpeas.sh -a 2>&1 | tee /var/log/linpeas-$(date +%Y%m%d).log

Important findings and their meaning:


# ─────────────────────────────────────────────────
# LinPEAS: Important findings
# ─────────────────────────────────────────────────

# 1. SUID binaries (red = critical)
# -> Binaries with SUID bit can inherit root privileges
# -> Risk: If a SUID binary has a vulnerability,
#   an attacker can use it to become root
# -> Action: Remove unnecessary SUID bits

# 2. Writable /etc/passwd (red = critical)
# -> Anyone can write to /etc/passwd
# -> Risk: Attacker can create own root user entry
# -> Action: Fix permissions (0644)

# 3. World-writable files (yellow = medium)
# -> Files that anyone can write to
# -> Risk: Attacker can modify configuration files
# -> Action: Fix permissions

# 4. Cleartext passwords in config files (red = critical)
# -> Passwords in plaintext in configuration files
# -> Risk: Immediately readable on compromise
# -> Action: Move passwords to secrets backends

# 5. Insecure SSH config (yellow = medium)
# -> PermitRootLogin yes, PasswordAuthentication yes
# -> Risk: Brute-force or root access via SSH
# -> Action: Harden SSH configuration

💡 LinPEAS vs. manual audit: LinPEAS automatically finds the most common vulnerabilities. But it does not replace a manual audit. It does not check for: unknown zero-days, company-specific requirements, or complex attack vectors. LinPEAS is the FIRST step, not the last.

Audit Combination: All Tools Together

A complete security audit combines multiple tools in phases:


┌─────────────────────────────────────────────────────────────┐
│               Security Audit: 3-Phase Model                  │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Phase 1: Automated (Duration: ~30 min)                     │
│  ────────────────────────────────────────                   │
│  - lynis audit system                                       │
│  - checksec procAll                                         │
│  - LinPEAS (privilege escalation scanner)                   │
│  - auditd: ausearch -ts today                               │
│  - Output: raw data, findings & vulnerabilities            │
│                                                             │
│  Phase 2: Manual (Duration: 2-4 hrs)                        │
│  ──────────────────────────────────                         │
│  - Check kernel hardening & sysctl                          │
│  - Audit open ports & TLS certificates                      │
│  - Check user permissions & SUID binaries                   │
│  - Verify backup integrity & restores                       │
│  - Validate auditd rule set & filters                       │
│                                                             │
│  Phase 3: Report (Duration: 1-2 hrs)                        │
│  ──────────────────────────────────                         │
│  - Rate findings by severity (Critical/High/...)            │
│  - Derive concrete measures & hardening plan                │
│  - Create audit-proof archive (3 years)                     │
│  - Schedule next audit date (+6 months)                     │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Phase 1: Automated


# Save Lynis output for report
sudo lynis audit system --no-colors 2>&1 | tee /tmp/lynis-report.txt

# checksec for all processes
checksec --no-banner --no-warnings procAll 2>&1 | tee /tmp/checksec-report.txt

# Save LinPEAS output for report
sudo /tmp/linpeas.sh -a 2>&1 | tee /tmp/linpeas-report.txt

# Audit events from the last 24 hours
sudo ausearch -ts today --interpret 2>&1 | tee /tmp/audit-report.txt

Phase 2: Manual (selection)


# Check kernel version
uname -r
# -> 7.1.9-hardened1-1-hardened ✓

# Check unused users
lastlog | grep "Never logged in"
# -> No unused users ✓

# Check open ports
ss -tlnp
# -> Only SSH (22), nginx (80, 443) ✓

# Check TLS certificates
openssl x509 -in /etc/nginx/certs/site.crt -checkend 2592000
# -> "Certificate will expire in 365 days" ✓

Practice: Semi-Annual Security Audit

Step 1: Preparation


# Set audit date and scope
AUDIT_DATE=$(date +%Y-%m-%d)
AUDIT_DIR="/root/audits/${AUDIT_DATE}"
mkdir -p "$AUDIT_DIR"

# Store audit metadata
echo "Security Audit started: $AUDIT_DATE" > "$AUDIT_DIR/README.md"
echo "Scope: Complete system" >> "$AUDIT_DIR/README.md"

Step 2: Automated tools


# Lynis
sudo lynis audit system --no-colors 2>&1 | tee "$AUDIT_DIR/lynis.txt"

# checksec
checksec --no-banner --no-warnings procAll 2>&1 | tee "$AUDIT_DIR/checksec.txt"

# LinPEAS
sudo /tmp/linpeas.sh -a 2>&1 | tee "$AUDIT_DIR/linpeas.txt"

Step 3: Manual checks


# Kernel
uname -r > "$AUDIT_DIR/kernel.txt"

# Users
lastlog > "$AUDIT_DIR/users.txt"
awk -F: '($2 == "") {print $1}' /etc/shadow >> "$AUDIT_DIR/users.txt"

# Network
ss -tlnp > "$AUDIT_DIR/ports.txt"
nft list ruleset > "$AUDIT_DIR/firewall.txt"

# Encryption
lsblk -f > "$AUDIT_DIR/encryption.txt"

# Backups
restic snapshots --repo /mnt/nas/backups/$(hostname) > "$AUDIT_DIR/backups.txt" 2>&1

# Audit
sudo auditctl -l > "$AUDIT_DIR/audit-rules.txt"
sudo aureport > "$AUDIT_DIR/audit-summary.txt"

Step 4: Create report


cat > "$AUDIT_DIR/bericht.md" << 'EOF'
# Security Audit Report

## Date
$(date +%Y-%m-%d)

## Summary

| Category | Status | Note |
|----------|--------|------|
| Kernel | OK | linux-hardened active |
| Users | OK | No unused accounts |
| Network | OK | Only necessary ports open |
| Encryption | OK | LUKS active, header backup present |
| Logging | OK | auditd + rsyslog active |
| Backups | OK | Restic active, integrity checked |

## Findings

### Critical
- None

### High
- None

### Medium
- LinPEAS: 2 SUID binaries identified as unnecessary

### Low
- Kernel update available (7.1.9 -> 7.1.10)

## Recommendations

1. [ ] Remove unnecessary SUID bits
2. [ ] Perform kernel update
3. [ ] Next audit: $(date -d "+6 months" +%Y-%m-%d)
EOF


# Archive audit directory
tar czf "/root/audits/${AUDIT_DATE}.tar.gz" "$AUDIT_DIR"
echo "Audit completed: $AUDIT_DIR"

💡 Audit archiving: Store every audit report for at least 3 years. In the event of a security incident or compliance audit, you can prove that regular audits were conducted.

Verification


# ─────────────────────────────────────────────────────
# Check audit status
# ─────────────────────────────────────────────────────

# 1. auditd active?
sudo systemctl status auditd
# active (running)

# 2. Audit rules loaded?
sudo auditctl -l
# -w /etc/shadow -p rwa -k shadow_access
# ...

# 3. Audit events present?
sudo ausearch -ts today --interpret | wc -l
# > 0

# 4. Lynis report present?
ls -la /root/audits/
# drwxr-x--- 2 root root 4096 Aug 27 12:34 2026-08-27

# 5. Next audit date?
echo "Next audit: $(date -d '+6 months' +%Y-%m-%d)"
# Next audit: 2027-02-27

Official Documentation

Topic	Documentation
Arch Linux Wiki	wiki.archlinux.org
AppArmor	apparmor.net
linux-hardened	github.com/anthraxx/linux-hardened
nftables Wiki	wiki.nftables.org
auditd	github.com/linux-audit/audit-userspace
Restic	restic.net
BorgBackup	borgbackup.readthedocs.io
checksec	github.com/slimm609/checksec.sh
LinPEAS	github.com/carlospolop/PEASS-ng
systemd-cryptenroll	freedesktop.org/software/systemd/man/systemd-cryptenroll.html

Further Articles in This Series

Article	Topic
System Hardening and Security	Fundamentals: user management, nftables, SSH, Fail2Ban
Best Practices and Tips	BTRFS, Snapper, snap-pac, grub-btrfs
Package Management with Pacman	Package sources, signatures, cleanup
Installation and Basic Configuration	UEFI, partitioning, BTRFS, encryption
Graphical User Interface	Wayland, PipeWire, SDDM, KDE Plasma 6

Conclusion: The Hardened Arch Linux Complete System

With the completion of this sixth part, you have mastered the entire journey from empty storage medium to a highly secured, production-grade Linux environment. Arch Linux has proven that it is far more than a mere tinkering distribution: through absolute transparency and minimal base, a security level can be realized that surpasses even many pre-configured server distributions.

We have hardened the system layer by layer:

Foundation: Clean package management with pacman, encrypted BTRFS subvolumes, and secure boot via GRUB and TPM2.
Operations: Low-maintenance automation with systemd timers and fail-safe rollbacks through snapper and grub-btrfs.
Defense in Depth: Mandatory Access Control with AppArmor, kernel hardening through linux-hardened, and ASLR optimization.
Isolation: Container network segmentation with nftables without dangerous Docker defaults.
Observability and Resilience: Tamper-proof central TLS logging, audit-proof auditd monitoring, and an immutable 3-2-1 backup strategy.

💡 Final practical tip: Security is not a static state, but a continuous process. Keep your audit routines semi-annual, regularly check audit logs for anomalies, and test your backup restores at least once a quarter under real conditions.

You now possess a deep understanding of the architecture of modern Linux systems and the tools to design, operate, and defend production systems to the highest security standards.