--- id: arch-linux-06 slug: arch-linux-advanced-security-features-and-maintenance title: "Arch Linux: Advanced Security Features and Maintenance" excerpt: "MAC with AppArmor, kernel hardening, rootless Podman containers, TLS-encrypted central logging, and immutable backups – the final step to a production-grade Arch Linux." date: 2026-08-27 updated: 2026-08-27 author: name: Sebastian Palencsar handle: spalencsar category: arch-linux-serie tags: - arch-linux - security - apparmor - hardened-kernel - podman - containers - rsyslog - tls - btrfs - restic - auditd - backup reading_time: 55 toc: true --- # Arch Linux Advanced Security Features and Maintenance ## Security: Mandatory Access Control with AppArmor ### Why AppArmor? Discretionary Access Control (DAC) – the standard Linux permission model with `rwx` for user, group, others – has a fundamental weakness: once a process runs as root, it can access everything. A compromised web server with `CAP_NET_BIND_SERVICE` and root access can read `/etc/shadow`, mount filesystems, or load kernel modules. DAC offers no barrier. Mandatory Access Control (MAC) closes this gap. The kernel enforces a policy that applies to **all** processes – including root. Even a fully compromised service can only access resources explicitly allowed in its profile. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ DAC vs. MAC: ACCESS CONTROL COMPARISON │ ├─────────────────────────────────────────────────────────────┤ │ DAC (Discretionary Access Control): │ │ • Owner decides who gets access │ │ • root bypasses all restrictions │ │ • Weakness: One compromised process = full system access │ │ │ │ MAC (Mandatory Access Control): │ │ • Kernel enforces policy for ALL processes │ │ • root is ALSO subject to MAC rules │ │ • Strength: root compromise -> limited damage │ └─────────────────────────────────────────────────────────────┘ ``` Arch Linux supports two MAC frameworks: **AppArmor** (profile-based, Ubuntu/Debian standard) and **SELinux** (label-based, Red Hat/Fedora standard). AppArmor is the better choice for Arch because: * Simpler configuration: text-based profiles instead of complex type enforcement * Faster learning curve: `aa-genprof` generates profiles from running processes * Arch Wiki has extensive AppArmor documentation * Kernel `linux-hardened` includes AppArmor support out of the box ### AppArmor in Practice **Installation and activation:** ```bash # Install AppArmor sudo pacman -S apparmor # Enable the service sudo systemctl enable --now apparmor # Verify status sudo aa-status # apparmor module is loaded. # 37 profiles are loaded. # 37 profiles are in enforce mode. ``` **First steps: Learn mode for existing services** Before blocking anything, AppArmor must learn what files and capabilities a service actually needs: ```bash # Start nginx in learn mode sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx # Change to complain mode (log violations, don't block) sudo aa-complain /etc/apparmor.d/usr.sbin.nginx # Restart the service and exercise normal operations sudo systemctl restart nginx curl -s https://localhost/ > /dev/null # Access all vhosts, static files, PHP applications # Check learn mode logs sudo journalctl -u apparmor --since "1 hour ago" | grep "profile" ``` **Create a profile from learn mode output:** ```bash # Generate profile from learn mode sudo aa-genprof /usr/sbin/nginx # Scanning system logs for AppArmor events... # # The following local profile file was generated: # /etc/apparmor.d/usr.sbin.nginx ``` **Verify and activate:** ```bash # Verify profile syntax sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx # Switch to enforce mode sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx # Verify sudo aa-status | grep nginx # 1 profiles are in enforce mode. # /usr/sbin/nginx ``` **Custom profile for a web application:** ```bash sudo nano /etc/apparmor.d/usr.sbin.mywebapp ``` ```bash #include /usr/sbin/mywebapp { #include #include #include # Binary /usr/sbin/mywebapp mr, # Configuration /etc/mywebapp/** r, /etc/mywebapp/config.yml r, # Data directory /var/lib/mywebapp/** rwk, /var/lib/mywebapp/tmp/ rw, # Logs /var/log/mywebapp/** w, /run/mywebapp.pid rw, # Network network inet stream, network inet6 stream, # Deny dangerous capabilities deny capability sys_admin, deny capability sys_rawio, deny capability sys_module, # Deny write to sensitive paths deny /etc/shadow w, deny /etc/passwd w, deny /etc/sudoers w, deny /proc/sys/** w, } ``` ```bash # Load and activate profile sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mywebapp sudo aa-enforce /etc/apparmor.d/usr.sbin.mywebapp # Verify sudo aa-status | grep mywebapp ```
⚠️ **Restart required:** AppArmor profiles are only loaded when the service starts. After profile changes, always restart the affected service: `sudo systemctl restart nginx`.
### AppArmor vs. SELinux Both frameworks solve the same problem, but with different approaches: | Feature | AppArmor | SELinux | |---------|----------|---------| | Configuration | Text-based profiles | Type enforcement (labels) | | Learning curve | Low | High | | Granularity | Per-binary | Per-process + per-file context | | Default on Arch | Optional (install required) | Optional (install required) | | `linux-hardened` support | ✅ | ✅ | | Real-world deployment | Ubuntu, SUSE, Debian | RHEL, Fedora, CentOS | | Debugging tools | `aa-status`, `aa-logprof` | `audit2allow`, `sealert` |
💡 **Recommendation for Arch:** AppArmor. The simpler configuration and faster debugging make it the better choice for small to medium Arch deployments. SELinux is better suited for enterprise environments with dedicated security teams.
## Kernel Hardening with linux-hardened ### Why a hardened kernel? The standard Linux kernel is optimized for compatibility and performance – not for maximum security. Many security-relevant features are either disabled or configured conservatively by default. The `linux-hardened` kernel patch set enables these features and adds additional protections. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ linux-hardened vs. standard kernel │ ├─────────────────────────────────────────────────────────────┤ │ STANDARD KERNEL: │ │ • Defaults for compatibility (weaker security settings) │ │ • Many hardening options disabled or configurable only │ │ • No compile-time restrictions │ │ │ │ linux-hardened: │ │ • Security features enabled by default │ │ • Compile-time restrictions (e.g., no module loading) │ │ • Additional patches (e.g., stronger ASLR, restricted │ │ /proc visibility) │ └─────────────────────────────────────────────────────────────┘ ``` **Installation:** ```bash # Install linux-hardened sudo pacman -S linux-hardened linux-hardened-headers # GRUB: Set as default kernel sudo nano /etc/default/grub # GRUB_DEFAULT=saved # GRUB_SAVEDEFAULT=true # Set linux-hardened as default sudo grub-set-default "Arch Linux, with linux-hardened linux" # Update GRUB sudo grub-mkconfig -o /boot/grub/grub.cfg # Reboot into hardened kernel sudo reboot # Verify uname -r # 7.1.9-hardened1-1-hardened ``` ### Kernel Parameters for Maximum Security **sysctl configuration:** ```bash sudo nano /etc/sysctl.d/99-hardening.conf ``` ```ini # Kernel Hardening Parameters # 1. ASLR (Address Space Layout Randomization) # Full randomization (0=off, 1=half, 2=full) kernel.randomize_va_space = 2 # 2. Process hiding # Hide kernel pointers from unprivileged users kernel.kptr_restrict = 2 # Restrict dmesg access kernel.dmesg_restrict = 1 # Restrict /proc visibility kernel.yama.ptrace_scope = 1 # 3. Network hardening # Disable IP forwarding (unless router) net.ipv4.ip_forward = 0 net.ipv6.conf.all.forwarding = 0 # Disable source routing net.ipv4.conf.all.accept_source_route = 0 net.ipv6.conf.all.accept_source_route = 0 # Disable ICMP redirects net.ipv4.conf.all.accept_redirects = 0 net.ipv6.conf.all.accept_redirects = 0 # SYN flood protection net.ipv4.tcp_syncookies = 1 # 4. File system hardening # Restrict symlinks fs.protected_symlinks = 1 fs.protected_hardlinks = 1 # Restrict FIFO and regular file in world-writable dirs fs.protected_fifos = 2 fs.protected_regular = 2 ``` ```bash # Apply sysctl changes sudo sysctl --system # Verify key settings cat /proc/sys/kernel/randomize_va_space # 2 cat /proc/sys/kernel/kptr_restrict # 2 ``` **Boot parameters for additional security:** ```bash sudo nano /etc/default/grub ``` ```ini # Add to GRUB_CMDLINE_LINUX_DEFAULT: # quiet loglevel=3 slab_nomerge init_on_alloc=1 init_on_free=1 # page_alloc.shuffle=1 randomize_kstack_offset=on # vsyscall=none debugfs=off ``` ```bash # Update GRUB sudo grub-mkconfig -o /boot/grub/grub.cfg # Reboot sudo reboot ``` | Parameter | Effect | |-----------|--------| | `slab_nomerge` | Prevents slab merging (hardens heap exploitation) | | `init_on_alloc=1` | Zero memory on allocation (prevents info leaks) | | `init_on_free=1` | Zero memory on free (prevents use-after-free) | | `page_alloc.shuffle=1` | Randomizes page allocator (hardens heap layout) | | `randomize_kstack_offset=on` | Randomizes kernel stack offset per syscall | | `vsyscall=none` | Disables legacy vsyscall page (attack surface reduction) | | `debugfs=off` | Disables debugfs (attack surface reduction) |
⚠️ **Test first:** Not all sysctl and boot parameters work with every hardware combination. Test on a non-production system first, especially `init_on_alloc=1` and `init_on_free=1`, which can impact performance.
## Container Security with Podman ### Rootless Containers Docker's daemon runs as root – a single container escape gives the attacker full host access. Podman eliminates this architecture with rootless containers: each user runs their own Podman process, with no central daemon and no root requirement. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ DOCKER vs. PODMAN: ARCHITECTURE │ ├─────────────────────────────────────────────────────────────┤ │ DOCKER: │ │ • dockerd runs as root (PID 1 or systemd-managed) │ │ • Container processes run under root's namespace │ │ • Single point of failure: compromised daemon = full host │ │ │ │ PODMAN (rootless): │ │ • No central daemon │ │ • Each user runs their own containers │ │ • User namespaces: container root != host root │ │ • No privilege escalation possible (in theory) │ └─────────────────────────────────────────────────────────────┘ ``` **Rootless setup:** ```bash # Install Podman sudo pacman -S podman # Create subuid/subgid mappings for your user sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER # Verify cat /etc/subuid # sebastian:100000:65536 cat /etc/subgid # sebastian:100000:65536 # Run a rootless container podman run --rm -it alpine sh # You are now "root" inside the container, but a regular user on the host id # uid=0(root) gid=0(root) # On the host: ps aux | grep container # sebastian 12345 ... podman run --rm -it alpine sh # (running as regular user, NOT root) ``` ### SELinux Integration If SELinux is available, Podman supports `--selinux-enabled` for additional isolation: ```bash # Check SELinux status getenforce # Enforcing (if configured) # Run with SELinux labels podman run --rm --security-opt label=type:container_t alpine sh # Verify label ps -eZ | grep container # system_u:system_r:container_t:s0:c123,c456 12345 ... podman run ... ``` ### Seccomp and AppArmor for Containers **Seccomp: Restrict system calls** ```bash # Default seccomp profile (already applied by Podman) podman run --rm alpine cat /proc/self/status | grep Seccomp # Seccomp: 2 # Seccomp_filters: 1 ``` **AppArmor for containers:** ```bash # Run with AppArmor profile podman run --rm --security-opt apparmor=my-custom-profile alpine sh # Check which AppArmor profiles are loaded podman info | grep -i apparmor # apparmorEnabled: true ``` ### Container Networking with nftables Docker creates default bridge networks with automatic NAT – convenient but dangerous. Containers can reach each other, the host, and external networks without restrictions. Podman's rootless networking is more restrictive by default, but explicit rules are still necessary for production. **nftables rules for container isolation:** ```bash sudo nano /etc/nftables.d/containers.conf ``` ```bash #!/usr/sbin/nft -f # Container Network Segmentation flush ruleset # 1. Define container networks # Docker/Podman default bridge define PODMAN_BRIDGE = 10.89.0.0/24 # Custom container networks define WEB_CONTAINERS = 10.89.1.0/24 define DB_CONTAINERS = 10.89.2.0/24 define MONITOR_CONTAINERS = 10.89.3.0/24 # 2. Default policy: deny all table inet container-filter { chain input { type filter hook input priority 0; policy drop; # Allow established connections ct state established,related accept # Allow loopback iif "lo" accept # Allow ICMP (ping) ip protocol icmp accept ip6 nexthdr icmpv6 accept } chain forward { type filter hook forward priority 0; policy drop; # Allow established connections ct state established,related accept # 3. Inter-container rules # Web -> DB (only specific ports) ip saddr $WEB_CONTAINERS ip daddr $DB_CONTAINERS \ tcp dport { 5432, 6379 } accept comment "web to db" # Web -> Internet (HTTP/HTTPS only) ip saddr $WEB_CONTAINERS \ tcp dport { 80, 443 } accept comment "web to internet" # DB -> NO internet # (default drop covers this) # Monitor -> all containers (read-only) ip saddr $MONITOR_CONTAINERS \ tcp dport { 9100, 9090, 3000 } accept comment "monitor scrape" # 4. Deny everything else } } ``` ```bash # Load rules sudo nft -f /etc/nftables.d/containers.conf # Verify sudo nft list ruleset | grep "container-filter" ``` ```markdown ┌─────────────────────────────────────────────────────────────┐ │ CONTAINER NETWORK SEGMENTATION │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Web │ │ DB │ │ Monitor │ │ │ │ 10.89.1.0/24│ │ 10.89.2.0/24│ │ 10.89.3.0/24│ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ │ ┌──────────────┼──────────────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────────────┐ │ │ │ nftables (inet container-filter) │ │ │ │ • Web -> DB: 5432, 6379 only │ │ │ │ • Web -> Internet: 80, 443 only │ │ │ │ • DB -> Internet: BLOCKED │ │ │ │ • Monitor -> all: 9100, 9090, 3000 │ │ │ │ • Everything else: DROP │ │ │ └─────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` ### Logging, Debugging and Troubleshooting **Audit logs for dropped container packets:** ```ini # /etc/rsyslog.d/10-container.conf :msg, contains, "[CONTAINER-" /var/log/container-firewall.log & stop ``` **Logrotate configuration:** ```ini # /etc/logrotate.d/container-firewall /var/log/container-firewall.log { daily rotate 7 compress missingok notifempty create 0640 root root } ```
💡 **Debugging with tcpdump:** When containers have connectivity issues, capture traffic directly on the interface: `sudo tcpdump -i docker0 -n`.
## Multi-Host Logging: Central Logs with TLS Forwarding ### Why Central Logs? An attacker who compromises a server will delete local log files first. `~/.bash_history`, `/var/log/auth.log`, `/var/log/syslog` – everything gets overwritten or deleted. Without central logs, you have no forensic foundation left. Central logs solve three problems: 1. **Prevent anti-forensics:** Logs on the log server are unreachable by the attacker 2. **Correlation:** Detect warnings across multiple hosts (e.g., simultaneous SSH logins on web and DB servers) 3. **Compliance:** GDPR and the German IT Security Act require auditable access logs with retention periods ```markdown ┌─────────────────────────────────────────────────────────────┐ │ MULTI-HOST LOGGING: TLS FORWARDING TOPOLOGY │ ├─────────────────────────────────────────────────────────────┤ │ HOST A (Web) HOST B (DB) HOST C (Monitor) │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ rsyslog │ │ rsyslog │ │ rsyslog │ │ │ │ (Sender) │ │ (Sender) │ │ (Sender) │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ │ mTLS (Port 6514)│ mTLS (Port 6514) │ │ │ └───────────────────┼────────────────────┘ │ │ │ │ │ ┌──────────┴──────────┐ │ │ │ LOG-SERVER (Host D) │ │ │ │ rsyslog Receiver │ │ │ │ /logs/{web,db,...} │ │ │ └─────────────────────┘ │ │ │ │ ADVANTAGE: Host compromise keeps logs on D untouchable. │ └─────────────────────────────────────────────────────────────┘ ``` ### Certificates for TLS Logging TLS forwarding protects logs from being read on the network. Each host needs a client certificate, the log server needs a server certificate. We create our own CA (Certificate Authority) for this. **Step 1: Create CA** ```bash # Create directory for certificates sudo mkdir -p /etc/rsyslog/ca cd /etc/rsyslog/ca # Generate CA private key (4096 bit, RSA) sudo openssl genrsa -out ca.key 4096 # Generate CA certificate (valid for 10 years) sudo openssl req -x509 -new -nodes -key ca.key \ -sha256 -days 3650 \ -out ca.crt \ -subj "/C=DE/ST=Bayern/L=Muenchen/O=Homelab/CN=Log-CA" # Verification sudo openssl x509 -in ca.crt -text -noout | grep -A2 "Issuer" # Issuer: C = DE, ST = Bayern, L = Muenchen, O = Homelab, CN = Log-CA ``` **Step 2: Server certificate for log receiver** ```bash cd /etc/rsyslog/ca # Generate server key sudo openssl genrsa -out server.key 4096 # Generate CSR (Certificate Signing Request) sudo openssl req -new -key server.key \ -out server.csr \ -subj "/C=DE/ST=Bayern/L=Muenchen/O=Homelab/CN=logserver.local" # Sign server certificate (valid for 5 years) sudo openssl x509 -req -in server.csr \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -out server.crt -days 1825 -sha256 # Add SAN (Subject Alternative Name) # WARNING: Without SAN, rsyslog refuses the connection! sudo openssl x509 -in server.crt -out server-san.crt \ -extfile <(printf "subjectAltName=DNS:logserver.local,DNS:logserver,IP:192.168.1.12") sudo mv server-san.crt server.crt ``` **Step 3: Client certificates for each host** ```bash # Generate a certificate for each host for HOST in webserver dbserver monitor; do sudo openssl genrsa -out ${HOST}.key 4096 sudo openssl req -new -key ${HOST}.key \ -out ${HOST}.csr \ -subj "/C=DE/ST=Bayern/L=Muenchen/O=Homelab/CN=${HOST}.local" sudo openssl x509 -req -in ${HOST}.csr \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -out ${HOST}.crt -days 1825 -sha256 done ``` **Step 4: Distribute certificates to hosts** ```bash # On each host: sudo mkdir -p /etc/rsyslog/certs # CA certificate (on all hosts) sudo cp ca.crt /etc/rsyslog/certs/ # Server certificate (only on log server) sudo cp server.key server.crt /etc/rsyslog/certs/ # Client certificate (on the respective host) sudo cp webserver.key webserver.crt /etc/rsyslog/certs/ ``` **Step 5: Verification** ```bash # Check if certificate is signed by CA sudo openssl verify -CAfile ca.crt server.crt # server.crt: OK sudo openssl verify -CAfile ca.crt webserver.crt # webserver.crt: OK # Display certificate details sudo openssl x509 -in server.crt -text -noout | grep -A2 "Subject:" # Subject: C = DE, ST = Bayern, L = Muenchen, O = Homelab, CN = logserver.local ```
⚠️ **SAN is mandatory:** Without `subjectAltName` in the server certificate, rsyslog refuses the TLS connection with `SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed`. The SAN must contain the hostname AND the IP address of the log server.
### Configure rsyslog Sender On each host that sends logs to the log server: ```bash sudo nano /etc/rsyslog.d/50-forward.conf ``` ```ini # rsyslog TLS Forwarding (Sender) # Load global modules module(load="imuxsock") # Local logs module(load="imklog") # Kernel logs module(load="imfile") # File monitoring module(load="omfwd") # Forwarding # TLS transport definition # TLS certificates $DefaultNetstreamDriverCAFile /etc/rsyslog/certs/ca.crt $DefaultNetstreamDriverCertFile /etc/rsyslog/certs/webserver.crt $DefaultNetstreamDriverKeyFile /etc/rsyslog/certs/webserver.key # Enable TLS transport $ActionSendStreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/certvalid # Forwarding rules # Send all local logs to log server *.* @@(omfwd)logserver.local:6514;RSYSLOG_SyslogProtocol23Format # Queueing for reliability # Queue type: Disk-assisted (overflow to disk) $ActionQueueType LinkedList $ActionQueueFileName fwd_queue $ActionQueueMaxDiskSpace 256m $ActionQueueSaveOnShutdown on $ActionQueueTimeoutEnqueue 100 $ActionResumeRetryCount -1 $ActionResumeInterval 30 # Rate limiting against log flooding # Maximum 1000 messages per second $NetstatsRateLimit 1000 ``` ```bash # Verification: Check syntax sudo rsyslogd -N1 # rsyslogd: version ..., config validation run took 0.001 seconds # Restart rsyslog sudo systemctl restart rsyslog # Check connection status sudo systemctl status rsyslog # active (running) ``` **What happens during network failure:** ```markdown ┌─────────────────────────────────────────────────────────────┐ │ QUEUEING BEHAVIOR DURING NETWORK FAILURE │ ├─────────────────────────────────────────────────────────────┤ │ 1. NETWORK FAILURE: │ │ • rsyslog buffers logs in /var/spool/rsyslog/fwd_queue │ │ • Max 256 MB disk-assisted spooling active │ │ │ │ 2. RECONNECTION: │ │ • TCP connection to port 6514 automatically reactivated │ │ • Local queue transmitted chronologically to log server │ │ │ │ 3. REBOOT SAFE: │ │ • Unsent logs survive a server restart │ │ • No data loss on connection drops │ └─────────────────────────────────────────────────────────────┘ ``` ### Configure rsyslog Receiver On the log server: ```bash sudo nano /etc/rsyslog.d/00-receiver.conf ``` ```ini # rsyslog TLS Receiver (Log Server) # Load module for TCP input module(load="imtcp" StreamDriver="gtls" StreamDriver.AuthMode="x509/certvalid" StreamDriver.Mode="1") # TLS certificates $DefaultNetstreamDriverCAFile /etc/rsyslog/certs/ca.crt $DefaultNetstreamDriverCertFile /etc/rsyslog/certs/server.crt $DefaultNetstreamDriverKeyFile /etc/rsyslog/certs/server.key # Open port input(type="imtcp" port="6514") # Log storage by host/tag # Template: Storage path with hostname template(name="HostLog" type="string" string="/logs/%programname%.log") # Template: Path with hostname + date template(name="HostLogRotated" type="string" string="/logs/%HOSTNAME%/%$year%-%$month%-%$day%.log") # Store received logs by host if $hostname != 'logserver' then { # Store logs of the respective host action(type="omfile" dynaFile="HostLogRotated" FileCreateMode="0640" DirCreateMode="0750") # Store auth logs separately (for audit) if $programname == "sshd" or $programname == "sudo" then { action(type="omfile" file="/logs/auth/audit.log" FileCreateMode="0640") } } ``` **Create log directory structure:** ```bash # Create directories sudo mkdir -p /logs/{auth,web,db,monitor} sudo mkdir -p /logs/{webserver,dbserver,monitor} # Set permissions sudo chown -R root:adm /logs sudo chmod -R 750 /logs # Configure log rotation sudo nano /etc/logrotate.d/remote-logs ``` ```ini /logs/*/*.log { daily rotate 90 compress delaycompress missingok notifempty create 0640 root adm dateext dateformat -%Y-%m-%d } ``` ### journald Remote Forwarding If you use `journald` instead of `rsyslog` as the primary logger (or additionally): ```bash # On the sender (Host A/B/C): sudo nano /etc/systemd/journald.conf ``` ```ini [Journal] # Remote forwarding to rsyslog ForwardToSyslog=yes ForwardToSyslogSocket=yes SyslogSocket=192.168.1.12:6514 # Keep local logs Storage=persistent SystemMaxUse=1G SystemMaxFileSize=100M MaxRetentionSec=30day ``` ```bash # On the receiver (log server): sudo nano /etc/rsyslog.d/10-journald.conf ``` ```ini # Accept journald logs (if in addition to rsyslog) module(load="imudp") input(type="imudp" port="514") ``` ```bash # Restart journald sudo systemctl restart systemd-journald ```
💡 **rsyslog vs. journald for multi-host:** rsyslog is better suited for multi-host logging. It supports TLS natively, has queueing, and can split logs by host/tag/program. journald is primarily designed for local logs. Recommendation: rsyslog as sender, journald only locally.
### Log Integrity Logs must be protected against manipulation. Two methods: signing and append-only mount. **Method 1: Log signing with rsyslog** ```bash # On the log server: sudo nano /etc/rsyslog.d/20-signing.conf ``` ```ini # Log signing # Generate signing key (one-time) # openssl genrsa -out /etc/rsyslog/signing.key 4096 module(load="omfile") # Sign logs after writing template(name="SignedLog" type="string" string="/logs/%HOSTNAME%/%programname%.log") action(type="omfile" dynaFile="SignedLog" FileCreateMode="0640" DirCreateMode="0750" cmd="/usr/bin/rsyslog-log-signer -s /etc/rsyslog/signing.key") ``` **Method 2: Append-only mount (simpler and more effective)** ```bash # Mount log partition (append-only) # WARNING: After this, logs can only be added, not deleted! sudo mount -o remount,append-only /logs # Verification mount | grep /logs # /dev/sdb1 on /logs type ext4 (rw,appendonly) ``` ```markdown ┌─────────────────────────────────────────────────────────────┐ │ LOG INTEGRITY: PROTECTION METHOD COMPARISON │ ├─────────────────────────────────────────────────────────────┤ │ APPEND-ONLY FILESYSTEM (mount -o append-only / chattr +a): │ │ • Prevents: Deleting and overwriting existing lines │ │ • Allows: Only continuously appending new log data │ │ • Practice: Low overhead, ideal for Linux log servers │ │ │ │ CRYPTOGRAPHIC SIGNING (rsyslog-log-signer / RFC 5424): │ │ • Prevents: Forgery and manipulation by attackers │ │ • Protection: Integrity check of every individual log block │ │ │ │ RECOMMENDATION: mTLS forwarding combined with append-only! │ └─────────────────────────────────────────────────────────────┘ ``` ### Practice: 3-Host Setup **Host overview:** | Host | IP | Role | rsyslog Role | |------|-----|------|--------------| | webserver | 192.168.1.10 | nginx, Docker | Sender | | dbserver | 192.168.1.11 | PostgreSQL | Sender | | logserver | 192.168.1.12 | Log aggregation | Receiver | **Complete configuration for each host:** ```bash # ───────────────────────────────────────────────────── # Log Server (192.168.1.12) # ───────────────────────────────────────────────────── # 1. Install certificates sudo cp ca.crt server.key server.crt /etc/rsyslog/certs/ # 2. Enable receiver configuration # /etc/rsyslog.d/00-receiver.conf # 3. Open firewall sudo nft add rule inet filter input \ tcp dport 6514 accept comment "rsyslog TLS" # 4. Create log directories sudo mkdir -p /logs/{auth,web,db,monitor} sudo chown -R root:adm /logs # 5. Start rsyslog sudo systemctl enable --now rsyslog # 6. Verification sudo ss -tlnp | grep 6514 # LISTEN 0 128 0.0.0.0:6514 0.0.0.0:* users:(("rsyslogd",pid=1234,fd=6)) ``` ```bash # ───────────────────────────────────────────────────── # Web Server (192.168.1.10) # ───────────────────────────────────────────────────── # 1. Install certificates sudo cp ca.crt webserver.key webserver.crt /etc/rsyslog/certs/ # 2. Store sender configuration # /etc/rsyslog.d/50-forward.conf # 3. Start rsyslog sudo systemctl enable --now rsyslog # 4. Verification sudo rsyslogd -N1 # config validation run took 0.001 seconds ``` ```bash # ───────────────────────────────────────────────────── # DB Server (192.168.1.11) # ───────────────────────────────────────────────────── # 1. Install certificates sudo cp ca.crt dbserver.key dbserver.crt /etc/rsyslog/certs/ # 2. Store sender configuration # /etc/rsyslog.d/50-forward.conf # 3. Additionally capture PostgreSQL logs # /etc/rsyslog.d/51-postgres.conf: # $AddUnixListenSocket /var/run/postgresql/.s.PGSQL.5432 # :programname, isequal, "postgres" /var/log/postgres.log # & stop # 4. Start rsyslog sudo systemctl enable --now rsyslog ``` **End-to-end verification:** ```bash # 1. On web server: Generate test log logger -t test "Multi-Host-Logging Test from webserver" # 2. On log server: Check sudo tail -f /logs/webserver/test.log # Aug 27 12:34:56 webserver test: Multi-Host-Logging Test from webserver # 3. Check auth logs sudo tail -f /logs/auth/audit.log # Aug 27 12:34:56 webserver sshd[1234]: Accepted publickey for user # 4. Check TLS connection openssl s_client -connect logserver:6514 \ -cert /etc/rsyslog/certs/webserver.crt \ -key /etc/rsyslog/certs/webserver.key \ -CAfile /etc/rsyslog/certs/ca.crt # Verify return code: 0 (ok) ``` ### Retention and Compliance **GDPR requirements:** * **Access logs:** 10 days (minimum retention) * **Complete logs:** 6 months (in case of data breach) * **Deletion concept:** Automatic deletion after expiry **German IT Security Act (IT-SiG):** * **Technical log data:** 12 months * **Audit logs:** 12 months * **For critical infrastructure (KRITIS):** 24 months **Log rotation with automatic deletion:** ```ini # /etc/logrotate.d/remote-logs /logs/*/*.log { daily rotate 90 compress delaycompress missingok notifempty create 0640 root adm dateext dateformat -%Y-%m-%d postrotate /usr/bin/systemctl kill -s HUP rsyslog.service > /dev/null 2>&1 || true endscript } ``` ```ini # /etc/cron.d/log-cleanup (Daily at 03:00: Delete logs older than 365 days) 0 3 * * * root find /logs -name "*.log.*" -mtime +365 -delete ```
💡 **Audit before deletion:** An admin should review the affected logs before automatic deletion.
### Verification ```bash # 1. On ALL hosts: Check rsyslog syntax sudo rsyslogd -N1 # Expectation: "config validation run took 0.001 seconds" # 2. Log server: Port 6514 open? sudo ss -tlnp | grep 6514 # LISTEN 0 128 0.0.0.0:6514 0.0.0.0:* users:(("rsyslogd",...)) # 3. Sender -> Receiver: TLS connection sudo openssl s_client -connect logserver:6514 \ -cert /etc/rsyslog/certs/webserver.crt \ -key /etc/rsyslog/certs/webserver.key \ -CAfile /etc/rsyslog/certs/ca.crt \ /dev/null | grep "Verify return code" # Verify return code: 0 (ok) # 4. Log arrival on log server logger -t verification "Test from $(hostname)" sleep 1 sudo tail -5 /logs/$(hostname)/$(date +%Y-%m-%d).log # Aug 27 12:34:56 webserver verification: Test from webserver # 5. Queue status (during network issues) sudo find /var/spool/rsyslog/ -name "fwd_queue*" -ls # 6. Log integrity check mount | grep /logs # /dev/sdb1 on /logs type ext4 (rw,appendonly) ```
⚠️ **Do not forget the firewall:** The log server must allow `Port 6514` (TCP) from each host: `nft add rule inet filter input tcp dport 6514 accept`. Without this rule, no logs can be received.
## Backup Security: Encryption, Integrity, and the 3-2-1 Rule ### Why Backup Security? An attacker who gains root access on a server has two priorities: first, cover tracks (delete logs), second, take data hostage. This means: encrypt backups, delete local snapshots, and issue a ransom demand. Without secured, encrypted offsite backups, you face a choice: data lost or pay ransom. GDPR Art. 34 requires notification to the supervisory authority in case of data breaches. If you can prove that backups exist and are restorable, the damage is considerably reduced. If not, a fine looms. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ ATTACK PLAYBOOK: RANSOMWARE ON LINUX SERVERS │ ├─────────────────────────────────────────────────────────────┤ │ 1. GAIN ROOT ACCESS: │ │ • SSH compromise, kernel exploit, or container escape │ │ │ │ 2. IDENTIFY BACKUPS & SNAPSHOTS: │ │ • Local BTRFS snapshots: `btrfs subvolume list /` │ │ • Local archives under /backup/ and /var/backup/ │ │ │ │ 3. DESTROY LOCAL BACKUPS & ENCRYPT FILES: │ │ • Delete snapshots: `btrfs subvolume delete /snapshots/*` │ │ • Encrypt production data & leave ransom note │ │ │ │ PROTECTION: Untouchable, immutable offsite backups! │ └─────────────────────────────────────────────────────────────┘ ``` ### LUKS Header Backup (Cross-Reference) The backup of the LUKS header was covered in detail in the section [LUKS2 Header Management and Offline Backups](/en/online-courses/arch-linux-serie/arch-linux-advanced-security-features-and-maintenance#luks2-header-management-offline-backups){.badge-link-text}. Summary: * **Header backup on USB stick** (store offline, not on the same server) * **Regular test:** `sudo cryptsetup luksHeaderRestore` from the USB stick * **Why critical:** Without the header, all data is inaccessible, even with a known password
💡 **LUKS header + offsite backup = complete protection.** The LUKS header protects the boot process, offsite backups protect the data from ransomware. Together they form a complete security concept.
### BTRFS Snapshot Security BTRFS snapshots are a powerful tool for quick rollbacks, but they are **not a backup**. A snapshot is a consistent state at a specific point in time – but it offers no protection against deletion by an attacker or hardware failure. **What happens during compromise:** ```bash # Attacker finds snapshots sudo btrfs subvolume list / # ID 257 gen 5 path @ # ID 258 gen 100 path @home # ID 300 gen 200 path @snapshots/2026-08-01 # ID 301 gen 210 path @snapshots/2026-08-15 # ID 302 gen 220 path @snapshots/2026-08-27 # Attacker deletes ALL snapshots sudo btrfs subvolume delete /@snapshots/2026-08-01 sudo btrfs subvolume delete /@snapshots/2026-08-15 sudo btrfs subvolume delete /@snapshots/2026-08-27 # -> All snapshots gone. No rollback possible. ``` **Protection: Read-only snapshots** ```bash # Create read-only snapshot (cannot be deleted, only via *) sudo btrfs subvolume snapshot -r / /@snapshots/2026-08-27-readonly # WARNING: read-only snapshots CANNOT be deleted # * Unless: attacker mounts the volume and makes it read-write # -> Therefore: secure snapshots on EXTERNAL volume! ``` **Secure snapshots to external volume:** ```bash # ───────────────────────────────────────────────────── # Snapper + Push-to-NAS (automated) # ───────────────────────────────────────────────────── # Mount NAS (e.g., via NFS) sudo mount -t nfs 192.168.1.200:/backup /mnt/nas # Adjust Snapper configuration sudo nano /etc/snapper/configs/root ``` ```bash # Snapper configuration SUBVOLUME="/" FSTYPE="btrfs" # Snapshot directory on NAS TIMELINE_MIN_AGE="1800" TIMELINE_LIMIT_HOURLY="5" TIMELINE_LIMIT_DAILY="7" TIMELINE_LIMIT_WEEKLY="4" TIMELINE_LIMIT_MONTHLY="6" # Post-snapshot script: push snapshot to NAS RUN_FREQUENCY=1 ``` ```bash # Create post-snapshot script sudo nano /etc/snapper/post-snapshot.d/10-push-to-nas.sh ``` ```bash #!/bin/bash # Synchronize snapshot to NAS (after each new snapshot) SNAPSHOT_DIR="/@snapshots" NAS_TARGET="/mnt/nas/snapshots/$(hostname)" # Create NAS target directory mkdir -p "$NAS_TARGET" # Push newest snapshot to NAS NEWEST=$(ls -td ${SNAPSHOT_DIR}/*/ | head -1) rsync -a --delete "${NEWEST}" "${NAS_TARGET}/" # Verification echo "Snapshot pushed: $(basename $NEWEST) -> $NAS_TARGET" ``` ```bash # Make executable sudo chmod +x /etc/snapper/post-snapshot.d/10-push-to-nas.sh ``` **BTRFS Scrub (integrity check):** ```bash # Regular integrity check of all snapshots sudo btrfs scrub start / # Check status sudo btrfs scrub status / # UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # Scrub started: Mon Aug 27 12:00:00 2026 # Status: finished # Duration: 0:05:23 # Total to scrub: 128.00GB # Rate: 400.00MB/s # Error summary: no errors found # Cron: Monthly automatic scrub echo "0 3 1 * * root btrfs scrub start /" | \ sudo tee /etc/cron.d/btrfs-scrub ```
⚠️ **Snapshots are not a backup.** A snapshot protects against accidental file deletion and enables quick rollbacks. But it does NOT protect against: hardware failure, ransomware (which deletes snapshots), server theft. For these scenarios, you need offsite backups.
**Cross-Reference: Snapper Basics** The installation and configuration of Snapper with BTRFS was covered in detail in the [previous article](/en/online-courses/arch-linux-serie/arch-linux-best-practices-and-maintenance-tips){.badge-link-text}. There you will find the setup for `snapper`, `snap-pac`, and `grub-btrfs`. ### Backup Encryption There are several tools for encrypted backups. Each has its own strengths: **Comparison of methods:** | Feature | Restic | BorgBackup | rsync + GPG | LUKS Container | |---------|--------|------------|-------------|----------------| | Encryption | AES-256 | AES-256 | GPG (varies) | AES-256 (LUKS) | | Deduplication | ✅ | ✅ | ❌ | ❌ | | Incremental | ✅ | ✅ | ✅ (rsync) | ❌ | | Anonymization | ❌ | ❌ | ❌ | ❌ | | Complexity | Medium | Medium | Low | Low | | Performance | Good | Very good | Good | Very good | | Multi-backend | ✅ (S3, B2, SFTP) | ✅ (SSH) | ✅ (SSH) | ❌ (local volume) | **Method 1: Restic (recommended for homelab)** ```bash # ───────────────────────────────────────────────────── # Restic: Encrypted backup with deduplication # ───────────────────────────────────────────────────── # Install sudo pacman -S restic # Create repository (encrypted) # Password is requested interactively restic init --repo /mnt/nas/backups/$(hostname) # Or: Use password file echo "MySecureBackupPassword123!" | \ restic init --repo /mnt/nas/backups/$(hostname) --password-file /dev/stdin # Create backup restic backup /home /etc /var/log \ --repo /mnt/nas/backups/$(hostname) \ --password-file /mnt/usb/backup-passwd.txt \ --verbose # Incremental backup (only changed data) restic backup /home /etc /var/log \ --repo /mnt/nas/backups/$(hostname) \ --password-file /mnt/usb/backup-passwd.txt # Display snapshots restic snapshots --repo /mnt/nas/backups/$(hostname) # Restore backup restic restore latest \ --repo /mnt/nas/backups/$(hostname) \ --target /restore/ ``` **Restic cron for automatic backups:** ```bash sudo nano /etc/cron.d/restic-backup ``` ```ini # Daily at 02:00: Create backup 0 2 * * * root restic backup /home /etc /var/log \ --repo /mnt/nas/backups/$(hostname) \ --password-file /mnt/usb/backup-passwd.txt \ --verbose >> /var/log/restic-backup.log 2>&1 # Weekly (Sunday 04:00): Delete old snapshots # (keeps: 7 daily, 4 weekly, 6 monthly) 0 4 * * 0 root restic forget --repo /mnt/nas/backups/$(hostname) \ --password-file /mnt/usb/backup-passwd.txt \ --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \ --prune --verbose >> /var/log/restic-forget.log 2>&1 ``` **Method 2: BorgBackup (Pull mode)** ```bash # ───────────────────────────────────────────────────── # BorgBackup: Deduplication + Encryption # ───────────────────────────────────────────────────── # On the backup server (NAS): sudo pacman -S borg # Create Borg repository borg init --encryption=repokey /mnt/nas/backups/$(hostname) # On the source server (pull mode): # The backup server pulls the data (no SSH key needed on source!) # Borg pull configuration on backup server sudo nano /etc/borg.d/pull-webserver.conf ``` ```ini # Borg Pull configuration SOURCE_HOST="webserver.local" SOURCE_USER="backup" SOURCE_PATH="/home /etc" REMOTE_BORG_REPO="/mnt/nas/backups/webserver" # SSH connection to source server borg create --stats --progress \ ssh://${SOURCE_USER}@${SOURCE_HOST}${REMOTE_BORG_REPO}::'{hostname}-{now}' \ ${SOURCE_PATH} ``` **Method 3: rsync + GPG (simple and flexible)** ```bash # ───────────────────────────────────────────────────── # rsync + GPG: Encrypted transfer # ───────────────────────────────────────────────────── # Generate GPG key (one-time) gpg --gen-key # Note the fingerprint! ``` ## Security Audits and Auditd ### Why Security Audits? A security audit is not a one-time action, but a recurring process. The question is not whether a system will be compromised, but when. Regular audits find vulnerabilities before an attacker exploits them. **When is an audit performed?** | Occasion | Frequency | Focus | |----------|-----------|-------| | Routine | Semi-annually | Complete check | | After changes | As needed | Affected component | | Incident response | After incident | Damage limitation | | Compliance | Annually | GDPR, IT-SiG, BSI IT-Grundschutz | | Before deploy | As needed | Configuration, dependencies | ### Audit Checklist The following checklist serves as a working template for every audit. Each item is documented with status (OK/missing/critical) and justification. **System:** | # | Check | Command | Status | |---|-------|---------|--------| | 1.1 | Kernel version (hardened?) | `uname -r` | | | 1.2 | Installed packages up to date? | `checkupdates` | | | 1.3 | Users without password? | `awk -F: '($2 == "") {print $1}' /etc/shadow` | | | 1.4 | Root without password? | `passwd -S root` | | | 1.5 | Unused users deactivated? | `lastlog \| grep -v "Never logged in"` | | | 1.6 | SUID binaries reduced? | `find / -perm -4000 -type f 2>/dev/null` | | **Network:** | # | Check | Command | Status | |---|-------|---------|--------| | 2.1 | Open ports only necessary? | `ss -tlnp` | | | 2.2 | Firewall active? | `nft list ruleset` | | | 2.3 | SSH: Root login disabled? | `grep PermitRootLogin /etc/ssh/sshd_config` | | | 2.4 | SSH: Password auth disabled? | `grep PasswordAuthentication /etc/ssh/sshd_config` | | | 2.5 | TLS on all services? | `openssl s_client -connect ...` | | **Encryption:** | # | Check | Command | Status | |---|-------|---------|--------| | 3.1 | LUKS active? | `lsblk -f` | | | 3.2 | LUKS header backup present? | Physically checked | | | 3.3 | SSH key encryption? | `ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key` | | | 3.4 | TLS certificates valid? | `openssl x509 -in ... -checkend 0` | | **Logging:** | # | Check | Command | Status | |---|-------|---------|--------| | 4.1 | auditd active? | `systemctl status auditd` | | | 4.2 | rsyslog active? | `systemctl status rsyslog` | | | 4.3 | Central logs present? | `ls /logs/` | | | 4.4 | Log rotation active? | `cat /etc/logrotate.d/remote-logs` | | **Backups:** | # | Check | Command | Status | |---|-------|---------|--------| | 5.1 | Backups present? | `restic snapshots --repo ...` | | | 5.2 | Integrity checked? | `restic check --read-data ...` | | | 5.3 | Restore test performed? | Documentation present | | | 5.4 | Offsite backup present? | Physically checked | | ### auditd: The Linux Audit Framework `auditd` is the kernel-level auditing for Linux. It logs system calls, file accesses, and user actions at kernel level – meaning even a root-compromised account cannot delete the audit logs without stopping the audit daemon (which is normally not possible). **Installation and activation:** ```bash # Install sudo pacman -S audit # Enable and start sudo systemctl enable --now auditd # Verify sudo systemctl status auditd # active (running) ``` **Write audit rules:** ```bash sudo nano /etc/audit/rules.d/10-security.rules ``` ```bash # Audit rules for security monitoring # 1. Monitor file access # /etc/shadow: Every access is logged -w /etc/shadow -p rwa -k shadow_access # /etc/sudoers: Changes are logged -w /etc/sudoers -p wa -k sudoers_mod -w /etc/sudoers.d/ -p wa -k sudoers_mod # /etc/ssh/sshd_config: Changes are logged -w /etc/ssh/sshd_config -p wa -k sshd_config # /etc/passwd, /etc/group: Changes are logged -w /etc/passwd -p wa -k user_mod -w /etc/group -p wa -k group_mod -w /etc/gshadow -p wa -k group_mod # 2. Monitor system calls # execve: Which programs are executed? -a always,exit -F arch=b64 -S execve -k program_exec # connect: Which network connections are established? -a always,exit -F arch=b64 -S connect -k network_connect # mount: Filesystems are mounted -a always,exit -F arch=b64 -S mount -k mount_op # umount: Filesystems are unmounted -a always,exit -F arch=b64 -S umount2 -k umount_op # 3. Monitor user actions # Login attempts (successful and failed) -w /var/log/lastlog -p wa -k login_events -w /var/run/faillock/ -p wa -k login_events # SSH logins -w /var/log/auth.log -p wa -k auth_log # sudo usage -w /var/log/sudo.log -p wa -k sudo_usage # 4. Monitor kernel modules # Load/unload modules -a always,exit -F arch=b64 -S init_module -k module_load -a always,exit -F arch=b64 -S delete_module -k module_unload # 5. Performance: Log only relevant events # Increase buffer size for audit events -b 8192 # Prevent loss of audit events -f 1 ``` **Load audit rules:** ```bash # Load rules sudo auditctl -R /etc/audit/rules.d/10-security.rules # Display loaded rules sudo auditctl -l # -w /etc/shadow -p rwa -k shadow_access # -w /etc/sudoers -p wa -k sudoers_mod # -a always,exit -F arch=b64 -S execve -k program_exec # ... ``` **Evaluate audit logs:** ```bash # ───────────────────────────────────────────────── # ausearch: Search for specific events # ───────────────────────────────────────────────── # All accesses to /etc/shadow sudo ausearch -k shadow_access --interpret # All execve events (program calls) sudo ausearch -k program_exec --interpret # All network connections sudo ausearch -k network_connect --interpret # Events from the last hour sudo ausearch -ts recent -k auth_log # Events of a specific user sudo ausearch -ua 1000 --interpret # ───────────────────────────────────────────────── # aureport: Create summaries # ───────────────────────────────────────────────── # Overall overview sudo aureport # Auth report (login attempts) sudo aureport --auth # Failed login report sudo aureport --failed # Login report by user sudo aureport --login --summary # File access report sudo aureport -f # Error report sudo aureport -e ``` **Example: Detecting suspicious activity** ```bash # All accesses to /etc/shadow in the last 24 hours sudo ausearch -k shadow_access -ts today --interpret # Expected output: # type=PROCTITLE msg=audit(2026-08-27 12:34:56.789:1234) : proctitle=cat /etc/shadow # type=SYSCALL msg=audit(2026-08-27 12:34:56.789:1234) : arch=c000003e syscall=257 success=yes ... # uid=0 root (expected: only root should read this) # exe=/usr/bin/cat # Or: Access by unknown user (CRITICAL!) # type=SYSCALL msg=audit(...) : uid=1001 webserver # exe=/usr/bin/cat -> WARNING: webserver reads /etc/shadow! ```
⚠️ **Audit performance:** Too many audit rules can impact performance. Every execve event creates an audit entry. On an active webserver with many HTTP requests, this can cause significant overhead. Restrict execve auditing to suspicious programs, not everything.
**Optimize audit rules:** ```bash # ───────────────────────────────────────────────── # Performance-optimized audit rules # ───────────────────────────────────────────────── # ONLY log execve for specific programs # (not for everything, only for suspicious ones) -a always,exit -F arch=b64 -S execve -F path=/usr/bin/sudo -k sudo_exec -a always,exit -F arch=b64 -S execve -F path=/usr/bin/su -k su_exec -a always,exit -F arch=b64 -S execve -F path=/usr/bin/passwd -k passwd_exec # Only log connect for specific UIDs -a always,exit -F arch=b64 -S connect -F uid=0 -k root_network # Excluded programs (performance) # These programs are NOT logged -a never,exit -F arch=b64 -S execve -F exe=/usr/bin/bash -a never,exit -F arch=b64 -S execve -F exe=/usr/bin/zsh ``` ### LinPEAS: Automated Vulnerability Scanning LinPEAS (Linux Privilege Escalation Awesome Scripts) is an automated tool that checks a system for known vulnerabilities and misconfigurations that could lead to privilege escalation. **Installation and execution:** ```bash # Download LinPEAS curl -L https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh -o /tmp/linpeas.sh chmod +x /tmp/linpeas.sh # Run as root (with colored output) sudo /tmp/linpeas.sh # Run as root (without colors, for log file) sudo /tmp/linpeas.sh -a 2>&1 | tee /var/log/linpeas-$(date +%Y%m%d).log ``` **Important findings and their meaning:** ```bash # ───────────────────────────────────────────────── # LinPEAS: Important findings # ───────────────────────────────────────────────── # 1. SUID binaries (red = critical) # -> Binaries with SUID bit can inherit root privileges # -> Risk: If a SUID binary has a vulnerability, # an attacker can use it to become root # -> Action: Remove unnecessary SUID bits # 2. Writable /etc/passwd (red = critical) # -> Anyone can write to /etc/passwd # -> Risk: Attacker can create own root user entry # -> Action: Fix permissions (0644) # 3. World-writable files (yellow = medium) # -> Files that anyone can write to # -> Risk: Attacker can modify configuration files # -> Action: Fix permissions # 4. Cleartext passwords in config files (red = critical) # -> Passwords in plaintext in configuration files # -> Risk: Immediately readable on compromise # -> Action: Move passwords to secrets backends # 5. Insecure SSH config (yellow = medium) # -> PermitRootLogin yes, PasswordAuthentication yes # -> Risk: Brute-force or root access via SSH # -> Action: Harden SSH configuration ```
💡 **LinPEAS vs. manual audit:** LinPEAS automatically finds the most common vulnerabilities. But it does not replace a manual audit. It does not check for: unknown zero-days, company-specific requirements, or complex attack vectors. LinPEAS is the FIRST step, not the last.
### Audit Combination: All Tools Together A complete security audit combines multiple tools in phases: ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Security Audit: 3-Phase Model │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Phase 1: Automated (Duration: ~30 min) │ │ ──────────────────────────────────────── │ │ - lynis audit system │ │ - checksec procAll │ │ - LinPEAS (privilege escalation scanner) │ │ - auditd: ausearch -ts today │ │ - Output: raw data, findings & vulnerabilities │ │ │ │ Phase 2: Manual (Duration: 2-4 hrs) │ │ ────────────────────────────────── │ │ - Check kernel hardening & sysctl │ │ - Audit open ports & TLS certificates │ │ - Check user permissions & SUID binaries │ │ - Verify backup integrity & restores │ │ - Validate auditd rule set & filters │ │ │ │ Phase 3: Report (Duration: 1-2 hrs) │ │ ────────────────────────────────── │ │ - Rate findings by severity (Critical/High/...) │ │ - Derive concrete measures & hardening plan │ │ - Create audit-proof archive (3 years) │ │ - Schedule next audit date (+6 months) │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Phase 1: Automated** ```bash # Save Lynis output for report sudo lynis audit system --no-colors 2>&1 | tee /tmp/lynis-report.txt # checksec for all processes checksec --no-banner --no-warnings procAll 2>&1 | tee /tmp/checksec-report.txt # Save LinPEAS output for report sudo /tmp/linpeas.sh -a 2>&1 | tee /tmp/linpeas-report.txt # Audit events from the last 24 hours sudo ausearch -ts today --interpret 2>&1 | tee /tmp/audit-report.txt ``` **Phase 2: Manual (selection)** ```bash # Check kernel version uname -r # -> 7.1.9-hardened1-1-hardened ✓ # Check unused users lastlog | grep "Never logged in" # -> No unused users ✓ # Check open ports ss -tlnp # -> Only SSH (22), nginx (80, 443) ✓ # Check TLS certificates openssl x509 -in /etc/nginx/certs/site.crt -checkend 2592000 # -> "Certificate will expire in 365 days" ✓ ``` ### Practice: Semi-Annual Security Audit **Step 1: Preparation** ```bash # Set audit date and scope AUDIT_DATE=$(date +%Y-%m-%d) AUDIT_DIR="/root/audits/${AUDIT_DATE}" mkdir -p "$AUDIT_DIR" # Store audit metadata echo "Security Audit started: $AUDIT_DATE" > "$AUDIT_DIR/README.md" echo "Scope: Complete system" >> "$AUDIT_DIR/README.md" ``` **Step 2: Automated tools** ```bash # Lynis sudo lynis audit system --no-colors 2>&1 | tee "$AUDIT_DIR/lynis.txt" # checksec checksec --no-banner --no-warnings procAll 2>&1 | tee "$AUDIT_DIR/checksec.txt" # LinPEAS sudo /tmp/linpeas.sh -a 2>&1 | tee "$AUDIT_DIR/linpeas.txt" ``` **Step 3: Manual checks** ```bash # Kernel uname -r > "$AUDIT_DIR/kernel.txt" # Users lastlog > "$AUDIT_DIR/users.txt" awk -F: '($2 == "") {print $1}' /etc/shadow >> "$AUDIT_DIR/users.txt" # Network ss -tlnp > "$AUDIT_DIR/ports.txt" nft list ruleset > "$AUDIT_DIR/firewall.txt" # Encryption lsblk -f > "$AUDIT_DIR/encryption.txt" # Backups restic snapshots --repo /mnt/nas/backups/$(hostname) > "$AUDIT_DIR/backups.txt" 2>&1 # Audit sudo auditctl -l > "$AUDIT_DIR/audit-rules.txt" sudo aureport > "$AUDIT_DIR/audit-summary.txt" ``` **Step 4: Create report** ```bash cat > "$AUDIT_DIR/bericht.md" << 'EOF' # Security Audit Report ## Date $(date +%Y-%m-%d) ## Summary | Category | Status | Note | |----------|--------|------| | Kernel | OK | linux-hardened active | | Users | OK | No unused accounts | | Network | OK | Only necessary ports open | | Encryption | OK | LUKS active, header backup present | | Logging | OK | auditd + rsyslog active | | Backups | OK | Restic active, integrity checked | ## Findings ### Critical - None ### High - None ### Medium - LinPEAS: 2 SUID binaries identified as unnecessary ### Low - Kernel update available (7.1.9 -> 7.1.10) ## Recommendations 1. [ ] Remove unnecessary SUID bits 2. [ ] Perform kernel update 3. [ ] Next audit: $(date -d "+6 months" +%Y-%m-%d) EOF ``` ```bash # Archive audit directory tar czf "/root/audits/${AUDIT_DATE}.tar.gz" "$AUDIT_DIR" echo "Audit completed: $AUDIT_DIR" ```
💡 **Audit archiving:** Store every audit report for at least 3 years. In the event of a security incident or compliance audit, you can prove that regular audits were conducted.
### Verification ```bash # ───────────────────────────────────────────────────── # Check audit status # ───────────────────────────────────────────────────── # 1. auditd active? sudo systemctl status auditd # active (running) # 2. Audit rules loaded? sudo auditctl -l # -w /etc/shadow -p rwa -k shadow_access # ... # 3. Audit events present? sudo ausearch -ts today --interpret | wc -l # > 0 # 4. Lynis report present? ls -la /root/audits/ # drwxr-x--- 2 root root 4096 Aug 27 12:34 2026-08-27 # 5. Next audit date? echo "Next audit: $(date -d '+6 months' +%Y-%m-%d)" # Next audit: 2027-02-27 ``` ## Official Documentation | Topic | Documentation | |-------|---------------| | Arch Linux Wiki | [wiki.archlinux.org](https://wiki.archlinux.org/){.badge-link-text} | | AppArmor | [apparmor.net](https://apparmor.net/){.badge-link-text} | | linux-hardened | [github.com/anthraxx/linux-hardened](https://github.com/anthraxx/linux-hardened){.badge-link-text} | | nftables Wiki | [wiki.nftables.org](https://wiki.nftables.org/){.badge-link-text} | | auditd | [github.com/linux-audit/audit-userspace](https://github.com/linux-audit/audit-userspace){.badge-link-text} | | Restic | [restic.net](https://restic.net/){.badge-link-text} | | BorgBackup | [borgbackup.readthedocs.io](https://borgbackup.readthedocs.io/){.badge-link-text} | | checksec | [github.com/slimm609/checksec.sh](https://github.com/slimm609/checksec.sh){.badge-link-text} | | LinPEAS | [github.com/carlospolop/PEASS-ng](https://github.com/carlospolop/PEASS-ng){.badge-link-text} | | systemd-cryptenroll | [freedesktop.org/software/systemd/man/systemd-cryptenroll.html](https://www.freedesktop.org/software/systemd/man/systemd-cryptenroll.html){.badge-link-text} | ## Further Articles in This Series | Article | Topic | |---------|-------| | [System Hardening and Security](/en/online-courses/arch-linux-serie/arch-linux-system-hardening-security-best-practices){.badge-link-text} | Fundamentals: user management, nftables, SSH, Fail2Ban | | [Best Practices and Tips](/en/online-courses/arch-linux-serie/arch-linux-best-practices-and-maintenance-tips){.badge-link-text} | BTRFS, Snapper, snap-pac, grub-btrfs | | [Package Management with Pacman](/en/online-courses/arch-linux-serie/arch-linux-comprehensive-guide-package-manager-pacman){.badge-link-text} | Package sources, signatures, cleanup | | [Installation and Basic Configuration](/en/online-courses/arch-linux-serie/archlinux-installation-and-basic-configuration){.badge-link-text} | UEFI, partitioning, BTRFS, encryption | | [Graphical User Interface](/en/online-courses/arch-linux-serie/arch-linux-installing-the-graphical-user-interface){.badge-link-text} | Wayland, PipeWire, SDDM, KDE Plasma 6 | ## Conclusion: The Hardened Arch Linux Complete System With the completion of this sixth part, you have mastered the entire journey from empty storage medium to a highly secured, production-grade Linux environment. Arch Linux has proven that it is far more than a mere tinkering distribution: through absolute transparency and minimal base, a security level can be realized that surpasses even many pre-configured server distributions. **We have hardened the system layer by layer:** 1. **Foundation:** Clean package management with `pacman`, encrypted BTRFS subvolumes, and secure boot via GRUB and TPM2. 2. **Operations:** Low-maintenance automation with systemd timers and fail-safe rollbacks through `snapper` and `grub-btrfs`. 3. **Defense in Depth:** Mandatory Access Control with `AppArmor`, kernel hardening through `linux-hardened`, and ASLR optimization. 4. **Isolation:** Container network segmentation with `nftables` without dangerous Docker defaults. 5. **Observability and Resilience:** Tamper-proof central TLS logging, audit-proof `auditd` monitoring, and an immutable 3-2-1 backup strategy.
💡 **Final practical tip:** Security is not a static state, but a continuous process. Keep your audit routines semi-annual, regularly check audit logs for anomalies, and test your backup restores at least once a quarter under real conditions.
You now possess a deep understanding of the architecture of modern Linux systems and the tools to design, operate, and defend production systems to the highest security standards.