---
id: 2026-08-26-linux-server-hardening-fido2-crowdsec
slug: linux-server-hardening-fido2-crowdsec
title: "Linux server hardening: SSH with FIDO2/YubiKey and CrowdSec"
excerpt: "Linux server hardening with SSH FIDO2/YubiKey for passwordless authentication and CrowdSec as a collaborative intrusion-prevention system on Ubuntu 24.04/26.04."
date: "2026-08-26 12:00:00"
updated: "2026-08-26 12:00:00"
author:
name: "Sebastian PalencsΓ‘r"
handle: "spalencsar"
category: "server-environments"
tags: ["linux", "security", "ssh", "fido2", "yubikey", "crowdsec", "hardening"]
reading_time: 40
toc: true
---
SSH brute-force attacks run continuously against every public server. Most Linux servers still default to passwords or static SSH keys β both have known weaknesses. Passwords fall to credential stuffing and GPU clusters in minutes; static keys stay unchanged until someone rotates them by hand.
On **Ubuntu 24.04 LTS** and **Ubuntu 26.04 LTS** the server is locked down with two concrete measures: **SSH with FIDO2 and YubiKey** for hardware-based, passwordless authentication, and **CrowdSec** as a collaborative intrusion-prevention system. Together they close the most common attack vectors without password rotation or key-management busywork.
The configuration works on a fresh server the same as on an existing setup β no assumptions about the current state.
π‘ **Prerequisites:** Basic knowledge of `SSH` and `systemd`. An Ubuntu 24.04 or 26.04 server with root or sudo access. For FIDO2 you need a YubiKey (or another FIDO2 device). For [CrowdSec](https://www.crowdsec.net/){.badge-link-text} a TCP port for SSH is enough. The audience is Linux administrators, DevOps engineers and homelab operators who want to harden existing servers or stand up new ones to a current standard.
β οΈ **Note:** Hardening here is at operating-system level. Application-specific hardening (web servers, databases) stays out of scope, because each application has its own requirements.
## SSH with FIDO2 and YubiKey
### Why FIDO2 for SSH?
Passwords are the weakest link in the SSH chain. Every server with a [public IP](/en/netzwerk/oeffentliche-vs-private-ip-adresse-die-hauptunterschiede){.badge-link-text} is permanently worked by brute-force scanners β `fail2ban` logs show that daily. Static SSH keys (Ed25519, RSA) are better, but they do not rotate. Anyone holding a compromised private key stays current unnoticed.
FIDO2 solves both problems:
The private key is stored on a physical device (YubiKey) and never leaves it. Without the YubiKey on the local machine, login is impossible β no matter how good the password or key is. FIDO2 additionally enforces user verification (PIN or biometrics) on every login.
**What changes:**
| Property | Password | Static SSH key | FIDO2 + YubiKey |
|-------------|----------|-------------------|-----------------|
| Physical component | No | No | Yes |
| Brute-force resistant | Conditional | Yes | Yes |
| Phishing resistant | No | Conditional | Yes |
| Rotation | Manual | Manual | Automatic (key on device) |
| Multi-factor | Optional | No | Standard |
The combination of physical token and PIN makes SSH logins nearly unattackable. An attacker would need both the YubiKey and the PIN β and both at the same physical station.
π‘ **FIDO2 vs. U2F:** FIDO2 is the successor to U2F and supports WebAuthn (browser) and CTAP2 (native apps). For SSH, CTAP2 is what matters β the YubiKey is addressed directly by the SSH client, with no browser detour.
### FIDO2 and WebAuthn: how it works
FIDO2 is challenge-response based. The server sends a random challenge, the YubiKey signs it with a private key stored on the device, and the server verifies the signature. The private key never leaves the YubiKey.
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β FIDO2 challenge-response β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β ββββββββββββββββ ββββββββββββ ββββββββββββββββ β
β β SSH client β β YubiKey β β SSH server β β
β ββββββββ¬ββββββββ ββββββ¬ββββββ ββββββββ¬ββββββββ β
β β β β β
β β Challenge (random) β β β
β β ββββββββββββββββββ>β β β
β β β PIN / Touch β β
β β β ββββββββββ (User) β β
β β β β β
β β Signature(Chall.) β β β
β β <βββββββββββββββββ β β β
β β β β β
β β Challenge + sig. β β β
β β ββββββββββββββββββββββββββββββββββββββ>β β
β β β β β
β β β Verify(PubKey,Sig)β β
β β β βββββββββββββββββ β β
β β β β β
β β Access granted β β β
β β <βββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
**What happens in detail:**
1. The SSH client detects that a FIDO2 key is configured
2. It asks the YubiKey to sign the challenge
3. The user enters their PIN or touches the sensor
4. The YubiKey signs the challenge with the ed25519-sk key pair
5. The server verifies the signature against the stored public key
6. Login is granted
The decisive difference from static keys: the challenge is different on every login attempt. Even if someone intercepts the signature, they cannot reuse it (replay protection).
### YubiKey hardware setup
For SSH with FIDO2, every YubiKey 5-series with USB-C or USB-A is suitable. What matters is that the key supports **FIDO2** and not only U2F β you see that from the product name "YubiKey 5" (not "YubiKey Security Key", which only does U2F).
**Initialise the YubiKey:**
```bash
# Install YubiKey Manager (Debian/Ubuntu)
sudo apt install yubikey-manager -y
# Set the FIDO2 PIN (default PIN is empty)
ykman fido access change-pin
```
The PIN protects the YubiKey against unauthorised use. Choose a PIN with at least 6 characters. Without a PIN, anyone with physical access to the YubiKey can read and use the FIDO2 keys.
**Verification:**
```bash
# Check whether the YubiKey is detected
ykman fido info
```
Expected output:
```bash
Applications: FIDO2
slots: 2
pin: true
pin retries: 3
```
β οΈ **Backup YubiKey and break-glass emergency anchor:** Always create at least a second YubiKey as a physical backup in a safe place. If you lose your only YubiKey, you are locked out β there is no reset mechanism for FIDO2 keys. For root and cloud servers, also protect the hoster's out-of-band web console (for example Hetzner Robot, Proxmox NoVNC or IPMI) with a separate 2FA as an emergency anchor.
### Generate SSH keys with FIDO2
OpenSSH 8.2+ supports FIDO2 keys natively. The `ssh-keygen` command with `-t ed25519-sk` creates a key that is stored on the YubiKey.
**Generate the key:**
```bash
# Generate a FIDO2 SSH key
ssh-keygen -t ed25519-sk -C "admin@server"
```
During generation you are asked to touch the YubiKey and enter your FIDO2 PIN. The private key is stored on the YubiKey, not on disk.
**What happens during generation:**
1. `ssh-keygen` creates an ed25519-sk key pair
2. The private key is placed in the YubiKey's secure element
3. The public key is stored in `~/.ssh/id_ed25519-sk.pub`
4. Every login attempt requires the YubiKey
**Options:**
```bash
# Without touch (PIN only)
ssh-keygen -t ed25519-sk -O no-touch-required
# With a custom filename
ssh-keygen -t ed25519-sk -f ~/.ssh/id_yubikey -C "admin@server"
```
π‘ **Resident vs. non-resident keys:** With `-O resident` the private key is stored fully on the YubiKey and can be exported from there. Without that option only a handle is stored. For SSH, non-resident (the default) is recommended, because the key is then not exportable.
**Copy the public key to the server:**
```bash
# Transfer the public key to the server
ssh-copy-id -i ~/.ssh/id_ed25519-sk.pub user@server
```
If `ssh-copy-id` has trouble with FIDO2 (sometimes the case on older OpenSSH versions), copy the key manually:
```bash
# Manual installation
cat ~/.ssh/id_ed25519-sk.pub | ssh user@server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
```
**Check on the server:**
```bash
# Check the installed key
cat ~/.ssh/authorized_keys | grep "sk@"
```
The line should start with `ssh-ed25519-cert-v01@openssh.com` or contain `sk@` β that confirms it is a FIDO2 key.
### Configure the SSH server for FIDO2
The SSH server must know that it accepts FIDO2 keys. That is already true in the default configuration, but we want to be explicit and set additional security settings.
**Server configuration:**
```bash
sudo nano /etc/ssh/sshd_config
```
**Relevant settings:**
```ini
# Allow FIDO2 keys (default: yes)
PubkeyAuthentication yes
# Allow SSH keys only (disable passwords)
PasswordAuthentication no
# Forbid root login
PermitRootLogin no
# Maximum of 3 authentication attempts
MaxAuthTries 3
# Cap grace time at 30 seconds
LoginGraceTime 30
# Minimise information disclosure
DebianBanner no
Banner none
# Keepalive for long-running sessions
ClientAliveInterval 300
ClientAliveCountMax 2
```
**Optional: allow only FIDO2 keys (no more static keys):**
If you want to use FIDO2 keys exclusively, you can restrict `PubkeyAuthOptions`:
```bash
# Accept only verified FIDO2 keys
PubkeyAuthOptions verify-required
```
β οΈ **Caution with `verify-required`:** This option accepts only FIDO2 keys with user verification (PIN or biometrics). Static Ed25519 and RSA keys are rejected. Test the change first in an additional session before you close the existing one.
**Restart the SSH server:**
```bash
sudo systemctl restart sshd
```
β οΈ **Never close the active session before you have tested!** Open a second terminal and test login with the new FIDO2 key. Only when that login works do you close the old session.
**Test:**
```bash
# Login with the FIDO2 key
ssh -i ~/.ssh/id_ed25519-sk user@server
# Check which method was used
ssh -v user@server 2>&1 | grep "Offering"
```
The verbose output should show `Offering public key: .../id_ed25519-sk` and `Accepted publickey` as the result.
### Certificate-based authentication
For larger environments an SSH certificate authority (CA) pays off. Instead of distributing individual keys on every server, a CA signs public keys and the server trusts only the CA.
**Create the CA on the signing host:**
```bash
# Generate the CA key pair
sudo mkdir -p /etc/ssh/ca
sudo ssh-keygen -t ed25519 -f /etc/ssh/ca/server-ca -C "SSH CA"
```
**Distribute the public CA key to all servers:**
```bash
# On every server: register the CA public key as a TrustedKey
echo "cert-authority $(cat /etc/ssh/ca/server-ca.pub)" >> /etc/ssh/authorized_keys.d/trusted-ca
```
**Sign a user key with the CA:**
```bash
# Sign the FIDO2 key (creates a certificate)
sudo ssh-keygen -s /etc/ssh/ca/server-ca \
-I "admin-fido2" \
-n admin \
-V +52w \
~/.ssh/id_ed25519-sk.pub
```
**Options:**
| Option | Meaning |
|--------|----------|
| `-s` | CA private key for signing |
| `-I` | Certificate ID (for logs) |
| `-n` | Allowed usernames (comma-separated) |
| `-V` | Validity (`+52w` = 52 weeks) |
The generated certificate (`id_ed25519-sk-cert.pub`) is recognised automatically by the SSH client and sent along at login.
π‘ **Advantage of the CA:** On every server you only need to place the public CA key. Individual user keys are managed centrally and can be revoked through the CA without editing `authorized_keys` on every server.
### SSH config and ssh-agent
For daily use a clean SSH configuration on the client pays off.
**`~/.ssh/config`:**
```ini
# Server definition with FIDO2 key
Host server
HostName 192.168.1.100
User admin
IdentityFile ~/.ssh/id_ed25519-sk
IdentitiesOnly yes
# Jump host for internal servers
Host internal-server
HostName 10.0.0.50
User admin
ProxyJump server
IdentityFile ~/.ssh/id_ed25519-sk
IdentitiesOnly yes
```
**`IdentitiesOnly yes`** forces the SSH client to use only the specified key. That prevents another key from the agent being offered by accident.
**ssh-agent with FIDO2:**
The `ssh-agent` can support FIDO2 keys, but because the private key stays on the YubiKey, only the handle is stored in the agent. For FIDO2 the agent makes little sense β the YubiKey has to be physically attached anyway.
```bash
# Start ssh-agent (optional for FIDO2)
eval $(ssh-agent -s)
# Add the key (touch is requested on every login)
ssh-add ~/.ssh/id_ed25519-sk
```
π‘ **Recommendation:** Skip ssh-agent for FIDO2 keys. The agent brings no advantage, because the YubiKey must be touched on every login. Use `IdentityFile` in `~/.ssh/config` instead.
### Troubleshooting: common SSH-FIDO2 problems
**Problem: "Key rejected by server"**
```bash
# Check whether FIDO2 is supported on the SSH server
ssh -V # OpenSSH_8.9 or higher
# Check the server configuration
sudo sshd -T | grep pubkeyauth
```
β οΈ **Cause: OpenSSH version too old (below 8.2).** FIDO2 support arrived with OpenSSH 8.2. Ubuntu 24.04 has OpenSSH 9.6, Ubuntu 26.04 has OpenSSH 9.7 β both support FIDO2.
**Problem: "Sign_and_send_pubkey: signing failed: agent refused operation"**
```bash
# ssh-agent is running and trying to sign the key
# Fix: specify the key explicitly or disable the agent
ssh -i ~/.ssh/id_ed25519-sk user@server
```
Cause: ssh-agent tries to sign the key with an old method. `IdentitiesOnly yes` in the SSH config solves the problem.
**Problem: "Touch your authenticator" does not appear**
```bash
# Check whether the YubiKey is detected
lsusb | grep Yubico
# Check whether the FIDO2 module is loaded
ls /dev/hidraw*
```
Cause: YubiKey not detected or drivers missing. On Ubuntu all required drivers are present by default.
**Problem: "PIN auth required" but the PIN is not prompted**
```bash
# Set or reset the FIDO2 PIN manually
ykman fido access change-pin
```
β οΈ **Cause:** The YubiKey was initialised without a PIN, or the PIN is outdated.
**Verification on the server:**
```bash
# Check logs
journalctl -u sshd -f
# Recognise successful FIDO2 logins
journalctl -u sshd | grep "Accepted publickey"
```
The logs show which key was used and whether it was a FIDO2 key (`sk@`).
## CrowdSec: collaborative intrusion prevention
### Fail2Ban vs. CrowdSec
Fail2Ban reads local log files and bans IPs through iptables/nftables rules. Each server acts in isolation β an attacker scanning from 1,000 IPs has to be banned on every server individually.
CrowdSec works differently: the agent analyses logs and decisions are shared centrally in a community database. When an attacker is banned on one community server, everyone else benefits. CrowdSec also separates analysis (agent) from enforcement (bouncer), which makes the architecture more flexible.
**What changes:**
| Property | Fail2Ban | CrowdSec |
|-------------|----------|----------|
| Log analysis | Local | Local + community |
| Enforcement | iptables directly | Bouncer (external) |
| Performance | Good with few jails | Scalable for many servers |
| Community | None | Shared decisions |
| Dashboard | None | Web dashboard + metrics |
| Architecture | Monolith | Agent + bouncer split |
π‘ **Core idea:** CrowdSec collects signatures of attack patterns (scenarios) and compares them with your logs. Hits produce decisions (bans) that bouncers enforce. Community intelligence supplements local detection.
### Installation and first-time setup
**Install the CrowdSec agent:**
```bash
# Add the repository (Debian/Ubuntu)
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
# Install CrowdSec
sudo apt install crowdsec -y
```
**Install the firewall bouncer:**
```bash
# Bouncer for nftables/iptables
sudo apt install crowdsec-firewall-bouncer-nftables -y
```
β οΈ **Watch the order:** Install the agent first, then the bouncer. The bouncer connects to the agent through the local API. Do not start the bouncer before the agent is running.
**Check status:**
```bash
sudo systemctl status crowdsec
sudo systemctl status crowdsec-firewall-bouncer
```
Expected output for the agent:
```bash
β crowdsec.service - CrowdSec SIEM
Loaded: loaded (/lib/systemd/system/crowdsec.service; enabled)
Active: active (running) since Mon 2026-08-26 12:00:00 UTC
```
**Check first decisions:**
```bash
sudo cscli decisions list
```
In the initial state the list is empty. As soon as the agent analyses logs and recognises attack patterns, decisions appear here.
### Scenarios and decisions
CrowdSec works with scenarios β signatures that describe particular attack patterns. The agent compares log entries with these scenarios and creates decisions on hits.
**Scenarios enabled by default:**
```bash
sudo cscli scenarios list
```
| Name | Service | Type |
|------|---------|------|
| `ssh-bf` | ssh | leak |
| `ssh-slow-bf` | ssh | leak |
| `http-bad-user-agent` | http | leak |
| `http-dos-bf-path` | http | leak |
**Scenarios relevant for SSH:**
| Scenario | Detection | Action |
|----------|-----------|--------|
| `ssh-bf` | 5 failed logins in 10 min | Ban for 1h |
| `ssh-slow-bf` | Slow brute force over hours | Ban for 24h |
| `ssh-crawl` | Behavioural analysis of login attempts | Ban for 4h |
**Create your own scenarios:**
If the default scenarios do not fit, you can define your own:
```bash
sudo nano /etc/crowdsec/scenarios/custom-ssh-aggressive.yaml
```
```yaml
type: leak
name: ssh-aggressive-scan
description: "Aggressive SSH scans with many failed logins"
labels:
service: ssh
remediation: true
filter:
LABELS.service == "ssh" && MSG.ContainerName == "sshd"
groupby: LABELS.source_ip
capacity: 5
leakspeed: 1m
blackhole: 1m
certainty: 75
for: 10m
```
**Verification:**
```bash
# Reload scenarios
sudo cscli scenarios list | grep custom
# Simulate a test scenario
sudo cscli scenarios install --local /etc/crowdsec/scenarios/custom-ssh-aggressive.yaml
```
### Bouncers (firewall integration)
Bouncers enforce the agent's decisions. There are different bouncer types for different use cases.
**nftables bouncer (default):**
```bash
# Check whether the bouncer is active
sudo cscli bouncers list
```
```bash
Name β IP Address β Valid Until β Type
ββββββββββββΌβββββββββββββΌββββββββββββββΌββββββ
firewall β 127.0.0.1 β 2027-01-01 β nftables
```
**Bouncer types:**
| Bouncer | Use | Advantage |
|---------|---------|---------|
| `nftables` | Server firewall | Fast, close to the kernel |
| `nginx` | Web server inline | No extra port |
| `caddy` | Caddy integration | Simple configuration |
| `crowdsec-openapi` | External APIs | Flexible |
**Check firewall rules:**
```bash
# Check the CrowdSec chain in nftables
sudo nft list chain inet crowdsec-crowdsec-block
```
```bash
table inet crowdsec-crowdsec {
chain crowdsec-block {
type filter hook input priority filter; policy accept;
ip saddr @crowdsec-blacklist-jump drop
ip6 saddr @crowdsec-blacklist6-jump drop
}
}
```
### Whitelisting
Not every IP should be banned. Local servers, monitoring systems or trusted networks need whitelisting.
**Whitelist file:**
```bash
sudo nano /etc/crowdsec/parsers/s02-enrich/whitelist.yaml
```
```yaml
filter: "Meta.IP != ''"
whitelist:
reason: "Local address"
expression: "Ip('127.0.0.1') || Ip('::1')"
expression: "Ip('192.168.0.0/16')"
expression: "Ip('10.0.0.0/8')"
```
**Alternatively: whitelist via cscli:**
```bash
# Whitelist a single IP
sudo cscli decisions add --ip 192.168.1.100 --duration 0 --type whitelist --reason "Monitoring"
# Whitelist an IP range
sudo cscli decisions add --range 192.168.1.0/24 --duration 0 --type whitelist --reason "Local network"
```
β οΈ **Check the whitelist before bans:** Always check whether an IP might come from a local monitoring system or a trusted network before you analyse bans. Otherwise CrowdSec accidentally bans your own systems.
### Dashboards and alerting
CrowdSec provides a web dashboard and metrics for monitoring systems.
**Web dashboard:**
```bash
# Show dashboard credentials
sudo cscli console status
```
The dashboard is reachable at `http://localhost:6060` (default port). Adjust the firewall for external access.
**Metrics for Prometheus:**
```bash
# Check the metrics endpoint
curl -s http://localhost:6060/metrics | head -20
```
```bash
# Important metrics
crowdsec_decisions_total{source_ip="...",scenario="..."}
crowdsec_alerts_total{scenario="..."}
crowdsec_bouncers_total{type="..."}
```
**Alerting via webhook:**
```bash
sudo nano /etc/crowdsec/notifications/email.yaml
```
```yaml
type: email
name: email_default
log_level: info
# Email configuration
smtp_host: "smtp.example.com"
smtp_port: 587
smtp_username: "alert@example.com"
smtp_password: "YOUR_PASSWORD"
smtp_from: "crowdsec@example.com"
smtp_to:
- "admin@example.com"
smtp_subject: "CrowdSec Alert: {scenario}"
format: |
CrowdSec detected an attack.
Scenario: {scenario}
Source IP: {source_ip}
Decision: {decision}
Time: {created_at}
```
**Verification:**
```bash
# Check notifications
sudo cscli notifications list
# Send a test notification
sudo cscli alert inspect --id 1
```
### Performance and scaling
For larger environments (more than 10 servers) a central CrowdSec instance pays off.
**Architecture for multiple servers:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CrowdSec multi-server architecture β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β βββββββββββ βββββββββββ βββββββββββ β
β β Server1 β β Server2 β β Server3 β β
β β Agent β β Agent β β Agent β β
β ββββββ¬βββββ ββββββ¬βββββ ββββββ¬βββββ β
β β β β β
β βΌ βΌ βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Local API (central) β β
β β http://central:8080 β β
β ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Community database β β
β β (shared decisions) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
**Point the local agent at the central API:**
```bash
sudo nano /etc/crowdsec/config.yaml
```
```yaml
api:
client:
insecure_skip_verify: false
server:
listen_uri: http://127.0.0.1:8080
```
π‘ **Performance:** A single CrowdSec agent can process 10,000+ log entries per second without trouble. For most setups a central agent on a dedicated server is enough.
### Troubleshooting
**Problem: "crowdsec service not running"**
```bash
# Check logs
sudo journalctl -u crowdsec -f
# Common cause: port 8080 occupied
sudo ss -tulnp | grep 8080
```
**Problem: "Bouncer not connected"**
```bash
# Check the bouncer API token
sudo cscli bouncers list
# Regenerate the token
sudo cscli bouncers add my-bouncer --auto
```
**Problem: own IPs are being banned**
```bash
# Check the whitelist
sudo cscli decisions list --type whitelist
# Add the local IP to the whitelist
sudo cscli decisions add --ip 192.168.1.100 --duration 0 --type whitelist --reason "Own server"
```
**Verification:**
```bash
# Check active decisions
sudo cscli decisions list
# Check the agent logs
sudo journalctl -u crowdsec | grep "decision"
```
## System hardening
### Kernel parameters (sysctl)
The Linux kernel exposes dozens of parameters through `sysctl` that shrink the attack surface. The defaults are tuned for compatibility, not security.
**Create the file:**
```bash
sudo nano /etc/sysctl.d/99-hardening.conf
```
**Network security:**
```ini
# Prevent IP spoofing
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
# Disable source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
# Enable SYN flood protection
net.ipv4.tcp_syncookies = 1
# Log martian packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
```
**Memory protection:**
```ini
# Address space layout randomisation
kernel.randomize_va_space = 2
# Restrict dmesg (root only)
kernel.dmesg_restrict = 1
# Hide kernel pointers
kernel.kptr_restrict = 2
# Restrict BPF
kernel.unprivileged_bpf_disabled = 1
# Restrict user namespaces (for containers)
kernel.unprivileged_userns_clone = 0
```
**Prevent kernel loading:**
```ini
# Prevent loading further modules (after boot)
kernel.modules_disabled = 1
```
β οΈ **`kernel.modules_disabled = 1`** prevents loading new kernel modules after boot. That can break NVIDIA drivers or VirtualBox modules, for example. Only enable it when you are sure every required module is loaded at boot.
**Load the parameters:**
```bash
sudo sysctl -p /etc/sysctl.d/99-hardening.conf
```
**Verification:**
```bash
# Check whether parameters are set
sysctl net.ipv4.conf.all.rp_filter
sysctl kernel.randomize_va_space
```
**Expected output:**
```bash
net.ipv4.conf.all.rp_filter = 1
kernel.randomize_va_space = 2
```
### Filesystem hardening
By default many files and directories have permissions that are too open. `/tmp`, `/var/tmp` and `/dev/shm` in particular are attack surface.
**Mount options for `/tmp`:**
```bash
# /tmp is often already a tmpfs; check with:
mount | grep tmp
```
If `/tmp` sits on the main disk, a dedicated partition with safe mount options is recommended:
```bash
# Put this in /etc/fstab (with a separate /tmp partition)
# UUID=... /tmp ext4 defaults,noexec,nosuid,nodev 0 2
```
**Set directory permissions strictly:**
```bash
# Lock down critical directories
sudo chmod 700 /root
sudo chmod 600 /etc/shadow
sudo chmod 600 /etc/gshadow
sudo chmod 644 /etc/passwd
sudo chmod 644 /etc/group
# SSH directory
sudo chmod 700 /root/.ssh
sudo chmod 600 /root/.ssh/authorized_keys
```
**Reduce SUID/SGID bits:**
```bash
# Check SUID bits on standard tools
find / -perm -4000 -type f 2>/dev/null
```
**Remove unnecessary SUID bits (examples):**
```bash
# Only if the programs do not need to be SUID
sudo chmod u-s /usr/bin/newgrp
sudo chmod u-s /usr/bin/chsh
sudo chmod u-s /usr/bin/chfn
```
π‘ **Tip:** `find / -perm -4000 -type f` shows every program with the SUID bit. Do not remove blindly β some programs need the bit for their function (for example `sudo`, `passwd`).
### Service minimisation
Every running service is potential attack surface. On a fresh Ubuntu server dozens of services run that you may not need.
**Check active services:**
```bash
# Show every running service
systemctl list-units --type=service --state=running
# Services often unnecessary on a server
systemctl list-units --type=service --state=running | grep -E "cups|bluetooth|avahi|modem"
```
**Disable unnecessary services:**
```bash
# Printer service (not needed on a server)
sudo systemctl stop cups
sudo systemctl disable cups
sudo systemctl mask cups
# Bluetooth (not needed on a server)
sudo systemctl stop bluetooth
sudo systemctl disable bluetooth
sudo systemctl mask bluetooth
# mDNS/Avahi (service discovery β unnecessary on a server)
sudo systemctl stop avahi-daemon
sudo systemctl disable avahi-daemon
sudo systemctl mask avahi-daemon
```
**`mask` vs. `disable`:**
| Action | Effect |
|--------|--------|
| `disable` | Service does not start automatically, but can still be started manually |
| `mask` | Service cannot be started at all (symlink to `/dev/null`) |
β οΈ **Only mask what you truly do not need.** If you later need a service after all, you first have to run `systemctl unmask`.
**Check the systemd security score:**
```bash
# Show the security score for a service
systemd-analyze security sshd.service
```
```bash
β Overall exposure level for sshd.service: 9.6 OK
```
The lower the value, the better. `sshd` normally has a good score when `PasswordAuthentication no` and `PermitRootLogin no` are set.
### Audit logging
`auditd` records security-relevant events and stores them in a tamper-resistant log file. That is the foundation for forensics and compliance.
**Installation:**
```bash
sudo apt install auditd -y
sudo systemctl enable auditd
```
**Rules for SSH and sudo:**
```bash
# Watch the SSH configuration
sudo auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config
# Watch authorized_keys
sudo auditctl -w /root/.ssh/authorized_keys -p wa -k ssh_keys
# Watch the sudo configuration
sudo auditctl -w /etc/sudoers -p wa -k sudoers
sudo auditctl -w /etc/sudoers.d/ -p wa -k sudoers
# Watch crontab
sudo auditctl -w /etc/crontab -p wa -k cron
sudo auditctl -w /var/spool/cron/ -p wa -k cron
```
**Persist the rules:**
```bash
sudo nano /etc/audit/rules.d/hardening.rules
```
```ini
# SSH
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /root/.ssh/authorized_keys -p wa -k ssh_keys
# Sudo
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
# Cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
# User changes
-w /etc/passwd -p wa -k user_change
-w /etc/shadow -p wa -k user_change
-w /etc/group -p wa -k group_change
```
**Verification:**
```bash
# List rules
sudo auditctl -l
# Search audits for a key
sudo ausearch -k sshd_config --interpret
# Create an audit report
sudo aureport --auth
```
### File-integrity monitoring (AIDE)
AIDE (Advanced Intrusion Detection Environment) compares files against a known state and reports changes.
**Installation:**
```bash
sudo apt install aide -y
```
**Initial scan:**
```bash
# Initialise the database
sudo aideinit
```
β οΈ **The initial scan can take time.** On a large server the first scan can take 10-30 minutes. Do not run it during production operation.
**Activate the database:**
```bash
# Move the new database into place
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
```
**Cron-based check:**
```bash
sudo nano /etc/cron.d/aide-check
```
```ini
# Daily check at 03:00
0 3 * * * root /usr/bin/aide --check | mail -s "AIDE Report" admin@example.com
```
**Verification:**
```bash
# Run a manual check
sudo aide --check
# Expected output on a clean system:
# AIDE found NO differences between database and filesystem.
```
### Automatic security patches (unattended upgrades)
A purely manual patch routine leaves zero-day holes and known CVEs unprotected for days. Automatic security patches close critical vulnerabilities promptly, while `needrestart` restarts stale processes after library updates.
**Installation and activation:**
```bash
# Install automatic updates and the restart checker
sudo apt install unattended-upgrades needrestart -y
# Activate the configuration
sudo dpkg-reconfigure -plow unattended-upgrades
```
**Adjust the configuration (`/etc/apt/apt.conf.d/50unattended-upgrades`):**
```ini
// Install security updates automatically only
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
};
// Automatically remove orphaned dependencies
Unattended-Upgrade::Remove-Unused-Dependencies "true";
// Automatic reboot on kernel updates at 04:00
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";
```
**Verification:**
```bash
# Dry run without changes
sudo unattended-upgrade --dry-run --debug
```
### Mandatory access control (AppArmor)
AppArmor restricts applications (such as Nginx, Unbound or PHP-FPM) to the absolutely necessary filesystem and network access. Even if a service has a remote-code-execution vulnerability, the AppArmor profile prevents breakout into the system.
**Check status and active profiles:**
```bash
sudo aa-status
```
**Expected output:**
```bash
apparmor module is loaded.
XX profiles are in enforce mode.
0 processes are unconfined but have a profile defined.
```
**Switch profiles to enforce mode:**
```bash
# Activate profiles and enforce them strictly
sudo apt install apparmor-utils -y
sudo aa-enforce /etc/apparmor.d/*
```
π‘ **Tip:** You can record new or custom profiles with `aa-genprof ` in interactive learning mode and then put them live with `aa-enforce`.
## Network hardening
### nftables: the modern firewall
`nftables` has replaced `iptables` as the default firewall manager. On Ubuntu 24.04 and 26.04, `nftables` is the kernel default. Anyone still using `iptables` is working through a compatibility layer β that works, but `nftables` is more efficient and has clearer syntax.
**Basic structure:**
```bash
# Show current tables
sudo nft list ruleset
```
```bash
table inet filter {
chain input {
type filter hook input priority filter; policy drop;
# Rules here
}
chain forward {
type filter hook forward priority filter; policy drop;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
```
**Create the firewall baseline:**
```bash
sudo nano /etc/nftables.conf
```
```bash
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority filter; policy drop;
# Allow established connections
ct state established,related accept
# Allow loopback
iif lo accept
# Allow ICMP (ping)
ip protocol icmp accept
ip6 nexthdr icmpv6 accept
# Allow SSH (port 22)
tcp dport 22 accept
# Allow HTTP/HTTPS (if a web server)
tcp dport { 80, 443 } accept
# Logging for dropped traffic
log prefix "nft-drop: " counter drop
}
chain forward {
type filter hook forward priority filter; policy drop;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
```
**Start the service:**
```bash
sudo systemctl enable nftables
sudo systemctl start nftables
```
β οΈ **Watch the order:** On remote servers always allow SSH (port 22) first, before you set the default policy to `drop`. Otherwise you lock yourself out.
**Rate limiting for SSH:**
```bash
# SSH rate limit: at most 5 connections per minute
sudo nft add rule inet filter input tcp dport 22 meter ssh-rate { limit rate 5/minute burst 5 packets } accept
```
**Verification:**
```bash
# Check active rules
sudo nft list ruleset
# Check connections
sudo ss -tulnp | grep -E "22|80|443"
# Check drops in the logs
sudo journalctl -k | grep "nft-drop"
```
### CrowdSec + nftables integration
CrowdSec works with `nftables` through the firewall bouncer. The agent analyses logs and the bouncer enforces bans through `nftables` rules.
**Check bouncer rules:**
```bash
# Show CrowdSec-specific chains
sudo nft list chain inet crowdsec-crowdsec-block
```
```bash
table inet crowdsec-crowdsec {
chain crowdsec-block {
type filter hook input priority filter; policy accept;
ip saddr @crowdsec-blacklist-jump drop
ip6 saddr @crowdsec-blacklist6-jump drop
}
}
```
**Traffic flow:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Firewall traffic flow β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β [Inbound traffic] β
β β β
β βΌ β
β ββββββββββββββββββββ β
β β CrowdSec check β βββ Whitelist? βββΆ ACCEPT β
β β (blacklist) β βββ Ban? βββββββββΆ DROP + LOG β
β ββββββββββ¬ββββββββββ β
β β β
β βΌ β
β ββββββββββββββββββββ β
β β Firewall rules β βββ SSH/HTTP βββββΆ ACCEPT β
β β (nftables) β βββ Rest βββββββββΆ DROP β
β ββββββββββ¬ββββββββββ β
β β β
β βΌ β
β ββββββββββββββββββββ β
β β CrowdSec agent β βββ Analyse logs β
β β (log analysis) β βββ Create decisions β
β ββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
### TLS configuration
For web servers, TLS 1.3 is the current standard. TLS 1.2 is still acceptable, but older versions (1.0, 1.1) should be disabled.
**Let's Encrypt with Certbot:**
```bash
# Install Certbot
sudo apt install certbot -y
# Request a certificate (Nginx)
sudo certbot --nginx -d example.com
# Request a certificate (Apache)
sudo certbot --apache -d example.com
```
**TLS configuration for Nginx:**
```bash
sudo nano /etc/nginx/snippets/ssl-params.conf
```
```nginx
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
# Security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options DENY always;
```
**Certificate auto-renewal:**
```bash
# Cron job for automatic renewal
sudo crontab -e
```
```ini
# Daily at 02:30, renew the certificate if needed
30 2 * * * certbot renew --quiet --post-hook "systemctl reload nginx"
```
**Verification:**
```bash
# Test the TLS configuration
openssl s_client -connect example.com:443 -tls1_3
# Check certificate details
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates
```
### DNS-over-HTTPS
By default the server sends DNS queries unencrypted over UDP port 53. Anyone on the path can see which domains you resolve. DNS-over-HTTPS (DoH) encrypts those queries.
**Unbound as a local resolver:**
```bash
sudo apt install unbound -y
```
**Configuration:**
```bash
sudo nano /etc/unbound/unbound.conf.d/hardening.conf
```
```ini
server:
interface: 127.0.0.1
access-control: 127.0.0.0/8 allow
# Enable DNSSEC
auto-trust-anchor-file: /var/lib/unbound/root.key
# Privacy
hide-identity: yes
hide-version: yes
# Performance
msg-cache-slabs: 8
rrset-cache-slabs: 8
infra-cache-slabs: 8
key-cache-slabs: 8
```
**Forward upstream with Cloudflare DoH:**
```ini
forward-zone:
name: "."
forward-addr: 1.1.1.1@853#cloudflare-dns.com
forward-addr: 9.9.9.9@853#dns.quad9.net
forward-tls-upstream: yes
```
**Start the service:**
```bash
sudo systemctl enable unbound
sudo systemctl start unbound
```
**Verification:**
```bash
# Test DNS resolution
dig example.com @127.0.0.1
# DNSSEC test
dig dnssec-failed.org @127.0.0.1
# Expectation: SERVFAIL (DNSSEC breaker recognised correctly)
# Test DNS-over-HTTPS
curl -v https://cloudflare-dns.com/dns-query?name=example.com 2>&1 | grep "HTTP/2"
```
π‘ **Recommendation:** For servers with many clients (for example a homelab), Unbound as a local resolver is the best choice. You configure clients with `127.0.0.1` as DNS server; Unbound resolves upstream over DoH/DoT.
## Automation
### Ansible playbook for server hardening
The previous sections showed which configurations are needed. Running individual commands on every server is labour-intensive and error-prone. Ansible automates that β you describe the desired state, and Ansible applies it on all servers.
**Project structure:**
```bash
mkdir -p ~/ansible-hardening/{roles,inventory,group_vars}
```
**Inventory:**
```bash
nano ~/ansible-hardening/inventory/hosts.ini
```
```ini
[production]
server1 ansible_host=192.168.1.100
server2 ansible_host=192.168.1.101
[homelab]
server3 ansible_host=192.168.10.50
[all:vars]
ansible_user=admin
ansible_python_interpreter=/usr/bin/python3
```
**Playbook:**
```bash
nano ~/ansible-hardening/hardening.yml
```
```yaml
---
- name: Linux server hardening
hosts: all
become: yes
vars:
ssh_port: 22
allowed_users:
- admin
enable_crowdsec: true
enable_aide: true
tasks:
# System updates
- name: Update the system
apt:
update_cache: yes
upgrade: dist
# SSH hardening
- name: Harden SSH configuration
lineinfile:
path: /etc/ssh/sshd_config
regexp: "{{ item.regexp }}"
line: "{{ item.line }}"
loop:
- { regexp: '^#?PermitRootLogin', line: 'PermitRootLogin no' }
- { regexp: '^#?PasswordAuthentication', line: 'PasswordAuthentication no' }
- { regexp: '^#?MaxAuthTries', line: 'MaxAuthTries 3' }
- { regexp: '^#?LoginGraceTime', line: 'LoginGraceTime 30' }
notify: restart sshd
# Sysctl hardening
- name: Set sysctl parameters
sysctl:
name: "{{ item.name }}"
value: "{{ item.value }}"
state: present
reload: yes
loop:
- { name: 'net.ipv4.conf.all.rp_filter', value: '1' }
- { name: 'net.ipv4.conf.all.accept_redirects', value: '0' }
- { name: 'kernel.randomize_va_space', value: '2' }
- { name: 'kernel.dmesg_restrict', value: '1' }
# Unnecessary services
- name: Disable unnecessary services
systemd:
name: "{{ item }}"
state: stopped
enabled: no
masked: yes
loop:
- cups
- bluetooth
- avahi-daemon
# Audit logging
- name: Install auditd
apt:
name: auditd
state: present
- name: Deploy audit rules
copy:
src: files/audit-rules.rules
dest: /etc/audit/rules.d/hardening.rules
notify: restart auditd
handlers:
- name: restart sshd
systemd:
name: sshd
state: restarted
- name: restart auditd
systemd:
name: auditd
state: restarted
```
**Run the playbook:**
```bash
cd ~/ansible-hardening
ansible-playbook -i inventory/hosts.ini hardening.yml
```
π‘ **Idempotence:** Ansible is idempotent β you can run the playbook multiple times without the system changing unnecessarily. Only divergent states are corrected.
**Verification:**
```bash
# Check whether every server is reachable
ansible all -i inventory/hosts.ini -m ping
# Check SSH configuration
ansible all -i inventory/hosts.ini -m shell -a "sshd -T | grep permitrootlogin"
```
Expected output:
```bash
server1 | CHANGED | rc=0 >>
permitrootlogin no
server2 | CHANGED | rc=0 >>
permitrootlogin no
```
### CIS scoring with OpenSCAP
CIS (Center for Internet Security) benchmarks define security standards for various systems. OpenSCAP checks your system against those benchmarks and produces a score.
**Installation:**
```bash
sudo apt install libopenscap8 scap-security-guide -y
```
**Run the Ubuntu 24.04/26.04 benchmark:**
```bash
# Show available profiles
oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2404-ds.xml | grep "Profile"
# CIS Level 1 scan
sudo oscap xccdf eval \
--profile cis_level1_server \
--results cis-results.xml \
--report cis-report.html \
/usr/share/xml/scap/ssg/content/ssg-ubuntu2404-ds.xml
```
**Open the report in the browser:**
```bash
# Copy the report to the local machine and open it
scp admin@server:/tmp/cis-report.html ~/Desktop/
```
**Interpret the score:**
| Score | Meaning |
|-------|-----------|
| 100% | All CIS recommendations met |
| 80-99% | Good state, smaller adjustments needed |
| 60-79% | Baseline configuration, some hardening missing |
| < 60% | Critical gaps, urgent action required |
**Automated remediation (caution):**
```bash
# Only if you understand the impact
sudo oscap xccdf generate fix \
--profile cis_level1_server \
--fix-type ansible \
--output cis-remediation.yml \
/usr/share/xml/scap/ssg/content/ssg-ubuntu2404-ds.xml
```
β οΈ **Remediation only with intent:** Automatic remediation can overwrite working configurations. Check every change before running it. When in doubt: manual and with judgement.
**Verification:**
```bash
# Check the score
oscap xccdf generate report cis-results.xml > /dev/null
grep -E "score|pass|fail" cis-results.xml | tail -10
```
## Hardening checklist
| Step | Measure | Effect |
|---------|----------|---------|
| 1 | SSH: `PasswordAuthentication no`, `PermitRootLogin no` | Brute force blocked |
| 2 | Set up FIDO2/YubiKey | Passwordless, hardware-based auth |
| 3 | Install CrowdSec agent + bouncer | Collaborative intrusion protection |
| 4 | Apply sysctl hardening | Lock down kernel parameters |
| 5 | Mask unnecessary services | Shrink attack surface |
| 6 | Enable auditd | Log security events |
| 7 | Initialise AIDE | File-integrity monitoring |
| 8 | Set up nftables firewall | Filter network traffic |
| 9 | TLS 1.3 + security headers | Harden the web server |
| 10 | Configure Unbound + DoH | Encrypt DNS queries |
Each step builds on the previous one. The order is deliberate: first the most critical weaknesses (SSH, CrowdSec), then system, then network layer.
## Further Resources
### Official documentation
[OpenSSH FIDO2](https://www.openssh.com/manual.html){.badge-link-text}
[CrowdSec documentation](https://doc.crowdsec.net/){.badge-link-text}
[nftables wiki](https://wiki.nftables.org/){.badge-link-text}
[CIS Benchmarks](https://www.cisecurity.org/cis-benchmarks){.badge-link-text}
[OpenSCAP documentation](https://www.open-scap.org/tools/){.badge-link-text}
### Tools and packages
[YubiKey Manager](https://developers.yubico.com/yubikey-manager/){.badge-link-text}
[CrowdSec Hub](https://hub.crowdsec.net/){.badge-link-text}
[Certbot](https://certbot.eff.org/){.badge-link-text}
[Unbound](https://nlnet.nl/projects/unbound/){.badge-link-text}
### Related articles on admindocs.de
| Topic | Article |
|-------|---------|
| VPN fundamentals | [Set up a WireGuard VPN server](/en/server-environments/set-up-a-wireguard-vpn-server-on-ubuntu){.badge-link-text} |
| Containers | [Ubuntu 26.04 Docker](/en/server-environments/install-and-use-docker-on-ubuntu-26-04){.badge-link-text} |
| Automation | [Ansible fundamentals](/en/devops/ansible-grundlagen-automatisierung-fuer-linux-administratoren){.badge-link-text} |
| Networking | [Analyse network problems](/en/netzwerk/netzwerkprobleme-unter-linux-systematisch-analysieren){.badge-link-text} |
| Fundamentals | [LPIC-1 series](/en/category/lpic-1-serie){.badge-link-text} |
| Server | [Ubuntu upgrade: 22.04 to 24.04 LTS](/en/server-environments/ubuntu-upgrade-from-22-04-lts-to-24-04-lts){.badge-link-text} |
## Conclusion
Server hardening is not a one-off project, but an ongoing process. The measures described here β SSH with FIDO2, CrowdSec, kernel hardening, nftables and TLS β form the foundation every production Linux system should build on.
What matters is the starting point: not everything at once, but step by step. The hardening checklist above sets the order. With each step the attack surface shrinks, and the combination of several measures creates a robust security profile that does not give individual weaknesses the same damage potential.
For teams, automation with Ansible pays off β what is configured once can be transferred to any number of servers. CIS benchmarks with OpenSCAP set the score and show where there is still room to improve.
**A hardened server is not a server that is secured against everything.** It is a server where you know what is secured and what is not β and why.