--- id: 2026-08-06-analyzing-network-problems-on-linux-systematically slug: analyzing-network-problems-on-linux-systematically title: "Analyzing network problems on Linux systematically" excerpt: "How you analyse and fix network problems on Linux in a structured way: IP addressing, routing, DNS and the essential diagnostic tools." date: "2026-08-06T09:00:00+02:00" updated: "2026-08-06T09:00:00+02:00" author: name: "Sebastian Palencsár" handle: "spalencsar" category: ["network"] tags: ["netzwerk", "linux-netzwerk", "troubleshooting", "linuxadmin", "systemanalyse", "iproute2", "tcpdump", "firewall"] reading_time: 45 toc: true --- Systematic analysis of network problems on Linux is not an optional extra skill. It is a core discipline of daily system administration. Faults rarely appear in isolation. They show up as unclear symptoms: a service does not answer, a host is unreachable at times, name resolution fails or latency rises without an obvious reason. Without a structured approach you quickly lose time in random checks and wrong assumptions. What follows is a clear, repeatable grid with which you narrow problems methodically and make statements that hold up. The goal is a practice-oriented reference. You do not get a theoretical textbook and no collection of isolated commands. You learn how you use the onboard tools of modern Linux systems in a targeted order. The focus is a bottom-up approach that runs from local host configuration through connectivity and path checks to packet and log analysis. Each step builds on the previous one and shrinks the search space. That way you avoid typical detours and reach a solid diagnosis faster. The path runs step by step from the basics to deep analysis. You work systematically from local host configuration through path and reachability tests to packet captures and kernel logs. Every section is built so that on an acute problem you can also jump straight into the matching chapter. Prerequisites for successful work with this material are solid Linux system knowledge, confident use of the shell and first practical experience in system administration. You should already understand the basic network concepts — interfaces, addressing, routing, DNS and firewall — and be able to move in the terminal without constant lookup. Shell-scripting knowledge helps automate recurring checks, but it is not mandatory. The text assumes that you can interpret the output of the tools used and place the results in an overall context.
⚠️ **Important:** This is aimed at Linux administrators and IT professionals who already have basic experience with network administration on Linux. You should be confident in the terminal and understand the most important network concepts. Anyone who brings those basics finds a tool that works in production. The text stays close to real admin conditions: limited time, unclear symptoms and the need to decide quickly whether the problem is local, on the path or on the far side. That is exactly what the systematics here is for.
The following method is the foundation for all further steps. ## Systematic method of network diagnosis ### Bottom-up approach along the network layers Network problems on Linux rarely arise on a single layer in isolation. Every higher layer assumes the ones below it work. That is why you always start diagnosis from the bottom and work up. This bottom-up approach stops you wasting time on complex tools while the real cause sits much closer to the physical or link layer. The relevant layers follow the classic model, adapted to Linux reality. At the bottom you check the physical link and driver status. Next is the data-link layer with MAC addresses, ARP and VLAN assignment. The network layer covers IP addressing and routing. Above that sit transport protocols and sockets, finally the application itself. ```markdown ┌─────────────────────────────────────┐ │ Application / service (logs, config)│ ├─────────────────────────────────────┤ │ Transport (TCP/UDP, ports, ss) │ ├─────────────────────────────────────┤ │ Network (IP, routing, ICMP) │ ├─────────────────────────────────────┤ │ Data-link (ARP, MAC, bridges) │ ├─────────────────────────────────────┤ │ Physical / link (cable, ethtool) │ └─────────────────────────────────────┘ ↑ Diagnosis starts here ``` Always start with link status. An interface that is DOWN or reports no carrier makes every further check pointless. Only when the link is up and a valid address is present do you check gateway reachability. After that you test forwarding and only then DNS, ports and application answers.
⚠️ **Note:** If you skip the lower layers, you risk misdiagnosis: you hunt firewall rules or service configs while the fault already sits in the driver or the cabling.
The approach forces discipline. You document every result before you go to the next layer. Only then do you see dependencies and avoid parallel, unstructured checks. In complex environments with bridges, bonding or containers this disciplined path is especially valuable, because several virtual layers sit on top of each other.
💡 **Tip:** The bottom-up path also fits when symptoms appear at application level. A web server that accepts no connections can fail just as well on a missing route or a down interface as on a wrong listen address.
🔧 **Practical example:** Imagine a host does not answer SSH requests. Instead of immediately starting `ss` or `tcpdump`, you begin with the interface: ```bash ip link show eth0 ethtool eth0 | grep -E 'Link detected|Speed|Duplex' ip addr show eth0 ip route show default ``` **Only when Link detected:** `yes`, a `valid IP` and a `default route` are present do you go on to `ping` and socket status. This sequence often saves half the diagnosis time in practice. ### Information gathering and reproducibility A solid diagnosis stands or falls with the quality of the information you collect. Before you go deep into individual layers, you save the current state of the system. Only then can you later reconstruct what changed and trigger the problem again under comparable conditions. Without that base many observations stay anecdotal and hard to verify. Start with a fixed set of baseline data. Note time, affected interfaces, current IP addresses, routing table and the status of the relevant services. Also capture the latest kernel messages and the output of the most important network tools. This snapshot is the reference point. Later you can run the same command set again and see the differences. Reproducibility means you can recreate the symptom under controlled conditions. That is decisive for intermittent faults. **Document exactly under which circumstances the problem appears:** after a reboot, only at certain times of day, only under load or only from certain source addresses. Record which steps you have already taken and which results they produced.
⚠️ **Note:** Without this discipline you quickly lose the overview and repeat the same checks several times.
A practical way is to redirect outputs into files and stamp them with a timestamp. That creates a traceable history. Use simple shell constructs and store the results in a dedicated directory. Make sure the files contain the relevant context — hostname, date and the exact command.
💡 **Tip:** Reproducibility gains extra value when several people work on the incident. A clearly documented baseline stops colleagues repeating the same steps or starting from stale assumptions.
:::legend **Diagnosis snapshot** 1. Time + host 2. Link and address status 3. Routing and neighbours 4. Socket and service status 5. Latest kernel/journal messages 6. Observed symptom + trigger ::: 🔧 **Practical example:** Create a directory and save the initial state: ```bash mkdir -p /tmp/netdiag/$(date +%Y%m%d-%H%M) cd /tmp/netdiag/$(date +%Y%m%d-%H%M) { echo "=== $(hostname) $(date -Iseconds) ===" ip -br link ip -br addr ip route ip neigh ss -tulpn } > 01-basis.txt journalctl -k -b --no-pager | tail -n 100 > 02-kernel.txt dmesg -T | tail -n 50 > 03-dmesg.txt ``` Run the same set later and compare the files. That way you see changes on interfaces, routes or socket states without relying on memory. Add targeted captures or extra tool output as soon as you have narrowed the affected layer. The collected data stay useful only if they are complete and consistent in time. Avoid running individual commands in isolation and without context. Always work with the full set and save the outputs before you change configurations or restart services.
❗ **Note:** A common error is overwriting older snapshots; number the files clearly and keep the original baseline.
### Typical entry mistakes and how to avoid them Even with clean information gathering and reproducible snapshots, diagnoses often go wrong when the first steps are chosen badly. The most common entry mistakes do not come from not knowing individual commands, but from the wrong order and from hasty assumptions. If you know these patterns, you save hours and avoid changing the symptom further with your own interventions. A classic error is to search at application level immediately. Many people start with port scans, firewall rules or service logs although the interface is still DOWN or has no carrier. That leads to long detours. Avoidance is simple: every diagnosis starts with link and address status. Only when that base is sound may the next layer be looked at. If you skip that, you later interpret symptoms that do not belong to the real cause. Name resolution and reachability are often mixed up too. A failed ping to a hostname is quickly treated as a network outage, although only the resolver is stuck. **The correct split is:** first ping the `IP address` directly, then the `hostname`. If only the name fails, the problem is `DNS` or `/etc/hosts`. If the IP already fails, the layer below is affected.
⚠️ **Note:** If you do not make that distinction, you waste time in the wrong place and suspect routing or firewall although the fault sits in resolution.
Blind trust in outdated tools is just as common. `ifconfig` and `netstat` give incomplete or misleading output on modern systems because they only show bridge, bonding and namespace information in a limited way. Avoidance is to use `ip` and `ss` consistently. These tools read directly from the kernel and show the actual state. Anyone who still uses the legacy commands often misses virtual interfaces or container networks. Equally problematic: changing the system before the baseline is saved. Restarting NetworkManager, flushing the ARP table or reloading firewall rules can make the symptom disappear temporarily and hide the cause. The discipline is: document first, then intervene. Every change must be traceable and ideally reversible. The viewpoint is often chosen wrongly too. Many people test only from the affected host and miss that the service listens locally but is not reachable from outside — or the reverse. Avoidance needs at least a second observation point: another host in the same segment or a targeted test from outside. Only then is it clear whether the problem is local, on the path or on the far side.
💡 **Tip:** A subtle error arises when temporary states are ignored. Stale ARP entries, expired DHCP leases or short-lived routes can produce symptoms that vanish on their own after a few minutes. Anyone who at that moment already goes deep into packet analysis looks for a cause that no longer exists. The countermeasure is to capture the time of the symptom exactly and repeat the same test soon after.
```markdown Wrong entry Correct entry ─────────── ───────────── Check service logs → Check link status Read firewall rules → Check address and route Start tcpdump → Split reachability Change configuration → Save a snapshot ``` 🔧 **Practical example:** Assume a host does not answer HTTP requests. The wrong entry often looks like this: ```bash ss -tulpn | grep 80 nft list ruleset tcpdump -i any port 80 systemctl restart nginx ``` **The correct entry starts lower and leaves the state unchanged:** ```bash ip -br link ip -br addr ip route get 8.8.8.8 ping -c 3 $(ip route | awk '/default/ {print $3}') ss -tlnp '( sport = :80 )' ``` Only when these outputs are consistent and unremarkable do you bring in firewall and packet capture. This order prevents a service restart from wiping valuable state messages. A last, often overlooked error is working with insufficient rights or in the wrong network namespace. Commands without enough privileges return incomplete socket lists or may not query certain interfaces. In container environments the visible network stack is not that of the host. Avoidance means checking the context explicitly: `ip netns list` and targeted execution inside the right namespace if one exists.
❗ **Note:** Without the correct namespace you may be checking an entirely different network setup than the affected one.
## Analysis of the local host configuration ### Interfaces, link status and addressing Checking interfaces is the real entry into local host analysis. Without a working link and without valid addressing, all further tests stay largely meaningless. Always capture the physical and logical state of the network interface first, before routing, DNS or even services. **The central command for the overview is `ip`.** With `-br` you get a compact view of all interfaces, their state and the assigned addresses. The output immediately shows whether an interface is administratively UP and whether the kernel sees a carrier. If the `LOWER_UP` flag is missing or `NO-CARRIER` appears, the problem sits below the IP layer. ```bash ip -br link ip -br addr ``` **For detailed inspection of individual interfaces use `ip link show`.** Here flags, MTU, queue discipline and the current state become visible. Especially relevant are the combinations `UP,LOWER_UP` (*link is up*). `UP,NO-CARRIER` (*interface enabled, but no signal*). MAC address and promiscuous mode can also give hints, for example in bridge or monitoring setups. **In addition `ethtool` provides information that `ip` does not cover.** That includes negotiated speed, duplex mode, link detection and error counters. A gigabit interface that only runs at 100 Mbit/s or in half-duplex almost always points to a cable, switch-port or autonegotiation problem. Rising error or drop counters point to physical faults or driver problems. ```bash ethtool eth0 ethtool -S eth0 | grep -E 'error|drop|crc|frame' ``` **Addressing is checked with `ip addr show`.** Besides the IP address itself, scope, valid lifetime and assignment mode matter. Dynamic addresses from `DHCP` always carry the hint `dynamic`; static ones do not. An address in `169.254.0.0/16` signals that no DHCP server was reached and the system fell back to APIPA. IPv6 addresses should be looked at too, because many modern environments run dual-stack.
💡 **Tip:** An interface can be administratively UP and still not allow useful communication if the address is missing, expired or assigned to the wrong scope. In those cases `ip` shows the link as functional, but the higher layer stays blocked.
**The interface names themselves deserve attention too.** Modern distributions use predictable names such as `ens3`, `enp0s3` or `enx` followed by the MAC. Older scripts that still expect `eth0` then fail silently. In virtualised or containerised environments virtual interfaces, VLANs and bonding devices are added. These must be included in the check explicitly, because they determine the actual exit path. ```markdown Link check in practice ────────────────────── 1. ip -br link → state of all interfaces 2. ip link show DEV → flags and details 3. ethtool DEV → speed, duplex, carrier 4. ethtool -S DEV → error counters 5. ip addr show DEV → addresses and scope ```
💡 **Tip:** The order is deliberate: first administrative and physical state, then addressing. That way a missing address is not mistaken for a routing or firewall problem.
🔧 **Practical example:** A host should reach external destinations but does not answer. The check starts with the full set: ```bash ip -br link ip -br addr ip link show eth0 ethtool eth0 | grep -E 'Speed|Duplex|Link detected' ethtool -S eth0 | grep -iE 'err|drop|crc' ip addr show eth0 ``` Expected normal values are `state UP`, `LOWER_UP`, `Link detected: yes`, a matching speed/duplex combination and at least one valid global address. If `NO-CARRIER` or `Link detected: no` appears, continue diagnosis at the physical or driver level. If the address is missing or only a link-local address is present, the problem is address assignment. **Watch temporary states** Interfaces can briefly show as `DOWN` after a driver reload or after a cable is unplugged and then go `UP` again without the address being renewed. In DHCP environments a lease renew often has to be forced then. Static configurations stay stable as long as the interface is not reconfigured explicitly.
❗ **Note:** A query of administrative status alone is not enough — without checking carrier and error counters you miss physical packet loss.
### Routing, neighbour tables and DNS resolvers After link and address status, forwarding and name resolution come into focus. This layer decides whether packets can leave the local host at all and whether destination addresses are resolved correctly. Errors here often produce symptoms that are wrongly blamed on the firewall or the remote service. The routing table is examined with `ip route` and targeted queries. First you check the default route. If it is missing or points at an unreachable gateway, external communication is impossible. Then you test the concrete path to a destination with `ip route get`. This command shows not only the route, but also the interface used and the source address. That is especially valuable on systems with several interfaces or policy routing. ```bash ip route ip route show default ip route get 8.8.8.8 ip route get 192.168.1.50 from 10.0.0.10 ``` Several routing tables or rules (`ip rule`) appear in more complex setups. For most diagnoses the main table is enough. What matters is that the route to the gateway goes via an interface that is actually UP and addressed. A route via a DOWN interface stays ineffective even if it sits in the table. The neighbour table (ARP for IPv4, NDP for IPv6) is viewed with `ip neigh`. It shows which layer-2 addresses the kernel has learned for known IPs. States such as `REACHABLE` and `STALE` are normal. `FAILED` or `INCOMPLETE` indicate that the gateway or neighbour does not answer. A permanently missing entry for the default gateway is a clear signal of a problem on the data-link layer or at the gateway itself. ```bash ip neigh ip neigh show dev eth0 ip neigh show to 192.168.1.1 ``` Stale entries can cause temporary faults. In those cases a targeted delete of the entry helps so the kernel resolves the address again. That should only happen after the baseline is documented.
⚠️ **Note:** A missing or wrong default route is often overlooked because `ping` to local addresses still works and only fails for external destinations. Checking with `ip route get` makes the actual exit path visible and prevents misdiagnosis.
The DNS resolver is the next checkpoint. Modern systems often use `systemd-resolved`. Status and configured servers are queried with `resolvectl`. `/etc/resolv.conf` should be checked as well, because older applications or containers still use it. A working resolver does not automatically mean resolution works for all applications — especially in container environments their own resolver configs can apply. ```bash resolvectl status resolvectl query example.com cat /etc/resolv.conf dig +short example.com dig @8.8.8.8 example.com ``` The split between reachability and name resolution remains decisive. A successful ping to the IP with a failed hostname isolates the problem clearly to DNS. Conversely a failed IP ping shows that the cause sits deeper and DNS is irrelevant for now.
💡 **Tip:** The combination of `ip route get` and a targeted `dig` against a known server tells you in a few seconds whether routing or resolution causes the fault. That saves the detour through unnecessary firewall or packet analysis.
🔧 **Practical example:** A host should reach an internal service under the name `app.internal`. The check runs in a clear order: ```bash # Routing and gateway ip route show default ip route get 10.20.30.40 ping -c 3 $(ip route | awk '/default/ {print $3}') # Neighbour table for the gateway GATEWAY=$(ip route | awk '/default/ {print $3}') ip neigh show to $GATEWAY # DNS split ping -c 2 10.20.30.40 ping -c 2 app.internal dig +short app.internal resolvectl query app.internal ``` If the ping to the IP fails while the route and neighbour entry look correct, the problem is beyond the local host. If only name resolution fails, you check the resolver and DNS-server reachability. An extra test against a public resolver (`dig @1.1.1.1`) shows whether the problem is local or at the internal DNS server. On hosts with several interfaces, policy routing also plays a role. On hosts with several addresses `ip route get` can pick a different source IP than expected. That causes replies to come back on the wrong path or to be dropped by firewalls. In those cases an explicit source address on the test helps. ```bash ip route get 8.8.8.8 from 192.168.10.50 ping -I 192.168.10.50 -c 3 8.8.8.8 ```
❗ **Note:** Anyone who skips neighbour table and gateway routes often suspects errors in the application although layer 2 or 3 is already blocked.
### Socket status and local services with `ss` Next you check whether local services listen on the expected sockets. This is where it is decided whether a service actually listens, which address it is bound to and whether connections can be built. `ss` is the tool of choice here, because it reads socket information directly from the kernel and works clearly faster and more completely than older `netstat`. **The first and most important check is listening sockets.** With `ss -tulpn` you see all TCP and UDP listeners, the numeric ports and the associated processes. The output immediately shows whether a service is bound to `0.0.0.0`, a specific IP or only `127.0.0.1`. A bind only to localhost is one of the most common reasons a service works locally but stays unreachable from outside. ```bash ss -tulpn ss -tlnp ss -ulnp ss -tlnp '( sport = :80 or sport = :443 )' ``` For more detailed filtering `ss` supports expression syntax. That lets you search by port, state or address without walking the whole output by hand. Restricting to a single port or to connections in a given state is especially useful. ```bash ss -tn state listening ss -tn state established ss -tn state syn-sent ss -tn '( dport = :443 or sport = :443 )' ss -tnp dst 10.20.30.40 ``` **The states themselves give important hints.** `LISTEN` confirms that the service has opened the port. `ESTABLISHED` shows active connections. `SYN-SENT` points to outbound connection attempts that are not answered. A high number of `TIME_WAIT` entries can be normal on heavily loaded servers, but becomes a problem when local ports are exhausted.
⚠️ **Note:** A service that runs in the process list but does not appear as a listener in `ss` is either bound to the wrong address, started in the wrong network namespace or already crashed without the process manager noticing.
Besides the pure socket list, `ss -s` gives a summary of socket statistics. Here you see the number of connections per state at a glance. That helps spot anomalous piles quickly, for example masses of `CLOSE-WAIT` or `SYN-RECV` entries that can point to resource problems or attacks. ```bash ss -s ss -tn state time-wait | wc -l ss -tnp | grep -E 'CLOSE-WAIT|FIN-WAIT' ``` In environments with containers or network namespaces the context must be watched. An `ss` on the host does not show sockets inside a container. Either `nsenter` or running `ss` directly in the container helps. Otherwise you diagnose the wrong network stack. ```bash # Example for namespace switch ip netns list ip netns exec myns ss -tulpn ```
💡 **Tip:** Combining a port filter with process display (`-p`) saves time, because you immediately see which process holds the port and whether it runs under the expected user. That prevents mix-ups when several instances of the same service exist.
🔧 **Practical example:** A web service should be reachable on port 8080 but does not answer. The systematic check with `ss` looks like this: ```bash # Show all listeners and filter specifically ss -tlnp | grep 8080 ss -tlnp '( sport = :8080 )' # Check which address it is bound to ss -tlnp '( sport = :8080 )' | awk '{print $4}' # Active connections to the port ss -tnp '( dport = :8080 or sport = :8080 )' # Summary of states ss -s ``` The expected result on a correct configuration is a `LISTEN` entry on `0.0.0.0:8080` or the desired interface IP plus the matching process name. If only `127.0.0.1:8080` appears, the bind is wrong. If the entry is missing entirely, the service is not running or listens on another port. In that case you check the process list and the service config before further measures. **Special points for UDP sockets** Unlike TCP, UDP has no connection state in the classic sense. `ss -ulnp` still shows the bound ports and processes. That is decisive for services such as DNS, NTP or your own UDP-based applications. If the entry is missing here while the process runs, there is almost always a wrong bind or a namespace problem. ```bash ss -ulnp ss -unp '( sport = :53 )' ```
❗ **Note:** Outdated tools such as `netstat` or missing `-p` flags often hide which process holds a port or in which namespace it runs.
## Connectivity and path analysis ### Reachability tests Once sockets and local services are established, diagnosis moves to the external network path. The question now is whether packets leave the host, reach the destination and come back. For that `ping`, `mtr` and `tracepath` give different views of the same path that complement each other. **`ping` remains the fastest reachability test.** It checks whether ICMP echo requests are answered and returns latency and packet loss. What matters is the consistent split of IP and name resolution. First the pure IP is tested, then the hostname. Only that isolates DNS problems from real connection faults. `-c` limits the number of packets, `-W` sets the timeout per packet and `-i` controls the interval. ```bash ping -c 5 8.8.8.8 ping -c 5 -W 2 192.168.1.1 ping -c 10 -i 0.2 10.20.30.40 ping -c 5 example.com ``` A successful ping to the IP with a failed hostname points clearly at the resolver. Conversely a failed IP ping shows that the problem is on the path or at the destination. High latency swings or rising loss point to overload or unstable links. `mtr` combines `ping` and `traceroute` and gives a running statistic for every hop. In report mode (`-r` or `-rw`) it produces a one-shot summary that documents well. `-c` sets the number of cycles. With `--tcp` or `--udp` you can test other protocols instead of ICMP, which is useful when ICMP is filtered. ```bash mtr -rwc 50 8.8.8.8 mtr -rwc 30 --tcp --port 443 example.com mtr -n -rwc 20 10.20.30.40 ``` The output shows loss and latency for every hop. Loss that jumps up from a certain hop and stays on the following hops marks the problematic point. Loss that appears only on a single hop and then vanishes is often ICMP rate limiting and not necessarily a real problem.
⚠️ **Note:** ICMP is filtered or limited on many paths. A failed ping or high loss in `mtr` therefore does not automatically mean that TCP or UDP traffic is blocked too. In those cases extra tests with TCP-based `mtr` or targeted port checks are required.
`tracepath` adds path-MTU discovery. It shows not only the hops, but also the maximum packet size that can pass without fragmentation. That is decisive for problems with large packets, VPN tunnels or certain cloud connections. Unlike classic `traceroute`, `tracepath` needs no root rights. ```bash tracepath 8.8.8.8 tracepath -n 10.20.30.40 tracepath -b example.com ``` The combination of the three tools gives a solid picture. `ping` gives the fast yes/no and the base latency. `mtr` shows where on the path loss or delay arises. `tracepath` clarifies whether the MTU restricts traffic. Together they stop a single symptom being over-interpreted.
💡 **Tip:** Choose the order deliberately: first a short `ping` to the IP, then `mtr` for hop statistics and finally `tracepath` if you suspect MTU problems. That keeps diagnosis effort low and the signal high.
```markdown Reachability test — useful order ──────────────────────────────── 1. ping -c 5 → basic decision 2. ping -c 5 → DNS split 3. mtr -rwc 30 → hop statistics 4. tracepath → MTU path ``` 🔧 **Practical example:** An internal server at `10.50.1.20` should be reached, the application reports timeouts. **The check runs as follows:** ```bash # Basic decision and DNS split ping -c 5 10.50.1.20 ping -c 5 app.internal.example # Hop analysis with a large enough sample mtr -rwc 50 -n 10.50.1.20 mtr -rwc 30 --tcp --port 443 10.50.1.20 # Check MTU path tracepath -n 10.50.1.20 # Extra test with smaller interval if you suspect intermittent loss ping -c 20 -i 0.2 10.50.1.20 ``` If loss in `mtr` rises permanently from hop 3, the fault sits there. If loss stays high on `ICMP` while the TCP test with `--tcp` runs clean, `ICMP` is filtered. If `tracepath` shows an unexpectedly low `pMTU`, large packets or tunnel configs must be investigated. On path analysis also watch asymmetric routes. `ping` and `mtr` measure the outbound and return path. If the return path differs and packets are dropped there, the destination appears unreachable although the outbound path is intact. In those cases comparing tests from both sides or looking at routing tables on the hosts involved helps.
❗ **Note:** A pure ICMP check easily misleads, because many firewalls or routers throttle or filter ICMP packets on purpose.
### Firewall and filter rules nftables, firewalld, ufw Local packet filters often decide whether an otherwise intact path ends at the destination host. The three common management paths — `nftables`, `firewalld` and `ufw` — must therefore be read systematically as soon as reachability tests suggest a break. **`nftables` is the lowest visible layer.** `nft list ruleset` dumps the entire active configuration. What matters most are the counters on the individual rules. If the counter of a drop or reject rule rises while accept rules for the same port stay at zero, the cause is found locally. Handle numbers (`-a`) later allow precise removal of single entries without reloading the whole base. ```bash nft list ruleset nft list ruleset -a nft list table inet filter nft list chain inet filter input nft list chain inet filter forward ``` On dual-stack systems IPv4 and IPv6 tables must be looked at separately. A rule that exists only in one family explains asymmetric symptoms. **`firewalld` organises access via zones.** Each interface sits in exactly one zone, and each zone brings its own default policies plus allowed ports and services. If the zone of the affected interface differs from the expected one, suddenly stricter rules apply. Runtime and permanent configuration can also diverge; only the permanent settings survive a service restart. ```bash firewall-cmd --state firewall-cmd --get-active-zones firewall-cmd --list-all firewall-cmd --list-ports firewall-cmd --query-port=8443/tcp firewall-cmd --permanent --list-all ``` **`ufw` gives a numbered, easy-to-read overview.** The default policies for incoming and outgoing sit at the top. Rule numbers allow targeted deletion without touching the rest of the configuration. ```bash ufw status verbose ufw status numbered ufw app list ``` Regardless of the frontend, the same check order applies: default policy of the relevant chain, explicit allow rules for port and protocol, assignment of the interface to a zone (with `firewalld`), finally the counters or log entries. Only then is it visible whether a packet even reached the rule base or was already dropped earlier.
⚠️ **Note:** The position of a rule inside the chain is decisive. A late accept rule stays ineffective if an earlier drop rule already matches the same traffic.
The difference between `drop` and `reject` strongly affects the symptoms. `drop` produces timeouts, `reject` gives the sender a clear reply. Many diagnostic tools treat a missing reject as a generic connection break and hide the actual filter effect. ```markdown Filter diagnosis — critical points ────────────────────────────────── * Policy of the input and forward chain * Explicit allow rule for port/protocol * Interface-to-zone assignment (firewalld) * Counter values on drop rules * Runtime vs permanent match ```
💡 **Tip:** Counters and log messages are the only objective proof that a packet was actually hit by a given rule. Without that information every guess about blocked traffic stays unconfirmed.
🔧 **Practical example:** Port `8443` is locally in listen state, but external connections fail. The filter check runs like this: ```bash # nftables — rules and counters nft list ruleset -a nft list chain inet filter input | grep -E '8443|drop|reject|accept' # firewalld — zone and port firewall-cmd --get-active-zones firewall-cmd --list-all firewall-cmd --query-port=8443/tcp firewall-cmd --permanent --query-port=8443/tcp # ufw — if active ufw status numbered | grep 8443 ``` An accept rule with a rising counter and a matching zone assignment speak for an intact filter configuration. If the rule is missing or the counter of a drop rule rises, the cause is in the local policy. With `firewalld` you must also check whether the rule is set both runtime and permanent. On systems that also route or bridge, the forward chain applies. An allow rule in the input chain is then not enough. Traffic that only traverses the host is filtered separately and must be allowed on its own. ```bash nft list chain inet filter forward firewall-cmd --list-all --zone=internal ```
❗ **Note:** Forward rules are easily overlooked because most diagnosis scripts only read the input chain. On router, bridge or container hosts that leads to false all-clears.
### Service and port reachability from outside Local confirmation of an open port and allowed filter rules is not enough. Only a test from a remote system shows whether the service is actually reachable from outside. Differences between local listen state and external reachability come from filters in front, asymmetric routes or binds that only allow local traffic. `nc` is a fast connection checker. In zero mode the tool only builds the TCP handshake and then closes the connection immediately. The reply is unambiguous: success or timeout / connection refused. For UDP services `-u` is set. Timeout can be limited with `-w` so hanging tests do not block for too long. ```bash nc -zv 10.50.1.20 8443 nc -zv -w 4 10.50.1.20 443 nc -zvu -w 3 10.50.1.20 53 nc -zv app.internal.example 8080 ``` If the attempt from a remote host fails while the same command succeeds on the target itself, the fault sits on the path in between. `nmap` adds state information besides the pure connection status. The connect scan (`-sT`) works without root and reports `open`, `closed` or `filtered`. The SYN scan (`-sS`) is more precise but needs elevated privileges. `-Pn` skips host discovery if ICMP is dropped on the way. Service detection (`-sV`) also tries to identify the protocol on the port. ```bash nmap -p 8443 10.50.1.20 nmap -sT -Pn -p 80,443,8443 10.50.1.20 nmap -sS -p 8443 10.50.1.20 nmap -sV -p 8443 10.50.1.20 ``` `closed` means a reset came back — the host answers, but the port is not occupied. `filtered` signals missing replies and thus a filter on the path or at the destination. For HTTP and HTTPS endpoints `curl` adds the application layer on top of transport. Connection errors, TLS problems and HTTP status codes become visible separately. `--connect-timeout` prevents long waits, `--resolve` forces a given IP and bypasses DNS influence on the test. ```bash curl -v --connect-timeout 5 https://10.50.1.20:8443/ curl -v -o /dev/null -w "%{http_code}\n" http://10.50.1.20:8080/health curl -v --resolve api.example.com:8443:10.50.1.20 https://api.example.com:8443/ ```
⚠️ **Note:** Local success with `ss` and simultaneous failure of `nc` or `nmap` from outside point to a fault that sits outside the target host. The further search must then continue on the path or on systems in between.
The choice of test system affects the result noticeably. A host in the same layer-2 segment sees different filters and routes than a system in a remote net or behind a NAT gateway. Tests should therefore be run from at least two different locations if the topology allows. The source address used can also be relevant when access-list-based rules apply on the target or in between. ```markdown External check — useful combination ────────────────────────────────── 1. nc -zv → fast connect test 2. nmap -sT/-sS → state information 3. curl -v → application layer (HTTP/S) 4. Comparison → same vs foreign subnet ```
💡 **Tip:** A port can appear `open` in `nmap` and still react wrongly at application level. Only the protocol-specific test with `curl` or a suitable client gives the final statement about service availability.
🔧 **Practical example:** A service on port 8443 is bound locally and allowed in the filter rules. Clients in the company net report connection errors. The external check is run from two different hosts: ```bash # Test from host A (same subnet) nc -zv -w 3 10.50.1.20 8443 nmap -sT -Pn -p 8443 10.50.1.20 # Test from host B (other net) nc -zv -w 3 10.50.1.20 8443 nmap -sT -Pn -p 8443 10.50.1.20 nmap -sV -p 8443 10.50.1.20 # Application test curl -v --connect-timeout 5 https://10.50.1.20:8443/health curl -v --resolve service.example.com:8443:10.50.1.20 https://service.example.com:8443/health ``` Successful connect tests from both sides confirm the transport layer. If the `curl` call then fails, the cause is TLS, certificate or application logic. If `nc` already stays without a connection, the path between the hosts involved must be examined again. On systems with several addresses or interfaces the reply can return on a different path than the outbound one. In those cases SYN packets arrive, but SYN-ACK packets leave the host via an interface the client does not expect. Comparing packet flows on the target host with `tcpdump` makes that asymmetry visible.
❗ **Note:** Tests that are started only from the target host itself systematically hide every problem that appears only on the path between client and server.
## Packet analysis, logs and extended diagnosis ### Targeted captures When external reachability tests and local checks do not give a clear cause, what remains is looking at the actual packets. `tcpdump` captures traffic on the chosen interface and shows whether SYN packets arrive, whether replies leave the host and which flags or options are set. The skill is to set capture filters so tightly that only the relevant traffic appears and the output stays usable. The basic call specifies the interface and a filter expression. Without a filter too much is captured quickly. With `-n` addresses and ports are shown numerically, which speeds output and prevents DNS lookups during the capture. `-nn` additionally suppresses resolution of port names. For later analysis in Wireshark or tshark the output is written to a file (`-w`). ```bash tcpdump -i eth0 -n port 8443 tcpdump -i any -nn host 10.50.1.20 and port 8443 tcpdump -i eth0 -nn -w /tmp/capture.pcap host 10.40.1.15 and port 443 ``` Filter expressions follow BPF syntax. They can be combined by host, net, port, protocol and direction. The operators `and`, `or` and `not` allow precise limits. For TCP handshakes flags are especially useful: `tcp[tcpflags] & tcp-syn != 0` captures SYN packets; further flags can be addressed analogously. ```bash tcpdump -i eth0 -nn 'tcp port 8443 and (tcp[tcpflags] & tcp-syn != 0)' tcpdump -i eth0 -nn 'host 10.50.1.20 and (port 80 or port 443)' tcpdump -i eth0 -nn 'src net 10.40.0.0/16 and dst port 8443' tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0' ``` `-c` limits the number of captured packets and prevents files that grow without control. `-s 0` or `-s 65535` ensures full packets are stored, not only headers. On suspicion of fragmentation or unusual options that is essential. For longer captures a ring buffer via `-C` and `-W` is recommended so older files are overwritten automatically. ```bash tcpdump -i eth0 -nn -c 100 -w /tmp/short.pcap port 8443 tcpdump -i eth0 -nn -s 0 -w /tmp/full.pcap host 10.50.1.20 tcpdump -i eth0 -nn -C 50 -W 5 -w /tmp/ring.pcap port 443 ```
⚠️ **Note:** Filters that are too broad quickly produce confusing amounts of packets and make analysis unnecessarily hard. Every capture should be limited to the concrete suspicion and the addresses and ports involved.
Reading a stored capture file is also done with `tcpdump`. The same filter syntax can be applied afterwards to narrow the stored data further. `-tttt` produces readable timestamps, `-X` or `-A` shows the payload in hex or ASCII. ```bash tcpdump -nn -r /tmp/capture.pcap tcpdump -nn -tttt -r /tmp/capture.pcap 'tcp[tcpflags] & tcp-syn != 0' tcpdump -nn -A -r /tmp/capture.pcap port 80 tcpdump -nn -X -c 20 -r /tmp/capture.pcap ``` On interpretation the order of packets counts. An incoming SYN without a following SYN-ACK points to filtering or a service that does not answer. A SYN-ACK that leaves the host but never arrives at the client points to problems on the return path. Duplicate SYN packets or unexpected RST flags give further hints of timeout behaviour or active rejection.
💡 **Tip:** Combining a tight filter, a limited packet count and then targeted evaluation of the pcap file prevents relevant information drowning in a flood of irrelevant packets.
🔧 **Practical example:** External clients cannot reach port 8443 although the service listens locally and the firewall allows the port. The capture is started on the target host while a client tries the connection: ```bash # Start capture (in a separate terminal) tcpdump -i eth0 -nn -s 0 -w /tmp/8443.pcap host 10.40.1.15 and port 8443 # In parallel from the client: nc -zv 10.50.1.20 8443 # Stop capture (Ctrl+C) and evaluate tcpdump -nn -tttt -r /tmp/8443.pcap tcpdump -nn -r /tmp/8443.pcap 'tcp[tcpflags] & tcp-syn != 0' tcpdump -nn -r /tmp/8443.pcap 'tcp[tcpflags] & tcp-ack != 0' ``` If SYN packets from the client appear but no SYN-ACK packets from the server, the cause sits on the target host itself (service, local filters or bind). If SYN-ACK packets leave the host but do not reach the client, the return path must be examined. If even the SYN packets are missing, a filter in front of the target host is in play. For UDP services the view changes because there is no handshake. Request and possible reply packets are compared directly. Packet size and possible ICMP errors (fragmentation needed, port unreachable) also give hints. ```bash tcpdump -i eth0 -nn -s 0 -w /tmp/dns.pcap port 53 tcpdump -nn -r /tmp/dns.pcap ```
❗ **Note:** Captures without prior restriction to host and port produce files that are hardly useful to evaluate in practice and can also contain sensitive data.
### Log and kernel analysis Packet captures show traffic on the wire. System and kernel logs explain why that traffic happens or breaks. Both sources belong together: a missing SYN-ACK in the capture file only becomes understandable when the kernel at the same time reports a driver error, a netfilter drop or a link loss. `journalctl` is the central entry on systems with systemd. The unit `NetworkManager`, `systemd-networkd` or the service in question can be queried specifically. Time filters limit the output to the relevant window. `-k` restricts the display to kernel messages, `-b` shows only the current boot. ```bash journalctl -u NetworkManager --since "10 min ago" journalctl -u systemd-networkd -b journalctl -k --since "2024-01-01 14:30:00" --until "2024-01-01 14:45:00" journalctl -k -b | grep -iE 'eth0|link|carrier|netfilter|nf_' ``` For network-specific events it is worth searching for terms such as `link`, `carrier`, `timeout`, `refused` or the interface name. Many drivers write the reasons for a link-down or rejected packets here. `dmesg` delivers the same kernel ring buffer, often a bit more directly and with readable timestamps via `-T`. The output can be filtered with the same pattern. On suspicion of driver problems or firmware messages `dmesg` is often the faster path. ```bash dmesg -T | tail -n 100 dmesg -T | grep -iE 'error|fail|timeout|eth0|enp' dmesg -T --level=err,warn ``` Netfilter itself writes its own entries when logging is enabled. Rules with the target `LOG` or `NFLOG` produce messages that become visible in the kernel logs or via `journalctl`. The priority and prefix of the log rule determine how easy the entries are to find later. ```bash journalctl -k | grep -i 'NETFILTER\|NFLOG\|DROP' dmesg -T | grep -i 'DROP\|REJECT' ```
⚠️ **Note:** Kernel messages about link losses or driver resets often appear only briefly and are pushed out of the ring buffer quickly when log volume is high. Capture must therefore happen close to the observed symptom.
Correlation with the packet capture is done via timestamps. A SYN packet that arrives in the capture file and a simultaneous kernel message about a drop in the input chain explain the missing SYN-ACK. Driver messages about CRC errors or overruns can also explain loss of packets that `tcpdump` already no longer saw. ```markdown Log sources and their focus ─────────────────────────── journalctl -u → service and NetworkManager events journalctl -k / dmesg → driver, link, netfilter, hardware Service logs → application errors, bind problems ```
💡 **Tip:** Combining a tight time filter and a targeted term search reduces the log volume to the entries that belong to the concrete incident. Without that limit, relevant messages drown in general system activity.
🔧 **Practical example:** A service on port 8443 intermittently does not answer. The packet capture shows incoming SYN packets without SYN-ACK. In parallel the logs of the relevant time window are checked: ```bash # Kernel and netfilter messages in the window journalctl -k --since "14:32:00" --until "14:35:00" | grep -iE 'eth0|drop|reject|link|error' # NetworkManager or networkd journalctl -u systemd-networkd --since "14:32:00" --until "14:35:00" journalctl -u NetworkManager --since "14:32:00" --until "14:35:00" # dmesg with readable timestamps dmesg -T | grep -iE 'eth0|enp|error|fail|timeout' | tail -n 50 # Service log in parallel journalctl -u myapi.service --since "14:32:00" --until "14:35:00" ``` If messages about carrier loss or netfilter drops appear in the same window, the cause is found. If such entries are missing while the service itself reports errors accepting connections, the problem sits in the application or in resource shortages. Driver and firmware messages deserve special attention. Many NICs write recurring entries on overload, faulty autonegotiation or damaged cables. These messages often correlate with rising error counters on the interface and explain intermittent losses that only show sporadically in pure connectivity tests. ```bash dmesg -T | grep -iE 'firmware|reset|watchdog|tx_|rx_|overrun' ethtool -S eth0 | grep -iE 'error|fail|drop|crc|over' ```
❗ **Note:** Logs that are read without reference to the concrete time and without a parallel packet capture easily lead to wrong cause assignment, because they contain events from completely different contexts.
### Error counters, performance and MTU problems Quantitative counters at interface and protocol level make visible what individual packet captures and log excerpts only hint at. They answer how often and to what extent errors, losses or limits occur. Hardware-near counters come from `ethtool -S`. Here CRC errors, alignment problems, overruns, missed packets and collision counters of the NIC appear. These values come directly from the controller registers and stay independent of kernel statistics. In parallel `ip -s link` holds the software-side RX and TX counters, including drop and error values that the kernel itself records. ```bash ethtool -S eth0 ethtool -S eth0 | grep -E 'crc|align|over|miss|error|drop|coll|fail' ip -s link show eth0 ip -s -s link show eth0 ``` The view only becomes meaningful through repeated queries. Two measurements one to five minutes apart and the difference of the relevant fields show whether the counters currently keep running. A high absolute value from the past is meaningless if it does not change in the observation window. At TCP level `ss` delivers retransmit and congestion information per connection. `-ti` or `-tai` show RTO, retransmits, congestion window and further variables. If retransmits pile up while interface counters stay stable, the loss is highly likely on the path and not on the local host. ```bash ss -ti ss -tai ss -ti state established ss -tai dst 10.50.1.20 | grep -E 'retrans|rto|cwnd|rtt' ``` MTU limits produce a class of symptoms of their own. Packets that are too large are dropped when intermediate systems do not allow fragmentation and ICMP messages are suppressed. `tracepath` finds the path MTU, `ping` with Don’t-Fragment set and a defined size checks concrete thresholds. ```bash tracepath -n 10.50.1.20 ping -c 4 -M do -s 1472 10.50.1.20 ping -c 4 -M do -s 1400 10.50.1.20 ping -c 4 -M do -s 1200 10.50.1.20 ip link show eth0 ``` The value 1472 tests the classic Ethernet MTU of 1500 bytes minus headers. If this test fails and smaller sizes pass, an MTU restriction exists on the path.
⚠️ **Note:** CRC and alignment errors usually rise only on physical problems: damaged cable, faulty connector, bad switch port or defective NIC. Pure software changes leave these counters untouched.
Further performance indicators sit in the softnet statistics and the queue disciplines. `/proc/net/softnet_stat` shows dropped packets in the receive queues of the CPUs. `tc qdisc show` lists the active queueing strategy and possible drop behaviour. `nstat` aggregates protocol counters such as TcpRetransSegs or IpInDiscards. ```bash cat /proc/net/softnet_stat tc qdisc show dev eth0 nstat -az | grep -E 'TcpRetrans|TcpLoss|IpInDiscards|Ip6InDiscards' ```
💡 **Tip:** The rate of increase of the counters inside a defined time window is the real indicator. Absolute values without reference to the observation period regularly lead to misinterpretation.
🔧 **Practical example:** Sporadic drops occur, captures show occasional retransmits. The counters are read before and during reproduction: ```bash # Measurement 1 ethtool -S eth0 | grep -E 'crc|error|drop|over|miss' > /tmp/e1.txt ip -s link show eth0 > /tmp/i1.txt ss -ti > /tmp/s1.txt nstat -az > /tmp/n1.txt # Reproduce the fault, then measurement 2 ethtool -S eth0 | grep -E 'crc|error|drop|over|miss' > /tmp/e2.txt ip -s link show eth0 > /tmp/i2.txt ss -ti > /tmp/s2.txt nstat -az > /tmp/n2.txt # Compare diff /tmp/e1.txt /tmp/e2.txt diff /tmp/i1.txt /tmp/i2.txt diff /tmp/n1.txt /tmp/n2.txt | grep -E 'TcpRetrans|Discards' # MTU check ping -M do -s 1472 -c 5 10.50.1.20 tracepath -n 10.50.1.20 ``` If CRC or drop counters rise during the fault, the cause is at the interface or the medium. If they stay constant and only TCP retransmits rise, the loss is further away. If the DF ping with 1472 bytes fails, the MTU on the path or at tunnel endpoints must be adjusted. On bonding or VLAN configurations the counters must be read on every involved layer. Errors that appear only on the logical bond device point to asymmetries between the slave interfaces or to problems in the bonding logic. ```bash ethtool -S bond0 ethtool -S eth0 ethtool -S eth1 ip -s link show bond0 ip -s link show eth0 ```
❗ **Note:** Counter values without a before and after measurement in the concrete time window of the problem do not allow a reliable statement about current error activity.
## Important resources and checklists ### Emergency checklist: bottom-up in 6 steps When a network problem is reported in production, walk the diagnosis without haste in this order: | Step | Diagnosis layer | Recommended commands | Check / focus | | :--- | :--- | :--- | :--- | | **Link and hardware** | Physical and driver | `ip -br link` / `ethtool eth0` | Carrier `LOWER_UP`? Duplex and speed correct? | | **IP and gateway** | Network layer and routing | `ip -br addr` / `ip route get ` | IP present? Valid default route set? | | **Local listener** | Transport and sockets | `ss -tulpn` / `ss -tlnp '( sport = : )'` | Does the service listen on `0.0.0.0` (not only `127.0.0.1`)? | | **Path and transport** | Reachability and port | `ping -c 3 ` / `nc -zv ` / `mtr -rwc 30 ` | Reachability split from DNS? Port open? | | **Firewall and rules** | Packet filter and policies | `nft list ruleset` / `firewall-cmd --list-all` | Do counters rise on drop/reject rules? | | **Captures and logs** | Packet flow and kernel | `tcpdump -i any -nn host ` / `journalctl -k -b` | Do SYN packets arrive? Driver / drop logs? | ### Reference tools: modern onboard tools vs legacy On modern Linux distributions you should avoid outdated commands, because they often do not show virtual network stacks, namespaces or bonding interfaces correctly: | Diagnosis area | Modern tool (recommended) | Legacy (avoid) | Important man page | | :--- | :--- | :--- | :--- | | **Interfaces and IP** | `ip link`, `ip addr` | `ifconfig` | `man ip-link`, `man ip-address` | | **Routing and tables** | `ip route`, `ip neigh` | `route`, `arp` | `man ip-route`, `man ip-neighbour` | | **Sockets** | `ss -tulpn` | `netstat` | `man ss` | | **Hardware and speed** | `ethtool eth0` | `mii-tool` | `man ethtool` | | **Hop and path analysis** | `mtr`, `tracepath` | `traceroute` | `man mtr`, `man tracepath` | | **Packet captures** | `tcpdump`, `tshark` | `ethereal` | `man tcpdump` | | **Filter and firewall** | `nft`, `firewall-cmd` | `iptables` | `man nft` | ### Recommended man pages and documentation For deeper configuration details and kernel parameters these references pay off: * **Man pages in the terminal:** * `man 8 ip` — the documentation of the `iproute2` suite. * `man 8 ss` — detailed description of all filter expressions for sockets. * `man 8 tcpdump` — BPF filter syntax and display options. * `man 8 nft` — syntax and table structure of modern `nftables` setups. **Kernel documentation:** * [Linux kernel networking documentation](https://www.kernel.org/doc/html/latest/networking/index.html){.badge-link-text} ### Practice checklists for daily use On acute incidents these short playbooks help you pin the fault on the right layer without searching: **Scenario 1: service runs but is not reachable from outside** * `ss -tlnp '( sport = : )'` — check whether the bind is on `0.0.0.0` (not only `127.0.0.1`). * `nft list ruleset` / `firewall-cmd --list-all` — check whether the port is allowed in the `input` chain / zone. * `nc -zv ` (from an external host) — check whether the handshake comes through. * `tcpdump -i any -nn host and port ` — check whether `SYN` arrives at the server. **Scenario 2: name resolution fails or is slow** * `ping -c 2 ` vs `ping -c 2 ` — split reachability from name resolution. * `cat /etc/resolv.conf` / `resolvectl status` — check configured nameservers and status. * `dig +short ` vs `dig @1.1.1.1 ` — compare local vs public DNS server. * `time dig ` — check latency and timeouts on the primary DNS. **Scenario 3: sporadic drops or packet loss** * `ethtool eth0` and `ethtool -S eth0 | grep -iE 'err|drop|crc'` — check physical link and error counters. * `mtr -rwc 50 -n ` — isolate lasting loss per hop (ICMP vs TCP `--tcp`). * `ping -M do -s 1472 ` — test path MTU for fragmentation problems. * `ss -ti` — watch TCP retransmits and RTO of the active connection. ### Further sources and community resources **Documentation and specifications:** [Linux kernel networking documentation](https://www.kernel.org/doc/html/latest/networking/index.html){.badge-link-text} [RFC 793 (TCP specification)](https://datatracker.ietf.org/doc/html/rfc793){.badge-link-text} [Red Hat Enterprise Linux network guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/index){.badge-link-text} **Blogs:** [Brendan Gregg’s blog](https://www.brendangregg.com/blog/){.badge-link-text} [Cloudflare networking blog](https://blog.cloudflare.com/tag/networking/){.badge-link-text} **Communities and mailing lists:** [Netdev mailing list](https://vger.kernel.org/vger-lists.html#netdev){.badge-link-text} [r/networking](https://www.reddit.com/r/networking/){.badge-link-text} [r/sysadmin](https://www.reddit.com/r/sysadmin/){.badge-link-text} ## Command Reference (Cheatsheet) The following reference collects the essential diagnosis commands in bottom-up order: | Layer | Command | Purpose | | :--- | :--- | :--- | | **Link** | `ip -br link` | Compact state of all interfaces | | **Link** | `ethtool eth0` | Speed, duplex, carrier | | **Link** | `ethtool -S eth0` | Hardware error counters | | **Address** | `ip -br addr` | Assigned IPv4/IPv6 addresses | | **Routing** | `ip route show default` | Default route | | **Routing** | `ip route get 8.8.8.8` | Concrete exit path and source IP | | **Neighbours** | `ip neigh show` | ARP/NDP table | | **DNS** | `resolvectl query example.com` | Resolver status and query | | **DNS** | `dig @8.8.8.8 example.com` | Direct query to a known server | | **Sockets** | `ss -tulpn` | Listeners with process | | **Path** | `ping -c 5 ` | Reachability without DNS | | **Path** | `mtr -rwc 30 ` | Hop statistics | | **Path** | `tracepath -n ` | Path MTU | | **Firewall** | `nft list ruleset -a` | Rules and counters | | **External** | `nc -zv ` | TCP connect from outside | | **Capture** | `tcpdump -i eth0 -nn -w /tmp/c.pcap host ` | Targeted pcap | | **Logs** | `journalctl -k -b` | Kernel messages of this boot | ## Further Resources | Resource | Description | Type | | :--- | :--- | :--- | | [OSI model](/en/network/the-osi-model-a-detailed-introduction-to-the-7-layers){.badge-link-text} | Layer model for the bottom-up path | Article | | [TCP/IP protocol family](/en/network/tcp-ip-structure-and-how-the-protocol-family-works){.badge-link-text} | TCP, UDP, ICMP and addressing | Article | | [Public vs private IP addresses](/en/network/public-vs-private-ip-addresses-the-main-differences){.badge-link-text} | NAT, RFC 1918 and routing context | Article | | [Linux kernel networking docs](https://www.kernel.org/doc/html/latest/networking/index.html){.badge-link-text} | Kernel networking reference | Documentation | | [RFC 793](https://datatracker.ietf.org/doc/html/rfc793){.badge-link-text} | TCP specification | RFC | | [RHEL 9 networking guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/index){.badge-link-text} | Distribution networking guide | Documentation | | [Brendan Gregg’s blog](https://www.brendangregg.com/blog/){.badge-link-text} | Performance analysis | Blog | | [Netdev mailing list](https://vger.kernel.org/vger-lists.html#netdev){.badge-link-text} | Kernel networking development | Community | ## Conclusion Network faults on Linux become solvable when you shrink the search space layer by layer instead of jumping between tools. Start at the link, save a snapshot, split IP from DNS, then check sockets, path, filters and only then packets and logs. With `ip`, `ss`, `ethtool`, `mtr`, `nft` and `tcpdump` you have everything you need on a current system — as long as you use them in that order. Most time is not lost because a command is unknown, but because the first check sits too high or the baseline was never saved. Keep the six-step emergency checklist close, number your snapshots and test from a second host when local listen state and external reachability disagree. Then you decide quickly whether the problem is local, on the path or on the far side — and you can prove it.