Analyzing network problems on Linux systematically

Systematic analysis of network problems on Linux is not an optional extra skill. It is a core discipline of daily system administration. Faults rarely appear in isolation. They show up as unclear symptoms: a service does not answer, a host is unreachable at times, name resolution fails or latency rises without an obvious reason. Without a structured approach you quickly lose time in random checks and wrong assumptions. What follows is a clear, repeatable grid with which you narrow problems methodically and make statements that hold up.

The goal is a practice-oriented reference. You do not get a theoretical textbook and no collection of isolated commands. You learn how you use the onboard tools of modern Linux systems in a targeted order. The focus is a bottom-up approach that runs from local host configuration through connectivity and path checks to packet and log analysis. Each step builds on the previous one and shrinks the search space. That way you avoid typical detours and reach a solid diagnosis faster.

The path runs step by step from the basics to deep analysis.

You work systematically from local host configuration through path and reachability tests to packet captures and kernel logs. Every section is built so that on an acute problem you can also jump straight into the matching chapter.

Prerequisites for successful work with this material are solid Linux system knowledge, confident use of the shell and first practical experience in system administration. You should already understand the basic network concepts — interfaces, addressing, routing, DNS and firewall — and be able to move in the terminal without constant lookup. Shell-scripting knowledge helps automate recurring checks, but it is not mandatory. The text assumes that you can interpret the output of the tools used and place the results in an overall context.

⚠️ Important: This is aimed at Linux administrators and IT professionals who already have basic experience with network administration on Linux. You should be confident in the terminal and understand the most important network concepts.

Anyone who brings those basics finds a tool that works in production. The text stays close to real admin conditions: limited time, unclear symptoms and the need to decide quickly whether the problem is local, on the path or on the far side. That is exactly what the systematics here is for.

The following method is the foundation for all further steps.

Systematic method of network diagnosis

Bottom-up approach along the network layers

Network problems on Linux rarely arise on a single layer in isolation. Every higher layer assumes the ones below it work. That is why you always start diagnosis from the bottom and work up. This bottom-up approach stops you wasting time on complex tools while the real cause sits much closer to the physical or link layer.

The relevant layers follow the classic model, adapted to Linux reality. At the bottom you check the physical link and driver status. Next is the data-link layer with MAC addresses, ARP and VLAN assignment. The network layer covers IP addressing and routing. Above that sit transport protocols and sockets, finally the application itself.

┌─────────────────────────────────────┐                        
│ Application / service (logs, config)│                        
├─────────────────────────────────────┤                        
│ Transport (TCP/UDP, ports, ss)      │                        
├─────────────────────────────────────┤                        
│ Network (IP, routing, ICMP)         │                        
├─────────────────────────────────────┤                        
│ Data-link (ARP, MAC, bridges)       │                        
├─────────────────────────────────────┤                        
│ Physical / link (cable, ethtool)    │                        
└─────────────────────────────────────┘                        
                ↑ Diagnosis starts here                        

Always start with link status. An interface that is DOWN or reports no carrier makes every further check pointless. Only when the link is up and a valid address is present do you check gateway reachability. After that you test forwarding and only then DNS, ports and application answers.

⚠️ Note: If you skip the lower layers, you risk misdiagnosis: you hunt firewall rules or service configs while the fault already sits in the driver or the cabling.

The approach forces discipline. You document every result before you go to the next layer. Only then do you see dependencies and avoid parallel, unstructured checks. In complex environments with bridges, bonding or containers this disciplined path is especially valuable, because several virtual layers sit on top of each other.

💡 Tip: The bottom-up path also fits when symptoms appear at application level. A web server that accepts no connections can fail just as well on a missing route or a down interface as on a wrong listen address.

🔧 Practical example:

Imagine a host does not answer SSH requests. Instead of immediately starting ss or tcpdump, you begin with the interface:

ip link show eth0
ethtool eth0 | grep -E 'Link detected|Speed|Duplex'
ip addr show eth0
ip route show default

Only when Link detected: yes, a valid IP and a default route are present do you go on to ping and socket status. This sequence often saves half the diagnosis time in practice.

Information gathering and reproducibility

A solid diagnosis stands or falls with the quality of the information you collect. Before you go deep into individual layers, you save the current state of the system. Only then can you later reconstruct what changed and trigger the problem again under comparable conditions. Without that base many observations stay anecdotal and hard to verify.

Start with a fixed set of baseline data.

Note time, affected interfaces, current IP addresses, routing table and the status of the relevant services. Also capture the latest kernel messages and the output of the most important network tools. This snapshot is the reference point. Later you can run the same command set again and see the differences.

Reproducibility means you can recreate the symptom under controlled conditions. That is decisive for intermittent faults. Document exactly under which circumstances the problem appears: after a reboot, only at certain times of day, only under load or only from certain source addresses. Record which steps you have already taken and which results they produced.

⚠️ Note: Without this discipline you quickly lose the overview and repeat the same checks several times.

A practical way is to redirect outputs into files and stamp them with a timestamp. That creates a traceable history. Use simple shell constructs and store the results in a dedicated directory. Make sure the files contain the relevant context — hostname, date and the exact command.

💡 Tip: Reproducibility gains extra value when several people work on the incident. A clearly documented baseline stops colleagues repeating the same steps or starting from stale assumptions.

🔧 Practical example:

Create a directory and save the initial state:

mkdir -p /tmp/netdiag/$(date +%Y%m%d-%H%M)
cd /tmp/netdiag/$(date +%Y%m%d-%H%M)

{
  echo "=== $(hostname) $(date -Iseconds) ==="
  ip -br link
  ip -br addr
  ip route
  ip neigh
  ss -tulpn
} > 01-basis.txt

journalctl -k -b --no-pager | tail -n 100 > 02-kernel.txt
dmesg -T | tail -n 50 > 03-dmesg.txt

Run the same set later and compare the files. That way you see changes on interfaces, routes or socket states without relying on memory. Add targeted captures or extra tool output as soon as you have narrowed the affected layer.

The collected data stay useful only if they are complete and consistent in time. Avoid running individual commands in isolation and without context. Always work with the full set and save the outputs before you change configurations or restart services.

❗ Note: A common error is overwriting older snapshots; number the files clearly and keep the original baseline.

Typical entry mistakes and how to avoid them

Even with clean information gathering and reproducible snapshots, diagnoses often go wrong when the first steps are chosen badly. The most common entry mistakes do not come from not knowing individual commands, but from the wrong order and from hasty assumptions. If you know these patterns, you save hours and avoid changing the symptom further with your own interventions.

A classic error is to search at application level immediately.

Many people start with port scans, firewall rules or service logs although the interface is still DOWN or has no carrier. That leads to long detours. Avoidance is simple: every diagnosis starts with link and address status. Only when that base is sound may the next layer be looked at. If you skip that, you later interpret symptoms that do not belong to the real cause.

Name resolution and reachability are often mixed up too.

A failed ping to a hostname is quickly treated as a network outage, although only the resolver is stuck. The correct split is: first ping the IP address directly, then the hostname. If only the name fails, the problem is DNS or /etc/hosts. If the IP already fails, the layer below is affected.

⚠️ Note: If you do not make that distinction, you waste time in the wrong place and suspect routing or firewall although the fault sits in resolution.

Blind trust in outdated tools is just as common.

ifconfig and netstat give incomplete or misleading output on modern systems because they only show bridge, bonding and namespace information in a limited way. Avoidance is to use ip and ss consistently. These tools read directly from the kernel and show the actual state. Anyone who still uses the legacy commands often misses virtual interfaces or container networks.

Equally problematic: changing the system before the baseline is saved. Restarting NetworkManager, flushing the ARP table or reloading firewall rules can make the symptom disappear temporarily and hide the cause. The discipline is: document first, then intervene. Every change must be traceable and ideally reversible.

The viewpoint is often chosen wrongly too. Many people test only from the affected host and miss that the service listens locally but is not reachable from outside — or the reverse. Avoidance needs at least a second observation point: another host in the same segment or a targeted test from outside. Only then is it clear whether the problem is local, on the path or on the far side.

💡 Tip: A subtle error arises when temporary states are ignored. Stale ARP entries, expired DHCP leases or short-lived routes can produce symptoms that vanish on their own after a few minutes. Anyone who at that moment already goes deep into packet analysis looks for a cause that no longer exists. The countermeasure is to capture the time of the symptom exactly and repeat the same test soon after.

Wrong entry                       Correct entry                
───────────                       ─────────────                
Check service logs          →     Check link status            
Read firewall rules         →     Check address and route      
Start tcpdump               →     Split reachability           
Change configuration        →     Save a snapshot              

🔧 Practical example:

Assume a host does not answer HTTP requests. The wrong entry often looks like this:

ss -tulpn | grep 80
nft list ruleset
tcpdump -i any port 80
systemctl restart nginx

The correct entry starts lower and leaves the state unchanged:

ip -br link
ip -br addr
ip route get 8.8.8.8
ping -c 3 $(ip route | awk '/default/ {print $3}')
ss -tlnp '( sport = :80 )'

Only when these outputs are consistent and unremarkable do you bring in firewall and packet capture. This order prevents a service restart from wiping valuable state messages.

A last, often overlooked error is working with insufficient rights or in the wrong network namespace. Commands without enough privileges return incomplete socket lists or may not query certain interfaces. In container environments the visible network stack is not that of the host. Avoidance means checking the context explicitly: ip netns list and targeted execution inside the right namespace if one exists.

❗ Note: Without the correct namespace you may be checking an entirely different network setup than the affected one.

Analysis of the local host configuration

Interfaces, link status and addressing

Checking interfaces is the real entry into local host analysis. Without a working link and without valid addressing, all further tests stay largely meaningless. Always capture the physical and logical state of the network interface first, before routing, DNS or even services.

The central command for the overview is ip.

With -br you get a compact view of all interfaces, their state and the assigned addresses. The output immediately shows whether an interface is administratively UP and whether the kernel sees a carrier. If the LOWER_UP flag is missing or NO-CARRIER appears, the problem sits below the IP layer.

ip -br link
ip -br addr

For detailed inspection of individual interfaces use ip link show.

Here flags, MTU, queue discipline and the current state become visible. Especially relevant are the combinations UP,LOWER_UP (link is up). UP,NO-CARRIER (interface enabled, but no signal). MAC address and promiscuous mode can also give hints, for example in bridge or monitoring setups.

In addition ethtool provides information that ip does not cover.

That includes negotiated speed, duplex mode, link detection and error counters. A gigabit interface that only runs at 100 Mbit/s or in half-duplex almost always points to a cable, switch-port or autonegotiation problem. Rising error or drop counters point to physical faults or driver problems.

ethtool eth0
ethtool -S eth0 | grep -E 'error|drop|crc|frame'

Addressing is checked with ip addr show.

Besides the IP address itself, scope, valid lifetime and assignment mode matter. Dynamic addresses from DHCP always carry the hint dynamic; static ones do not. An address in 169.254.0.0/16 signals that no DHCP server was reached and the system fell back to APIPA. IPv6 addresses should be looked at too, because many modern environments run dual-stack.

💡 Tip: An interface can be administratively UP and still not allow useful communication if the address is missing, expired or assigned to the wrong scope. In those cases ip shows the link as functional, but the higher layer stays blocked.

The interface names themselves deserve attention too.

Modern distributions use predictable names such as ens3, enp0s3 or enx followed by the MAC. Older scripts that still expect eth0 then fail silently. In virtualised or containerised environments virtual interfaces, VLANs and bonding devices are added. These must be included in the check explicitly, because they determine the actual exit path.

Link check in practice                                         
──────────────────────                                         
1. ip -br link          → state of all interfaces              
2. ip link show DEV     → flags and details                    
3. ethtool DEV          → speed, duplex, carrier               
4. ethtool -S DEV       → error counters                       
5. ip addr show DEV     → addresses and scope                  

💡 Tip: The order is deliberate: first administrative and physical state, then addressing. That way a missing address is not mistaken for a routing or firewall problem.

🔧 Practical example:

A host should reach external destinations but does not answer. The check starts with the full set:

ip -br link
ip -br addr
ip link show eth0
ethtool eth0 | grep -E 'Speed|Duplex|Link detected'
ethtool -S eth0 | grep -iE 'err|drop|crc'
ip addr show eth0

Expected normal values are state UP, LOWER_UP, Link detected: yes, a matching speed/duplex combination and at least one valid global address. If NO-CARRIER or Link detected: no appears, continue diagnosis at the physical or driver level. If the address is missing or only a link-local address is present, the problem is address assignment.

Watch temporary states

Interfaces can briefly show as DOWN after a driver reload or after a cable is unplugged and then go UP again without the address being renewed. In DHCP environments a lease renew often has to be forced then. Static configurations stay stable as long as the interface is not reconfigured explicitly.

❗ Note: A query of administrative status alone is not enough — without checking carrier and error counters you miss physical packet loss.

Routing, neighbour tables and DNS resolvers

After link and address status, forwarding and name resolution come into focus. This layer decides whether packets can leave the local host at all and whether destination addresses are resolved correctly. Errors here often produce symptoms that are wrongly blamed on the firewall or the remote service.

The routing table is examined with ip route and targeted queries. First you check the default route. If it is missing or points at an unreachable gateway, external communication is impossible. Then you test the concrete path to a destination with ip route get. This command shows not only the route, but also the interface used and the source address. That is especially valuable on systems with several interfaces or policy routing.

ip route
ip route show default
ip route get 8.8.8.8
ip route get 192.168.1.50 from 10.0.0.10

Several routing tables or rules (ip rule) appear in more complex setups. For most diagnoses the main table is enough. What matters is that the route to the gateway goes via an interface that is actually UP and addressed. A route via a DOWN interface stays ineffective even if it sits in the table.

The neighbour table (ARP for IPv4, NDP for IPv6) is viewed with ip neigh. It shows which layer-2 addresses the kernel has learned for known IPs. States such as REACHABLE and STALE are normal. FAILED or INCOMPLETE indicate that the gateway or neighbour does not answer. A permanently missing entry for the default gateway is a clear signal of a problem on the data-link layer or at the gateway itself.

ip neigh
ip neigh show dev eth0
ip neigh show to 192.168.1.1

Stale entries can cause temporary faults. In those cases a targeted delete of the entry helps so the kernel resolves the address again. That should only happen after the baseline is documented.

⚠️ Note: A missing or wrong default route is often overlooked because ping to local addresses still works and only fails for external destinations. Checking with ip route get makes the actual exit path visible and prevents misdiagnosis.

The DNS resolver is the next checkpoint. Modern systems often use systemd-resolved. Status and configured servers are queried with resolvectl. /etc/resolv.conf should be checked as well, because older applications or containers still use it. A working resolver does not automatically mean resolution works for all applications — especially in container environments their own resolver configs can apply.

resolvectl status
resolvectl query example.com
cat /etc/resolv.conf
dig +short example.com
dig @8.8.8.8 example.com

The split between reachability and name resolution remains decisive. A successful ping to the IP with a failed hostname isolates the problem clearly to DNS. Conversely a failed IP ping shows that the cause sits deeper and DNS is irrelevant for now.

💡 Tip: The combination of ip route get and a targeted dig against a known server tells you in a few seconds whether routing or resolution causes the fault. That saves the detour through unnecessary firewall or packet analysis.

🔧 Practical example:

A host should reach an internal service under the name app.internal. The check runs in a clear order:

# Routing and gateway
ip route show default
ip route get 10.20.30.40
ping -c 3 $(ip route | awk '/default/ {print $3}')

# Neighbour table for the gateway
GATEWAY=$(ip route | awk '/default/ {print $3}')
ip neigh show to $GATEWAY

# DNS split
ping -c 2 10.20.30.40
ping -c 2 app.internal
dig +short app.internal
resolvectl query app.internal

If the ping to the IP fails while the route and neighbour entry look correct, the problem is beyond the local host. If only name resolution fails, you check the resolver and DNS-server reachability. An extra test against a public resolver (dig @1.1.1.1) shows whether the problem is local or at the internal DNS server.

On hosts with several interfaces, policy routing also plays a role. On hosts with several addresses ip route get can pick a different source IP than expected. That causes replies to come back on the wrong path or to be dropped by firewalls. In those cases an explicit source address on the test helps.

ip route get 8.8.8.8 from 192.168.10.50
ping -I 192.168.10.50 -c 3 8.8.8.8

❗ Note: Anyone who skips neighbour table and gateway routes often suspects errors in the application although layer 2 or 3 is already blocked.

Socket status and local services with ss

Next you check whether local services listen on the expected sockets. This is where it is decided whether a service actually listens, which address it is bound to and whether connections can be built. ss is the tool of choice here, because it reads socket information directly from the kernel and works clearly faster and more completely than older netstat.

The first and most important check is listening sockets.

With ss -tulpn you see all TCP and UDP listeners, the numeric ports and the associated processes. The output immediately shows whether a service is bound to 0.0.0.0, a specific IP or only 127.0.0.1. A bind only to localhost is one of the most common reasons a service works locally but stays unreachable from outside.

ss -tulpn
ss -tlnp
ss -ulnp
ss -tlnp '( sport = :80 or sport = :443 )'

For more detailed filtering ss supports expression syntax. That lets you search by port, state or address without walking the whole output by hand. Restricting to a single port or to connections in a given state is especially useful.

ss -tn state listening
ss -tn state established
ss -tn state syn-sent
ss -tn '( dport = :443 or sport = :443 )'
ss -tnp dst 10.20.30.40

The states themselves give important hints.

LISTEN confirms that the service has opened the port. ESTABLISHED shows active connections. SYN-SENT points to outbound connection attempts that are not answered. A high number of TIME_WAIT entries can be normal on heavily loaded servers, but becomes a problem when local ports are exhausted.

⚠️ Note: A service that runs in the process list but does not appear as a listener in ss is either bound to the wrong address, started in the wrong network namespace or already crashed without the process manager noticing.

Besides the pure socket list, ss -s gives a summary of socket statistics. Here you see the number of connections per state at a glance. That helps spot anomalous piles quickly, for example masses of CLOSE-WAIT or SYN-RECV entries that can point to resource problems or attacks.

ss -s
ss -tn state time-wait | wc -l
ss -tnp | grep -E 'CLOSE-WAIT|FIN-WAIT'

In environments with containers or network namespaces the context must be watched. An ss on the host does not show sockets inside a container. Either nsenter or running ss directly in the container helps. Otherwise you diagnose the wrong network stack.

# Example for namespace switch
ip netns list
ip netns exec myns ss -tulpn

💡 Tip: Combining a port filter with process display (-p) saves time, because you immediately see which process holds the port and whether it runs under the expected user. That prevents mix-ups when several instances of the same service exist.

🔧 Practical example:

A web service should be reachable on port 8080 but does not answer. The systematic check with ss looks like this:

# Show all listeners and filter specifically
ss -tlnp | grep 8080
ss -tlnp '( sport = :8080 )'

# Check which address it is bound to
ss -tlnp '( sport = :8080 )' | awk '{print $4}'

# Active connections to the port
ss -tnp '( dport = :8080 or sport = :8080 )'

# Summary of states
ss -s

The expected result on a correct configuration is a LISTEN entry on 0.0.0.0:8080 or the desired interface IP plus the matching process name. If only 127.0.0.1:8080 appears, the bind is wrong. If the entry is missing entirely, the service is not running or listens on another port. In that case you check the process list and the service config before further measures.

Special points for UDP sockets

Unlike TCP, UDP has no connection state in the classic sense. ss -ulnp still shows the bound ports and processes. That is decisive for services such as DNS, NTP or your own UDP-based applications. If the entry is missing here while the process runs, there is almost always a wrong bind or a namespace problem.

ss -ulnp
ss -unp '( sport = :53 )'

❗ Note: Outdated tools such as netstat or missing -p flags often hide which process holds a port or in which namespace it runs.

Connectivity and path analysis

Reachability tests

Once sockets and local services are established, diagnosis moves to the external network path. The question now is whether packets leave the host, reach the destination and come back. For that ping, mtr and tracepath give different views of the same path that complement each other.

ping remains the fastest reachability test.

It checks whether ICMP echo requests are answered and returns latency and packet loss. What matters is the consistent split of IP and name resolution. First the pure IP is tested, then the hostname. Only that isolates DNS problems from real connection faults. -c limits the number of packets, -W sets the timeout per packet and -i controls the interval.

ping -c 5 8.8.8.8
ping -c 5 -W 2 192.168.1.1
ping -c 10 -i 0.2 10.20.30.40
ping -c 5 example.com

A successful ping to the IP with a failed hostname points clearly at the resolver. Conversely a failed IP ping shows that the problem is on the path or at the destination. High latency swings or rising loss point to overload or unstable links.

mtr combines ping and traceroute and gives a running statistic for every hop. In report mode (-r or -rw) it produces a one-shot summary that documents well. -c sets the number of cycles. With --tcp or --udp you can test other protocols instead of ICMP, which is useful when ICMP is filtered.

mtr -rwc 50 8.8.8.8
mtr -rwc 30 --tcp --port 443 example.com
mtr -n -rwc 20 10.20.30.40

The output shows loss and latency for every hop. Loss that jumps up from a certain hop and stays on the following hops marks the problematic point. Loss that appears only on a single hop and then vanishes is often ICMP rate limiting and not necessarily a real problem.

⚠️ Note: ICMP is filtered or limited on many paths. A failed ping or high loss in mtr therefore does not automatically mean that TCP or UDP traffic is blocked too. In those cases extra tests with TCP-based mtr or targeted port checks are required.

tracepath adds path-MTU discovery. It shows not only the hops, but also the maximum packet size that can pass without fragmentation. That is decisive for problems with large packets, VPN tunnels or certain cloud connections. Unlike classic traceroute, tracepath needs no root rights.

tracepath 8.8.8.8
tracepath -n 10.20.30.40
tracepath -b example.com

The combination of the three tools gives a solid picture. ping gives the fast yes/no and the base latency. mtr shows where on the path loss or delay arises. tracepath clarifies whether the MTU restricts traffic. Together they stop a single symptom being over-interpreted.

💡 Tip: Choose the order deliberately: first a short ping to the IP, then mtr for hop statistics and finally tracepath if you suspect MTU problems. That keeps diagnosis effort low and the signal high.

Reachability test — useful order                               
────────────────────────────────                               
1. ping -c 5 <IP>           → basic decision                   
2. ping -c 5 <hostname>     → DNS split                        
3. mtr -rwc 30 <target>     → hop statistics                   
4. tracepath <target>       → MTU path                         

🔧 Practical example:

An internal server at 10.50.1.20 should be reached, the application reports timeouts.

The check runs as follows:

# Basic decision and DNS split
ping -c 5 10.50.1.20
ping -c 5 app.internal.example

# Hop analysis with a large enough sample
mtr -rwc 50 -n 10.50.1.20
mtr -rwc 30 --tcp --port 443 10.50.1.20

# Check MTU path
tracepath -n 10.50.1.20

# Extra test with smaller interval if you suspect intermittent loss
ping -c 20 -i 0.2 10.50.1.20

If loss in mtr rises permanently from hop 3, the fault sits there. If loss stays high on ICMP while the TCP test with --tcp runs clean, ICMP is filtered. If tracepath shows an unexpectedly low pMTU, large packets or tunnel configs must be investigated.

On path analysis also watch asymmetric routes. ping and mtr measure the outbound and return path. If the return path differs and packets are dropped there, the destination appears unreachable although the outbound path is intact. In those cases comparing tests from both sides or looking at routing tables on the hosts involved helps.

❗ Note: A pure ICMP check easily misleads, because many firewalls or routers throttle or filter ICMP packets on purpose.

Firewall and filter rules

nftables, firewalld, ufw

Local packet filters often decide whether an otherwise intact path ends at the destination host. The three common management paths — nftables, firewalld and ufw — must therefore be read systematically as soon as reachability tests suggest a break.

nftables is the lowest visible layer.

nft list ruleset dumps the entire active configuration. What matters most are the counters on the individual rules. If the counter of a drop or reject rule rises while accept rules for the same port stay at zero, the cause is found locally. Handle numbers (-a) later allow precise removal of single entries without reloading the whole base.

nft list ruleset
nft list ruleset -a
nft list table inet filter
nft list chain inet filter input
nft list chain inet filter forward

On dual-stack systems IPv4 and IPv6 tables must be looked at separately. A rule that exists only in one family explains asymmetric symptoms.

firewalld organises access via zones.

Each interface sits in exactly one zone, and each zone brings its own default policies plus allowed ports and services. If the zone of the affected interface differs from the expected one, suddenly stricter rules apply. Runtime and permanent configuration can also diverge; only the permanent settings survive a service restart.

firewall-cmd --state
firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --list-ports
firewall-cmd --query-port=8443/tcp
firewall-cmd --permanent --list-all

ufw gives a numbered, easy-to-read overview.

The default policies for incoming and outgoing sit at the top. Rule numbers allow targeted deletion without touching the rest of the configuration.

ufw status verbose
ufw status numbered
ufw app list

Regardless of the frontend, the same check order applies: default policy of the relevant chain, explicit allow rules for port and protocol, assignment of the interface to a zone (with firewalld), finally the counters or log entries. Only then is it visible whether a packet even reached the rule base or was already dropped earlier.

⚠️ Note: The position of a rule inside the chain is decisive. A late accept rule stays ineffective if an earlier drop rule already matches the same traffic.

The difference between drop and reject strongly affects the symptoms. drop produces timeouts, reject gives the sender a clear reply. Many diagnostic tools treat a missing reject as a generic connection break and hide the actual filter effect.

Filter diagnosis — critical points                             
──────────────────────────────────                             
* Policy of the input and forward chain                        
* Explicit allow rule for port/protocol                        
* Interface-to-zone assignment (firewalld)                     
* Counter values on drop rules                                 
* Runtime vs permanent match                                   

💡 Tip: Counters and log messages are the only objective proof that a packet was actually hit by a given rule. Without that information every guess about blocked traffic stays unconfirmed.

🔧 Practical example:

Port 8443 is locally in listen state, but external connections fail. The filter check runs like this:

# nftables — rules and counters
nft list ruleset -a
nft list chain inet filter input | grep -E '8443|drop|reject|accept'

# firewalld — zone and port
firewall-cmd --get-active-zones
firewall-cmd --list-all
firewall-cmd --query-port=8443/tcp
firewall-cmd --permanent --query-port=8443/tcp

# ufw — if active
ufw status numbered | grep 8443

An accept rule with a rising counter and a matching zone assignment speak for an intact filter configuration. If the rule is missing or the counter of a drop rule rises, the cause is in the local policy. With firewalld you must also check whether the rule is set both runtime and permanent.

On systems that also route or bridge, the forward chain applies. An allow rule in the input chain is then not enough. Traffic that only traverses the host is filtered separately and must be allowed on its own.

nft list chain inet filter forward
firewall-cmd --list-all --zone=internal

❗ Note: Forward rules are easily overlooked because most diagnosis scripts only read the input chain. On router, bridge or container hosts that leads to false all-clears.

Service and port reachability from outside

Local confirmation of an open port and allowed filter rules is not enough. Only a test from a remote system shows whether the service is actually reachable from outside. Differences between local listen state and external reachability come from filters in front, asymmetric routes or binds that only allow local traffic.

nc is a fast connection checker. In zero mode the tool only builds the TCP handshake and then closes the connection immediately. The reply is unambiguous: success or timeout / connection refused. For UDP services -u is set. Timeout can be limited with -w so hanging tests do not block for too long.

nc -zv 10.50.1.20 8443
nc -zv -w 4 10.50.1.20 443
nc -zvu -w 3 10.50.1.20 53
nc -zv app.internal.example 8080

If the attempt from a remote host fails while the same command succeeds on the target itself, the fault sits on the path in between.

nmap adds state information besides the pure connection status. The connect scan (-sT) works without root and reports open, closed or filtered. The SYN scan (-sS) is more precise but needs elevated privileges. -Pn skips host discovery if ICMP is dropped on the way. Service detection (-sV) also tries to identify the protocol on the port.

nmap -p 8443 10.50.1.20
nmap -sT -Pn -p 80,443,8443 10.50.1.20
nmap -sS -p 8443 10.50.1.20
nmap -sV -p 8443 10.50.1.20

closed means a reset came back — the host answers, but the port is not occupied. filtered signals missing replies and thus a filter on the path or at the destination.

For HTTP and HTTPS endpoints curl adds the application layer on top of transport. Connection errors, TLS problems and HTTP status codes become visible separately. --connect-timeout prevents long waits, --resolve forces a given IP and bypasses DNS influence on the test.

curl -v --connect-timeout 5 https://10.50.1.20:8443/
curl -v -o /dev/null -w "%{http_code}\n" http://10.50.1.20:8080/health
curl -v --resolve api.example.com:8443:10.50.1.20 https://api.example.com:8443/

⚠️ Note: Local success with ss and simultaneous failure of nc or nmap from outside point to a fault that sits outside the target host. The further search must then continue on the path or on systems in between.

The choice of test system affects the result noticeably. A host in the same layer-2 segment sees different filters and routes than a system in a remote net or behind a NAT gateway. Tests should therefore be run from at least two different locations if the topology allows.

The source address used can also be relevant when access-list-based rules apply on the target or in between.

External check — useful combination                            
──────────────────────────────────                             
1. nc -zv          → fast connect test                         
2. nmap -sT/-sS    → state information                         
3. curl -v         → application layer (HTTP/S)                
4. Comparison      → same vs foreign subnet                    

💡 Tip: A port can appear open in nmap and still react wrongly at application level. Only the protocol-specific test with curl or a suitable client gives the final statement about service availability.

🔧 Practical example:

A service on port 8443 is bound locally and allowed in the filter rules. Clients in the company net report connection errors. The external check is run from two different hosts:

# Test from host A (same subnet)
nc -zv -w 3 10.50.1.20 8443
nmap -sT -Pn -p 8443 10.50.1.20

# Test from host B (other net)
nc -zv -w 3 10.50.1.20 8443
nmap -sT -Pn -p 8443 10.50.1.20
nmap -sV -p 8443 10.50.1.20

# Application test
curl -v --connect-timeout 5 https://10.50.1.20:8443/health
curl -v --resolve service.example.com:8443:10.50.1.20 https://service.example.com:8443/health

Successful connect tests from both sides confirm the transport layer. If the curl call then fails, the cause is TLS, certificate or application logic. If nc already stays without a connection, the path between the hosts involved must be examined again.

On systems with several addresses or interfaces the reply can return on a different path than the outbound one. In those cases SYN packets arrive, but SYN-ACK packets leave the host via an interface the client does not expect. Comparing packet flows on the target host with tcpdump makes that asymmetry visible.

❗ Note: Tests that are started only from the target host itself systematically hide every problem that appears only on the path between client and server.

Packet analysis, logs and extended diagnosis

Targeted captures

When external reachability tests and local checks do not give a clear cause, what remains is looking at the actual packets. tcpdump captures traffic on the chosen interface and shows whether SYN packets arrive, whether replies leave the host and which flags or options are set. The skill is to set capture filters so tightly that only the relevant traffic appears and the output stays usable.

The basic call specifies the interface and a filter expression. Without a filter too much is captured quickly. With -n addresses and ports are shown numerically, which speeds output and prevents DNS lookups during the capture. -nn additionally suppresses resolution of port names. For later analysis in Wireshark or tshark the output is written to a file (-w).

tcpdump -i eth0 -n port 8443
tcpdump -i any -nn host 10.50.1.20 and port 8443
tcpdump -i eth0 -nn -w /tmp/capture.pcap host 10.40.1.15 and port 443

Filter expressions follow BPF syntax. They can be combined by host, net, port, protocol and direction. The operators and, or and not allow precise limits. For TCP handshakes flags are especially useful: tcp[tcpflags] & tcp-syn != 0 captures SYN packets; further flags can be addressed analogously.

tcpdump -i eth0 -nn 'tcp port 8443 and (tcp[tcpflags] & tcp-syn != 0)'
tcpdump -i eth0 -nn 'host 10.50.1.20 and (port 80 or port 443)'
tcpdump -i eth0 -nn 'src net 10.40.0.0/16 and dst port 8443'
tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'

-c limits the number of captured packets and prevents files that grow without control. -s 0 or -s 65535 ensures full packets are stored, not only headers. On suspicion of fragmentation or unusual options that is essential. For longer captures a ring buffer via -C and -W is recommended so older files are overwritten automatically.

tcpdump -i eth0 -nn -c 100 -w /tmp/short.pcap port 8443
tcpdump -i eth0 -nn -s 0 -w /tmp/full.pcap host 10.50.1.20
tcpdump -i eth0 -nn -C 50 -W 5 -w /tmp/ring.pcap port 443

⚠️ Note: Filters that are too broad quickly produce confusing amounts of packets and make analysis unnecessarily hard. Every capture should be limited to the concrete suspicion and the addresses and ports involved.

Reading a stored capture file is also done with tcpdump. The same filter syntax can be applied afterwards to narrow the stored data further. -tttt produces readable timestamps, -X or -A shows the payload in hex or ASCII.

tcpdump -nn -r /tmp/capture.pcap
tcpdump -nn -tttt -r /tmp/capture.pcap 'tcp[tcpflags] & tcp-syn != 0'
tcpdump -nn -A -r /tmp/capture.pcap port 80
tcpdump -nn -X -c 20 -r /tmp/capture.pcap

On interpretation the order of packets counts. An incoming SYN without a following SYN-ACK points to filtering or a service that does not answer. A SYN-ACK that leaves the host but never arrives at the client points to problems on the return path. Duplicate SYN packets or unexpected RST flags give further hints of timeout behaviour or active rejection.

💡 Tip: Combining a tight filter, a limited packet count and then targeted evaluation of the pcap file prevents relevant information drowning in a flood of irrelevant packets.

🔧 Practical example:

External clients cannot reach port 8443 although the service listens locally and the firewall allows the port. The capture is started on the target host while a client tries the connection:

# Start capture (in a separate terminal)
tcpdump -i eth0 -nn -s 0 -w /tmp/8443.pcap host 10.40.1.15 and port 8443

# In parallel from the client:
nc -zv 10.50.1.20 8443

# Stop capture (Ctrl+C) and evaluate
tcpdump -nn -tttt -r /tmp/8443.pcap
tcpdump -nn -r /tmp/8443.pcap 'tcp[tcpflags] & tcp-syn != 0'
tcpdump -nn -r /tmp/8443.pcap 'tcp[tcpflags] & tcp-ack != 0'

If SYN packets from the client appear but no SYN-ACK packets from the server, the cause sits on the target host itself (service, local filters or bind). If SYN-ACK packets leave the host but do not reach the client, the return path must be examined. If even the SYN packets are missing, a filter in front of the target host is in play.

For UDP services the view changes because there is no handshake. Request and possible reply packets are compared directly. Packet size and possible ICMP errors (fragmentation needed, port unreachable) also give hints.

tcpdump -i eth0 -nn -s 0 -w /tmp/dns.pcap port 53
tcpdump -nn -r /tmp/dns.pcap

❗ Note: Captures without prior restriction to host and port produce files that are hardly useful to evaluate in practice and can also contain sensitive data.

Log and kernel analysis

Packet captures show traffic on the wire. System and kernel logs explain why that traffic happens or breaks. Both sources belong together: a missing SYN-ACK in the capture file only becomes understandable when the kernel at the same time reports a driver error, a netfilter drop or a link loss.

journalctl is the central entry on systems with systemd. The unit NetworkManager, systemd-networkd or the service in question can be queried specifically. Time filters limit the output to the relevant window. -k restricts the display to kernel messages, -b shows only the current boot.

journalctl -u NetworkManager --since "10 min ago"
journalctl -u systemd-networkd -b
journalctl -k --since "2024-01-01 14:30:00" --until "2024-01-01 14:45:00"
journalctl -k -b | grep -iE 'eth0|link|carrier|netfilter|nf_'

For network-specific events it is worth searching for terms such as link, carrier, timeout, refused or the interface name. Many drivers write the reasons for a link-down or rejected packets here.

dmesg delivers the same kernel ring buffer, often a bit more directly and with readable timestamps via -T. The output can be filtered with the same pattern. On suspicion of driver problems or firmware messages dmesg is often the faster path.

dmesg -T | tail -n 100
dmesg -T | grep -iE 'error|fail|timeout|eth0|enp'
dmesg -T --level=err,warn

Netfilter itself writes its own entries when logging is enabled. Rules with the target LOG or NFLOG produce messages that become visible in the kernel logs or via journalctl. The priority and prefix of the log rule determine how easy the entries are to find later.

journalctl -k | grep -i 'NETFILTER\|NFLOG\|DROP'
dmesg -T | grep -i 'DROP\|REJECT'

⚠️ Note: Kernel messages about link losses or driver resets often appear only briefly and are pushed out of the ring buffer quickly when log volume is high. Capture must therefore happen close to the observed symptom.

Correlation with the packet capture is done via timestamps. A SYN packet that arrives in the capture file and a simultaneous kernel message about a drop in the input chain explain the missing SYN-ACK. Driver messages about CRC errors or overruns can also explain loss of packets that tcpdump already no longer saw.

Log sources and their focus                                    
───────────────────────────                                    
journalctl -u <service>  → service and NetworkManager events   
journalctl -k / dmesg    → driver, link, netfilter, hardware   
Service logs             → application errors, bind problems   

💡 Tip: Combining a tight time filter and a targeted term search reduces the log volume to the entries that belong to the concrete incident. Without that limit, relevant messages drown in general system activity.

🔧 Practical example:

A service on port 8443 intermittently does not answer. The packet capture shows incoming SYN packets without SYN-ACK. In parallel the logs of the relevant time window are checked:

# Kernel and netfilter messages in the window
journalctl -k --since "14:32:00" --until "14:35:00" | grep -iE 'eth0|drop|reject|link|error'

# NetworkManager or networkd
journalctl -u systemd-networkd --since "14:32:00" --until "14:35:00"
journalctl -u NetworkManager --since "14:32:00" --until "14:35:00"

# dmesg with readable timestamps
dmesg -T | grep -iE 'eth0|enp|error|fail|timeout' | tail -n 50

# Service log in parallel
journalctl -u myapi.service --since "14:32:00" --until "14:35:00"

If messages about carrier loss or netfilter drops appear in the same window, the cause is found. If such entries are missing while the service itself reports errors accepting connections, the problem sits in the application or in resource shortages.

Driver and firmware messages deserve special attention. Many NICs write recurring entries on overload, faulty autonegotiation or damaged cables. These messages often correlate with rising error counters on the interface and explain intermittent losses that only show sporadically in pure connectivity tests.

dmesg -T | grep -iE 'firmware|reset|watchdog|tx_|rx_|overrun'
ethtool -S eth0 | grep -iE 'error|fail|drop|crc|over'

❗ Note: Logs that are read without reference to the concrete time and without a parallel packet capture easily lead to wrong cause assignment, because they contain events from completely different contexts.

Error counters, performance and MTU problems

Quantitative counters at interface and protocol level make visible what individual packet captures and log excerpts only hint at. They answer how often and to what extent errors, losses or limits occur.

Hardware-near counters come from ethtool -S. Here CRC errors, alignment problems, overruns, missed packets and collision counters of the NIC appear. These values come directly from the controller registers and stay independent of kernel statistics. In parallel ip -s link holds the software-side RX and TX counters, including drop and error values that the kernel itself records.

ethtool -S eth0
ethtool -S eth0 | grep -E 'crc|align|over|miss|error|drop|coll|fail'
ip -s link show eth0
ip -s -s link show eth0

The view only becomes meaningful through repeated queries. Two measurements one to five minutes apart and the difference of the relevant fields show whether the counters currently keep running. A high absolute value from the past is meaningless if it does not change in the observation window.

At TCP level ss delivers retransmit and congestion information per connection. -ti or -tai show RTO, retransmits, congestion window and further variables. If retransmits pile up while interface counters stay stable, the loss is highly likely on the path and not on the local host.

ss -ti
ss -tai
ss -ti state established
ss -tai dst 10.50.1.20 | grep -E 'retrans|rto|cwnd|rtt'

MTU limits produce a class of symptoms of their own. Packets that are too large are dropped when intermediate systems do not allow fragmentation and ICMP messages are suppressed. tracepath finds the path MTU, ping with Don’t-Fragment set and a defined size checks concrete thresholds.

tracepath -n 10.50.1.20
ping -c 4 -M do -s 1472 10.50.1.20
ping -c 4 -M do -s 1400 10.50.1.20
ping -c 4 -M do -s 1200 10.50.1.20
ip link show eth0

The value 1472 tests the classic Ethernet MTU of 1500 bytes minus headers. If this test fails and smaller sizes pass, an MTU restriction exists on the path.

⚠️ Note: CRC and alignment errors usually rise only on physical problems: damaged cable, faulty connector, bad switch port or defective NIC. Pure software changes leave these counters untouched.

Further performance indicators sit in the softnet statistics and the queue disciplines. /proc/net/softnet_stat shows dropped packets in the receive queues of the CPUs. tc qdisc show lists the active queueing strategy and possible drop behaviour. nstat aggregates protocol counters such as TcpRetransSegs or IpInDiscards.

cat /proc/net/softnet_stat
tc qdisc show dev eth0
nstat -az | grep -E 'TcpRetrans|TcpLoss|IpInDiscards|Ip6InDiscards'

💡 Tip: The rate of increase of the counters inside a defined time window is the real indicator. Absolute values without reference to the observation period regularly lead to misinterpretation.

🔧 Practical example:

Sporadic drops occur, captures show occasional retransmits. The counters are read before and during reproduction:

# Measurement 1
ethtool -S eth0 | grep -E 'crc|error|drop|over|miss' > /tmp/e1.txt
ip -s link show eth0 > /tmp/i1.txt
ss -ti > /tmp/s1.txt
nstat -az > /tmp/n1.txt

# Reproduce the fault, then measurement 2
ethtool -S eth0 | grep -E 'crc|error|drop|over|miss' > /tmp/e2.txt
ip -s link show eth0 > /tmp/i2.txt
ss -ti > /tmp/s2.txt
nstat -az > /tmp/n2.txt

# Compare
diff /tmp/e1.txt /tmp/e2.txt
diff /tmp/i1.txt /tmp/i2.txt
diff /tmp/n1.txt /tmp/n2.txt | grep -E 'TcpRetrans|Discards'

# MTU check
ping -M do -s 1472 -c 5 10.50.1.20
tracepath -n 10.50.1.20

If CRC or drop counters rise during the fault, the cause is at the interface or the medium. If they stay constant and only TCP retransmits rise, the loss is further away. If the DF ping with 1472 bytes fails, the MTU on the path or at tunnel endpoints must be adjusted.

On bonding or VLAN configurations the counters must be read on every involved layer. Errors that appear only on the logical bond device point to asymmetries between the slave interfaces or to problems in the bonding logic.

ethtool -S bond0
ethtool -S eth0
ethtool -S eth1
ip -s link show bond0
ip -s link show eth0

❗ Note: Counter values without a before and after measurement in the concrete time window of the problem do not allow a reliable statement about current error activity.

Important resources and checklists

Emergency checklist: bottom-up in 6 steps

When a network problem is reported in production, walk the diagnosis without haste in this order:

Reference tools: modern onboard tools vs legacy

On modern Linux distributions you should avoid outdated commands, because they often do not show virtual network stacks, namespaces or bonding interfaces correctly:

Recommended man pages and documentation

For deeper configuration details and kernel parameters these references pay off:

• Man pages in the terminal:
• man 8 ip — the documentation of the iproute2 suite.
• man 8 ss — detailed description of all filter expressions for sockets.
• man 8 tcpdump — BPF filter syntax and display options.
• man 8 nft — syntax and table structure of modern nftables setups.

Kernel documentation:

• Linux kernel networking documentation

Practice checklists for daily use

On acute incidents these short playbooks help you pin the fault on the right layer without searching:

Scenario 1: service runs but is not reachable from outside

• ss -tlnp '( sport = :<port> )' — check whether the bind is on 0.0.0.0 (not only 127.0.0.1).
• nft list ruleset / firewall-cmd --list-all — check whether the port is allowed in the input chain / zone.
• nc -zv <server-IP> <port> (from an external host) — check whether the handshake comes through.
• tcpdump -i any -nn host <client-IP> and port <port> — check whether SYN arrives at the server.

Scenario 2: name resolution fails or is slow

• ping -c 2 <IP> vs ping -c 2 <hostname> — split reachability from name resolution.
• cat /etc/resolv.conf / resolvectl status — check configured nameservers and status.
• dig +short <hostname> vs dig @1.1.1.1 <hostname> — compare local vs public DNS server.
• time dig <hostname> — check latency and timeouts on the primary DNS.

Scenario 3: sporadic drops or packet loss

• ethtool eth0 and ethtool -S eth0 | grep -iE 'err|drop|crc' — check physical link and error counters.
• mtr -rwc 50 -n <dest-IP> — isolate lasting loss per hop (ICMP vs TCP --tcp).
• ping -M do -s 1472 <dest-IP> — test path MTU for fragmentation problems.
• ss -ti — watch TCP retransmits and RTO of the active connection.

Further sources and community resources

Documentation and specifications:

Linux kernel networking documentation RFC 793 (TCP specification) Red Hat Enterprise Linux network guide

Blogs:

Brendan Gregg’s blog Cloudflare networking blog

Communities and mailing lists:

Netdev mailing list r/networking r/sysadmin

Command Reference (Cheatsheet)

The following reference collects the essential diagnosis commands in bottom-up order:

Further Resources

Conclusion

Network faults on Linux become solvable when you shrink the search space layer by layer instead of jumping between tools. Start at the link, save a snapshot, split IP from DNS, then check sockets, path, filters and only then packets and logs. With ip, ss, ethtool, mtr, nft and tcpdump you have everything you need on a current system — as long as you use them in that order.

Most time is not lost because a command is unknown, but because the first check sits too high or the baseline was never saved. Keep the six-step emergency checklist close, number your snapshots and test from a second host when local listen state and external reachability disagree. Then you decide quickly whether the problem is local, on the path or on the far side — and you can prove it.