---
id: linux-administration-processes-and-resource-management
slug: linux-administration-processes-and-resource-management
title: "Linux Administration #3: Processes and Resource Management"
excerpt: "In this third part of our Linux administration series, you will learn everything about processes and resource management. From basic concepts to practical application."
date: "2024-10-31T09:00:00+01:00"
updated: "2024-10-31T10:00:00+01:00"
author:
  name: "Sebastian Palencsar"
  handle: "AdminDocs"
category: "linux-administration"
tags: ["process-management", "top", "htop", "kill", "nice", "jobs", "background-processes", "linux-administration"]
toc: true
reading_time: 45
---

**Welcome to the third part of our technical wiki series on Linux administration!**

After we covered the [fundamentals of system administration](/en/linux-administration/linux-administration-advanced-user-management){.badge-link-text} and advanced user management in the previous articles, we now turn to another fundamental aspect:

<span class="nb-accent">Managing processes and system resources.</span>

As a Linux administrator, it is important to understand how processes work and how you effectively manage system resources. Think of your Linux system as a large company where many different employees (processes) perform different tasks and must share resources (CPU, RAM, etc.).

<blockquote class="infobox infobox--warn">
⚠️ **Note:** In this article, we use Ubuntu/Debian as the example distribution. The basic concepts are the same on all Linux systems, but package installation and some configuration paths may vary depending on the distribution. If you use a different distribution, please consult the relevant documentation for the specific installation commands and paths.
</blockquote>

## What is a Process?

A process is a running program on your system. To better understand this, here is an analogy:

* A program is like an employee handbook with work instructions
* A process is like an active employee executing those instructions
* Every process has a unique ID (PID), like an employee number
* Processes can communicate with each other, like employees in teams

### Types of Processes

In Linux, there are different types of processes, similar to how different employee types exist in a company. Let's learn about the most important process types:

**Process types:**

```markdown
┌─────────── System Services ─────────────────────────────────┐
│  • Run in the background                                    │
│  • Start at system boot                                     │
│  • Example: Print service                                   │
├─────────── User Processes ──────────────────────────────────┤
│  • Started by you                                           │
│  • Run under your name                                      │
│  • Example: Firefox                                         │
├─────────── Kernel Processes ────────────────────────────────┤
│  • Managed by the system                                    │
│  • Important for basic functions                            │
│  • Example: Device drivers                                  │
└─────────────────────────────────────────────────────────────┘
```

### System Processes (Daemons)

**Properties of system processes:**

```markdown
┌─────────── Startup Behavior ───────────────────────────────┐
│  • Start at system boot                                     │
│  • Run in the background                                    │
│  • No terminal connection                                   │
├─────────── Examples ────────────────────────────────────────┤
│  • Apache (web server)                                      │
│  • MySQL (database)                                         │
│  • SSH (Secure Shell)                                       │
└─────────────────────────────────────────────────────────────┘
```

### User Processes

**Properties of user processes:**

```markdown
┌─────────── Interactive ─────────────────────────────────────┐
│  • Started by users                                         │
│  • Terminal-based                                           │
│  • Foreground or background                                 │
├─────────── Examples ────────────────────────────────────────┤
│  • Firefox (browser)                                        │
│  • LibreOffice (office software)                            │
│  • Terminal commands                                        │
└─────────────────────────────────────────────────────────────┘
```

### Kernel Processes

**Properties of kernel processes:**

```markdown
┌─────────── System Level ───────────────────────────────────┐
│  • Managed by the kernel                                    │
│  • Highest priority                                         │
│  • Direct hardware access                                   │
├─────────── Examples ────────────────────────────────────────┤
│  • kthreadd (thread manager)                                │
│  • kswapd (memory management)                               │
│  • ksoftirqd (interrupt handler)                            │
└─────────────────────────────────────────────────────────────┘
```

### How a Process is Created and Works

When you start a program, many important steps happen in the background. Let's understand how a process works from creation to completion.

<span class="nb-accent">The Process Lifecycle</span>

**Process lifecycle:**

```markdown
┌─────────── Start Program ───────────────────────────────────┐
│  1. Program is loaded into RAM                               │
│  2. PID is assigned                                         │
│  3. Resources are reserved                                  │
├─────────── Process Running ─────────────────────────────────┤
│  4. CPU time is allocated                                   │
│  5. Memory is used                                          │
│  6. Input/Output is executed                                │
├─────────── Process Ends ────────────────────────────────────┤
│  7. Resources are released                                  │
│  8. Exit status is returned                                 │
│  9. Process is terminated                                   │
└─────────────────────────────────────────────────────────────┘
```

<span class="nb-accent">Practical Example: Starting Firefox</span>

```bash
# Start Firefox in foreground
firefox

# Start Firefox in background
firefox &
```

**What happens here?**

* Shell creates a new process
* Firefox program is loaded into RAM
* New PID is assigned
* Process starts execution

### Checking Process Status

A process goes through various states during its lifetime. Let's understand what states exist and how they relate:

**Process states:**

```markdown
┌─────────── Running ─────────────────────────────────────────┐
│  • Process is being executed                                │
│  • Actively using CPU time                                  │
├─────────── Sleeping ────────────────────────────────────────┤
│  • Waiting for event/resource                               │
│  • Releases CPU                                             │
├─────────── Stopped ─────────────────────────────────────────┤
│  • Suspended (e.g., by SIGSTOP)                             │
│  • Can be resumed                                           │
├─────────── Zombie ──────────────────────────────────────────┤
│  • Process terminated                                       │
│  • Waiting for parent process                               │
└─────────────────────────────────────────────────────────────┘
```

**Detailed process information**

```bash
# Display status of a process
ps -l

Output:
F S   UID   PID  PPID  C PRI   NI ADDR SZ WCHAN  TTY          TIME     CMD
0 S  1000  1234  1000  0  80   0  -    234 -     pts/0        00:00:01 bash

# Meaning of status codes:
# R = Running
# S = Sleeping
# T = Stopped
```

**Let's understand this information:**

* `UID`: User who started the process
* `PID`: Unique process ID
* `PPID`: Parent process ID
* `C`: CPU usage in percent
* `SZ`: Size in memory
* `RSS`: Actually used RAM
* `PSR`: Processor core used
* `STIME`: Start time
* `TTY`: Connected terminal
* `TIME`: CPU time consumed
* `CMD`: Executed command

## Process Communication and Signals

Processes in Linux frequently need to communicate with each other. This works similarly to how employees in a company use various communication channels. Let's understand the most important communication methods.

### 1. Understanding Process Signals

**Signal types:**

```markdown
┌─────────── Control Signals ─────────────────────────────────┐
│  SIGTERM (15) - Terminate                                    │
│  SIGKILL (9)  - Immediate termination                        │
│  SIGSTOP (19) - Suspend                                      │
├─────────── Error Signals ───────────────────────────────────┤
│  SIGSEGV (11) - Memory error                                 │
│  SIGFPE (8)   - Arithmetic error                             │
│  SIGABRT (6)  - Abort                                        │
├─────────── Info Signals ────────────────────────────────────┤
│  SIGUSR1 (10) - User-defined 1                               │
│  SIGUSR2 (12) - User-defined 2                               │
└─────────────────────────────────────────────────────────────┘
```

### 2. Signals in Practice

A process can receive various signals and react to them:

```bash
# Terminate process normally (recommended)
kill -TERM 1234    

# or: kill -15 1234
# The process can still clean up
# Terminate process immediately (emergencies only!)
kill -KILL 1234    

# or: kill -9 1234
# The process is terminated immediately
# Suspend and resume process
kill -STOP 1234    

# Suspend process
kill -CONT 1234    

# Resume process
```

<span class="nb-accent">Inter-Process Communication (IPC)</span>

In a Linux system, processes frequently need to communicate with each other, similar to employees in a company. Here you will learn about the different communication methods.

### 1. Understanding Communication Methods

**IPC methods:**

```markdown
┌─────────── Pipes ───────────────────────────────────────────┐
│  • Data transfer between                                    │
│  related processes                                          │
│  • Like a pipe for data                                     │
├─────────── Signals ─────────────────────────────────────────┤
│  • Short messages                                           │
│  • Like hand signals between                                 │
│  processes                                                  │
├─────────── Shared Memory ───────────────────────────────────┤
│  • Shared memory area                                       │
│  • Like a shared bulletin board                             │
└─────────────────────────────────────────────────────────────┘
```

### 2. Pipes in Practice

Pipes are like tubes through which data flows from one process to another:

**Simple pipe examples**

```bash
ls -l | grep ".txt"    

# Lists only .txt files
```

**What happens here?**

* `ls -l` creates a list of all files
* The pipe (`|`) forwards this list
* `grep` filters for ".txt"

**Multiple pipes**

```bash
cat logfile.txt | grep "Error" | sort | uniq

# 1. Reads the log file
# 2. Filters for "Error"
# 3. Sorts the lines
# 4. Removes duplicates
```

<span class="nb-accent">More Inter-Process Communication Methods</span>

In addition to pipes, there are other important methods for processes to communicate with each other. Let's look at these in detail.

### 1. Shared Memory

**Shared Memory concept:**

```markdown
┌─────────── Process A ───────────────────────────────────────┐
│  Reads and writes data                                      │
├─────────── Shared Memory ───────────────────────────────────┤
│  Shared memory area                                         │
│  Fast data exchange                                         │
├─────────── Process B ───────────────────────────────────────┐
│  Reads and writes data                                      │
└─────────────────────────────────────────────────────────────┘
```

Shared Memory is like a shared bulletin board that multiple processes can access:

**Create Shared Memory Segment**

```bash
ipcs -m    

# Shows all Shared Memory segments
```

**Example output**

* key        shmid      owner      perms      bytes      nattch     status
* 0x00000000 0          root       644        16384      2          dest

**Check Shared Memory Status**

```bash
ipcs -m -i [shmid] 

# Shows detailed information
```

### 2. Message Queues

**Message Queue concept:**

```markdown
┌─────────── Sender ──────────────────────────────────────────┐
│  Process sends messages                                     │
├─────────── Queue ───────────────────────────────────────────┤
│  1. Message                                                 │
│  2. Message                                                 │
│  3. Message                                                 │
├─────────── Receiver ────────────────────────────────────────┤
│  Process receives messages                                  │
└─────────────────────────────────────────────────────────────┘
```

<span class="nb-accent">Semaphores and Inter-Process Synchronization</span>

Semaphores are like traffic lights – they regulate access to shared resources. Let's understand how they work and how you use them in practice.

### 1. Understanding Semaphores

**Semaphore types:**

```markdown
┌─────────── Binary Semaphores ───────────────────────────────┐
│  • Like a traffic light: red or green                        │
│  • Only two states: 0 or 1                                  │
│  • For simple synchronization                                │
├─────────── Counting Semaphores ─────────────────────────────┤
│  • Like a parking garage with counter                        │
│  • Multiple resources available                              │
│  • Counts available units                                    │
└─────────────────────────────────────────────────────────────┘
```

### 2. Practical Application

**Create and initialize semaphore**

```bash
ipcs -s    

# Shows all semaphores in the system
```

**Example output**

* key        semid      owner      perms      nsems
* 0x00000000 0          root       600        1

**Check semaphore status**

```bash
ipcs -s -i [semid] 

# Shows detailed information about a semaphore
```

<span class="nb-accent">Example: Shared Resource Usage</span>

```bash
# Python example for semaphore usage
from multiprocessing import Semaphore

# Semaphore for 3 simultaneous accesses
sem = Semaphore(3)

def protected_area():
	sem.acquire()    

# "Set traffic light to red"
	try:

# Here comes the protected code
		print("Access to resource")
	finally:
		sem.release()    

# "Set traffic light back to green"
```

## Practical Use Cases

<span class="nb-accent">Process and Resource Management</span>

Let's learn with real-world examples how you manage processes and resources in typical situations. These scenarios are common in the daily life of a Linux administrator.

**Typical scenario:**

```markdown
┌─────────── Apache/Nginx ────────────────────────────────────┐
│  • High visitor numbers                                      │
│  • Many parallel connections                                 │
│  • Limited server resources                                  │
├─────────── Optimization ────────────────────────────────────┤
│  • Adjust process priority                                   │
│  • Assign CPU cores                                          │
│  • Set resource limits                                       │
└─────────────────────────────────────────────────────────────┘
```

### Optimizing Web Server Processes

<span class="nb-accent">1. Set higher priority</span>

```bash
sudo renice -n -5 $(pgrep apache2)
```

<span class="nb-accent">2. Assign dedicated CPU cores</span>

```bash
sudo taskset -pc 4-7 $(pgrep mysqld)
```

<span class="nb-accent">3. Set I/O priority</span>

```bash
sudo ionice -c 1 -n 0 -p $(pgrep mysqld)
```

### Web Server Optimization

Imagine you're running an Apache web server under high load:

**Optimization goals:**

```markdown
┌─────────── Performance ─────────────────────────────────────┐
│  • Fast response times                                       │
│  • Efficient resource usage                                  │
│  • Stable availability                                       │
├─────────── Measures ────────────────────────────────────────┤
│  • Process prioritization                                    │
│  • CPU affinity                                              │
│  • Resource limits                                           │
└─────────────────────────────────────────────────────────────┘
```

<span class="nb-accent">1. Identify Apache processes</span>

```bash
ps aux | grep apache2

# Shows all Apache processes and their PIDs
```

**Example output**

* www-data  1234  2.5  1.2  ... /usr/sbin/apache2
* www-data  1235  1.8  1.1  ... /usr/sbin/apache2

<span class="nb-accent">2. Optimize process priority</span>

```bash
sudo renice -n -5 $(pgrep apache2)

# Gives Apache higher priority
```

<span class="nb-accent">3. Set CPU affinity</span>

```bash
sudo taskset -pc 0-2 $(pgrep apache2)

# Binds Apache to the first three CPU cores
```

<span class="nb-accent">Optimizing Database Server</span>

For a MySQL/MariaDB database, performance is especially important:

<span class="nb-accent">1. Identify MySQL process</span>

```bash
ps aux | grep mysql

# Shows the MySQL process and its PID
```

<span class="nb-accent">2. Monitor resources</span>

```bash
top -p $(pgrep mysqld)

# Shows real-time resource usage
```

<span class="nb-accent">3. Set I/O priority</span>

```bash
ionice -c 1 -n 0 -p $(pgrep mysqld)

# Gives MySQL highest I/O priority
```

## Resource Management

As a Linux administrator, it is important to understand how you effectively manage system resources. Think of your system as a large company where different departments (processes) must share the available resources (CPU, RAM, etc.).

### Understanding System Resources

**Resource types:**

```markdown
┌─────────── Processor (CPU) ─────────────────────────────────┐
│  • Compute time                                             │
│  • Processor cores                                          │
│  • Load                                                     │
├─────────── RAM ─────────────────────────────────────────────┤
│  • Physical RAM                                             │
│  • Swap memory                                              │
│  • Cache & Buffer                                           │
├─────────── Input/Output (I/O) ──────────────────────────────┤
│  • Disk access                                              │
│  • Network bandwidth                                        │
│  • System bus                                               │
└─────────────────────────────────────────────────────────────┘
```

### Monitoring CPU Resources

The CPU is the heart of your system. Effective monitoring helps you recognize and resolve performance problems early.

<span class="nb-accent">Understanding CPU Information</span>

**CPU metrics:**

```markdown
┌─────────── Load ────────────────────────────────────────────┐
│  %user   - User applications                                │
│  %system - Kernel processes                                 │
│  %iowait - Waiting for I/O                                  │
│  %idle   - CPU idle                                          │
├─────────── Load Average ────────────────────────────────────┤
│  1 min   - Last minute                                      │
│  5 min   - Last 5 minutes                                   │
│  15 min  - Last 15 minutes                                  │
└─────────────────────────────────────────────────────────────┘
```

### CPU Monitoring in Practice

```bash
# Basic CPU information
cat /proc/cpuinfo

# Shows:
# - Processor type
# - Number of cores
# - Clock frequency
# - Cache sizes
# CPU usage in real time
top

# Important lines:
# - Load Average: 0.15, 0.25, 0.30
# - %Cpu(s): 5.1 us, 2.3 sy, 0.0 ni, 91.3 id
# Detailed CPU statistics
mpstat -P ALL 1

# Shows load per CPU core
# Monitor CPU temperature
sensors

# Shows current CPU temperature
```

### Understanding CPU Load

The Load Average is like a thermometer for your system:

* < 1: System is not overloaded
* = 1: System is optimally loaded

### System may be overloaded

**Display Load Average**

* uptime

**Example output:**

```bash
14:30:05 up 5 days, load average: 0.15, 0.25, 0.30
									│     │     └── 15 min
									│     └──────  5 min
									└────────────  1 min
```

<span class="nb-accent">Memory Management and Resource Limits</span>

Effective management of RAM is crucial for the stability and performance of your system. Let's learn how you monitor and control memory.

### Understanding Memory Types

**Memory types:**

```markdown
┌─────────── Physical RAM ───────────────────────────────────┐
│  • Working memory                                           │
│  • Fast access                                              │
│  • Limited size                                             │
├─────────── Swap Memory ────────────────────────────────────┤
│  • Paging storage                                           │
│  • On hard disk                                              │
│  • Slower than RAM                                          │
├─────────── Cache & Buffer ──────────────────────────────────┤
│  • Temporary fast storage                                   │
│  • Automatically managed                                    │
│  • Improves performance                                     │
└─────────────────────────────────────────────────────────────┘
```

### Monitoring Memory Usage

<span class="nb-accent">Display total memory status</span>

* free -h

**Example output**:

* total       used       free     shared    buffers     cache
* Mem:          15Gi       5.2Gi      6.8Gi    1.2Gi      428Mi      2.3Gi
* Swap:         4.0Gi      128Mi      3.9Gi

**What does this mean?**

* `total`: Total available RAM
* `used`: Currently used memory
* `free`: Completely free memory
* `shared`: Memory shared by multiple processes
* `buffers/cache`: Used by the system as temporary storage

<span class="nb-accent">Memory Limits and Resource Management</span>

Controlling memory usage is crucial for the stability of your system. Here you will learn how you set and monitor memory limits.

### Understanding Memory Limits

**Memory types and limits:**

```markdown
┌─────────── Physical RAM ───────────────────────────────────┐
│  • RAM limitation                                           │
│  • Process-specific limits                                  │
│  • System-wide limits                                       │
├─────────── Virtual Memory ──────────────────────────────────┤
│  • Swap usage                                               │
│  • Virtual memory boundaries                                │
│  • Overcommit settings                                      │
├─────────── OOM Killer ──────────────────────────────────────┤
│  • Out-of-Memory Management                                 │
│  • Process prioritization                                   │
│  • Protection of important services                         │
└─────────────────────────────────────────────────────────────┘
```

### Setting Memory Limits

<span class="nb-accent">Display system-wide limits</span>

```bash
ulimit -a

# Shows all current limits:
# - max memory size
# - max stack size
# - max open files
# - etc.
```

<span class="nb-accent">Set process-specific limits</span>

🔧 **Practical Example: MySQL Server Process**

* sudo nano /etc/security/limits.conf

**Add these lines:**

```bash
mysql soft memlock 524288    # Soft limit: 512MB

mysql hard memlock 524288    # Hard limit: 512MB
```

<span class="nb-accent">Understanding and Configuring OOM Killer</span>

The OOM Killer (Out Of Memory Killer) is an important protective mechanism in Linux. It prevents your system from completely freezing due to memory shortage. Let's understand how it works and how you configure it.

### OOM Killer Basics

**OOM Killer function:**

```markdown
┌─────────── Monitoring ──────────────────────────────────────┐
│  • Observes memory usage                                    │
│  • Detects critical situations                              │
│  • Evaluates processes                                      │
├─────────── Evaluation ──────────────────────────────────────┤
│  • Process size                                              │
│  • Runtime                                                   │
│  • Importance (Score)                                        │
├─────────── Action ──────────────────────────────────────────┤
│  • Selects process for termination                          │
│  • Terminates process                                        │
│  • Logs action                                               │
└─────────────────────────────────────────────────────────────┘
```

### Understanding and Adjusting OOM Score

<span class="nb-accent">Display OOM score of a process</span>

```bash
cat /proc/1234/oom_score

# Higher value = will be terminated first
```

<span class="nb-accent">Display OOM score adjustment</span>

```bash
cat /proc/1234/oom_score_adj

# Value range: -1000 to 1000
```

<span class="nb-accent">Protect important service</span>

```bash
echo -1000 > /proc/$(pgrep mysql)/oom_score_adj

# -1000 = process will never be terminated
```

<span class="nb-accent">Mark unimportant process</span>

```bash
echo 500 > /proc/$(pgrep firefox)/oom_score_adj

# Positive values = will be terminated first
```

## I/O Limits and Resource Control

Controlling input and output (I/O) is crucial for system performance. Here you will learn how you set and monitor I/O limits.

### Understanding I/O Scheduling and Limits

**I/O control:**

```markdown
┌─────────── I/O Scheduler ───────────────────────────────────┐
│  • Manages disk access                                       │
│  • Prioritizes requests                                      │
│  • Optimizes performance                                     │
├─────────── I/O Limits ──────────────────────────────────────┤
│  • Limits throughput                                         │
│  • Controls bandwidth                                        │
│  • Prevents overload                                        │
├─────────── Monitoring ──────────────────────────────────────┤
│  • Monitors activity                                         │
│  • Detects bottlenecks                                       │
│  • Logs access                                               │
└─────────────────────────────────────────────────────────────┘
```

### I/O Limits in Practice

<span class="nb-accent">Set I/O priority for processes</span>

```bash
ionice -c 2 -n 7 -p $(pgrep firefox)
```

**Classes:**

* 1 = Realtime (root only)
* 2 = Best-Effort (default)
* 3 = Idle (lowest)

<span class="nb-accent">Display I/O statistics</span>

```bash
iostat -x 1

# Shows:
# - Disk load in real time
# - Throughput and wait times
# - Load per device
```

### Practical Examples for I/O Limits

In practice, it is often important to control I/O usage, especially when multiple services or users share the system.

<span class="nb-accent">Understanding I/O Scheduling and Priorities</span>

**I/O scheduling classes:**

```markdown
┌─────────── Realtime ────────────────────────────────────────┐
│  • Highest priority                                         │
│  • For critical system processes                            │
│  • Only for root users                                      │
├─────────── Best-Effort ─────────────────────────────────────┤
│  • Default priority                                         │
│  • For normal applications                                  │
│  • Fair resource distribution                               │
├─────────── Idle ────────────────────────────────────────────┤
│  • Lowest priority                                          │
│  • For unimportant processes                                 │
│  • Only when system is free                                 │
└─────────────────────────────────────────────────────────────┘
```

<span class="nb-accent">Applying I/O Limits in Practice</span>

**Set I/O priority for backup process:**

```bash
ionice -c 3 ./backup.sh

# Class 3 (Idle) means:
# - Only runs when no other I/O requests
# - Ideal for backups and archiving
# - Does not disturb other processes
```

**Monitor I/O statistics:**

```bash
iostat -x 1

# Shows:
# - Disk load in real time
# - Throughput and wait times
# - Load per device
```

## System Monitoring

As a Linux administrator, it is important to continuously monitor your system. Think of it like a control center where you keep track of all important measurements.

### Basic System Monitoring

**Monitoring areas:**

```markdown
┌─────────── System Resources ────────────────────────────────┐
│  • CPU load                                                 │
│  • Memory usage                                             │
│  • Disk load                                                │
├─────────── Processes ───────────────────────────────────────┤
│  • Running processes                                         │
│  • Process status                                           │
│  • Resource consumption                                     │
├─────────── System Logs ─────────────────────────────────────┤
│  • System events                                            │
│  • Error messages                                           │
│  • Security warnings                                        │
└─────────────────────────────────────────────────────────────┘
```

### Important Monitoring Tools

The most important tool for system monitoring is `top`. It shows you in real time what is happening on your system:

```bash
top

# Start system monitoring with top
```

**Example output:**

```bash
top - 19:09:30 up  9:46,  2 users,  load average: 1.39, 1.30, 1.35
Tasks: 282 total,   1 running, 281 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.3 us,  4.0 sy,  0.0 ni, 87.2 id,  0.0 wa,  0.4 hi,  0.1 si,  0.0 st

MiB Mem:  23930.2 total,   2294.4 free,  11825.2 used,  10645.2 buff/cache
MiB Swap:  50253.9 total,  50253.9 free,      0.0 used.  12105.0 avail Mem

PID   USER   PR  NI   VIRT    RES    SHR    S   %CPU  %MEM     TIME+    COMMAND
16377 demo+  20   0   28.2g   1.1g   506144 S   5.3   4.6      71:37.38 firefox

# What do the most important lines mean?
# First line: System time, uptime, users, load average
# Second line: Process summary
# Third line: CPU usage
# Fourth/Fifth line: Memory usage
```

<span class="nb-accent">Advanced Monitoring Features</span>

As a Linux administrator, you need detailed insights into your system. Advanced monitoring features help you recognize and resolve problems early.

### System Monitoring with SAR (System Activity Reporter)

**SAR functions:**

```markdown
┌─────────── CPU Monitoring ──────────────────────────────────┐
│  • Processor load                                           │
│  • System/User/IO-Wait                                      │
│  • Process statistics                                       │
├─────────── Memory Monitoring ───────────────────────────────┤
│  • RAM usage                                                │
│  • Swap activity                                            │
│  • Page Faults                                              │
├─────────── I/O Monitoring ──────────────────────────────────┤
│  • Disk activity                                            │
│  • Network traffic                                          │
│  • System load                                              │
└─────────────────────────────────────────────────────────────┘
```

### SAR in Practice

<span class="nb-accent">Install and activate SAR</span>

* sudo apt install sysstat
* sudo systemctl enable sysstat
* sudo systemctl start sysstat

<span class="nb-accent">Monitor CPU usage</span>

```bash
sar -u 1 5    

# Every 1 second, 5 measurements
```

**Example output:**

```bash
19:16:27        CPU      %user     %nice     %system   %iowait   %steal   %idle
19:16:28        all      4.00      0.00      2.00      0.00      0.00     94.00
19:16:29        all      4.02      0.00      1.76      0.00      0.00     94.22
19:16:30        all      4.04      0.00      1.77      0.00      0.00     94.19
19:16:31        all      5.00      0.00      3.00      0.00      0.00     92.00
19:16:32        all      3.50      0.00      1.50      0.00      0.00     95.00
Average:        all      4.11      0.00      2.01      0.00      0.00     93.88

# - %user (user applications)
# - %system (kernel processes)
# - %iowait (wait time for I/O)
# - %idle (CPU idle)
```

<span class="nb-accent">Monitor memory usage</span>

```bash
sar -r    

# RAM statistics
sar -S    

# Swap statistics
```

## Troubleshooting

As a Linux administrator, you will frequently encounter performance problems. Here you will learn how you systematically recognize, analyze, and resolve problems.

### Systematic Problem Analysis

**Troubleshooting steps:**

```markdown
┌─────────── Recognize Problem ───────────────────────────────┐
│  • Identify symptoms                                        │
│  • Find affected resources                                  │
│  • Document impact                                          │
├─────────── Analysis ────────────────────────────────────────┤
│  • Check system load                                        │
│  • Measure resource consumption                             │
│  • Evaluate logs                                            │
├─────────── Solution ────────────────────────────────────────┤
│  • Fix cause                                                │
│  • Document measures                                        │
│  • Plan prevention                                          │
└─────────────────────────────────────────────────────────────┘
```

<span class="nb-accent">Common Problems and Solutions</span>

### Recognizing and Resolving CPU Problems:

<span class="nb-accent">1. Check CPU usage</span>

```bash
top

# High values at %us or %sy indicate problems
```

<span class="nb-accent">2. Identify problematic processes</span>

```bash
ps aux --sort=-%cpu | head -5

# Shows the 5 most CPU-intensive processes
```

<span class="nb-accent">3. Adjust process priority</span>

```bash
renice +10 $(pgrep firefox)

# Gives Firefox lower priority
```

### Diagnosing Memory Problems:

<span class="nb-accent">1. Check memory usage</span>

```bash
free -h

# Important indicators:
# - Low free RAM
# - High swap usage
```

<span class="nb-accent">2. Find memory hogs</span>

```bash
ps aux --sort=-%mem | head -5

# Shows the 5 most memory-intensive processes
```

<span class="nb-accent">3. Clear cache (if necessary)</span>

```bash
sync && echo 3 > /proc/sys/vm/drop_caches
```

### Recognizing and Resolving I/O Problems

<span class="nb-accent">Understanding I/O Problems</span>

**I/O problem indicators:**

```markdown
┌─────────── Performance ─────────────────────────────────────┐
│  • System responds sluggishly                               │
│  • High disk activity                                       │
│  • Programs "freeze"                                        │
├─────────── Diagnosis ───────────────────────────────────────┤
│  • Observe I/O wait                                         │
│  • Measure throughput                                       │
│  • Check queues                                             │
└─────────────────────────────────────────────────────────────┘
```

<span class="nb-accent">1. Diagnose I/O problems</span>

```bash
iostat -x 1

# Important values:
# %util   - Device load
# await   - Average wait time
# svctm   - Average service time
```

<span class="nb-accent">2. Find top I/O culprits</span>

```bash
iotop

# Shows:
# - Processes with highest I/O load
# - Read and write rates
# - I/O priorities
```

<span class="nb-accent">3. Resolve I/O problems</span>

```bash
# Optimize I/O scheduler
echo deadline > /sys/block/sda/queue/scheduler

# Deadline is often better for servers
# Adjust I/O priority
ionice -c 2 -n 7 -p $(pgrep backup)

# Backup process gets low priority
```

## Exercises

Let's deepen what you've learned through practical exercises. Here are real-world scenarios you may encounter in your daily work as a Linux administrator.

### Exercise 1: Web Server Optimization

**Scenario:** You manage an Apache web server running under high load. Users complain about slow loading times, and monitoring shows high CPU and memory usage.

**Requirements:**

* Identify Apache processes
* Analyze resource usage
* Optimize performance
* Adjust process priority
* Set resource limits

**Possible solution:**

```bash
# 1. Identify Apache processes
ps aux | grep apache2

# Shows all Apache processes and their PIDs
# 2. Analyze resource usage
top -p $(pgrep -d',' apache2)

# Observe CPU and RAM usage
# 3. Optimize process priority
sudo renice -5 $(pgrep apache2)

# Gives Apache higher priority
# 4. Set resource limits
sudo nano /etc/security/limits.conf

# Add:
www-data soft nproc 150    # Maximum number of processes

www-data soft nofile 8192  # Maximum open files
```

### Exercise 2: Performance Optimization of a Database Server

**Scenario:** You manage a MySQL database server that manages the customer database of an online shop. During peak times, there are performance drops and slow response times.

**Requirements:**

* Identify MySQL processes
* Analyze resource usage
* Optimize CPU and I/O priority
* Adjust memory limits
* Monitor performance

**Possible solution:**

```bash
# 1. Identify MySQL processes
ps aux | grep mysql

# Shows all MySQL processes and their PIDs
# 2. Analyze resources
top -p $(pgrep mysqld)

# Observe:
# - CPU usage
# - Memory consumption
# - Load Average
# 3. Optimize CPU priority
sudo renice -10 $(pgrep mysqld)

# Gives MySQL higher priority
# 4. Set I/O priority
sudo ionice -c 1 -n 0 -p $(pgrep mysqld)

# Highest I/O priority for MySQL
# 5. Activate OOM Killer protection
echo -1000 > /proc/$(pgrep mysqld)/oom_score_adj

# Protects MySQL from OOM Killer
```

### Exercise 3: Process Prioritization and Resource Control

**Scenario:** You manage a Linux system where multiple developers work simultaneously. A compute-intensive compilation process (gcc) from one developer loads the system so heavily that other users complain about slow response times.

**Requirements:**

* Identify compilation processes
* Adjust process priority
* Distribute CPU resources fairly
* Ensure system performance for other users
* Configure OOM Killer correctly

**Possible solution:**

```bash
# 1. Identify compilation processes
ps aux | grep gcc

# Shows all gcc processes and their PIDs
# 2. Adjust process priority
renice +10 $(pgrep gcc)

# Gives compilation lower priority
# 3. Set CPU limits
# Restrict process to specific CPU cores
taskset -cp 2,3 $(pgrep gcc)

# Restricts gcc to cores 2 and 3
# 4. Adjust OOM score
echo 500 > /proc/$(pgrep gcc)/oom_score_adj

# Process will be terminated first on memory shortage
```

## Command Reference (Cheatsheet)

For quick access during analysis, monitoring, and control of processes, the following reference table summarizes the most important Linux commands:

| Command / Syntax | Category | Function & Description |
|---|---|---|
| `ps aux` | Snapshot | Shows all running processes in the system in BSD format. |
| `ps aux --sort=-%cpu` | Snapshot | Sorts processes descending by CPU usage. |
| `ps -ef --forest` | Process Tree | Visualizes parent-child relationships of processes in ASCII tree. |
| `top` / `htop` | Interactive | Real-time monitoring of CPU, RAM, load average, and processes. |
| `pgrep -l <name>` | Search | Searches for process IDs (PIDs) matching the specified name. |
| `kill -15 <PID>` | Signals | Sends default termination signal (SIGTERM) for orderly shutdown. |
| `kill -9 <PID>` | Signals | Forces immediate, non-interceptable process termination (SIGKILL). |
| `kill -1 <PID>` | Signals | Sends SIGHUP to reload daemon configurations without restart. |
| `pkill -15 <name>` | Signals | Terminates processes targeted by name pattern. |
| `nice -n 10 <cmd>` | Priority | Starts program with reduced CPU priority (higher nice value). |
| `renice -n 5 -p <PID>` | Priority | Adjusts the scheduling priority of a running process. |
| `ionice -c 3 -p <PID>` | I/O Priority | Sets I/O scheduling class to 'Idle' (only when disk resources are free). |
| `jobs -l` | Job Control | Lists all background jobs of the current shell session with PID. |
| `nohup <cmd> &` | Job Control | Starts process immune to SIGHUP when closing SSH session. |
| `systemd-cgtop` | cgroups | Real-time monitoring of resource usage per systemd cgroup. |
| `systemd-run --scope -p CPUQuota=30% <cmd>` | cgroups | Executes command with strict CPU limit via systemd cgroups v2. |

## Further Resources

The following guides, kernel documentations, and internal course modules deepen process and resource management:

| Resource | Description |
|---|---|
| [systemd Resource Control Guide](https://systemd.io/RESOURCE_MANAGEMENT/){.badge-link-text} | Official guide for systemd cgroups v2 resource control. |
| [Linux Kernel Procfs Documentation](https://www.kernel.org/doc/html/latest/filesystems/proc.html){.badge-link-text} | Documentation of the virtual kernel interfaces under `/proc/[PID]`. |
| [Linux Performance Tools (Brendan Gregg)](https://www.brendangregg.com/linuxperf.html){.badge-link-text} | Comprehensive methodology and tool overview for Linux performance analysis. |
| [Linux Administration #2: User Management](/en/linux-administration/linux-administration-advanced-user-management){.badge-link-text} | The previous module: PAM stacks, quotas, POSIX ACLs & Sudoers. |
| [Linux Administration #4: Networking](/en/linux-administration/linux-administration-network-configuration-and-management){.badge-link-text} | The next module: IP routing, netplan, nmcli, DNS & UFW. |
| [Command Line Processor in Linux](/en/linux-beginners/command-line-processor-in-linux){.badge-link-text} | Fundamental knowledge about shells, I/O streams, and pipes. |

## Conclusion

Mastering process and resource management is a core competency of every Linux system administrator. Through understanding the process lifecycle, the controlled use of POSIX signals, targeted CPU scheduling with nice values, and modern resource limits via systemd and cgroups v2, you ensure that your Linux servers perform stably, responsively, and predictably even under extreme load peaks.

<blockquote class="infobox infobox--info">
💡 **Practical Tip:** For resource-intensive periodic background jobs (such as nightly database dumps or archiving), always use the combination of `nice -n 19` and `ionice -c 3`. This way, the job runs with the lowest CPU and disk priority and does not affect regular productive operations at any time.
</blockquote>

In the next module of our course, we connect our servers with the outside world:
👉 **Next up:** [Linux Administration #4: Network Configuration and Management](/en/linux-administration/linux-administration-network-configuration-and-management){.badge-link-text}

👉 **Course Overview:** [All Linux Administration Articles & Modules](/en/category/linux-administration){.badge-link-text}
