LPIC-1: Text processing with shell commands

In the first three parts of our LPIC-1 series, we covered the fundamentals of the Linux command line, filesystem navigation, and editing file contents. Now it's time to take a decisive step further: exploring the powerful world of shell-based text processing.

Text processing via shell commands is one of the most characteristic and powerful features of Linux and Unix systems. While other operating systems often rely on graphical tools or specialized programs, Linux offers a collection of highly efficient command-line tools that can be seamlessly combined with each other.

❗ Important note: As emphasized in previous articles, this series does not replace an official exam preparation course for the LPIC-1 certification. It serves as a practice-oriented, didactically prepared supplement for your self-study and is designed to help you better understand and apply the sometimes complex content.

Why shell-based text processing is so central

┌─────────────────────────────────────────────────────────────┐
│            UNIX TEXT PROCESSING PIPELINE                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Raw data (files / streams / standard input)               │
│        │                                                    │
│        ▼                                                    │
│   ┌─────────────────────────────────────────────────────┐   │
│   │ 1. Filtering (grep, egrep, fgrep)                   │   │
│   │    Extracts lines based on patterns / regex          │   │
│   └────┬────────────────────────────────────────────────┘   │
│        │ [ stdout | stdin ] (pipe)                          │
│        ▼                                                    │
│   ┌─────────────────────────────────────────────────────┐   │
│   │ 2. Transforming & Extracting (cut, tr, sed, awk)    │   │
│   │    Cut columns, swap characters, substitute          │   │
│   └────┬────────────────────────────────────────────────┘   │
│        │ [ stdout | stdin ] (pipe)                          │
│        ▼                                                    │
│   ┌─────────────────────────────────────────────────────┐   │
│   │ 3. Sorting & Aggregating (sort | uniq -c | wc)      │   │
│   │    Sorts, deduplicates, counts frequencies           │   │
│   └────┬────────────────────────────────────────────────┘   │
│        │                                                    │
│        ▼                                                    │
│   Structured output / standard output / analysis report     │
│                                                             │
└─────────────────────────────────────────────────────────────┘

In Linux administration, text is ubiquitous: configuration files, log files, system outputs, reports, and scripts – everything is based on text data. The ability to efficiently search, filter, transform, and analyze this text data distinguishes a competent Linux administrator from a beginner.

Shell-based text processing tools follow the classic Unix philosophy:

• Each tool does one thing, but it does it very well
• Tools can be combined with each other via pipes
• Simple tools can be assembled into complex solutions
• Text data is the universal interface between tools

This approach makes it possible to perform complex data analyses and transformations with just a few well-mastered commands – tasks that would require specialized software in other environments.

Importance for the LPIC-1 certification

Shell-based text processing holds an important place in the LPIC-1 exam, particularly in the following examination objectives:

103.2:

• Process text streams using filters

103.3:

• Perform basic file manipulation

103.7:

• Search for and extract data from files using regular expressions

Together, these topics account for approximately 20–25% of the points in the LPIC-1 Exam 101, underscoring the importance of these skills.

What to expect in this article

In the upcoming sections, we will cover the following core topics:

• Basic text processing with the "big three": grep, sed, and awk
• Text sorting functions with sort and uniq to organize data and manage duplicates
• Counting and formatting text files with tools like wc, fmt, nl, and cut
• Regular expressions as a powerful tool for pattern matching and substitution

Each section will not only explain the technical aspects but also highlight practical application scenarios from real-world system administration.

Practical relevance for everyday administration

The commands covered in this article are part of every Linux administrator's daily toolkit.

Typical application scenarios include:

• Analyzing log files for troubleshooting and system monitoring
• Extracting specific information from configuration files
• Automated reports on system states
• Data cleansing and transformation for further processing
• Monitoring system resources and user activities

💡 Note: As usual, you'll find special markers throughout the article:

• 💡 Tips and hints for more efficient workflows
• ⚠️ Warnings and pitfalls to save you trouble
• 🔧 Practical examples to follow along directly
• ❗ Common sources of error and their solutions

How to get the most out of this article:

For optimal learning, I strongly recommend trying out the commands and techniques in your own Linux environment. Create test text files with various contents, experiment with the presented commands, and observe the results.

Particularly with text processing, hands-on practice is essential – combining different commands via pipes requires practice and experimentation. The more you work with these tools, the more intuitive their application becomes.

Let's now dive into the fascinating world of shell-based text processing – a field that will revolutionize your efficiency as a Linux administrator and at the same time secure you important points in the LPIC-1 exam.

Basic text processing

The three pillars of Linux text processing – grep, sed, and awk – form the core of shell-based data manipulation. These tools complement each other perfectly and make it possible to handle complex text processing tasks with elegant, efficient solutions. Each tool has its specific strengths and areas of application that we will explore in detail in this section.

grep – The powerful text searcher

What is grep and why is it indispensable?

grep (Global Regular Expression Print) is the fundamental search tool in Linux systems. It searches text files or data streams for lines matching a specific pattern and outputs them. The name already reveals its origin: it comes from the ed editor command g/re/p (global/regular expression/print).

The strength of grep lies in its simplicity and speed. It is optimized for one task: quickly finding text patterns in large data sets.

Basic syntax and usage

The basic syntax of grep is intuitive:

grep [options] "search_pattern" file(s)

🔧 Simple examples:

# Search for the word "error" in a log file
grep "error" /var/log/syslog

# Search in multiple files simultaneously
grep "Failed" /var/log/auth.log /var/log/secure

# Search via pipes (very commonly used)
ps aux | grep apache

💡 The last example shows one of the most common uses of grep: filtering the output of other commands via pipes.

Important options and their practical application

grep offers a variety of options that significantly extend its functionality:

🔧 Practical application examples for system administration:

a) Error analysis in log files:

# Find all critical errors from the last hour
grep -i "critical\|fatal\|emergency" /var/log/syslog

# Show context around SSH connection errors
grep -C 3 "Failed password" /var/log/auth.log

b) Analyzing configuration files:

# Find all active configuration directives
grep -v "^#\|^$" /etc/apache2/apache2.conf

# Find specific settings in SSH config
grep -i "PermitRootLogin\|PasswordAuthentication" /etc/ssh/sshd_config

c) System monitoring:

# Count failed login attempts
grep -c "Failed password" /var/log/auth.log

# Find all processes of a specific user
ps aux | grep -v grep | grep username

💡 Tip for practice: Combine grep with other commands to create powerful analysis pipelines. For example: grep "error" /var/log/syslog | wc -l counts all error lines in a log file.

sed – The stream editor

What is sed and how does it work?

sed (Stream Editor) is a non-interactive stream editor that processes text line by line. It is one of the most powerful tools for text manipulation on the command line and is particularly valuable for automated text transformations in scripts and pipelines.

The name "Stream Editor" already indicates its function: it processes data streams (stdin or files) without requiring user interaction.

Basic syntax and operations

The basic syntax of sed:

sed [options] 'command' file

🔧 Simple examples:

# Replace "old" with "new" in a file (display only)
sed 's/old/new/g' file.txt

# Replace "old" with "new" in a file (in-place editing)
sed -i 's/old/new/g' file.txt

# Delete all lines containing "pattern"
sed '/pattern/d' file.txt

# Display only lines 5-10
sed -n '5,10p' file.txt

Important sed commands and options

🔧 Practical examples for system administration:

a) Configuration file modification:

# Comment out all lines not starting with #
sed -i 's/^[^#]/# &/' config.conf

# Replace a specific line in a config file
sed -i '/^Port /c\Port 2222' /etc/ssh/sshd_config

# Remove empty lines from a file
sed '/^$/d' config.txt

b) Log file analysis:

# Extract only timestamp and error message
sed -n '/ERROR/p' /var/log/app.log | sed 's/.*\[ERROR\] //'

# Remove ANSI escape codes from log files
sed 's/\x1b\[[0-9;]*m//g' colored_log.txt

c) Text transformation:

# Convert Windows line endings to Unix
sed -i 's/\r$//' file.txt

# Remove trailing whitespace from each line
sed 's/[[:space:]]*$//' file.txt

⚠️ Be careful with the -i option: It modifies the original file irreversibly!

Always create backups for important files:

# Safe approach with backup
sed -i.bak 's/old/new/g' important_file.conf

Advanced sed techniques

Line addressing:

# Replace only in lines 10-20
sed '10,20s/old/new/g' file.txt

# From the first occurrence of a pattern to end of file
sed '/START/,/old/new/g' file.txt

# Edit every other line
sed '1~2s/old/new/g' file.txt

Combining multiple commands:

# Multiple substitutions in one pass
sed -e 's/http/https/g' -e 's/80/443/g' -e '/^#/d' config.txt

# Separate with semicolons
sed 's/old/new/g; /^#/d; s/[[:space:]]*//' file.txt

💡 Tip for complex operations: For very complex sed operations, it may be more practical to create a sed script.

# Create sed-script.sed
cat > sed-script.sed << 'EOF'
s/http/https/g
s/80/443/g
/^#/d
/^$/d
EOF

# Use the script
sed -f sed-script.sed configuration.conf

awk – Introduction to text processing

Core concept of awk as a pattern-action language

awk is more than just a text processing tool – it is a complete programming language specifically designed for processing structured text data. The name comes from the surnames of its developers: Aho, Weinberger, and Kernighan.

The core concept of awk is based on the pattern-action paradigm:

┌─────────────────────────────────────────────────────────────┐
│               AWK PATTERN-ACTION MODEL                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│    Input line ──► [ Pattern / Condition? ]                  │
│                            │                                │
│                   YES ─────┴────── NO                       │
│                   │                 │                        │
│                   ▼                 ▼                        │
│           { Action to          (Line is                      │
│           {  execute }          ignored)                     │
│                   │                                          │
│                   ▼                                          │
│            Output (print)                                    │
│                                                             │
│    Structure:   awk '{ pattern { action }' file             │
│                     │        │                               │
│                     │        └─ Action (e.g. print $1)       │
│                     └─ Filter condition (e.g. $3 >= 1000)    │
│                                                             │
└─────────────────────────────────────────────────────────────┘

For each input line, awk checks whether it matches the pattern. If so, the corresponding action is executed.

The basic structure of an awk program:

awk 'BEGIN { initialization }
     pattern1 { action1 }
     pattern2 { action2 }
     END { finalization }' input_file

Simple awk commands for column processing

awk automatically splits each input line into fields that can be accessed via $1, $2, $3, etc. $0 represents the entire line.

🔧 Basic examples:

# Print first column of all lines
awk '{print $1}' file.txt

# Print first and third column
awk '{print $1, $3}' file.txt

# Show lines with more than 3 fields
awk 'NF > 3 {print $0}' file.txt

# Show line number and content
awk '{print NR ": " $0}' file.txt

Important built-in variables:

Practical use cases in system administration

1. Analyzing the /etc/passwd file:

# Display all usernames
awk -F: '{print $1}' /etc/passwd

# Users with UID greater than 1000 (regular users)
awk -F: '$3 >= 1000 {print $1 " (UID: " $3 ")"}' /etc/passwd

# Users with bash as default shell
awk -F: '$7 == "/bin/bash" {print $1}' /etc/passwd

2. Log analysis:

# Sum up log file sizes
ls -l /var/log/*.log | awk '{sum += $5} END {print "Total: " sum " bytes"}'

# Apache access log: IP addresses and status codes
awk '{print $1, $9}' /var/log/apache2/access.log

# Number of requests per IP address
awk '{count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log

3. Monitoring system resources:

# Memory usage of processes
ps aux | awk 'NR > 1 {mem += $4} END {print "Total RAM usage: " mem "%"}'

# Top 5 CPU-consuming processes
ps aux | awk 'NR > 1 {print $3, $11}' | sort -nr | head -5

4. Network analysis:

# Active connections per port
netstat -an | awk '/LISTEN/ {count[$4]++} END {for (port in count) print port, count[port]}'

# Count unique IP addresses in an access log
awk '{ips[$1]++} END {print "Unique IPs:", length(ips)}' access.log

Advanced awk functions

Conditional processing:

# Process lines with specific conditions
awk '$3 > 100 && $4 < 50 {print "Warning:", $0}' measurement_data.txt

# If-else constructs
awk '{if ($1 > 0) print "Positive"; else print "Zero or negative"}' numbers.txt

Mathematical operations:

# Calculate average
awk '{sum += $1; count++} END {print "Average:", sum/count}' numbers.txt

# Find minimum and maximum
awk 'BEGIN{min=999999; max=-999999} {if($1<min) min=$1; if($1>max) max=$1} END{print "Min:", min, "Max:", max}' numbers.txt

💡 Tip for complex awk programs: For longer awk programs, it is often more practical to write them in separate files:

# Create script.awk
cat > analyse.awk << 'EOF'
BEGIN {
    print "Starting analysis..."
    FS = ":"
}
NR > 1 {
    if ($3 >= 1000) {
        users++
        print $1 " is a regular user"
    }
}
END {
    print "Found:", users, "regular users"
}
EOF

# Use the script
awk -f analyse.awk /etc/passwd

Differences and use cases of grep, sed, and awk

To make the right choice among these three tools, it is important to understand their respective strengths:

Decision guide:

• Use grep when you only want to search for text or filter lines
• Use sed when you want to substitute, delete, or perform simple manipulations
• Use awk when working with structured data or needing complex logic

🔧 Practical example – all three tools in one pipeline:

# Analyze Apache logs: find 404 errors, extract IP and time, sort by frequency
grep "404" /var/log/apache2/access.log |
awk '{print $1, $4}' |
sed 's/\[//g' |
sort | uniq -c | sort -nr | head -10

This pipeline demonstrates the strengths of all three tools:

• grep filters relevant lines (404 errors)
• awk extracts specific fields (IP address and time)
• sed cleans up the formatting
• sort and uniq count and sort the results

⚠️ Important note for the LPIC-1 exam: You should not only know the syntax of these commands but also understand when to use which tool. Exam questions often test the understanding of proper tool selection.

Common sources of errors:

• Ignoring field separators: awk uses whitespace as default delimiter
• Forgetting quotes: especially with awk programs containing spaces
• Confusing line and field numbers: NR vs. NF in awk
• Mixing up regex syntax: sed and awk have slightly different regex dialects

With this fundamental understanding of grep, sed, and awk, you have learned the most important tools of Linux text processing. These three commands form the foundation for virtually all text-based operations in Linux systems and are essential for the LPIC-1 certification.

Text sorting functions

The ability to sort text data and manage duplicates is a fundamental building block of Linux text processing. The sort and uniq commands work hand in hand and make it possible to extract structured, cleansed information from chaotic data sets. These tools are particularly valuable for analyzing log files, preparing system data, and creating meaningful reports.

sort – Systematically ordering text

What is sort and why is it indispensable?

The sort command arranges text lines according to various criteria and is one of the most frequently used tools in Linux text processing. Its strength lies not only in simple alphabetical sorting but in the variety of sorting options and seamless integration into complex processing pipelines.

💡 sort follows the Unix principle of specialization: It does one thing – sorting – but it does it exceptionally well and flexibly.

Basic sorting of text files

The simplest use of sort is lexicographical (alphabetical) sorting:

sort file.txt

By default, sort sorts line by line based on the entire line content, starting with the first character of each line.

🔧 Practical example:

Create an unsorted list:

cat > names.txt << EOF
Müller
Schmidt
Bauer
Abel
Zimmermann
EOF

Sort the list:

sort names.txt
Abel
Bauer
Müller
Schmidt
Zimmermann

⚠️ Important note on localization: The sort order is influenced by the system locale (LOCALE). German umlauts are treated differently depending on the setting.

# Display current locale
locale | grep LC_COLLATE

# Sort with C locale (ASCII order)
LC_COLLATE=C sort names.txt

# Sort with German locale
LC_COLLATE=de_DE.UTF-8 sort names.txt

Important sort options and their practical usage

sort offers a variety of options for different sorting requirements:

Sorting by different criteria

1. Numerical sorting:

# Create a list of numbers
cat > numbers.txt << EOF
100
20
3
1000
5
EOF

# Alphabetical sorting (default) - often not desired!
sort numbers.txt
100
1000
20
3
5

# Numerical sorting - the desired result
sort -n numbers.txt
3
5
20
100
1000

2. Reverse sorting:

# Largest numbers first
sort -nr numbers.txt
1000
100
20
5
3

3. Sorting by columns:

The -k option enables sorting by specific fields or columns:

# Example: Sort /etc/passwd by UID (3rd field)
sort -t: -k3 -n /etc/passwd | head -5
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync

Here this means:

• -t: sets the colon as field separator
• -k3 sorts by the 3rd field (UID)
• -n performs numerical sorting

4. Multi-criteria sorting:

# Create a table with employee data
cat > employees.txt << EOF
Müller 1500 IT
Schmidt 2000 IT
Bauer 1500 HR
Abel 2500 IT
Zimmermann 1500 Sales
EOF

Sort first by salary (2nd column), then by name (1st column):

sort -k2 -n -k1 employees.txt
Bauer 1500 HR
Müller 1500 IT
Zimmermann 1500 Sales
Schmidt 2000 IT
Abel 2500 IT

Practical application examples for log analysis

1. Sort Apache access log by IP addresses:

# Extract and sort IP addresses
awk '{print $1}' /var/log/apache2/access.log | sort | head -10

2. System load analysis:

# Sort processes by CPU usage
ps aux | sort -k3 -nr | head -10

3. Analyze memory usage:

# Sort directories by size
du -h /var/log/* | sort -hr | head -10

4. Analyze network connections:

# Sort active connections by port
netstat -an | grep ESTABLISHED | sort -k4

💡 Tip for complex sorts: When using multiple sort criteria, the -k options are specified in order of priority. The first -k has the highest priority.

Advanced sort functions

Sorting with custom field ranges:

# Sort only by part of a field
sort -k1.2,1.4 file.txt  # Sorts by characters 2-4 of the first field

Sorting IP addresses:

# Correct sorting of IPv4 addresses
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n ip_addresses.txt

Verifying the sort order:

# Check if a file is already sorted
sort -c file.txt
sort: file.txt:3: disorder: Abel

# Silent check (exit code only)
sort -C file.txt && echo "Sorted" || echo "Not sorted"

uniq – Detecting and managing duplicates

What is uniq and how does it work with sort?

The uniq command removes or identifies consecutive identical lines in a file. An important point: uniq only detects consecutive duplicates! Therefore, it is almost always used in combination with sort to find all duplicates in a file.

The typical combination is:

sort file.txt | uniq

Removing duplicate lines

Basic duplicate removal:

# Create a file with duplicates
cat > duplicates.txt << EOF
Apple
Banana
Apple
Orange
Banana
Apple
EOF

Remove only consecutive duplicates (not the desired result):

uniq duplicates.txt
Apple
Banana
Apple
Orange
Banana
Apple

Correct method: sort first, then remove duplicates

sort duplicates.txt | uniq
Apple
Banana
Orange

🔧 Practical example for system administration:

# Extract unique IP addresses from an access log
awk '{print $1}' /var/log/apache2/access.log | sort | uniq > unique_ips.txt

# Count number of unique visitors
awk '{print $1}' /var/log/apache2/access.log | sort | uniq | wc -l

Important uniq options

1. Counting frequencies with -c:

cat > browsers.txt << EOF
Firefox
Chrome
Firefox
Safari
Chrome
Firefox
Edge
EOF

sort browsers.txt | uniq -c

1 Chrome
1 Edge
3 Firefox
1 Safari

# Sorted by frequency (most frequent first)
sort browsers.txt | uniq -c | sort -nr
3 Firefox
1 Safari
1 Edge
1 Chrome

2. Show only duplicates with -d:

sort browsers.txt | uniq -d
Firefox

3. Show only unique lines with -u:

sort browsers.txt | uniq -u
Chrome
Edge
Safari

Combination with sort for effective data analysis

The combination of sort and uniq is a powerful tool for data analysis:

1. Top-10 analysis of log files:

# Most frequent IP addresses in Apache logs
awk '{print $1}' /var/log/apache2/access.log |
     sort | uniq -c | sort -nr | head -10

# Most frequent HTTP status codes
awk '{print $9}' /var/log/apache2/access.log |
    sort | uniq -c | sort -nr

2. System user analysis:

# Most frequent login shells
awk -F: '{print $7}' /etc/passwd | sort | uniq -c | sort -nr
    15 /bin/bash
     8 /usr/sbin/nologin
     3 /bin/false
     2 /bin/sync
     1 /usr/bin/git-shell

3. Network traffic analysis:

# Most frequent destination ports in netstat output
netstat -an | awk '{print $4}' | grep -E ':[0-9]+$' |
sed 's/.*://' | sort -n | uniq -c | sort -nr | head -10

Application examples for system monitoring

1. Monitoring failed login attempts:

# Most frequent failed login attempts by IP
grep "Failed password" /var/log/auth.log |
awk '{print $11}' | sort | uniq -c | sort -nr | head -10

2. Analyzing system errors:

# Most frequent error messages in syslog
grep -i error /var/log/syslog |
awk '{for(i=6;i<=NF;i++) printf "%s ", $i; print ""}' |
sort | uniq -c | sort -nr | head -5

3. Monitoring storage usage:

# Directories with the most files
find /var -type f 2>/dev/null |
dirname | sort | uniq -c | sort -nr | head -10

4. Analyzing configuration files:

# Most frequent configuration parameters in Apache
grep -v "^#|^$" /etc/apache2/apache2.conf |
awk '{print $1}' | sort | uniq -c | sort -nr

💡 Tip for practice: The combination sort | uniq -c | sort -nr is so common that many administrators define it as an alias.

# In ~/.bashrc
alias frequency='sort | uniq -c | sort -nr'

# Usage:
awk '{print $1}' access.log | frequency | head -10

⚠️ Important performance consideration: For very large files, the combination of sort and uniq can be memory-intensive. For extremely large data sets, there are specialized tools or you can split the data into smaller chunks.

Common error sources:

Forgetting to sort before uniq:

# WRONG - does not find all duplicates:
uniq file.txt

# RIGHT:
sort file.txt | uniq

Confusing -d and -u:

# -d shows only duplicates
sort file.txt | uniq -d

# -u shows only unique (non-duplicated) lines
sort file.txt | uniq -u

Not considering that uniq is case-sensitive:

# Without -i, "Test" and "test" are treated as different
echo -e "Test\ntest\nTest" | sort | uniq
test
Test

# With -i, they are treated as identical
echo -e "Test\ntest\nTest" | sort | uniq -i
Test

Advanced combinations of sort and uniq

The true strength of sort and uniq shines in complex data processing pipelines:

Complex log analysis:

# Analyze Apache logs: Find the top-10 user agents
awk -F'"' '{print $6}' /var/log/apache2/access.log |
  sort | uniq -c | sort -nr | head -10

# Analyze error distribution by hour
grep "$(date +%d/%b/%Y)" /var/log/apache2/access.log |
  awk '{print substr($4,14,2)}' | sort | uniq -c | sort -k2n

System resource monitoring:

# Find processes that are started most frequently
ps aux | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

# Analyze memory usage by process name
ps aux | awk '{print $11, $4}' | sort |
  awk '{proc[$1]+=$2} END {for(p in proc) print proc[p], p}' | sort -nr

Network security analysis:

# Find suspicious connection attempts
grep "Failed password" /var/log/auth.log |
  awk '{print $11}' | sort | uniq -c |
  awk '$1 > 10 {print $1, $2}' | sort -nr

The sort and uniq commands are fundamental tools for every Linux administrator. Mastering them makes it possible to extract structured information from chaotic data sets and is essential for the LPIC-1 certification. Combining both commands in pipelines with other tools like awk, grep, and cut opens up nearly unlimited possibilities for data analysis and system monitoring.

Counting and formatting text files

Analyzing and formatting text files is one of the daily tasks of a Linux administrator. Whether it's about monitoring the size of log files, creating reports, or preparing data for further processing – the commands covered in this section (wc, fmt, nl, and cut) are indispensable tools. They perfectly complement the text processing tools already covered and enable precise statistics as well as professional formatting.

wc – Counting words, lines, and characters

What is wc and why is it fundamentally important?

The wc (word count) command is one of the most basic and frequently used tools for text analysis on Linux systems. It counts lines, words, characters, and bytes in text files or data streams, providing essential statistics for system administration.

The strength of wc lies in its simplicity and versatility: it can be used for both quick checks and complex data analyses in pipelines.

Basic functions of wc (-l, -w, -c, -m)

The basic syntax of wc is intuitive:

wc [options] file(s)

Without options, wc outputs three values: number of lines, words, and bytes:

wc beispiel.txt
    15    42   256 beispiel.txt

💡 This output means: 15 lines, 42 words, 256 bytes.

The most important wc options:

⚠️ Important difference between -c and -m: In UTF-8 encoded files, characters can span multiple bytes. While -c counts bytes, -m counts actual characters:

echo "Café" | wc -c
5
echo "Café" | wc -m
4

The "é" requires two bytes in UTF-8 but is counted as a single character.

Practical application examples for system statistics:

Log file monitoring:

# Count lines in a log file
wc -l /var/log/apache2/access.log
45823 /var/log/apache2/access.log

# Monitor log growth
watch "wc -l /var/log/syslog"

Counting system users:

# Total number of system users
wc -l /etc/passwd
42 /etc/passwd

# Number of regular users (UID >= 1000)
awk -F: '$3 >= 1000 {print}' /etc/passwd | wc -l

Analyzing configuration files:

# Active configuration lines (without comments and empty lines)
grep -v "^\s*#\|^\s*$" /etc/apache2/apache2.conf | wc -l

# Average line length in a configuration file
awk '{total += length($0); count++} END {print total/count}' /etc/ssh/sshd_config

Combination with other commands for complex analyses

The true strength of wc shows in combination with other commands via pipes:

Process analysis:

# Number of running processes
ps aux | wc -l

# Number of Apache processes
ps aux | grep apache | grep -v grep | wc -l

# Memory usage of all processes (simplified)
ps aux | awk 'NR>1 {sum+=$4} END {print sum "%"}'

Network monitoring:

# Number of active connections
netstat -an | grep ESTABLISHED | wc -l

# Number of listening ports
netstat -ln | grep LISTEN | wc -l

Filesystem analysis:

# Number of files in a directory (recursive)
find /var/log -type f | wc -l

# Number of directories
find /etc -type d | wc -l

# Largest files by line count
find /var/log -name "*.log" -exec wc -l {} \; | sort -nr | head -5

Log analysis:

# Error rate in log files
grep -c "ERROR" /var/log/application.log

# Ratio of success to errors in Apache logs
awk '$9 ~ /^2/ {success++} $9 ~ /^[^45]/ {error++} END {print "Success:", success, "Errors:", error}' /var/log/apache2/access.log

💡 Tip for monitoring scripts: wc is excellent for simple monitoring scripts:

#!/bin/bash
# Simple log monitoring

LOGFILE="/var/log/application.log"
THRESHOLD=1000

# Current line count
CURRENT_LINES=$(wc -l < "$LOGFILE")

if [ "$CURRENT_LINES" -gt "$THRESHOLD" ]; then
    echo "WARNING: Log file has $CURRENT_LINES lines (threshold: $THRESHOLD)"
    # Trigger notification or rotation here
fi

fmt – Text formatting and line wrapping

What is fmt and when is it needed?

The fmt (format) command is a text formatter that wraps long lines and converts text into uniform paragraphs. It is particularly useful for preparing documentation, emails, or other texts that require consistent formatting.

fmt follows intelligent rules when wrapping: it tries not to split words and considers punctuation and whitespace.

Automatic text formatting

The basic usage of fmt is simple:

fmt file.txt

By default, fmt formats text to a line length of 75 characters.

🔧 Practical example:

# Create a file with long lines
cat > long_text.txt << 'EOF'
This is a very long text that should span multiple lines but was written in a single long line, which significantly impairs readability.
EOF

Format the text:

fmt long_text.txt
This is a very long text that should span multiple lines
but was written in a single long line, which significantly
impairs readability.

Important options for line length and indentation

🔧 Practical use cases:

Email formatting:

# Format an email with quoted lines
fmt -p "> " -w 72 email.txt

Formatting code comments:

# Format comments in a configuration file
fmt -p "# " -w 80 config.conf

Preparing documentation:

# Format README files
fmt -w 80 README.txt > README_formatted.txt

Use cases for documentation and emails

Automatic documentation formatting:

#!/bin/bash
# Script for formatting Markdown files

for file in *.md; do
    # Format only text paragraphs, not code blocks
    sed '/^```/!s/.*/fmt -w 80 <<< "&"/e' "$file" > "${file}.formatted"
done

Email template creation:

# Create formatted email templates
cat > email_template.txt << 'EOF'
Dear Sir or Madam, we would like to inform you about important system maintenance work that will be carried out this coming weekend.
EOF

fmt -w 72 email_template.txt

💡 Tip for practice: fmt can be integrated into Vim/Emacs to format text directly in the editor. In Vim: :%!fmt -w 80

nl – Line numbering

What is nl and what is it used for?

The nl (number lines) command adds line numbers to text files. Unlike simple solutions like cat -n, nl offers extended options for numbering and can apply various numbering styles.

Various numbering options

The basic syntax:

nl file.txt

Important nl options:

🔧 Practical examples:

a) Numbering code files:

# Python script with line numbers
nl -b a -w 3 -s " | " script.py
  1 | #!/usr/bin/env python3
  2 | import sys
  3 |
  4 | def main():
  5 |     print("Hello World")

b) Configuration files for debugging:

# Apache configuration with line numbers for troubleshooting
nl -b t -w 4 -s ": " /etc/apache2/apache2.conf

Practical application for code and configuration files

Error analysis in configuration files:

# Show configuration file with line numbers for error messages
nl -b a -w 3 /etc/ssh/sshd_config | grep -A 2 -B 2 "Port"

Log analysis with line numbers:

# Add line numbers to log files for better referencing
nl -b a /var/log/apache2/error.log | tail -20

Creating documentation:

# Create numbered checklists
nl -b a -s ". " checklist.txt

cut – Extracting columns and fields

What is cut and why is it indispensable?

The cut command extracts specific columns or fields from structured text files. It is particularly valuable for working with CSV files, log files, or other column-based data and allows targeted extraction of specific information.

Extracting specific columns from text files

The basic syntax of cut:

cut [options] file

Important cut options:

Important options (-d, -f, -c)

Field-based extraction with -d and -f:

# Extract usernames from /etc/passwd
cut -d: -f1 /etc/passwd | head -5
root
daemon
bin
sys
sync

# Extract username and UID
cut -d: -f1,3 /etc/passwd | head -5
root:0
daemon:1
bin:2
sys:3
sync:4

Character-based extraction with -c:

# Extract the first 10 characters of each line
cut -c1-10 /var/log/syslog | head -5

# Extract characters 1-3 and 8-12
cut -c1-3,8-12 file.txt

Ranges and lists:

# Various range notations
cut -d: -f1-3 /etc/passwd    # Fields 1 to 3
cut -d: -f1,3,5- /etc/passwd # Fields 1, 3 and from 5 onwards
cut -d: -f-3 /etc/passwd     # Fields from start to 3

Practical applications for CSV files and structured logs

CSV data analysis:

# Create a sample CSV
cat > sales_data.csv << 'EOF'
Date,Product,Quantity,Price
2023-05-01,Laptop,5,999.99
2023-05-01,Mouse,20,29.99
2023-05-02,Keyboard,15,79.99
EOF

# Extract only product and price
cut -d, -f2,4 sales_data.csv
Product,Price
Laptop,999.99
Mouse,29.99
Keyboard,79.99

# Calculate total revenue (simplified)
cut -d, -f3,4 sales_data.csv | tail -n +2 | awk -F, '{sum += $1 * $2} END {print "Total revenue:", sum}'

Apache log analysis:

# Extract IP addresses and HTTP status codes
cut -d' ' -f1,9 /var/log/apache2/access.log | head -5

# Extract only the time from log entries
cut -d' ' -f4 /var/log/apache2/access.log | cut -d: -f2-4 | head -5

Analyzing system configuration:

# Extract all used shells
cut -d: -f7 /etc/passwd | sort | uniq -c

# Show users with specific UIDs
cut -d: -f1,3 /etc/passwd | awk -F: '$2 >= 1000 {print $1}'

Combination with other commands for data extraction

Complex data processing:

# Analyze process memory usage
ps aux | cut -c12-15,65- | sort -nr | head -10

# Extract and analyze network ports
netstat -an | grep LISTEN | cut -d' ' -f4 | cut -d: -f2 | sort -n | uniq -c

Log rotation and analysis:

# Extract timestamps for rotation
cut -d' ' -f1-3 /var/log/syslog | tail -1

# Analyze error distribution by hours
grep "ERROR" /var/log/application.log | cut -d' ' -f2 | cut -d: -f1 | sort | uniq -c

Configuration management:

# Extract active Apache modules
apache2ctl -M | cut -d' ' -f1 | sort

# Analyze SSH configuration
grep -v "^#\|^$" /etc/ssh/sshd_config | cut -d' ' -f1 | sort | uniq -c

💡 Tip for complex extraction: For complex data structures, awk can be more flexible than cut, but cut is often faster and simpler for straightforward column extraction.

⚠️ Important note: cut can only work with consistent delimiters. For irregular spacing, awk is often the better choice.

Common sources of errors:

Wrong field numbering:

• cut starts at 1, not at 0

Inconsistent delimiters:

• Mix of tabs and spaces

Forgetting the delimiter:

• Without -d, cut uses tabs as default

The commands wc, fmt, nl, and cut extend your toolkit with important functions for statistics, formatting, and data extraction. They are essential for daily work as a Linux administrator and important components of the LPIC-1 exam. Combining these tools with the previously covered commands makes it possible to solve virtually any text processing task efficiently.

Regular Expressions – Basics

Regular expressions (RegEx) are one of the most powerful and simultaneously most feared technologies in the Linux world. They make it possible to describe and find complex text patterns, making them an indispensable tool for every Linux administrator. Although their syntax may initially appear cryptic, regular expressions open up completely new dimensions of text processing once the basics are learned.

Concept and importance for text processing

Regular expressions are a formal language for describing string patterns[^2]. They serve to search, analyze, and manipulate text according to specific patterns, going far beyond simple text search. While a normal search only looks for exact matches, regular expressions can define flexible patterns that capture various variations of a search term.

🔧 Practical example:

Instead of searching individually for "Error", "ERROR", "error", and "Err", a regular expression can capture all these variations with a single pattern:

grep -E "[Ee]rr(or)?" /var/log/syslog

The strength of regular expressions lies in their ability to describe abstract patterns:

• Find all email addresses in a text
• Identify IP addresses in log files
• Recognize date values in various formats
• Extract configuration parameters with specific structures

Differences between regex variants (BRE, ERE)

Linux primarily uses two variants of regular expressions, which differ in their syntax and scope of features:

Basic Regular Expressions (BRE):

• Default format in traditional Unix tools like grep and sed
• More conservative syntax with fewer metacharacters
• Certain characters must be "escaped" to retain their special meaning

Extended Regular Expressions (ERE):

• Extended syntax with more features
• Available via grep -E, egrep, awk, and modern tools
• More intuitive syntax for complex expressions

🔧 Comparison example:

# BRE syntax (traditional grep)
grep 'colou\?r' file.txt          # Does not work as expected in BRE
grep 'colou*r' file.txt           # Searches for "colo" + any number of "u" + "r"

# ERE syntax (extended grep)
grep -E 'colou?r' file.txt        # Finds "color" and "colour"
grep -E 'colou+r' file.txt        # Finds "colouur", "colouuur", etc.

Where are regular expressions used in Linux?

Regular expressions are integrated into virtually all important Linux tools:

Command-line tools:

• grep, egrep, fgrep – text search
• sed – stream editing and text substitution
• awk – structured text processing
• find – file search with name patterns
• less, more – search in pagers

Editors:

• vim, emacs – search and replace
• nano – basic regex support

Programming languages and scripts:

• Bash, Python, Perl, PHP – built-in regex support
• Shell scripts for system administration

System configuration:

• Apache, Nginx – URL rewriting and configuration
• Logrotate – filename patterns
• Firewall rules – pattern-based filtering

⚠️ Important note: Not all tools use the same regex syntax. It is important to know which variant a particular tool supports.

Basic regex syntax

Literal characters and metacharacters

Regular expressions consist of two types of characters:

Literal characters:

• Normal letters, numbers, and most special characters
• Represent themselves
• Example: "abc" finds exactly the string "abc"

Metacharacters:

• Special characters with special meaning
• Control the behavior of the regular expression
• Basic metacharacters: . ^ $ * + ? [ ] { } ( ) | \

🔧 Practical examples for metacharacters:

The dot (.) – any character:

echo -e "cat\ncar\ncup" | grep "ca."
cat
car

The caret (^) – start of line:

grep "^root" /etc/passwd
root:x:0:0:root:/root:/bin/bash

The dollar sign ($) – end of line:

grep "bash$" /etc/passwd
root:x:0:0:root:/root:/bin/bash
username:x:1000:1000:User:/home/username:/bin/bash

Escaping metacharacters: When you want to search for a metacharacter as a literal character, you must escape it with a backslash:

# Search for a literal dot
grep "192\.168\.1\.1" /etc/hosts

# Search for a dollar sign
grep "\$HOME" script.sh

Character classes and ranges ([a-z], [0-9], \d, \w)

Character classes make it possible to define a group of characters, one of which can appear at a specific position.

Basic character classes:

# Single characters
[abc]           # Matches 'a', 'b', or 'c'
[aeiou]         # Matches any vowel

# Ranges
[a-z]           # Any lowercase letter
[A-Z]           # Any uppercase letter
[0-9]           # Any digit
[a-zA-Z0-9]     # Alphanumeric characters

Negated character classes:

[^0-9]          # Anything except digits
[^aeiou]        # All consonants
[^a-zA-Z]       # All non-letters

🔧 Practical application examples:

Finding IP addresses:

grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" /var/log/apache2/access.log

Email addresses (simplified):

grep -E "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" contacts.txt

Hexadecimal values:

grep -E "[0-9a-fA-F]+" system.log

Predefined character classes (in extended regex engines):

⚠️ Note: These predefined classes are not available in all Linux tools. grep and sed do not support them by default, while awk and modern programming languages use them.

Quantifiers (*, +, ?, {n,m})

Quantifiers determine how often a character or group of characters can be repeated:

🔧 Practical examples:

Log file analysis:

# Find lines with multiple consecutive spaces
grep -E " {2,}" /var/log/syslog

# Find IP addresses with optional leading zeros
grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" access.log

Validating configuration files:

# Find lines with at least one equals sign
grep -E ".+=.+" config.conf

# Find comment lines (optionally with spaces before #)
grep -E "^ *#" apache2.conf

System monitoring:

# Find processes with PID patterns
ps aux | grep -E "[0-9]{4,}"  # PIDs with at least 4 digits

Anchors (^, $) for start and end of line

Anchors define positions in text, not characters:

Start of line (^):

# Find lines starting with "Error"
grep "^Error" /var/log/application.log

# Find configuration lines not starting with #
grep "^[^#]" /etc/ssh/sshd_config

End of line ($):

# Find lines ending with a period
grep "\.$" document.txt

# Find shell scripts (lines ending with .sh)
ls -la | grep "\.sh$"

Combining anchors:

# Find empty lines
grep "^$" file.txt

# Find lines consisting only of whitespace
grep "^ *$" file.txt

# Find lines with exactly one word
grep "^[a-zA-Z]*$" wordlist.txt

Practical application of regular expressions

Usage with grep for extended searches

grep is the primary tool for applying regular expressions in text search:

Basic grep regex options:

grep "pattern" file          # BRE (Basic Regular Expressions)
grep -E "pattern" file       # ERE (Extended Regular Expressions)
grep -P "pattern" file       # PCRE (Perl Compatible, if available)
grep -F "pattern" file       # Fixed strings (no regex)

🔧 Extended search examples:

Log analysis with complex patterns:

# Find all HTTP errors (4xx and 5xx status codes)
grep -E " [45][0-9]{2} " /var/log/apache2/access.log

# Find timestamps in various formats
grep -E "[0-9]{4}-[0-9]{2}-[0-9]{2}|[0-9]{2}/[0-9]{2}/[0-9]{4}" logfile.txt

# Find critical system events
grep -E "(CRITICAL|FATAL|EMERGENCY)" /var/log/syslog

Network and security analysis:

# Find suspicious IP addresses (exclude private ranges)
grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" access.log | grep -vE "^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)"

# Find failed SSH logins with IP addresses
grep -E "Failed password.*from ([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/auth.log

Usage in sed for complex replacements

sed uses regular expressions for both search and replacement:

Extended sed operations:

# Replace all IP addresses with "XXX.XXX.XXX.XXX"
sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/XXX.XXX.XXX.XXX/g' logfile.txt

# Extract domain from email addresses
sed -E 's/.*@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*/\1/' email_list.txt

# Reformat dates (YYYY-MM-DD to DD.MM.YYYY)
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3.\2.\1/g' file.txt

Using backreferences:

# Swap first and last name
sed -E 's/([A-Za-z]+) ([A-Za-z]+)/\2, \1/' names.txt

# Add quotation marks around words
sed -E 's/([a-zA-Z]+)/"\1"/g' wordlist.txt

Typical patterns for system administration

IP addresses and networking:

# IPv4 address (simplified)
^([0-9]{1,3}\.){3}[0-9]{1,3}$

# IPv4 with port specification
^([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]{1,5}$

# MAC address
^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$

Email addresses:

# Simple email validation
^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

# Email extraction from text
[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

Date formats:

# ISO date (YYYY-MM-DD)
^[0-9]{4}-[0-9]{2}-[0-9]{2}$

# German date (DD.MM.YYYY)
^[0-9]{2}\.[0-9]{2}\.[0-9]{4}$

# American date (MM/DD/YYYY)
^[0-9]{2}/[0-9]{2}/[0-9]{4}$

System-specific patterns:

# Linux username
^[a-z_][a-z0-9_-]*$

# Absolute paths
^/([a-zA-Z0-9._-]+/?)*$

# Process IDs
^[0-9]+$

🔧 Practical example – complete log analysis:

#!/bin/bash
# Comprehensive log analysis script with regex

LOGFILE="/var/log/apache2/access.log"

echo "=== Apache Log Analysis ==="

# 1. Count unique IP addresses
echo "Unique visitors:"
grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" "$LOGFILE" | sort | uniq | wc -l

# 2. Top 10 IP addresses
echo -e "\nTop 10 IP addresses:"
grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" "$LOGFILE" | sort | uniq -c | sort -nr | head -10

# 3. Analyze HTTP errors
echo -e "\nHTTP errors (4xx and 5xx):"
grep -E " [45][0-9]{2} " "$LOGFILE" | awk '{print $9}' | sort | uniq -c | sort -nr

# 4. Find suspicious user agents
echo -e "\nSuspicious user agents (bot patterns):"
grep -iE "(bot|crawler|spider|scraper)" "$LOGFILE" | awk -F'"' '{print $6}' | sort | uniq -c | sort -nr | head -5

Common pitfalls and how to avoid them

Typical regex errors:

a) Forgetting to escape:

# WRONG – dot is interpreted as "any character"
grep "192.168.1.1" /etc/hosts

# RIGHT – dot is treated as a literal dot
grep "192\.168\.1\.1" /etc/hosts

b) Greedy vs. non-greedy quantifiers:

# Greedy – finds the longest possible match
echo '<tag>content</tag>' | sed 's/<.*>/REPLACED/'
# Result: REPLACED

# Solution: Use a more specific pattern
echo '<tag>content</tag>' | sed 's/<[^>]*>/REPLACED/g'
# Result: REPLACEDcontent</tag>

c) Confusing BRE and ERE:

# BRE (does not work as expected)
grep 'error+' logfile.txt        # Searches for "error" followed by "+"

# ERE (correct)
grep -E 'error+' logfile.txt     # Searches for "erro" + one or more "r"

d) Anchor misunderstandings:

# WRONG – ^ and $ refer to the entire line
echo "start middle end" | grep "^middle$"  # Finds nothing

# RIGHT – do not use anchors for substrings
echo "start middle end" | grep "middle"    # Finds "middle"

Tips for effective regex usage:

Test your regex step by step:

• Start with simple patterns and gradually extend them

Use online regex testers:

• Tools like regex101.com help with debugging

Document complex expressions:

• Add comments to explain the logic

Prefer readability:

• Multiple simple regex are often better than one complex one

⚠️ Performance considerations: Very complex regular expressions can be slow, especially with large files. In such cases, several simple grep commands in a pipeline can be more efficient.

Regular expressions are a powerful tool that becomes more natural with practice and experience. They are essential for the LPIC-1 certification and indispensable for efficient system administration. The investment in learning the basics pays off with considerable time savings and expanded possibilities in text processing.

💡 Tip for the LPIC-1 exam: For the LPIC-1 exam, it is not memorization but understanding and practical application of text processing commands that matters. Practice intensively with tools like grep, sed, awk, sort, and cut on real examples and regularly work with the man pages.

Many exam questions test typical invocation options, regex basics, and combination possibilities – practical experience helps you answer even detailed questions confidently.

Why is this important for LPIC-1?

Mastering shell-based text processing tools is a central examination objective of the LPIC-1 certification. Many tasks in the exam – and in everyday work as an administrator – require you to quickly analyze, filter, and specifically process large amounts of system and log data. The ability to use tools like grep, sed, awk, sort, or cut purposefully forms the foundation for efficient error analysis, system monitoring, and automation. Without this knowledge, sustainable Linux administration is hardly possible.

💡 Note: Resources and exam information can be found in our fundamentals module: LPIC-1: Basic navigation and filesystem commands.

Command Reference (Cheatsheet)

Further Resources

Conclusion

Mastering UNIX text tools and regular expressions forms the foundation for efficient system analysis, log file evaluation, and automation in Linux. Through the seamless interplay of grep, sed, awk, sort, uniq, and cut via pipes, you can transform even complex data streams and configurations with just a few precise command lines without external scripting languages.

💡 Practice tip: When building complex pipelines, always think modularly: test each filter step individually on a small sample (e.g., head -n 20) before writing the result with sed -i or redirecting it to production files.

In the next part of our LPIC-1 series, we tackle process management and system monitoring: LPIC-1: Managing and monitoring processes – where you'll get to know ps, top, kill, nice, and signal handling in detail.

👉 To the course overview: All LPIC-1 articles & modules