--- id: 2025-05-28-lpic-1-textverarbeitung-mit-shell-kommandos slug: lpic-1-text-processing-with-shell-commands title: "LPIC-1: Text processing with shell commands" excerpt: "Learn how to efficiently analyze, filter, sort, and edit text files in Linux using grep, sed, awk, sort, uniq, wc, cut, and regular expressions." date: "2025-05-28T07:14:54+02:00" updated: "2025-07-12T17:59:14+02:00" author: name: "Sebastian Palencsár" handle: "spalencsar" category: "lpic-1-serie" tags: ["lpic-1", "lpic-1-serie", "grep", "sed", "awk", "sort", "uniq", "wc", "cut", "regex", "shellscripting", "textverarbeitung"] reading_time: 25 toc: true --- In the [first three parts of our LPIC-1 series](/en/category/lpic-1-serie){.badge-link-text}, we covered the [fundamentals of the Linux command line](/en/online-courses/lpic-1-serie/2025/2025-05-13-lpic-1-understanding-the-linux-command-line-shell-terminal-and-first-commands){.badge-link-text}, [filesystem navigation](/en/lpic-1-serie/lpic-1-basic-navigation-and-filesystem-commands){.badge-link-text}, and [editing file contents](/en/online-courses/lpic-1-serie/2025/2025-05-16-lpic-1-viewing-and-editing-file-contents){.badge-link-text}. Now it's time to take a decisive step further: exploring the powerful world of shell-based text processing. Text processing via shell commands is one of the most characteristic and powerful features of Linux and Unix systems. While other operating systems often rely on graphical tools or specialized programs, Linux offers a collection of highly efficient command-line tools that can be seamlessly combined with each other.

❗ **Important note:** As emphasized in previous articles, this series does not replace an official exam preparation course for the [LPIC-1 certification](https://www.lpi.org/our-certifications/lpic-1-overview/){.badge-link-text}. It serves as a practice-oriented, didactically prepared supplement for your self-study and is designed to help you better understand and apply the sometimes complex content.

## Why shell-based text processing is so central ```markdown ┌─────────────────────────────────────────────────────────────┐ │ UNIX TEXT PROCESSING PIPELINE │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Raw data (files / streams / standard input) │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ 1. Filtering (grep, egrep, fgrep) │ │ │ │ Extracts lines based on patterns / regex │ │ │ └────┬────────────────────────────────────────────────┘ │ │ │ [ stdout | stdin ] (pipe) │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ 2. Transforming & Extracting (cut, tr, sed, awk) │ │ │ │ Cut columns, swap characters, substitute │ │ │ └────┬────────────────────────────────────────────────┘ │ │ │ [ stdout | stdin ] (pipe) │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ 3. Sorting & Aggregating (sort | uniq -c | wc) │ │ │ │ Sorts, deduplicates, counts frequencies │ │ │ └────┬────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Structured output / standard output / analysis report │ │ │ └─────────────────────────────────────────────────────────────┘ ``` In Linux administration, text is ubiquitous: configuration files, log files, system outputs, reports, and scripts – everything is based on text data. The ability to efficiently search, filter, transform, and analyze this text data distinguishes a competent Linux administrator from a beginner. **Shell-based text processing tools follow the classic Unix philosophy:** * Each tool does one thing, but it does it very well * Tools can be combined with each other via pipes * Simple tools can be assembled into complex solutions * Text data is the universal interface between tools This approach makes it possible to perform complex data analyses and transformations with just a few well-mastered commands – tasks that would require specialized software in other environments. ### Importance for the LPIC-1 certification Shell-based text processing holds an important place in the LPIC-1 exam, particularly in the following examination objectives: **103.2:** * Process text streams using filters **103.3:** * Perform basic file manipulation **103.7:** * Search for and extract data from files using regular expressions Together, these topics account for approximately **20–25%** of the points in the LPIC-1 Exam 101, underscoring the importance of these skills. What to expect in this article **In the upcoming sections, we will cover the following core topics:** * Basic text processing with the "big three": `grep`, `sed`, and `awk` * Text sorting functions with `sort` and `uniq` to organize data and manage duplicates * Counting and formatting text files with tools like `wc`, `fmt`, `nl`, and `cut` * Regular expressions as a powerful tool for pattern matching and substitution Each section will not only explain the technical aspects but also highlight practical application scenarios from real-world system administration. Practical relevance for everyday administration The commands covered in this article are part of every Linux administrator's daily toolkit. **Typical application scenarios include:** * Analyzing log files for troubleshooting and system monitoring * Extracting specific information from configuration files * Automated reports on system states * Data cleansing and transformation for further processing * Monitoring system resources and user activities

💡 **Note:** As usual, you'll find special markers throughout the article: * 💡 **Tips and hints** for more efficient workflows * ⚠️ **Warnings and pitfalls** to save you trouble * 🔧 **Practical examples** to follow along directly * ❗ **Common sources of error** and their solutions

How to get the most out of this article: For optimal learning, I strongly recommend trying out the commands and techniques in your own Linux environment. Create test text files with various contents, experiment with the presented commands, and observe the results. Particularly with text processing, hands-on practice is essential – combining different commands via pipes requires practice and experimentation. The more you work with these tools, the more intuitive their application becomes. Let's now dive into the fascinating world of shell-based text processing – a field that will revolutionize your efficiency as a Linux administrator and at the same time secure you important points in the LPIC-1 exam. ## Basic text processing The three pillars of Linux text processing – grep, sed, and awk – form the core of shell-based data manipulation. These tools complement each other perfectly and make it possible to handle complex text processing tasks with elegant, efficient solutions. Each tool has its specific strengths and areas of application that we will explore in detail in this section. ### grep – The powerful text searcher What is grep and why is it indispensable? grep (Global Regular Expression Print) is the fundamental search tool in Linux systems. It searches text files or data streams for lines matching a specific pattern and outputs them. The name already reveals its origin: it comes from the ed editor command g/re/p (global/regular expression/print). The strength of grep lies in its simplicity and speed. It is optimized for one task: quickly finding text patterns in large data sets. Basic syntax and usage The basic syntax of grep is intuitive: ```bash grep [options] "search_pattern" file(s) ``` 🔧 **Simple examples:** ```bash # Search for the word "error" in a log file grep "error" /var/log/syslog # Search in multiple files simultaneously grep "Failed" /var/log/auth.log /var/log/secure # Search via pipes (very commonly used) ps aux | grep apache ```

💡 **The last example shows one of the most common uses of grep:** filtering the output of other commands via pipes.

Important options and their practical application grep offers a variety of options that significantly extend its functionality: | Option | Description | Practical Example | | --- | --- | --- | | `-i` | Ignores case | `grep -i "error" logfile.txt` | | `-v` | Inverts the search (shows lines WITHOUT the pattern) | `grep -v "^#" config.conf` | | `-n` | Shows line numbers | `grep -n "TODO" script.sh` | | `-r` or `-R` | Recursive search in directories | `grep -r "password" /etc/` | | `-l` | Shows only filenames with matches | `grep -l "function" *.py` | | `-c` | Counts the number of matches | `grep -c "GET" access.log` | | `-A n` | Shows n lines after the match | `grep -A 3 "Exception" error.log` | | `-B n` | Shows n lines before the match | `grep -B 2 "FATAL" system.log` | | `-C n` | Shows n lines before and after the match | `grep -C 5 "segfault" kernel.log` | 🔧 **Practical application examples for system administration:** **a) Error analysis in log files:** ```bash # Find all critical errors from the last hour grep -i "critical\|fatal\|emergency" /var/log/syslog # Show context around SSH connection errors grep -C 3 "Failed password" /var/log/auth.log ``` **b) Analyzing configuration files:** ```bash # Find all active configuration directives grep -v "^#\|^$" /etc/apache2/apache2.conf # Find specific settings in SSH config grep -i "PermitRootLogin\|PasswordAuthentication" /etc/ssh/sshd_config ``` **c) System monitoring:** ```bash # Count failed login attempts grep -c "Failed password" /var/log/auth.log # Find all processes of a specific user ps aux | grep -v grep | grep username ```

💡 **Tip for practice:** Combine grep with other commands to create powerful analysis pipelines. For example: `grep "error" /var/log/syslog | wc -l` counts all error lines in a log file.

### sed – The stream editor What is sed and how does it work? sed (Stream Editor) is a non-interactive stream editor that processes text line by line. It is one of the most powerful tools for text manipulation on the command line and is particularly valuable for automated text transformations in scripts and pipelines. The name "Stream Editor" already indicates its function: it processes data streams (stdin or files) without requiring user interaction. Basic syntax and operations The basic syntax of sed: ```bash sed [options] 'command' file ``` 🔧 **Simple examples:** ```bash # Replace "old" with "new" in a file (display only) sed 's/old/new/g' file.txt # Replace "old" with "new" in a file (in-place editing) sed -i 's/old/new/g' file.txt # Delete all lines containing "pattern" sed '/pattern/d' file.txt # Display only lines 5-10 sed -n '5,10p' file.txt ``` Important sed commands and options | Command | Description | Practical Example | | --- | --- | --- | | `s/alt/neu/g` | Substitution (replace) | `sed 's/error/warning/g' logfile.txt` | | `d` | Delete lines | `sed '/^#/d' config.conf` | | `p` | Print lines | `sed -n '10,20p' file.txt` | | `a\` | Append text after line | `sed '/pattern/a\New text' file.txt` | | `i\` | Insert text before line | `sed '/pattern/i\New text' file.txt` | | `c\` | Replace entire line | `sed '/pattern/c\New line' file.txt` | | `-i` | In-place editing | `sed -i.bak 's/alt/neu/g' file.txt` | | `-n` | Suppress automatic output | `sed -n '5p' file.txt` | 🔧 **Practical examples for system administration:** **a) Configuration file modification:** ```bash # Comment out all lines not starting with # sed -i 's/^[^#]/# &/' config.conf # Replace a specific line in a config file sed -i '/^Port /c\Port 2222' /etc/ssh/sshd_config # Remove empty lines from a file sed '/^$/d' config.txt ``` **b) Log file analysis:** ```bash # Extract only timestamp and error message sed -n '/ERROR/p' /var/log/app.log | sed 's/.*\[ERROR\] //' # Remove ANSI escape codes from log files sed 's/\x1b\[[0-9;]*m//g' colored_log.txt ``` **c) Text transformation:** ```bash # Convert Windows line endings to Unix sed -i 's/\r$//' file.txt # Remove trailing whitespace from each line sed 's/[[:space:]]*$//' file.txt ```

⚠️ **Be careful with the `-i` option:** It modifies the original file irreversibly!

**Always create backups for important files:** ```bash # Safe approach with backup sed -i.bak 's/old/new/g' important_file.conf ``` Advanced sed techniques **Line addressing:** ```bash # Replace only in lines 10-20 sed '10,20s/old/new/g' file.txt # From the first occurrence of a pattern to end of file sed '/START/,/old/new/g' file.txt # Edit every other line sed '1~2s/old/new/g' file.txt ``` **Combining multiple commands:** ```bash # Multiple substitutions in one pass sed -e 's/http/https/g' -e 's/80/443/g' -e '/^#/d' config.txt # Separate with semicolons sed 's/old/new/g; /^#/d; s/[[:space:]]*//' file.txt ```

💡 **Tip for complex operations:** For very complex `sed` operations, it may be more practical to create a `sed` script.

```bash # Create sed-script.sed cat > sed-script.sed << 'EOF' s/http/https/g s/80/443/g /^#/d /^$/d EOF # Use the script sed -f sed-script.sed configuration.conf ``` ### awk – Introduction to text processing Core concept of awk as a pattern-action language `awk` is more than just a text processing tool – it is a complete programming language specifically designed for processing structured text data. The name comes from the surnames of its developers: Aho, Weinberger, and Kernighan. The core concept of awk is based on the pattern-action paradigm: ```markdown ┌─────────────────────────────────────────────────────────────┐ │ AWK PATTERN-ACTION MODEL │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Input line ──► [ Pattern / Condition? ] │ │ │ │ │ YES ─────┴────── NO │ │ │ │ │ │ ▼ ▼ │ │ { Action to (Line is │ │ { execute } ignored) │ │ │ │ │ ▼ │ │ Output (print) │ │ │ │ Structure: awk '{ pattern { action }' file │ │ │ │ │ │ │ └─ Action (e.g. print $1) │ │ └─ Filter condition (e.g. $3 >= 1000) │ │ │ └─────────────────────────────────────────────────────────────┘ ``` For each input line, `awk` checks whether it matches the pattern. If so, the corresponding action is executed. **The basic structure of an awk program:** ```bash awk 'BEGIN { initialization } pattern1 { action1 } pattern2 { action2 } END { finalization }' input_file ``` Simple awk commands for column processing `awk` automatically splits each input line into fields that can be accessed via `$1`, `$2`, `$3`, etc. `$0` represents the entire line. 🔧 **Basic examples:** ```bash # Print first column of all lines awk '{print $1}' file.txt # Print first and third column awk '{print $1, $3}' file.txt # Show lines with more than 3 fields awk 'NF > 3 {print $0}' file.txt # Show line number and content awk '{print NR ": " $0}' file.txt ``` **Important built-in variables:** | Variable | Description | Example | | --- | --- | --- | | `NR` | Current line number | `awk 'NR == 5 {print}' file.txt` | | `NF` | Number of fields in current line | `awk '{print NF}' file.txt` | | `FS` | Field separator | `awk 'BEGIN{FS=":"} {print $1}' /etc/passwd` | | `RS` | Record separator | `awk 'BEGIN{RS=";"} {print}' file.txt` | | `OFS` | Output field separator | `awk 'BEGIN{OFS=","} {print $1,$2}' file.txt` | Practical use cases in system administration **1. Analyzing the `/etc/passwd` file:** ```bash # Display all usernames awk -F: '{print $1}' /etc/passwd # Users with UID greater than 1000 (regular users) awk -F: '$3 >= 1000 {print $1 " (UID: " $3 ")"}' /etc/passwd # Users with bash as default shell awk -F: '$7 == "/bin/bash" {print $1}' /etc/passwd ``` **2. Log analysis:** ```bash # Sum up log file sizes ls -l /var/log/*.log | awk '{sum += $5} END {print "Total: " sum " bytes"}' # Apache access log: IP addresses and status codes awk '{print $1, $9}' /var/log/apache2/access.log # Number of requests per IP address awk '{count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log ``` **3. Monitoring system resources:** ```bash # Memory usage of processes ps aux | awk 'NR > 1 {mem += $4} END {print "Total RAM usage: " mem "%"}' # Top 5 CPU-consuming processes ps aux | awk 'NR > 1 {print $3, $11}' | sort -nr | head -5 ``` **4. Network analysis:** ```bash # Active connections per port netstat -an | awk '/LISTEN/ {count[$4]++} END {for (port in count) print port, count[port]}' # Count unique IP addresses in an access log awk '{ips[$1]++} END {print "Unique IPs:", length(ips)}' access.log ``` Advanced awk functions **Conditional processing:** ```bash # Process lines with specific conditions awk '$3 > 100 && $4 < 50 {print "Warning:", $0}' measurement_data.txt # If-else constructs awk '{if ($1 > 0) print "Positive"; else print "Zero or negative"}' numbers.txt ``` **Mathematical operations:** ```bash # Calculate average awk '{sum += $1; count++} END {print "Average:", sum/count}' numbers.txt # Find minimum and maximum awk 'BEGIN{min=999999; max=-999999} {if($1max) max=$1} END{print "Min:", min, "Max:", max}' numbers.txt ```

💡 **Tip for complex awk programs:** For longer awk programs, it is often more practical to write them in separate files:

```bash # Create script.awk cat > analyse.awk << 'EOF' BEGIN { print "Starting analysis..." FS = ":" } NR > 1 { if ($3 >= 1000) { users++ print $1 " is a regular user" } } END { print "Found:", users, "regular users" } EOF # Use the script awk -f analyse.awk /etc/passwd ``` Differences and use cases of grep, sed, and awk To make the right choice among these three tools, it is important to understand their respective strengths: | Tool | Main Purpose | Ideal For | Complexity | | --- | --- | --- | --- | | **grep** | Text search and filtering | Finding lines with specific patterns | Low | | **sed** | Stream editing | Simple text substitutions and manipulations | Medium | | **awk** | Structured data processing | Column-based operations and calculations | High | **Decision guide:** * Use `grep` when you only want to search for text or filter lines * Use `sed` when you want to substitute, delete, or perform simple manipulations * Use `awk` when working with structured data or needing complex logic 🔧 **Practical example – all three tools in one pipeline:** ```bash # Analyze Apache logs: find 404 errors, extract IP and time, sort by frequency grep "404" /var/log/apache2/access.log | awk '{print $1, $4}' | sed 's/\[//g' | sort | uniq -c | sort -nr | head -10 ``` **This pipeline demonstrates the strengths of all three tools:** * grep filters relevant lines (404 errors) * awk extracts specific fields (IP address and time) * sed cleans up the formatting * sort and uniq count and sort the results

⚠️ **Important note for the LPIC-1 exam:** You should not only know the syntax of these commands but also understand when to use which tool. Exam questions often test the understanding of proper tool selection.

**Common sources of errors:** * Ignoring field separators: awk uses whitespace as default delimiter * Forgetting quotes: especially with awk programs containing spaces * Confusing line and field numbers: NR vs. NF in awk * Mixing up regex syntax: sed and awk have slightly different regex dialects With this fundamental understanding of `grep`, `sed`, and `awk`, you have learned the most important tools of Linux text processing. These three commands form the foundation for virtually all text-based operations in Linux systems and are essential for the LPIC-1 certification. ## Text sorting functions The ability to sort text data and manage duplicates is a fundamental building block of Linux text processing. The `sort` and `uniq` commands work hand in hand and make it possible to extract structured, cleansed information from chaotic data sets. These tools are particularly valuable for analyzing log files, preparing system data, and creating meaningful reports. ### sort – Systematically ordering text What is sort and why is it indispensable? The `sort` command arranges text lines according to various criteria and is one of the most frequently used tools in Linux text processing. Its strength lies not only in simple alphabetical sorting but in the variety of sorting options and seamless integration into complex processing pipelines.

💡 **sort follows the Unix principle of specialization:** It does one thing – sorting – but it does it exceptionally well and flexibly.

Basic sorting of text files The simplest use of `sort` is lexicographical (alphabetical) sorting: ```bash sort file.txt ``` By default, `sort` sorts line by line based on the entire line content, starting with the first character of each line. 🔧 **Practical example:** Create an unsorted list: ```bash cat > names.txt << EOF Müller Schmidt Bauer Abel Zimmermann EOF ``` Sort the list: ```bash sort names.txt Abel Bauer Müller Schmidt Zimmermann ```

⚠️ **Important note on localization:** The sort order is influenced by the system locale (LOCALE). German umlauts are treated differently depending on the setting.

```bash # Display current locale locale | grep LC_COLLATE # Sort with C locale (ASCII order) LC_COLLATE=C sort names.txt # Sort with German locale LC_COLLATE=de_DE.UTF-8 sort names.txt ``` Important sort options and their practical usage `sort` offers a variety of options for different sorting requirements: | Option | Description | Practical Example | | --- | --- | --- | | `-n` | Numerical sorting | `sort -n numbers.txt` | | `-r` | Reverse order | `sort -r file.txt` | | `-k` | Sort by specific column | `sort -k2 table.txt` | | `-t` | Define field separator | `sort -t: -k3 /etc/passwd` | | `-u` | Removes duplicates during sorting | `sort -u list.txt` | | `-f` | Ignores case | `sort -f mixed_list.txt` | | `-M` | Sorts by month names | `sort -M months.txt` | | `-h` | "Human-readable" sizes (1K, 2M, 3G) | `sort -h filesizes.txt` | | `-c` | Checks if already sorted | `sort -c file.txt` | | `-o` | Output to file | `sort -o sorted.txt input.txt` | Sorting by different criteria **1. Numerical sorting:** ```bash # Create a list of numbers cat > numbers.txt << EOF 100 20 3 1000 5 EOF # Alphabetical sorting (default) - often not desired! sort numbers.txt 100 1000 20 3 5 # Numerical sorting - the desired result sort -n numbers.txt 3 5 20 100 1000 ``` **2. Reverse sorting:** ```bash # Largest numbers first sort -nr numbers.txt 1000 100 20 5 3 ``` **3. Sorting by columns:** The `-k` option enables sorting by specific fields or columns: ```bash # Example: Sort /etc/passwd by UID (3rd field) sort -t: -k3 -n /etc/passwd | head -5 root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync ``` **Here this means:** * `-t`: sets the colon as field separator * `-k3` sorts by the 3rd field (UID) * `-n` performs numerical sorting **4. Multi-criteria sorting:** ```bash # Create a table with employee data cat > employees.txt << EOF Müller 1500 IT Schmidt 2000 IT Bauer 1500 HR Abel 2500 IT Zimmermann 1500 Sales EOF ``` Sort first by salary (2nd column), then by name (1st column): ```bash sort -k2 -n -k1 employees.txt Bauer 1500 HR Müller 1500 IT Zimmermann 1500 Sales Schmidt 2000 IT Abel 2500 IT ``` Practical application examples for log analysis **1. Sort Apache access log by IP addresses:** ```bash # Extract and sort IP addresses awk '{print $1}' /var/log/apache2/access.log | sort | head -10 ``` **2. System load analysis:** ```bash # Sort processes by CPU usage ps aux | sort -k3 -nr | head -10 ``` **3. Analyze memory usage:** ```bash # Sort directories by size du -h /var/log/* | sort -hr | head -10 ``` **4. Analyze network connections:** ```bash # Sort active connections by port netstat -an | grep ESTABLISHED | sort -k4 ```

💡 **Tip for complex sorts:** When using multiple sort criteria, the `-k` options are specified in order of priority. The first `-k` has the highest priority.

Advanced sort functions **Sorting with custom field ranges:** ```bash # Sort only by part of a field sort -k1.2,1.4 file.txt # Sorts by characters 2-4 of the first field ``` **Sorting IP addresses:** ```bash # Correct sorting of IPv4 addresses sort -t. -k1,1n -k2,2n -k3,3n -k4,4n ip_addresses.txt ``` **Verifying the sort order:** ```bash # Check if a file is already sorted sort -c file.txt sort: file.txt:3: disorder: Abel # Silent check (exit code only) sort -C file.txt && echo "Sorted" || echo "Not sorted" ``` ### uniq – Detecting and managing duplicates What is uniq and how does it work with sort? The `uniq` command removes or identifies consecutive identical lines in a file. An important point: `uniq` only detects consecutive duplicates! Therefore, it is almost always used in combination with `sort` to find all duplicates in a file. **The typical combination is:** ```bash sort file.txt | uniq ``` Removing duplicate lines **Basic duplicate removal:** ```bash # Create a file with duplicates cat > duplicates.txt << EOF Apple Banana Apple Orange Banana Apple EOF ``` Remove only consecutive duplicates (not the desired result): ```bash uniq duplicates.txt Apple Banana Apple Orange Banana Apple ``` Correct method: sort first, then remove duplicates ```bash sort duplicates.txt | uniq Apple Banana Orange ``` 🔧 **Practical example for system administration:** ```bash # Extract unique IP addresses from an access log awk '{print $1}' /var/log/apache2/access.log | sort | uniq > unique_ips.txt # Count number of unique visitors awk '{print $1}' /var/log/apache2/access.log | sort | uniq | wc -l ``` ### Important uniq options | Option | Description | Practical Example | | --- | --- | --- | | `-c` | Counts occurrences of each line | `sort file.txt \| uniq -c` | | `-d` | Shows only duplicate lines | `sort file.txt \| uniq -d` | | `-u` | Shows only unique lines | `sort file.txt \| uniq -u` | | `-i` | Ignores case | `sort file.txt \| uniq -i` | | `-f n` | Ignores first n fields | `sort file.txt \| uniq -f 1` | | `-s n` | Ignores first n characters | `sort file.txt \| uniq -s 3` | **1. Counting frequencies with `-c`:** ```bash cat > browsers.txt << EOF Firefox Chrome Firefox Safari Chrome Firefox Edge EOF ``` ```bash sort browsers.txt | uniq -c ``` ```bash 1 Chrome 1 Edge 3 Firefox 1 Safari ``` ```bash # Sorted by frequency (most frequent first) sort browsers.txt | uniq -c | sort -nr 3 Firefox 1 Safari 1 Edge 1 Chrome ``` **2. Show only duplicates with `-d`:** ```bash sort browsers.txt | uniq -d Firefox ``` **3. Show only unique lines with `-u`:** ```bash sort browsers.txt | uniq -u Chrome Edge Safari ``` Combination with sort for effective data analysis The combination of `sort` and `uniq` is a powerful tool for data analysis: **1. Top-10 analysis of log files:** ```bash # Most frequent IP addresses in Apache logs awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -10 # Most frequent HTTP status codes awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c | sort -nr ``` **2. System user analysis:** ```bash # Most frequent login shells awk -F: '{print $7}' /etc/passwd | sort | uniq -c | sort -nr 15 /bin/bash 8 /usr/sbin/nologin 3 /bin/false 2 /bin/sync 1 /usr/bin/git-shell ``` **3. Network traffic analysis:** ```bash # Most frequent destination ports in netstat output netstat -an | awk '{print $4}' | grep -E ':[0-9]+$' | sed 's/.*://' | sort -n | uniq -c | sort -nr | head -10 ``` Application examples for system monitoring **1. Monitoring failed login attempts:** ```bash # Most frequent failed login attempts by IP grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr | head -10 ``` **2. Analyzing system errors:** ```bash # Most frequent error messages in syslog grep -i error /var/log/syslog | awk '{for(i=6;i<=NF;i++) printf "%s ", $i; print ""}' | sort | uniq -c | sort -nr | head -5 ``` **3. Monitoring storage usage:** ```bash # Directories with the most files find /var -type f 2>/dev/null | dirname | sort | uniq -c | sort -nr | head -10 ``` **4. Analyzing configuration files:** ```bash # Most frequent configuration parameters in Apache grep -v "^#|^$" /etc/apache2/apache2.conf | awk '{print $1}' | sort | uniq -c | sort -nr ```

💡 **Tip for practice:** The combination sort | uniq `-c` | sort `-nr` is so common that many administrators define it as an alias.

```bash # In ~/.bashrc alias frequency='sort | uniq -c | sort -nr' # Usage: awk '{print $1}' access.log | frequency | head -10 ```

⚠️ **Important performance consideration:** For very large files, the combination of `sort` and `uniq` can be memory-intensive. For extremely large data sets, there are specialized tools or you can split the data into smaller chunks.

**Common error sources:** **Forgetting to sort before uniq:** ```bash # WRONG - does not find all duplicates: uniq file.txt # RIGHT: sort file.txt | uniq ``` **Confusing `-d` and `-u`:** ```bash # -d shows only duplicates sort file.txt | uniq -d # -u shows only unique (non-duplicated) lines sort file.txt | uniq -u ``` **Not considering that uniq is case-sensitive:** ```bash # Without -i, "Test" and "test" are treated as different echo -e "Test\ntest\nTest" | sort | uniq test Test # With -i, they are treated as identical echo -e "Test\ntest\nTest" | sort | uniq -i Test ``` Advanced combinations of sort and uniq The true strength of `sort` and `uniq` shines in complex data processing pipelines: **Complex log analysis:** ```bash # Analyze Apache logs: Find the top-10 user agents awk -F'"' '{print $6}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -10 # Analyze error distribution by hour grep "$(date +%d/%b/%Y)" /var/log/apache2/access.log | awk '{print substr($4,14,2)}' | sort | uniq -c | sort -k2n ``` **System resource monitoring:** ```bash # Find processes that are started most frequently ps aux | awk '{print $11}' | sort | uniq -c | sort -nr | head -10 # Analyze memory usage by process name ps aux | awk '{print $11, $4}' | sort | awk '{proc[$1]+=$2} END {for(p in proc) print proc[p], p}' | sort -nr ``` **Network security analysis:** ```bash # Find suspicious connection attempts grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | awk '$1 > 10 {print $1, $2}' | sort -nr ``` The `sort` and `uniq` commands are fundamental tools for every Linux administrator. Mastering them makes it possible to extract structured information from chaotic data sets and is essential for the LPIC-1 certification. Combining both commands in pipelines with other tools like `awk`, `grep`, and `cut` opens up nearly unlimited possibilities for data analysis and system monitoring. ## Counting and formatting text files Analyzing and formatting text files is one of the daily tasks of a Linux administrator. Whether it's about monitoring the size of log files, creating reports, or preparing data for further processing – the commands covered in this section (`wc`, `fmt`, `nl`, and `cut`) are indispensable tools. They perfectly complement the text processing tools already covered and enable precise statistics as well as professional formatting. ### wc – Counting words, lines, and characters What is wc and why is it fundamentally important? The `wc` (word count) command is one of the most basic and frequently used tools for text analysis on Linux systems. It counts lines, words, characters, and bytes in text files or data streams, providing essential statistics for system administration. The strength of `wc` lies in its simplicity and versatility: it can be used for both quick checks and complex data analyses in pipelines. Basic functions of wc (`-l`, `-w`, `-c`, `-m`) **The basic syntax of `wc` is intuitive:** ```bash wc [options] file(s) ``` **Without options, `wc` outputs three values: number of lines, words, and bytes:** ```bash wc beispiel.txt 15 42 256 beispiel.txt ```

💡 **This output means:** 15 lines, 42 words, 256 bytes.

**The most important wc options:** | Option | Description | Practical Example | | --- | --- | --- | | `-l` | Counts only lines | `wc -l /var/log/syslog` | | `-w` | Counts only words | `wc -w document.txt` | | `-c` | Counts only bytes | `wc -c binary_file.dat` | | `-m` | Counts only characters | `wc -m unicode_text.txt` | | `-L` | Shows the length of the longest line | `wc -L configuration.conf` |

⚠️ **Important difference between `-c` and `-m`:** In UTF-8 encoded files, characters can span multiple bytes. While `-c` counts bytes, `-m` counts actual characters:

```bash echo "Café" | wc -c 5 echo "Café" | wc -m 4 ``` The "é" requires two bytes in UTF-8 but is counted as a single character. **Practical application examples for system statistics:** **Log file monitoring:** ```bash # Count lines in a log file wc -l /var/log/apache2/access.log 45823 /var/log/apache2/access.log # Monitor log growth watch "wc -l /var/log/syslog" ``` **Counting system users:** ```bash # Total number of system users wc -l /etc/passwd 42 /etc/passwd # Number of regular users (UID >= 1000) awk -F: '$3 >= 1000 {print}' /etc/passwd | wc -l ``` **Analyzing configuration files:** ```bash # Active configuration lines (without comments and empty lines) grep -v "^\s*#\|^\s*$" /etc/apache2/apache2.conf | wc -l # Average line length in a configuration file awk '{total += length($0); count++} END {print total/count}' /etc/ssh/sshd_config ``` Combination with other commands for complex analyses The true strength of `wc` shows in combination with other commands via pipes: **Process analysis:** ```bash # Number of running processes ps aux | wc -l # Number of Apache processes ps aux | grep apache | grep -v grep | wc -l # Memory usage of all processes (simplified) ps aux | awk 'NR>1 {sum+=$4} END {print sum "%"}' ``` **Network monitoring:** ```bash # Number of active connections netstat -an | grep ESTABLISHED | wc -l # Number of listening ports netstat -ln | grep LISTEN | wc -l ``` **Filesystem analysis:** ```bash # Number of files in a directory (recursive) find /var/log -type f | wc -l # Number of directories find /etc -type d | wc -l # Largest files by line count find /var/log -name "*.log" -exec wc -l {} \; | sort -nr | head -5 ``` **Log analysis:** ```bash # Error rate in log files grep -c "ERROR" /var/log/application.log # Ratio of success to errors in Apache logs awk '$9 ~ /^2/ {success++} $9 ~ /^[^45]/ {error++} END {print "Success:", success, "Errors:", error}' /var/log/apache2/access.log ```

💡 **Tip for monitoring scripts:** `wc` is excellent for simple monitoring scripts:

```bash #!/bin/bash # Simple log monitoring LOGFILE="/var/log/application.log" THRESHOLD=1000 # Current line count CURRENT_LINES=$(wc -l < "$LOGFILE") if [ "$CURRENT_LINES" -gt "$THRESHOLD" ]; then echo "WARNING: Log file has $CURRENT_LINES lines (threshold: $THRESHOLD)" # Trigger notification or rotation here fi ``` ### fmt – Text formatting and line wrapping What is fmt and when is it needed? The `fmt` (format) command is a text formatter that wraps long lines and converts text into uniform paragraphs. It is particularly useful for preparing documentation, emails, or other texts that require consistent formatting. `fmt` follows intelligent rules when wrapping: it tries not to split words and considers punctuation and whitespace. Automatic text formatting The basic usage of `fmt` is simple: ```bash fmt file.txt ``` By default, `fmt` formats text to a line length of 75 characters. 🔧 **Practical example:** ```bash # Create a file with long lines cat > long_text.txt << 'EOF' This is a very long text that should span multiple lines but was written in a single long line, which significantly impairs readability. EOF ``` **Format the text:** ```bash fmt long_text.txt This is a very long text that should span multiple lines but was written in a single long line, which significantly impairs readability. ``` **Important options for line length and indentation** | Option | Description | Example | | --- | --- | --- | | `-w n` | Sets maximum line length to n characters | `fmt -w 60 file.txt` | | `-s` | Only splits long lines, does not join short ones | `fmt -s file.txt` | | `-u` | Uniform spacing (one space between words) | `fmt -u file.txt` | | `-p PREFIX` | Formats only lines with the given prefix | `fmt -p "> " email.txt` | | `-t` | Preserves indentation of the first line | `fmt -t list.txt` | 🔧 **Practical use cases:** **Email formatting:** ```bash # Format an email with quoted lines fmt -p "> " -w 72 email.txt ``` **Formatting code comments:** ```bash # Format comments in a configuration file fmt -p "# " -w 80 config.conf ``` **Preparing documentation:** ```bash # Format README files fmt -w 80 README.txt > README_formatted.txt ``` Use cases for documentation and emails **Automatic documentation formatting:** ```bash #!/bin/bash # Script for formatting Markdown files for file in *.md; do # Format only text paragraphs, not code blocks sed '/^```/!s/.*/fmt -w 80 <<< "&"/e' "$file" > "${file}.formatted" done ``` **Email template creation:** ```bash # Create formatted email templates cat > email_template.txt << 'EOF' Dear Sir or Madam, we would like to inform you about important system maintenance work that will be carried out this coming weekend. EOF fmt -w 72 email_template.txt ```

💡 **Tip for practice:** fmt can be integrated into Vim/Emacs to format text directly in the editor. In Vim: :%!fmt -w 80

### nl – Line numbering What is nl and what is it used for? The `nl` (number lines) command adds line numbers to text files. Unlike simple solutions like `cat -n`, `nl` offers extended options for numbering and can apply various numbering styles. Various numbering options The basic syntax: ```bash nl file.txt ``` **Important nl options:** | Option | Description | Example | | --- | --- | --- | | `-b a` | Numbers all lines | `nl -b a file.txt` | | `-b t` | Numbers only non-empty lines (default) | `nl -b t file.txt` | | `-b n` | No numbering | `nl -b n file.txt` | | `-n ln` | Left-aligned numbering | `nl -n ln file.txt` | | `-n rn` | Right-aligned numbering | `nl -n rn file.txt` | | `-n rz` | Right-aligned with leading zeros | `nl -n rz file.txt` | | `-w n` | Width of the numbering field | `nl -w 4 file.txt` | | `-s STRING` | Separator between number and text | `nl -s ": " file.txt` | | `-v n` | Starting number | `nl -v 100 file.txt` | | `-i n` | Increment (step size) | `nl -i 5 file.txt` | 🔧 **Practical examples:** **a) Numbering code files:** ```bash # Python script with line numbers nl -b a -w 3 -s " | " script.py 1 | #!/usr/bin/env python3 2 | import sys 3 | 4 | def main(): 5 | print("Hello World") ``` **b) Configuration files for debugging:** ```bash # Apache configuration with line numbers for troubleshooting nl -b t -w 4 -s ": " /etc/apache2/apache2.conf ``` Practical application for code and configuration files **Error analysis in configuration files:** ```bash # Show configuration file with line numbers for error messages nl -b a -w 3 /etc/ssh/sshd_config | grep -A 2 -B 2 "Port" ``` **Log analysis with line numbers:** ```bash # Add line numbers to log files for better referencing nl -b a /var/log/apache2/error.log | tail -20 ``` **Creating documentation:** ```bash # Create numbered checklists nl -b a -s ". " checklist.txt ``` ### cut – Extracting columns and fields What is cut and why is it indispensable? The `cut` command extracts specific columns or fields from structured text files. It is particularly valuable for working with CSV files, log files, or other column-based data and allows targeted extraction of specific information. Extracting specific columns from text files The basic syntax of `cut`: ```bash cut [options] file ``` **Important cut options:** | Option | Description | Example | | --- | --- | --- | | `-d CHAR` | Sets field delimiter | `cut -d: -f1 /etc/passwd` | | `-f LIST` | Selects fields | `cut -f1,3,5 table.txt` | | `-c LIST` | Selects character positions | `cut -c1-10 file.txt` | | `-b LIST` | Selects byte positions | `cut -b1-5 file.txt` | | `--complement` | Inverts the selection | `cut -d: -f1 --complement /etc/passwd` | | `--output-delimiter=STRING` | Sets output delimiter | `cut -d: -f1,3 --output-delimiter=" " /etc/passwd` | Important options (`-d`, `-f`, `-c`) **Field-based extraction with `-d` and `-f`:** ```bash # Extract usernames from /etc/passwd cut -d: -f1 /etc/passwd | head -5 root daemon bin sys sync # Extract username and UID cut -d: -f1,3 /etc/passwd | head -5 root:0 daemon:1 bin:2 sys:3 sync:4 ``` **Character-based extraction with `-c`:** ```bash # Extract the first 10 characters of each line cut -c1-10 /var/log/syslog | head -5 # Extract characters 1-3 and 8-12 cut -c1-3,8-12 file.txt ``` **Ranges and lists:** ```bash # Various range notations cut -d: -f1-3 /etc/passwd # Fields 1 to 3 cut -d: -f1,3,5- /etc/passwd # Fields 1, 3 and from 5 onwards cut -d: -f-3 /etc/passwd # Fields from start to 3 ``` Practical applications for CSV files and structured logs **CSV data analysis:** ```bash # Create a sample CSV cat > sales_data.csv << 'EOF' Date,Product,Quantity,Price 2023-05-01,Laptop,5,999.99 2023-05-01,Mouse,20,29.99 2023-05-02,Keyboard,15,79.99 EOF # Extract only product and price cut -d, -f2,4 sales_data.csv Product,Price Laptop,999.99 Mouse,29.99 Keyboard,79.99 # Calculate total revenue (simplified) cut -d, -f3,4 sales_data.csv | tail -n +2 | awk -F, '{sum += $1 * $2} END {print "Total revenue:", sum}' ``` **Apache log analysis:** ```bash # Extract IP addresses and HTTP status codes cut -d' ' -f1,9 /var/log/apache2/access.log | head -5 # Extract only the time from log entries cut -d' ' -f4 /var/log/apache2/access.log | cut -d: -f2-4 | head -5 ``` **Analyzing system configuration:** ```bash # Extract all used shells cut -d: -f7 /etc/passwd | sort | uniq -c # Show users with specific UIDs cut -d: -f1,3 /etc/passwd | awk -F: '$2 >= 1000 {print $1}' ``` Combination with other commands for data extraction **Complex data processing:** ```bash # Analyze process memory usage ps aux | cut -c12-15,65- | sort -nr | head -10 # Extract and analyze network ports netstat -an | grep LISTEN | cut -d' ' -f4 | cut -d: -f2 | sort -n | uniq -c ``` **Log rotation and analysis:** ```bash # Extract timestamps for rotation cut -d' ' -f1-3 /var/log/syslog | tail -1 # Analyze error distribution by hours grep "ERROR" /var/log/application.log | cut -d' ' -f2 | cut -d: -f1 | sort | uniq -c ``` **Configuration management:** ```bash # Extract active Apache modules apache2ctl -M | cut -d' ' -f1 | sort # Analyze SSH configuration grep -v "^#\|^$" /etc/ssh/sshd_config | cut -d' ' -f1 | sort | uniq -c ```

💡 **Tip for complex extraction:** For complex data structures, awk can be more flexible than cut, but cut is often faster and simpler for straightforward column extraction.

⚠️ **Important note:** cut can only work with consistent delimiters. For irregular spacing, awk is often the better choice.

**Common sources of errors:** **Wrong field numbering:** * cut starts at 1, not at 0 **Inconsistent delimiters:** * Mix of tabs and spaces **Forgetting the delimiter:** * Without `-d`, cut uses tabs as default The commands `wc`, `fmt`, `nl`, and `cut` extend your toolkit with important functions for statistics, formatting, and data extraction. They are essential for daily work as a Linux administrator and important components of the LPIC-1 exam. Combining these tools with the previously covered commands makes it possible to solve virtually any text processing task efficiently. ## Regular Expressions – Basics Regular expressions (RegEx) are one of the most powerful and simultaneously most feared technologies in the Linux world. They make it possible to describe and find complex text patterns, making them an indispensable tool for every Linux administrator. Although their syntax may initially appear cryptic, regular expressions open up completely new dimensions of text processing once the basics are learned. ### Concept and importance for text processing Regular expressions are a formal language for describing string patterns[^2]. They serve to search, analyze, and manipulate text according to specific patterns, going far beyond simple text search. While a normal search only looks for exact matches, regular expressions can define flexible patterns that capture various variations of a search term. 🔧 **Practical example:** Instead of searching individually for "Error", "ERROR", "error", and "Err", a regular expression can capture all these variations with a single pattern: ```bash grep -E "[Ee]rr(or)?" /var/log/syslog ``` **The strength of regular expressions lies in their ability to describe abstract patterns:** * Find all email addresses in a text * Identify IP addresses in log files * Recognize date values in various formats * Extract configuration parameters with specific structures Differences between regex variants (BRE, ERE) Linux primarily uses two variants of regular expressions, which differ in their syntax and scope of features: **Basic Regular Expressions (BRE):** * Default format in traditional Unix tools like `grep` and `sed` * More conservative syntax with fewer metacharacters * Certain characters must be "escaped" to retain their special meaning **Extended Regular Expressions (ERE):** * Extended syntax with more features * Available via `grep -E`, `egrep`, `awk`, and modern tools * More intuitive syntax for complex expressions | Feature | BRE (Basic) | ERE (Extended) | | --- | --- | --- | | **Grouping** | `$` `$` | `(pattern)` | | **OR operator** | Not available | `pattern1\|pattern2` | | **Plus quantifier** | `\+` | `pattern+` | | **Question mark quantifier** | `\?` | `pattern?` | | **Curly braces** | `\{n,m\}` | `{n,m}` | 🔧 **Comparison example:** ```bash # BRE syntax (traditional grep) grep 'colou\?r' file.txt # Does not work as expected in BRE grep 'colou*r' file.txt # Searches for "colo" + any number of "u" + "r" # ERE syntax (extended grep) grep -E 'colou?r' file.txt # Finds "color" and "colour" grep -E 'colou+r' file.txt # Finds "colouur", "colouuur", etc. ``` Where are regular expressions used in Linux? Regular expressions are integrated into virtually all important Linux tools: **Command-line tools:** * `grep`, `egrep`, `fgrep` – text search * `sed` – stream editing and text substitution * `awk` – structured text processing * `find` – file search with name patterns * `less`, `more` – search in pagers **Editors:** * `vim`, `emacs` – search and replace * `nano` – basic regex support **Programming languages and scripts:** * Bash, Python, Perl, PHP – built-in regex support * Shell scripts for system administration **System configuration:** * Apache, Nginx – URL rewriting and configuration * Logrotate – filename patterns * Firewall rules – pattern-based filtering

⚠️ **Important note:** Not all tools use the same regex syntax. It is important to know which variant a particular tool supports.

### Basic regex syntax Literal characters and metacharacters Regular expressions consist of two types of characters: **Literal characters:** * Normal letters, numbers, and most special characters * Represent themselves * Example: "abc" finds exactly the string "abc" **Metacharacters:** * Special characters with special meaning * Control the behavior of the regular expression * Basic metacharacters: `. ^ $ * + ? [ ] { } ( ) | \` 🔧 **Practical examples for metacharacters:** **The dot (.) – any character:** ```bash echo -e "cat\ncar\ncup" | grep "ca." cat car ``` **The caret (^) – start of line:** ```bash grep "^root" /etc/passwd root:x:0:0:root:/root:/bin/bash ``` **The dollar sign ($) – end of line:** ```bash grep "bash$" /etc/passwd root:x:0:0:root:/root:/bin/bash username:x:1000:1000:User:/home/username:/bin/bash ``` **Escaping metacharacters:** When you want to search for a metacharacter as a literal character, you must escape it with a backslash: ```bash # Search for a literal dot grep "192\.168\.1\.1" /etc/hosts # Search for a dollar sign grep "\$HOME" script.sh ``` Character classes and ranges ([a-z], [0-9], \d, \w) Character classes make it possible to define a group of characters, one of which can appear at a specific position. **Basic character classes:** ```bash # Single characters [abc] # Matches 'a', 'b', or 'c' [aeiou] # Matches any vowel # Ranges [a-z] # Any lowercase letter [A-Z] # Any uppercase letter [0-9] # Any digit [a-zA-Z0-9] # Alphanumeric characters ``` **Negated character classes:** ```bash [^0-9] # Anything except digits [^aeiou] # All consonants [^a-zA-Z] # All non-letters ``` 🔧 **Practical application examples:** **Finding IP addresses:** ```bash grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" /var/log/apache2/access.log ``` **Email addresses (simplified):** ```bash grep -E "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" contacts.txt ``` **Hexadecimal values:** ```bash grep -E "[0-9a-fA-F]+" system.log ``` **Predefined character classes (in extended regex engines):** | Class | Meaning | Equivalent | | --- | --- | --- | | `\d` | Digit | `[0-9]` | | `\w` | Word character | `[a-zA-Z0-9_]` | | `\s` | Whitespace | `[ \t\n\r\f]` | | `\D` | Non-digit | `[^0-9]` | | `\W` | Non-word character | `[^a-zA-Z0-9_]` | | `\S` | Non-whitespace | `[^ \t\n\r\f]` |

⚠️ **Note:** These predefined classes are not available in all Linux tools. grep and sed do not support them by default, while awk and modern programming languages use them.

Quantifiers (*, +, ?, {n,m}) Quantifiers determine how often a character or group of characters can be repeated: | Quantifier | Meaning | Example | | --- | --- | --- | | `*` | Zero or more | `ab*` finds "a", "ab", "abb", "abbb" | | `+` | One or more (ERE only) | `ab+` finds "ab", "abb", "abbb" | | `?` | Zero or one (ERE only) | `ab?` finds "a", "ab" | | `{n}` | Exactly n times | `a{3}` finds "aaa" | | `{n,}` | At least n times | `a{2,}` finds "aa", "aaa", "aaaa" | | `{n,m}` | Between n and m times | `a{2,4}` finds "aa", "aaa", "aaaa" | 🔧 **Practical examples:** **Log file analysis:** ```bash # Find lines with multiple consecutive spaces grep -E " {2,}" /var/log/syslog # Find IP addresses with optional leading zeros grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" access.log ``` **Validating configuration files:** ```bash # Find lines with at least one equals sign grep -E ".+=.+" config.conf # Find comment lines (optionally with spaces before #) grep -E "^ *#" apache2.conf ``` **System monitoring:** ```bash # Find processes with PID patterns ps aux | grep -E "[0-9]{4,}" # PIDs with at least 4 digits ``` Anchors (^, $) for start and end of line Anchors define positions in text, not characters: **Start of line (^):** ```bash # Find lines starting with "Error" grep "^Error" /var/log/application.log # Find configuration lines not starting with # grep "^[^#]" /etc/ssh/sshd_config ``` **End of line ($):** ```bash # Find lines ending with a period grep "\.$" document.txt # Find shell scripts (lines ending with .sh) ls -la | grep "\.sh$" ``` **Combining anchors:** ```bash # Find empty lines grep "^$" file.txt # Find lines consisting only of whitespace grep "^ *$" file.txt # Find lines with exactly one word grep "^[a-zA-Z]*$" wordlist.txt ``` ### Practical application of regular expressions Usage with grep for extended searches `grep` is the primary tool for applying regular expressions in text search: **Basic grep regex options:** ```bash grep "pattern" file # BRE (Basic Regular Expressions) grep -E "pattern" file # ERE (Extended Regular Expressions) grep -P "pattern" file # PCRE (Perl Compatible, if available) grep -F "pattern" file # Fixed strings (no regex) ``` 🔧 **Extended search examples:** **Log analysis with complex patterns:** ```bash # Find all HTTP errors (4xx and 5xx status codes) grep -E " [45][0-9]{2} " /var/log/apache2/access.log # Find timestamps in various formats grep -E "[0-9]{4}-[0-9]{2}-[0-9]{2}|[0-9]{2}/[0-9]{2}/[0-9]{4}" logfile.txt # Find critical system events grep -E "(CRITICAL|FATAL|EMERGENCY)" /var/log/syslog ``` **Network and security analysis:** ```bash # Find suspicious IP addresses (exclude private ranges) grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" access.log | grep -vE "^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)" # Find failed SSH logins with IP addresses grep -E "Failed password.*from ([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/auth.log ``` Usage in `sed` for complex replacements `sed` uses regular expressions for both search and replacement: **Extended sed operations:** ```bash # Replace all IP addresses with "XXX.XXX.XXX.XXX" sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/XXX.XXX.XXX.XXX/g' logfile.txt # Extract domain from email addresses sed -E 's/.*@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*/\1/' email_list.txt # Reformat dates (YYYY-MM-DD to DD.MM.YYYY) sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3.\2.\1/g' file.txt ``` **Using backreferences:** ```bash # Swap first and last name sed -E 's/([A-Za-z]+) ([A-Za-z]+)/\2, \1/' names.txt # Add quotation marks around words sed -E 's/([a-zA-Z]+)/"\1"/g' wordlist.txt ``` Typical patterns for system administration **IP addresses and networking:** ```bash # IPv4 address (simplified) ^([0-9]{1,3}\.){3}[0-9]{1,3}$ # IPv4 with port specification ^([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]{1,5}$ # MAC address ^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$ ``` **Email addresses:** ```bash # Simple email validation ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ # Email extraction from text [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} ``` **Date formats:** ```bash # ISO date (YYYY-MM-DD) ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ # German date (DD.MM.YYYY) ^[0-9]{2}\.[0-9]{2}\.[0-9]{4}$ # American date (MM/DD/YYYY) ^[0-9]{2}/[0-9]{2}/[0-9]{4}$ ``` **System-specific patterns:** ```bash # Linux username ^[a-z_][a-z0-9_-]*$ # Absolute paths ^/([a-zA-Z0-9._-]+/?)*$ # Process IDs ^[0-9]+$ ``` 🔧 **Practical example – complete log analysis:** ```bash #!/bin/bash # Comprehensive log analysis script with regex LOGFILE="/var/log/apache2/access.log" echo "=== Apache Log Analysis ===" # 1. Count unique IP addresses echo "Unique visitors:" grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" "$LOGFILE" | sort | uniq | wc -l # 2. Top 10 IP addresses echo -e "\nTop 10 IP addresses:" grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" "$LOGFILE" | sort | uniq -c | sort -nr | head -10 # 3. Analyze HTTP errors echo -e "\nHTTP errors (4xx and 5xx):" grep -E " [45][0-9]{2} " "$LOGFILE" | awk '{print $9}' | sort | uniq -c | sort -nr # 4. Find suspicious user agents echo -e "\nSuspicious user agents (bot patterns):" grep -iE "(bot|crawler|spider|scraper)" "$LOGFILE" | awk -F'"' '{print $6}' | sort | uniq -c | sort -nr | head -5 ``` Common pitfalls and how to avoid them **Typical regex errors:** **a) Forgetting to escape:** ```bash # WRONG – dot is interpreted as "any character" grep "192.168.1.1" /etc/hosts # RIGHT – dot is treated as a literal dot grep "192\.168\.1\.1" /etc/hosts ``` **b) Greedy vs. non-greedy quantifiers:** ```bash # Greedy – finds the longest possible match echo 'content' | sed 's/<.*>/REPLACED/' # Result: REPLACED # Solution: Use a more specific pattern echo 'content' | sed 's/<[^>]*>/REPLACED/g' # Result: REPLACEDcontent ``` **c) Confusing BRE and ERE:** ```bash # BRE (does not work as expected) grep 'error+' logfile.txt # Searches for "error" followed by "+" # ERE (correct) grep -E 'error+' logfile.txt # Searches for "erro" + one or more "r" ``` **d) Anchor misunderstandings:** ```bash # WRONG – ^ and $ refer to the entire line echo "start middle end" | grep "^middle$" # Finds nothing # RIGHT – do not use anchors for substrings echo "start middle end" | grep "middle" # Finds "middle" ``` **Tips for effective regex usage:** **Test your regex step by step:** * Start with simple patterns and gradually extend them **Use online regex testers:** * Tools like regex101.com help with debugging **Document complex expressions:** * Add comments to explain the logic **Prefer readability:** * Multiple simple regex are often better than one complex one

⚠️ **Performance considerations:** Very complex regular expressions can be slow, especially with large files. In such cases, several simple grep commands in a pipeline can be more efficient.

Regular expressions are a powerful tool that becomes more natural with practice and experience. They are essential for the LPIC-1 certification and indispensable for efficient system administration. The investment in learning the basics pays off with considerable time savings and expanded possibilities in text processing.

💡 **Tip for the LPIC-1 exam:** For the LPIC-1 exam, it is not memorization but understanding and practical application of text processing commands that matters. Practice intensively with tools like `grep`, `sed`, `awk`, `sort`, and `cut` on real examples and regularly work with the man pages. Many exam questions test typical invocation options, regex basics, and combination possibilities – practical experience helps you answer even detailed questions confidently.

## Why is this important for LPIC-1? Mastering shell-based text processing tools is a central examination objective of the LPIC-1 certification. Many tasks in the exam – and in everyday work as an administrator – require you to quickly analyze, filter, and specifically process large amounts of system and log data. The ability to use tools like `grep`, `sed`, `awk`, `sort`, or `cut` purposefully forms the foundation for efficient error analysis, system monitoring, and automation. Without this knowledge, sustainable Linux administration is hardly possible.

💡 **Note:** Resources and exam information can be found in our fundamentals module: [LPIC-1: Basic navigation and filesystem commands](/en/lpic-1-serie/lpic-1-basic-navigation-and-filesystem-commands){.badge-link-text}.

## Command Reference (Cheatsheet) | Command | Key Syntax / Options | Description & LPIC-1 Exam Relevance | |---|---|---| | `grep` | `grep -i -v -n -E "pattern" file` | Searches text for patterns (`-i` ignores case, `-v` inverts, `-n` line number) | | `egrep` | `egrep "pattern1\|pattern2" file` | Extended regular expressions (ERE), equivalent to `grep -E` | | `fgrep` | `fgrep "fixed.string" file` | Fast search for fixed strings (no regex), equivalent to `grep -F` | | `sed` | `sed -i 's/old/new/g' file` | Non-interactive stream editor (`s` substitute, `d` delete, `p` print, `-i` in-place) | | `awk` | `awk -F: '$3 >= 1000 {print $1}' file` | Pattern-driven programming language for column-based text processing (`-F` delimiter) | | `cut` | `cut -d: -f1,7 /etc/passwd` | Cuts columns (`-f`) with delimiter (`-d`) or character ranges (`-c`) from lines | | `paste` | `paste -d, file1 file2` | Joins lines of corresponding files horizontally with delimiter | | `sort` | `sort -n -r -k2 -t: file` | Sorts lines (`-n` numeric, `-r` reverse, `-k` field number, `-t` delimiter) | | `uniq` | `uniq -c -d -u file` | Filters consecutive duplicates (`-c` counts, `-d` only duplicates, `-u` only uniques) | | `wc` | `wc -l -w -c file` | Counts lines (`-l`), words (`-w`) and bytes/characters (`-c`/`-m`) | | `tr` | `tr 'a-z' 'A-Z' < file` | Translates or deletes characters (`-d` delete, `-s` squeeze consecutive) | | `fmt` | `fmt -w 75 file` | Formats and wraps paragraphs to a maximum line width (`-w`) | | `nl` | `nl -ba file` | Numbers lines of a file (`-ba` all lines including empty lines) | | `pr` | `pr -3 -h "Title" file` | Prepares text files for printing and column layout | | `expand` | `expand -t 4 file` | Converts tabs to spaces (`unexpand` converts spaces to tabs) | ## Further Resources | Resource | Description | |---|---| | [GNU Grep Manual](https://www.gnu.org/software/grep/manual/){.badge-link-text} | Official documentation on standard, extended, and Perl-compatible regular expressions | | [GNU Sed Stream Editor](https://www.gnu.org/software/sed/manual/){.badge-link-text} | Complete reference manual for addressing, pattern matching, and transformation with `sed` | | [GNU Awk User's Guide](https://www.gnu.org/software/gawk/manual/){.badge-link-text} | The authoritative handbook for the AWK programming language and structured data processing | | [GNU Coreutils Text Utilities](https://www.gnu.org/software/coreutils/manual/html_node/Text-utilities.html){.badge-link-text} | GNU specification for `sort`, `uniq`, `cut`, `paste`, `tr`, `wc`, `fmt`, and `nl` | | [LPI: LPIC-1 Exam 101 Objectives](https://www.lpi.org/our-certifications/exam-101-objectives/){.badge-link-text} | Official learning objectives for Topic 103: GNU and Unix Commands (103.2, 103.3, 103.7) | ## Conclusion Mastering UNIX text tools and regular expressions forms the foundation for efficient system analysis, log file evaluation, and automation in Linux. Through the seamless interplay of `grep`, `sed`, `awk`, `sort`, `uniq`, and `cut` via pipes, you can transform even complex data streams and configurations with just a few precise command lines without external scripting languages.

💡 **Practice tip:** When building complex pipelines, always think modularly: test each filter step individually on a small sample (e.g., `head -n 20`) before writing the result with `sed -i` or redirecting it to production files.

In the next part of our LPIC-1 series, we tackle process management and system monitoring: [LPIC-1: Managing and monitoring processes](/en/online-courses/lpic-1-serie/2025/2025-05-29-lpic-1-managing-and-monitoring-processes){.badge-link-text} – where you'll get to know `ps`, `top`, `kill`, `nice`, and signal handling in detail. 👉 **To the course overview:** [All LPIC-1 articles & modules](/en/category/lpic-1-serie){.badge-link-text}