---
id: 2024-11-07-devops-fundamentals-in-modern-software-development
slug: devops-fundamentals-in-modern-software-development
title: "DevOps fundamentals: an introduction to modern software development"
excerpt: "Core concepts, tools and best practices for DevOps, from continuous integration through to automated deployment pipelines."
date: "2024-11-07T09:00:00+02:00"
updated: "2024-11-07T09:00:00+02:00"
author:
name: "Erik van Hooven"
handle: "evanhooven"
category: ["devops"]
tags: ["devops", "linuxadmin", "systemadmin", "ubuntu", "docker", "docker-compose", "github-actions", "prometheus", "grafana", "loki", "promtail", "nginx", "git", "pull-requests", "code-reviews", "ci-cd", "observability", "calms"]
reading_time: 79
toc: true
---
DevOps represents a fundamental shift in IT. For a long time developers and operations teams worked in separate silos, which led to slow release cycles, finger-pointing when things broke, and inefficient processes. Today DevOps joins development and operations into a shared responsibility across the entire lifecycle of an application. Teams ship more often, more reliably, and with far fewer manual interventions in the process.
π‘ **Tip:** Many teams report markedly shorter lead times and higher stability as soon as they start applying the principles consistently and putting automation at the centre.
The approach grew out of real pain in software development and is now standard in modern organisations of every size. The sections below give you a clear, practical introduction to the fundamentals: how you as a Linux administrator or junior DevOps engineer can fold the concepts into day-to-day work, and which concrete benefits that creates for you.
β οΈ **Important:** Before you dive into the world of DevOps, you should have basic experience with Linux and containers. For that we recommend this article: [Linux Administration: Virtualisation and VM Management](/en/linux-administration/linux-administration-virtualisierung-und-vm-management){.badge-link-text}.
You will work through the DevOps fundamentals step by step. You will see how modern software development and IT operations work hand in hand, and how you can apply the key tools and practices in your project. The material builds on the foundations of container technology and shows how it is used effectively in a DevOps context.
π§ **Practical example:**
A practical start often comes with Docker. Create a simple test environment by running the following command:
```bash
docker run -d --name webtest -p 8080:80 nginx:alpine
```
You can then reach the running web server at http://localhost:8080. This simple step already demonstrates the reproducibility and isolation that characterise DevOps workflows. You see immediately how containers can keep development, test and production consistent.
β **Watch out:** A widespread mistake is to treat DevOps as a purely technical topic and ignore the cultural changes.
With those first contact points you already have a solid foundation for the next steps. The next section gets to the core of what DevOps is and which principles sit behind it.
## What is DevOps?
### Definition and how it differs from classic development
DevOps is the close coupling of software development and IT operations into a shared area of responsibility across the entire lifecycle of an application. The term is formed from the English words Development and Operations and describes a way of working in which the previously strictly separated teams no longer work in isolation. Instead, developers and administrators share the same goals, tools and processes so that code reaches production faster, more safely and more reliably. In practice that means you as a Linux administrator or junior DevOps engineer think from the start about how the code will later run on servers, and developers understand the requirements that operations imposes.
The distinction from classic development and operations is fundamental.
In the traditional world, often organised around the waterfall model, there are clear phases and handovers. Developers write the code in a separate environment, test it locally and hand the finished package to the operations team by ticket or email. That team then installs it by hand on physical or virtual servers, adjusts configurations and starts the service. Every step is sequential, long waits are normal, and defects are only found late. That often leads to the infamous βit works on my machineβ problem, because the development environment never matches production exactly. Operations here is reactive β problems are fixed when they appear, and responsibility sits clearly separated with the respective teams.
DevOps dissolves exactly these silos.
Instead of handovers there are shared pipelines in which build, test and deployment run automatically. Every commit in the version-control system automatically triggers processes that keep the code in a deployable state. Responsibility is shared: a developer already thinks about scalability and monitoring while writing code, an administrator helps early with the definition of infrastructure as code. The result is shorter release cycles, fewer manual interventions and higher stability. For beginners it is important to understand that DevOps is not a pure technology, but a culture of shared learning and continuous improvement.
π‘ **Tip:** Many beginners make the mistake of seeing DevOps only as a tool set. In reality everything starts with the realisation that collaboration delivers more than isolated specialisation.
**A simple diagram makes the difference tangible:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CLASSIC VS. DEVOPS PROCESS β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Classic flow (waterfall model and silos): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Dev (Code) βββΊ Handover (Ticket) βββΊ Ops (Deploy) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β * High latency, manual handovers and many errors β
β β
β Modern DevOps flow (shared responsibility): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Dev + Ops βββΊ Automated CI/CD pipeline βββΊ Prod β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β * Continuous, reproducible and automated β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram shows you clearly where the classic split costs `time` and `quality`. In practice you see that immediately once you compare a manual deploy with an automated one.
β οΈ **Important:** Pay particular attention to the fact that the classic split often leads to blame β DevOps replaces that with shared metrics and learning loops.
For beginners it is worth looking briefly at the historical development, even if it is not part of the pure definition. Classic operations focused on stability and availability, while development was after features and speed. Those opposing goals created conflict. DevOps harmonises them by putting automation and standardisation at the centre. You as a Linux administrator benefit especially, because you suddenly no longer only fight fires, but shape infrastructure proactively.
β **Watch out:** A widespread beginner mistake is to believe DevOps simply means βuse more Dockerβ. The distinction sits deeper: it is about the whole mindset shift from manual processes to reproducible, measurable workflows.
π§ **Practical example:**
To experience the difference concretely, start a small test on your Linux system. First the classic manual variant, as it used to be common. Create a simple HTML file and copy it by hand onto a test server:
```bash
echo 'Hello from classic Ops
' > index.html
scp index.html admin@your-test-server:/var/www/html/
ssh admin@your-test-server 'systemctl restart nginx'
```
That works, but it is error-prone and not reproducible.
**Now the DevOps variant with Docker, which ensures consistency from the start:**
```bash
cat > Dockerfile <it works differently on my machine, because everything is containerised and versioned. This small step shows why the new way of working is so powerful for beginners.
With this understanding of the definition and the clear distinction from the classic world you are ready for the further principles that make up DevOps.
### Historical development: the three waves of DevOps
The historical development of DevOps shows you as a beginner very clearly why todayβs way of working emerged and how it has changed step by step. Long before the term was even coined, rigid dividing lines defined day-to-day IT. In the 1980s and 1990s development and operations worked in strict separation. Developers wrote code in their own world, handed the finished product to the operations team by ticket or by hand, and that team then looked after installation, configuration and maintenance on physical servers on its own. Every switch between the teams took days or weeks.
Defects only appeared in production, because the environments were never identical. That led to frustration, blame and extremely long release cycles. As a Linux administrator you have probably already experienced how a new web server was set up and configured by hand β with lots of small scripts and a great deal of copy-and-paste work.
The Agile movement from 2001 brought the first changes.
With the Agile Manifesto, shorter iterations and more frequent releases moved into the foreground. Yet the silos between Dev and Ops remained. Only in 2008 did something start to change. [Patrick Debois](https://www.patrickdebois.com/){.badge-link-text}, a Belgian IT expert, organised the first DevOpsDays conference in Ghent in 2009. Developers and operations people met there deliberately to talk about better collaboration. The term DevOps arose there almost by accident and spread rapidly via blogs and conferences. Suddenly it was no longer only about new tools, but about rethinking the entire collaboration. For you as a beginner it is important to understand that DevOps was not a sudden invention, but a logical answer to years of real problems in daily work.
This development can be divided usefully into three waves that build on one another and help you place the current state. Each wave has its own focus and has shaped practice lastingly.
The first wave was almost exclusively about technology and automation.
From around 2010, teams concentrated on accelerating repeatable processes with tools. Configuration-management systems such as [Puppet](https://puppet.com/){.badge-link-text} and [Chef](https://www.chef.io/){.badge-link-text} came onto the market, later [Ansible](https://www.ansible.com/){.badge-link-text}. Instead of configuring servers by hand, you described the desired state in code and let the tools do the work. CI/CD pipelines with [Jenkins](https://www.jenkins.io/){.badge-link-text} or later [GitLab CI](https://docs.gitlab.com/ee/ci/){.badge-link-text} made sure every commit was built and tested automatically.
[Docker](/en/tag/docker){.badge-link-text} appeared in 2013 and made containers popular, so applications finally ran identically in development, test and production. For many Linux admins this wave was a genuine liberation, because they got away from hours of manual tuning. The underlying silo problem remained, though β the tools helped, but the teams still were not really working together.
The second wave, which developed from around 2014, put cultural change at the centre.
Suddenly it was no longer only about tools, but about people and collaboration. Books such as [The Phoenix Project](https://www.thephoenixproject.com/){.badge-link-text} and [The DevOps Handbook](https://itrevolution.com/the-devops-handbook/){.badge-link-text} made it clear that shared responsibility, common goals and Lean principles such as continuous improvement are decisive. Teams began to take joint responsibility for the entire lifecycle. Shared on-call rotations, joint retrospectives and the idea You build it, you run it became standard. For beginners that means you learn not only to manage servers, but to talk actively with developers and improve processes together. The second wave showed that technology alone is not enough if the culture is missing.
The third wave, running since around 2018, integrates measurement, continuous improvement and a tight connection to the business.
Observability with [Prometheus](https://prometheus.io/){.badge-link-text}, [Grafana](https://grafana.com/){.badge-link-text} and [Distributed Tracing](https://opentelemetry.io/){.badge-link-text} became central, so teams could not only react but anticipate problems. DevSecOps brought security into the process from the start. GitOps with tools such as [ArgoCD](https://argoproj.github.io/){.badge-link-text} made infrastructure declarative and versioned. Cloud-native architectures and serverless models widened the spectrum. This wave ensures that DevOps does not only work internally, but creates real value for the organisation β faster time-to-market, lower downtime and better scalability.
π‘ **Tip:** Many beginners skip the historical perspective and jump straight to tools. Yet it is precisely the understanding of the waves that helps you see why certain practices matter today and where you still have gaps in your own environment.
The three waves build on one another and show a clear progression from pure technology through culture to measurable business success. As a junior DevOps engineer or Linux administrator you often see elements of all waves at the same time in projects today. The first wave gave you the tools, the second wave the right stance, and the third wave the ability to work in a truly sustainable way.
β οΈ **Important:** Watch that you do not get stuck in the first wave. Many teams install Jenkins and Docker and believe they are already doing DevOps. Without the cultural changes of the second wave the silos remain, and the real benefit never arrives.
A simple diagram makes the historical shift even more tangible:
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β HISTORICAL DEVELOPMENT: THE THREE WAVES β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Before DevOps (until 2008): rigid silos and long cycles β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Silos βββΊ Waterfall βββΊ Manual handovers β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Wave 1 (2010β2013): tools and automation β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β CI/CD + IaC + Docker βββΊ Faster deployments β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Wave 2 (2014β2017): culture and collaboration β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Shared responsibility + Lean βββΊ Shared goals β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Wave 3 (from 2018): measurement, observability, GitOps β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β DevSecOps + GitOps + Observability βββΊ Continuity β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram makes it clear how each wave builds on the previous one and extends practice.
β **Watch out:** A typical beginner mistake is to treat the waves as closed phases that you walk through one after another. In reality they often run in parallel, and you have to consider all aspects at the same time to succeed.
π§ **Practical example:**
To retrace the development yourself, you can run a small historical comparison on your Linux system. First you simulate the manual world before the first wave. Create a simple configuration file and copy it by hand onto a container that represents an old web server:
```bash
echo 'Manual deploy before DevOps
' > index.html
docker run -d --name oldstyle -p 8081:80 -v $(pwd)/index.html:/usr/share/nginx/html/index.html nginx:alpine
```
Now you see how error-prone that is β every change requires manual copying and restarting. For the first wave you build a simple Dockerfile that shows the automation:
```bash
cat > Dockerfile <speed, stability, security and scalability β are not abstract; they arise from the combination of culture, automation and shared responsibility. As a beginner it helps enormously to understand each promise on its own, so you see why the effort is worth it and how you can implement it in your environment. Working through each promise in turn makes the benefits concrete so you can apply them directly.
Speed is the first and often the most noticeable promise.
In the classic operations world you as an admin often waited for hours until developers had a build ready, then copied it by hand, adjusted configurations and restarted the service. With DevOps that changes radically. A developer pushes code into `Git`, the `CI/CD pipeline` builds automatically, tests, packs into a container and `deploys`. The time from idea to production drops from days to minutes. For you that means less firefighting and more time for strategic work. You can roll out features faster and gather feedback from the business sooner. The pipeline becomes your daily helper that takes over routine tasks. You learn how a single commit triggers a chain of steps that used to be coordinated by hand. As a Linux admin you configure the pipeline so that it runs on your server or in the cloud and reports immediately whether everything is green.
π‘ **Tip:** Many beginners underestimate how much this speed lifts team morale, because feedback loops become much shorter and successes become visible faster. You will see how satisfying it is when a fix goes live in minutes instead of only the next day.
Stability comes as the second promise.
DevOps makes sure that fast deploys do not lead to instability. Through automated tests in the pipeline you know before the deploy whether the code works. Techniques such as [Blue-Green Deployment](https://www.redhat.com/en/topics/devops/what-is-blue-green-deployment){.badge-link-text} allow the new version to run in parallel with the old one and to switch immediately if there are problems. Canary releases roll out only to a small part of the users. As a Linux admin you learn to configure health checks and rolling updates so that the service always stays available.
Stability arises through reproducibility β the same Docker image in every environment. That reduces downtime considerably and gives you confidence in every change. Previously you may have waited at night until a manual deploy was finished and then hoped that nothing would break. Today everything runs in a controlled way and you can go to sleep.
Then comes security, which is often underestimated.
Previously security was a separate team that checked at the end. DevSecOps brings it into every step. On every build the pipeline scans the code for vulnerabilities, checks dependencies and container images. As a beginner you can build tools such as [Trivy](https://aquasecurity.github.io/trivy/latest/){.badge-link-text} or [OWASP ZAP](https://www.zaproxy.org/){.badge-link-text} into your pipelines. Shift Left means finding problems early, before they reach production. It protects not only the app, but the entire system against attacks. You learn that security is not an add-on, but part of the daily workflow. As a Linux admin you integrate scans directly into your Docker builds and make sure that no known CVEs land in the images. That gives you a good feeling when the service goes online.
Scalability rounds off the promises.
DevOps makes it easy to grow with a growing user base. With [Infrastructure as Code](https://www.redhat.com/en/topics/devops/what-is-infrastructure-as-code){.badge-link-text} you describe the required infrastructure in files that you version and roll out automatically. Containers make it possible to replicate services on demand. In practice that means you scale from one server to ten without building new [VMs](https://www.redhat.com/en/topics/virtualization/what-is-a-virtual-machine){.badge-link-text} by hand. For beginners that is especially helpful, because you test locally with `Docker Compose` and later scale in the cloud. Costs stay controllable, performance adapts. You see how a simple command increases the number of containers and the load balancer automatically follows. That is the moment when you realise how DevOps enables real scalability without chaos.
π‘ **Tip:** A common stumbling block for beginners is to believe that one of the promises can be reached in isolation. In reality they reinforce each other β fast deploys without stability lead to chaos, and scalability without security is risky.
To see the promises in practice and how they reinforce each other, a concrete example follows.
π§ **Practical example:**
A simple way to test the four promises is a scalable Nginx service with Docker Compose. Create the file docker-compose.yml as follows to show speed through a fast start, stability through health checks, security through the image, and scalability through replicas:
```yaml
version: '3.8'
services:
web:
image: nginx:alpine
ports:
- "8080:80"
deploy:
replicas: 1
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost || exit 1"]
interval: 10s
timeout: 5s
retries: 3
```
**Start the environment with the command:**
```bash
docker compose up -d
```
You can reach the service immediately at `http://localhost:8080`. To demonstrate scalability, you scale up:
```bash
docker compose up --scale web=4 -d
```
For speed you change the configuration and redeploy with a single command.
**To check security, you scan the image:**
```bash
docker run --rm aquasec/trivy image nginx:alpine
```
The example shows you how, in a few minutes, you build an environment that fulfils all four promises. You see the reproducibility, the speed of the start and the ability to scale under load. As a beginner you can try this on your laptop and understand the principles immediately. Try it and watch how stable the service stays when you change the replicas.
β **Watch out:** A typical mistake is to ignore the pipeline and believe that manual scaling is enough. That leads to inconsistent environments and lost time.
With these four core promises in mind it becomes clear why DevOps is so valuable for beginners like you. It is not about more work, but about smarter, more effective work. You save time, reduce stress and build systems that grow with you, without you lying awake every night.
With those promises in mind, the core principles come next in detail.
## The core principles
### The CALMS framework in detail
The CALMS framework is the actual engine of DevOps and starts deliberately with Culture, because without that cultural change all further steps remain ineffective. As a beginner or junior DevOps engineer who has so far mainly managed Linux servers, you learn the decisive difference here: silos are replaced by genuine collaboration. Instead of developers writing their code and handing it to you as the admin by ticket, you jointly take responsibility for the entire lifecycle of an application. You bring your knowledge of servers, resources and stability in early, while the developers understand the requirements that operations imposes. That dissolves the classic βit works on my machineβ problem and creates a shared goal: stable, fast and secure software.
In the classic silo world everyone has their own world.
The developer concentrates on features, you as a Linux administrator on availability and performance. The handover often happens late and is error-prone. You receive an artefact that suddenly needs different dependencies on your test server or causes memory problems. The result is long debugging sessions, night-time deploys and mutual blame.
**Culture breaks these walls down.** You work in a shared rhythm, talk regularly about current tasks and share knowledge actively. That does not mean everyone can do everything, but that everyone understands and respects the otherβs perspective. You learn to review pull requests not only for code, but also for Dockerfiles or configuration files. The developer learns why a non-root user in containers matters and how health checks stabilise operations.
Culture creates psychological safety.
Teams can talk openly about mistakes without fear of blame. Instead of Who made the mistake? you ask What can we learn from the incident?. Blameless postmortems become the standard. Together you reconstruct the timeline of an incident, identify systemic causes and derive concrete improvements. For beginners that is especially valuable, because you can suggest without fear that a particular Docker command should be adjusted or that logging needs to be integrated into the app. The culture encourages continuous learning and turns isolated specialists into a real team.
In practice you start the implementation with small, concrete steps.
Set up a shared chat channel in which quick questions are possible instead of long tickets. Organise weekly sync meetings in which you talk about current challenges and prioritise together. Use a single Git repository for code and infrastructure. That way the developer can propose a change to the Dockerfile and you give direct feedback on security or performance. Pairing sessions in which you set up a new service configuration together accelerate knowledge exchange enormously. You show how to monitor resources with Linux commands, the developer explains the business logic of the app to you.
The benefits for you as a Linux administrator are immediately noticeable. You spend less time firefighting, because problems are recognised earlier and solved together. Your expertise already flows into development, so the systems are more robust from the start. The work becomes more varied and you constantly learn new aspects of the applications. At the same time the developers give you insight into their world, so you can make better decisions for the infrastructure.
π‘ **Tip:** Many beginners underestimate how much Culture makes the difference between a working tool set and a truly living DevOps process. It is the invisible glue that holds everything together.
Collaboration also makes dealing with containers and CI/CD pipelines much more effective. When you jointly decide which base images are used or how secrets are handled, consistent environments arise from development through to production. You as the admin can contribute early which Linux-specific optimisations make sense, for example special volume mounts or resource limits.
β οΈ **Important:** Watch that the cultural change takes time and does not work overnight. Many teams jump onto new tools without changing the collaboration, and later wonder why the results fail to appear.
**To make the difference between silos and genuine collaboration even clearer:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ORGANISATION: SILOS VS. SHARED RESPONSIBILITY β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Classic silo structure: β
β βββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββ β
β β Development (Dev) β Operations (Ops) β β
β βββββββββββββββββββββββββββββΌββββββββββββββββββββββββββ€ β
β β β’ Write and test code β β’ Receive the ticket β β
β β β’ Push features β β’ Deploy manually β β
β β β’ No server ownership β β’ Configure servers β β
β βββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββ β
β β
β DevOps culture (shared team and ownership): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Dev + Ops (shared team with shared ownership) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ β
β β β’ Code and infrastructure in shared Git repos β β
β β β’ Shared CI/CD pipelines and code reviews β β
β β β’ Shared on-call rotation and blameless postmortems β β
β β β’ Continuous knowledge sharing and feedback β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram shows clearly how the flow moves from separated to connected and where friction losses disappear.
β **Watch out:** A widespread beginner mistake is to believe that a shared tool such as Slack or a shared repo already creates Culture. Without the stance of shared responsibility the silos remain and the tools are only used half-heartedly.
π§ **Practical example:**
A concrete entry into Culture succeeds with a shared Docker project in a shared Git repository. You work with a developer on a simple Node.js application. First you clone the repo together on your Linux machine:
```bash
git clone https://github.com/your-team/devops-culture-example.git
cd devops-culture-example
```
The developer has created a basic Dockerfile. You open the file and add your Linux and Ops experience β a non-root user, a health check and optimised resource limits.
**The finished file looks like this:**
```dockerfile
FROM node:20-alpine
RUN addgroup -g 1001 -S nodejs && adduser -S -u 1001 nodejs
COPY --chown=nodejs:nodejs . /app
WORKDIR /app
RUN npm ci --only=production
USER nodejs
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -f http://localhost:3000/health || exit 1
CMD ["node", "server.js"]
```
**You commit the change with a clear message and push it:**
```bash
git add Dockerfile
git commit -m "Ops improvement: added non-root user, health check and resource limits"
git push origin main
```
The developer sees the pull request, you discuss the changes in the shared chat and they merge. Then you build the container together:
```bash
docker build -t culture-example .
docker run -d --name culture-app -p 3000:3000 --restart unless-stopped culture-example
```
**You test the health check together with:**
```bash
curl -f http://localhost:3000/health
```
This simple shared workflow shows you live how knowledge is exchanged, responsibility is shared and the app becomes more stable from the start. The developer learns important operations aspects, you understand the application better. That is exactly how genuine collaboration starts instead of silos.
Through such practical steps Culture becomes lived reality and you quickly notice how much more smoothly everything runs.
### Automation: why everything repeatable must be automated
Once Culture has created the foundation for collaboration, the Automation principle becomes the next logical step in the CALMS framework. Automation means that you pack all repeatable tasks into `scripts`, `pipelines` or `declarative configurations` so that they run automatically and consistently. As a beginner you take the time to identify manual steps and replace them with code. That not only saves time, it also eliminates many error sources that arise in daily Linux administration. In practice you quickly see why automation is indispensable. Many tasks such as setting up a new server, building Docker images, deploying applications or checking logs are repetitive.
If you execute these by hand, you risk inconsistencies and human error. A forgotten command or a wrong configuration can lead to outages. Automation provides reproducibility. The code defines the desired state, and the tool makes sure it is reached. That is why everything repeatable must be automated. It creates consistency across development, test and production and makes the process traceable for the entire team.
For you as a Linux administrator, automation is a game changer.
Instead of configuring every server individually, you use tools such as Ansible to write playbooks that install packages, create users and start services. You store the configuration in `Git` and can roll it out on demand. That reduces the time from hours to minutes and makes the process auditable. As a beginner you start with simple [Bash scripts](/en/category/bash-grundlagen){.badge-link-text} to automate routine tasks. You learn how to work with `if` conditions and `loops` to check conditions. Later you integrate that into `CI/CD pipelines` that are triggered on every `commit`.
The benefits are many. First you save time for more important tasks such as optimising the infrastructure or introducing new technologies. Second, reliability rises because automated tests run and only valid changes get through. Third, scalability becomes easier, because you can start several instances with one command. Fourth, the code serves as documentation, so new team members can learn quickly. Integrating automation into your workflow starts with an analysis of your daily work.
Note down which tasks you do more than once a week. That can be updates, backups or log rotation. Then write a script for it. On Linux you use `cron` for time-controlled execution or `systemd` for modern services. In a DevOps context you go further and connect it with `Git hooks` or `webhooks` in your `CI/CD toolchain`.
A good example is automating a Docker deployment.
You write a script that runs the build, tags the image and starts the container. That ensures every `deploy` is identical. Automating `Infrastructure as Code` is a milestone. With tools such as [Terraform](#){.badge-link-text} you describe your VMs, networks and storage in HCL and apply them with one command. That is declarative, which means you say what you want, not how. That is ideal for beginners, because it leaves less room for error. You can version the files and roll back when needed. In the Linux world you also automate classic tasks such as firewall rules with `ufw` or `firewalld`, user management with `useradd` commands in scripts, or backup routines with `rsync` and `tar`. Each of these tasks becomes reproducible and runs without your constant intervention.
π‘ **Tip:** A helpful hint for beginners is always to consider idempotence. That means the script can be executed several times without changing the state. That provides safety on repeated runs and prevents unwanted side effects.
**To illustrate the difference:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PROCESS COMPARISON: MANUAL VS. AUTOMATED β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Manual process (error-prone and slow): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β SSH βββΊ apt update βββΊ edit config βββΊ restart βββΊ ?β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β * High time cost, configuration drift, no audit β
β β
β Automated process (reproducible and fast): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β βGit push βββΊ CI/CD βββΊ Build βββΊ Test βββΊ Healthcheckβ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β * Fully versioned, auditable and automatically tested β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram makes it clear how many steps are automated and how the process is simplified. Practical implementation is the best way to understand the principle. As a beginner it is important to proceed step by step. Start with a simple task such as automating a Docker build. Write a Dockerfile and integrate it into a script that runs the build and the run. That gives you an immediate sense of success and shows the benefit. Later you extend it with tests and health checks so that the pipeline becomes complete.
Automation also supports compliance requirements.
Every change is traceable in the `Git log`, which makes audits easier. You no longer have to keep manual protocols. In practice you see automation in tools such as Docker, where you start a container with one command that is identical to other environments. Or in CI/CD, where `GitHub Actions` or `GitLab CI` take over the rest. All of that builds on Culture, because the team jointly decides which processes are automated and how the pipelines should look.
β οΈ **Important:** Watch that you never automate something you do not fully understand. Otherwise you can create problems that are hard to debug, hard to find, and in the worst case lead to longer outages.
In Linux administration you also automate monitoring setup. With Prometheus and Grafana you can store configurations as code and deploy them automatically. That ensures metrics are always available without manual configuration. For beginners it is advisable to test local scripts first before you put them into production pipelines. That way you learn the error sources and improve them early.
β **Watch out:** A typical beginner mistake is to automate too much at once. Start small so that you keep an overview and learn step by step. Many drop out because the first project becomes too complex and frustration arises.
π§ **Practical example:**
A concrete implementation follows. The next script automates provisioning of an Nginx web server. Create the file auto-deploy.sh with this content:
```bash
#!/bin/bash
set -e # Exit on error
echo "Starting automated deployment..."
# Pull the image
docker pull nginx:alpine
# Stop old container
docker stop nginx-auto 2>/dev/null || true
docker rm nginx-auto 2>/dev/null || true
# Run new container with best practices
docker run -d --name nginx-auto -p 8080:80 --restart unless-stopped --read-only --tmpfs /tmp nginx:alpine
echo "Deployment successful. Access at http://localhost:8080"
# Health check
sleep 3
if curl -f http://localhost:8080 > /dev/null; then
echo "Health check passed"
else
echo "Health check failed"
exit 1
fi
```
**Make it executable:**
```bash
chmod +x auto-deploy.sh
```
Run it:
```bash
./auto-deploy.sh
```
This script automates the entire process. It pulls the image, removes the old container and starts a new one with security features such as a read-only filesystem. As a beginner you see how you avoid manual SSH sessions and make the process reproducible. You can put the script into your `Git repo` and share it with team members.
That is the entry into real automation. Later you extend it with `Ansible` for several servers or integrate it into `GitHub Actions` for automatic deploys on `push`. Along the way you learn how parameters are passed and how you build in error handling. The script can also be extended to rotate logs or collect metrics. Try it on your Linux machine and watch how reliably the container starts every time.
With this understanding of automation you can take on the next pillar of the CALMS framework.
### Core principles in detail: Lean, Measurement and Sharing
Once automation has taken over the repetitive part of the work, the three remaining pillars of the CALMS framework move into the centre: Lean, Measurement and Sharing. Many beginners skip these aspects because they feel less tangible than containers or pipelines. Yet this is exactly where the difference lies between a working setup and a truly mature DevOps practice. As a Linux administrator or junior DevOps engineer you learn to integrate these pillars into your daily work step by step. They ensure that you not only work faster, but also smarter and more sustainably.
Lean comes from Japanese production philosophy and means eliminating waste consistently.
In DevOps that translates into a focus on value-creating activities. Every manual task, every unnecessary meeting and every superfluous configuration is questioned. For you as a beginner, Lean means that you analyse your daily Linux routines. Why do you copy configuration files by hand when you can describe them declaratively with `Ansible` or `Terraform`? Why do you wait for manual tests when automated checks run in the pipeline? Lean asks you to slim down the entire workflow. You build in small, incremental improvements and orient yourself on the pull principle: produce only what is actually needed.
In practice you start with a simple value-stream analysis of your current deploy processes. You note every step, measure the time and identify bottlenecks. The result is a leaner process that consumes fewer resources and runs faster. You learn that Lean does not mean thrift at any price, but intelligent simplification.
β **Watch out:** A good tip for beginners is to apply Lean first to small, personal workflows. Once you have halved the time for a recurring task, you will feel the effect immediately and be more motivated to tackle larger processes.
Measurement is about data-driven decisions.
Without measurements you fly blind. You need clear metrics for performance, availability, error rates and lead times. As a Linux admin you already know tools such as [top](https://linuxcommand.dev/cmd/top){.badge-link-text} or [htop](https://linuxcommand.dev/cmd/htop){.badge-link-text}. In a DevOps context you go further and build observability. You collect logs, metrics and traces systematically. Prometheus and Grafana become your constant companions. You define service level objectives and watch whether they are met. Measurement makes progress visible and shows deviations early.
You learn to set alerts sensibly so that you are not woken by every small thing. Instead you only receive relevant notifications. For beginners it is important to understand that good measurements do not only deliver technical data, but also map business value. How long does a feature take from idea to production? How high is the change-failure rate? These figures help you prioritise improvements and prove successes.
Sharing ensures knowledge transfer in the team.
Knowledge no longer stays stuck in individual heads, but is shared. You document `runbooks`, share `playbooks` and run `pairing sessions`. In practice that means you do not only store your `Ansible roles` or `Docker Compose files` locally, but put them into a shared `Git repository`. Other team members can inspect them, improve them and reuse them. `Sharing` creates a collective memory and reduces the bus factor.
As a beginner you start small. After every incident you write a short `postmortem` and put it into the `wiki`. You take developers along when you set up a new service, and explain your `Linux commands` as you go. That fosters understanding on both sides. `Sharing` is not a nice-to-have, but essential for continuous improvement.
These three pillars complement each other. Lean shows you where you have waste. Measurement delivers the data to prove it. Sharing ensures that the knowledge gained benefits everyone. Together they form the invisible foundation that makes `Culture` and `Automation` truly effective. Without them `DevOps` remains a pure tool set without a lasting effect.
β οΈ **Important:** Many beginners make the mistake of ignoring these pillars because they do not feel as technical as `Docker` or `Jenkins`. That leads to `pipelines` running, but real progress failing to appear and teams becoming frustrated.
**To make the difference between isolated tools and a complete CALMS approach clear:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β THE CALMS FRAMEWORK AT A GLANCE β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β C β’ Culture βββΊ Shared responsibility β β
β β A β’ Automation βββΊ Repeatable processes in code β β
β β L β’ Lean βββΊ Eliminate waste β β
β β M β’ Measurement βββΊ Metrics and data-driven calls β β
β β S β’ Sharing βββΊ Share knowledge and transparencyβ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Result: β
β * Stable software, fast releases and a learning culture β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram shows how the three pillars reinforce the other principles and close the loop of continuous improvement.
β **Watch out:** A typical beginner mistake is to run Measurement only technically and forget the business metrics. Then you measure CPU load, but not time-to-market, and miss the actual value of DevOps.
π§ **Practical example:**
A practical start succeeds by building a simple observability environment that connects Measurement and Sharing directly. You create a docker-compose.yml that starts Prometheus, Grafana and a Node Exporter for Linux metrics.
**The file looks like this:**
```yaml
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
depends_on:
- prometheus
node-exporter:
image: prom/node-exporter:latest
ports:
- "9100:9100"
```
**First you create a simple `prometheus.yml`:**
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['node-exporter:9100']
```
**Start everything with:**
```bash
docker compose up -d
```
Open `http://localhost:3000`, log in with `admin/admin` and add Prometheus as a data source.
Import a ready-made dashboard for Node Exporter. You now have immediately visible metrics of your Linux host. You share the entire setup via `Git` with the team, so everyone can inspect the metrics and add their own queries. That is Lean, because you no longer need manual checks, Measurement, because you see real data, and Sharing, because the repository distributes the knowledge. As a beginner you can try this on your laptop and later extend it into production.
Along the way you learn how to configure alerts in [Prometheus](https://prometheus.io/){.badge-link-text} and version [Grafana dashboards](https://grafana.com/grafana/dashboards/){.badge-link-text}. The entire stack runs reproducibly and shows you live how the three pillars work together.
With this understanding of the often underestimated pillars, the theoretical basis of the CALMS framework is complete. You can now move on to the practical DevOps practices.
## Practical DevOps practices
**Continuous Integration & Continuous Delivery/Deployment (CI/CD)**
The practices of `CI/CD` are the point at which the theoretical CALMS principles flow into your daily workflow. You have learned how Culture fosters collaboration, Automation takes over the repetitions, and Lean, Measurement and Sharing form the pillars for sustainability. Now it is about implementing these in a concrete pipeline. `Continuous Integration (CI)` ensures that every developer commit is automatically integrated, built and tested. `Continuous Delivery (CD)` ensures that the software is always in a deployable state. `Continuous Deployment` goes one step further and rolls the change out to production automatically when all checks have passed.
As a beginner you take the time to understand the differences precisely. `CI` is the first and foundational building block. As soon as you push a `commit` into the version-control system, usually `Git`, a pipeline starts on a runner. The runner checks out the code, installs all dependencies, runs unit tests, lints the code and builds the artefact, for example a Docker image.
If something goes wrong, the pipeline aborts immediately and notifies the team by email or chat. The `fail fast` principle helps you recognise defects early, long before they land in production. For you as a Linux administrator that means you no longer have to run manual builds on test servers. The pipeline runs reproducibly on dedicated runners or in the cloud and delivers the same result every time.
The benefits of CI are enormous.
You reduce integration problems because changes are merged several times a day. The team works on a shared code base, without anyone working in isolation in a feature branch for days. As a beginner you quickly see how much calmer day-to-day work becomes. Instead of night-time hotfixes you get immediate feedback. The pipeline becomes your daily helper that takes over routine tasks such as compiling or building Docker images. You learn that CI is not only there for developers, but also for you as an admin, because you use the same pipeline to test infrastructure changes.
Continuous Delivery goes one step further.
Here the successfully built artefact is automatically deployed into a staging or test environment. Further tests run there, for example integration tests, end-to-end tests or manual quality assurance. The final `deploy` into production can then happen at the press of a button or with a manual approval. You keep control, but you are always ready. Continuous Deployment removes this manual step completely. When all tests are green, the new version is switched live automatically.
That is suitable for teams with high maturity that have very good test coverage and monitoring. The difference lies in the appetite for risk and the depth of automation.
In practice a typical CI/CD pipeline looks like this.
It consists of several stages that run one after another. The first stage is `Checkout`, in which the code is pulled from the `Git` repository. Then comes `Build`, in which the code is compiled or the `Docker` image is created. The `Test` stage runs `unit tests`, static `code analysis` and `security scans`. In the `Package` stage the artefact is produced and pushed into a `registry`. The `Deploy` stage rolls the change out, often with `blue-green` or `canary` strategies. Each stage can fail and stops the entire process. That ensures high quality.
π‘ **Tip:** Many beginners start with [GitHub Actions](https://github.com/features/actions){.badge-link-text} because it is available for free in every GitHub repository and needs no extra server. You simply put a YAML file into the .github/workflows folder and the pipeline already runs on every push.
**To make the flow even clearer:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STRUCTURE OF A COMPLETE CI/CD PIPELINE β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Developer: git push origin main β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Continuous Integration (CI) β β
β β ββ Checkout code and install dependencies β β
β β ββ Build (build Docker image) β β
β β ββ Test (unit and integration tests) β β
β β ββ Security scan and artefact registry push β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Continuous Delivery / Deployment (CD) β β
β β ββ Automatic deploy to the staging environment β β
β β ββ Run end-to-end and integration tests β β
β β ββ Production deploy (after approval / automatic) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram shows the complete flow from your commit to the running application. You see how many manual steps disappear and how everything becomes automated and traceable.
As a Linux administrator you integrate CI/CD especially well with Docker.
You can build your existing container workflows directly into the pipeline. Instead of working via `SSH` on a server, you let the pipeline build, test and deploy the container. That saves time and reduces errors. You learn to handle secrets safely, for example with [GitHub Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets){.badge-link-text} or [HashiCorp Vault](https://www.hashicorp.com/products/vault){.badge-link-text}. The pipeline becomes your central tool that maps the entire lifecycle.
β οΈ **Important:** Watch that a good pipeline does not become too complex. Many beginners pack everything into a single YAML file and quickly lose the overview. Split the pipeline into reusable jobs and workflows so that it stays maintainable.
Integrating CI/CD into your daily work starts with a small project. Take a simple application that you already run with [Docker](/en/tag/docker){.badge-link-text}. Create a pipeline that runs the build and the tests on every push. Gradually you add the deploy. You will quickly notice how much calmer your work becomes. Instead of deploying by hand, you only watch the dashboard of your CI/CD toolchain.
β **Watch out:** A typical beginner mistake is to build the pipeline only for the code and leave the infrastructure aside. Integrate IaC checks as well, so that you have consistent environments from the start.
The benefits for you as a Linux admin are concrete. You spend less time on manual deployments and more time on optimising the systems. The pipeline gives you logs and metrics for every build, so you quickly see where bottlenecks sit. You can assign roles and permissions in a fine-grained way so that developers may only trigger certain stages. That further strengthens the collaboration from the Culture pillar.
π§ **Practical example:**
A real start succeeds with a complete GitHub Actions pipeline for a simple Nginx application. In your repository create the folder `.github/workflows` and the file `ci-cd.yml` with the following content:
```yaml
name: CI/CD Pipeline
on:
push:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ghcr.io/${{ github.repository_owner }}/myapp:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref: ghcr.io/${{ github.repository_owner }}/myapp:latest
format: 'table'
exit-code: '1'
severity: 'CRITICAL,HIGH'
deploy:
needs: build
runs-on: ubuntu-latest
steps:
- name: Deploy to production server
uses: appleboy/ssh-action@v1.0.3
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_KEY }}
script: |
docker pull ghcr.io/${{ github.repository_owner }}/myapp:latest
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp -p 8080:80 --restart unless-stopped ghcr.io/${{ github.repository_owner }}/myapp:latest
echo "Deployment completed successfully"
```
**Add the following minimal configuration to your Dockerfile:**
```dockerfile
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/
EXPOSE 80
HEALTHCHECK --interval=30s CMD curl -f http://localhost || exit 1
```
Create a simple `index.html` with test content. Commit everything and push to the `main` branch. The pipeline starts automatically. In the GitHub repository under Actions you see how the stages run. On your Linux server you store the secrets in the repository settings. The deploy step uses `SSH` to update the container on your server. The entire example is ready in less than five minutes and shows you live how CI takes over the build and the tests and CD takes care of the deploy.
You can call the health check to see whether everything is running.
```bash
curl -f http://your-server:8080
```
This example is deliberately kept simple so that you as a beginner understand the entire mechanism. Later you extend it with tests, further stages or GitLab CI if you work self-hosted. Along the way you learn how secrets are handled safely and how you keep the pipeline maintainable.
With a solid CI/CD setup you are ready to take on the next practice.
### Infrastructure as Code (IaC)
Infrastructure as Code is the logical further development of the automation you already know from the CI/CD pipeline. Instead of configuring servers by hand via `SSH` or editing configuration files by hand, you describe the entire infrastructure in code files. These files are versioned, tested and applied automatically. As a beginner you learn that infrastructure is no longer a static, hard-to-reproduce construct, but a living, changeable part of your project. You treat `servers`, `networks`, `storage` and `services` as if they were part of your application code. That means you can push changes in `Git`, review `pull requests` and the pipeline takes over the rest.
The environment is always consistent, whether on your laptop, in staging or in production. For you as a Linux administrator that is the end of the infamous snowflake servers. Those are the machines that, after months of manual adjustments, are unique and almost impossible to reconstruct when something goes wrong. With IaC every server is the result of the same code base. You reduce errors dramatically, speed up deploys and make rollbacks a simple operation.
The core idea behind IaC is the declarative approach.
You describe the desired end state in a configuration file and the tool takes care of establishing that state. In contrast there are imperative approaches, in which you specify commands step by step. Tools such as `Terraform` are declarative and ideal for beginners, because the HCL language is easy to read. `Ansible` is more imperative and is excellent for configuration management on existing servers. Both complement each other well in a DevOps stack. You quickly learn that IaC builds the bridge between your Linux administration and the modern DevOps world. You can translate your existing knowledge of commands such as `apt`, `systemctl` or `docker` directly into declarative definitions. That saves time and makes the process traceable for the entire team.
The benefits are especially tangible for beginners.
**First**, reproducibility. The same code always produces the same environment. **Second**, versioning. Every change to the infrastructure is traceable in the Git log. **Third**, collaboration. The whole team can work on the infrastructure without you as the admin being the only expert. **Fourth**, speed. A new test server is ready with one command. **Fifth**, auditability. You can see exactly who changed what when, which is decisive for compliance. **Sixth**, the self-service capability.
Developers can provision their own test environments themselves without asking you every time. That takes load off you and fosters the culture of collaboration you know from the CALMS framework. **Seventh**, scalability. You can bring up ten identical instances with one plan, without copying by hand. **Eighth**, drift detection. The tool detects when someone has changed something by hand and corrects it automatically. That prevents creeping inconsistencies that are so common in classic environments.
As a beginner you start best with a local setup.
Install Terraform on your Linux system from the official repository or the binary download. Then you create your first main.tf file. You learn to use variables to parameterise environments, outputs for important information and modules for reusable components. State is central. Terraform stores in a tfstate file which state it knows. In teams you use a remote backend such as S3 with locking to avoid conflicts. That prevents two people from running apply at the same time. You also learn how to handle secrets. Use environment variables or tools such as HashiCorp Vault so that passwords or keys are not stored in the code. That is decisive for security and prevents sensitive data from landing in Git.
Integration with your CI/CD pipeline is the next step.
The pipeline runs terraform init, terraform validate, terraform plan and, on approval, terraform apply. That makes IaC a seamless part of your workflow. You can even add policy as code with Sentinel or OPA to forbid certain changes. In Linux practice IaC replaces classic scripts. Instead of a Bash script that runs apt install, you define the packages declaratively. For container environments you use Terraformβs Docker provider to manage containers, networks and volumes. That is perfect for extending your existing Docker knowledge. You can also define systemd services, firewall rules or users declaratively. That makes your work scalable and less error-prone.
**Here the difference becomes clear:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β INFRASTRUCTURE MANAGEMENT: MANUAL VS. IAC β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Traditional administration: β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β SSH login βββΊ apt install βββΊ vim config βββΊ restartβ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β * No versioning, hard rollback, stale documentation β
β β
β Infrastructure as Code (IaC with Terraform): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Git repo βββΊ terraform init βββΊ plan βββΊ apply β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β * Declarative, versioned, auditable and destroyable β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram makes it clear how much friction disappears and how the process becomes more professional.
In practice you see IaC everywhere in large organisations. It is the standard for cloud infrastructure, but also for on-prem Linux servers. You can also manage local resources with Terraform, such as files or Docker containers. That is the perfect entry. You learn how to import existing environments when you switch from manual setups to IaC. The tool analyses the current reality and brings it into the code. That is especially helpful when you migrate legacy servers.
π‘ **Tip:** A helpful hint for beginners is always to start with small projects. Take a simple Docker setup and describe it in Terraform. The success motivates you for more complex stacks and shows you the benefit immediately.
You also learn how to keep the code maintainable. Split it into modules early so that it stays clear and you can reuse it. That prevents monoliths that are hard to maintain.
β οΈ **Important:** Pay particular attention to the state. If you lose the state, Terraform loses the overview of your resources and can no longer manage them correctly. Always store it in a remote backend.
In daily work as a Linux admin you combine IaC with your existing expertise. You define systemd services, firewall rules or users in code and let the tool do the work. That makes your work scalable and less error-prone. The combination with CI/CD makes IaC a powerful tool. The pipeline can run plan and deploy on approval. That gives you safety and speed at the same time.
β **Watch out:** A typical beginner mistake is to pack everything into one large main.tf. Split the code into modules early to keep it clear and reusable.
The benefits for scalability are obvious. With IaC you can provision ten identical servers with one command. That is ideal for load-balanced environments or test environments in CI. You see how IaC complements the pipeline and makes everything versionable.
π§ **Practical example:**
A real, beginner-friendly example is provisioning an Nginx container environment with Terraform on your Linux machine. First install Terraform if you have not already done so. Then create a folder for the project and the file main.tf with this content:
```hcl
terraform {
required_providers {
docker = {
source = "kreuzwerker/docker"
version = "~> 3.0.2"
}
}
}
provider "docker" {
host = "unix:///var/run/docker.sock"
}
resource "docker_image" "nginx" {
name = "nginx:alpine"
}
resource "docker_container" "nginx_iac" {
name = "nginx-iac-example"
image = docker_image.nginx.name
ports {
internal = 80
external = 8081
}
restart = "unless-stopped"
command = ["nginx", "-g", "daemon off;"]
}
output "container_url" {
value = "http://localhost:8081"
}
```
**Run the initialisation:**
```bash
terraform init
```
That downloads the Docker provider and prepares everything. Then create the plan to see what Terraform will do:
```bash
terraform plan
```
The plan shows exactly which resources will be created. If everything looks good, apply the plan:
```bash
terraform apply -auto-approve
```
Terraform builds the image, starts the container and gives you the output URL. Open `http://localhost:8081` in your browser to see Nginx. The container runs exactly as in your manual Docker command, but defined declaratively. To check whether everything is running, use:
```bash
docker ps
```
And to stop and remove the container:
```bash
terraform destroy -auto-approve
```
This example is deliberately kept simple so that you as a beginner can walk through the entire flow in a few minutes. You see the declarative code, the plan, the apply and the destroy. The state is stored in the file `terraform.tfstate`. In a team you would move this state into a remote backend such as an S3 bucket with locking. The example shows you how IaC improves your Docker workflows and makes them versionable.
You can extend the example by adding volumes for persistent data or defining networks. Try it on your system and experiment with changes to the file to see how Terraform only makes the necessary changes. That is the moment when you feel the real value of IaC.
With IaC as a firm practice your infrastructure is treated just like your application code.
### Observability: monitoring, logging and tracing
Observability starts exactly where `IaC` and `CI/CD` have made the infrastructure and the code stable. It gives you the ability to understand the inner state of your system without having to inspect every single component by hand.
**As a beginner you learn here that observability consists of three pillars:**
Monitoring for metrics, Logging for events and Tracing for the path of a request through your system. Together they enable you not only to discover problems, but also to find their cause quickly. In classic Linux administration you may have used top or journalctl to observe individual servers. Observability extends that to the entire distributed environment, whether `containers`, `microservices` or `cloud resources`. You see not only whether something is running, but why it is slow or where an error arises. That is the practical benefit you as a junior DevOps engineer or Linux administrator feel immediately in daily work.
**Monitoring** gives you continuous metrics for CPU, memory, network, disk and application-specific values such as request latency or error rate. You configure exporters that collect this data and send it to a central system such as Prometheus. As a beginner you start with the Node Exporter on your Linux host, which provides all kernel metrics. The metrics are stored in time series so that you can see trends over days or weeks. You learn to define alerts that only wake you for real problems.
Instead of reacting to a blanket high CPU load, you see exactly which containers or services are responsible. That reduces unnecessary alarm floods and gives you clear recommended actions. In practice you integrate monitoring directly into your IaC definitions so that new servers report their metrics automatically.
**Logging** collects all events, errors and debug information from your applications and systems. Instead of searching for logs locally on the server, you centralise them in a system such as [Loki](https://grafana.com/oss/loki/){.badge-link-text} or the [ELK Stack](https://www.elastic.co/what-is/elk-stack){.badge-link-text}. Every container writes its logs in a structured way, often in JSON format. You can then search specifically for certain strings, timestamps or correlation IDs. As a beginner you learn that good logging is not just **echo** in scripts, but structured output with levels such as `INFO`, `WARN` or `ERROR`. You configure log rotation and retention so that storage does not overflow.
Logging complements monitoring by providing the context for a metric. When CPU is high, you see in the log which request was responsible.
**Tracing** closes the gap between monitoring and logging. It follows a single request through all services. Each service adds a Trace-ID so that you can see the path from the frontend through the API to the database. Tools such as [Jaeger](https://www.jaegertracing.io/){.badge-link-text} or [Tempo](https://grafana.com/oss/tempo/){.badge-link-text} show you the latency per service and where delays arise. For beginners that is especially valuable when you work with microservices. A slow login process suddenly becomes transparent: you see that the auth service needs `800 ms` because the database is waiting.
Tracing makes distributed systems debuggable and helps you find bottlenecks that remain invisible in individual logs or metrics.
The three pillars together produce genuine observability.
You can see an `anomaly` in Grafana, find the exact point in time in the `log` and reconstruct the complete path in the `trace`. All of that happens in a unified interface. As a Linux administrator you benefit especially, because you can integrate your existing `systemd logs` and `Docker logs` seamlessly. You learn that observability does not only replace reactive firefighting, but enables proactive action. You recognise trends before they become outages, and you can build capacity planning on real data.
In DevOps practice you integrate observability into your pipelines and IaC from the start.
The CI/CD stage can automatically test metrics and logs. New containers start with built-in exporters. That is the moment when you see how all the previous practices work together. Culture ensures that the team jointly decides which metrics matter. Automation ensures consistent configuration. Measurement delivers the data that Lean and Sharing use to improve continuously.
π‘ **Tip:** A practical tip for beginners is to start with a single stack that covers all three pillars. You learn faster if you do not have to configure three separate tools, but can put everything together in one docker-compose.yml.
**A diagram makes the relationship clear:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β OBSERVABILITY: METRICS, LOGS AND TRACING β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Incoming user request (HTTP) β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Frontend (generate Trace-ID) β β
β β β β β
β β βΌ β β
β β API gateway / backend service (pass Trace-ID) β β
β β β β β
β β βΌ β β
β β Database / cache (carry Trace-ID) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β
β βΌ β
β Observability stack (Prometheus, Loki, Jaeger/Tempo) β
β * Full latency and error transparency per hop β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
This diagram shows how a request is followed from the surface through to the trace and how the three pillars deliver the data.
**As a beginner you build observability step by step.** First you enable metrics on your containers, then you centralise logs and finally you add tracing. You see immediately how much clearer your system becomes. The dashboards in Grafana become your daily tool. You learn to write queries and refine alerts. That reduces the time you spend debugging and makes you more productive.
β οΈ **Important:** Watch that observability does not start with too many metrics. Collect only what you really need, otherwise the system becomes slow and confusing. Start with the most important 10 metrics per service.
Integration with Docker is especially simple for you as a Linux admin. Every container can be given a label that automatically registers it in Prometheus. Logs are sent to Loki via a Docker logging driver. Tracing uses OpenTelemetry, which you can run as a sidecar container. All of that runs declaratively in your IaC definition.
β **Watch out:** A widespread beginner mistake is to introduce observability only after the first major outage. Then historical data is missing and you cannot learn what happened before. Build it in from day one.
The combination of the three pillars makes your system transparent and reliable. You can analyse incidents in minutes instead of hours. That strengthens the trust of the entire team and supports the blameless culture.
π§ **Practical example:**
A complete, runnable example for beginners is a docker-compose setup with Prometheus, Grafana, Loki and a simple Nginx container. Create the file `docker-compose.yml`:
```yaml
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
depends_on:
- prometheus
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
command:
- '-config.file=/etc/loki/local-config.yaml'
promtail:
image: grafana/promtail:latest
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers
command:
- '-config.file=/etc/promtail/config.yml'
nginx:
image: nginx:alpine
ports:
- "8080:80"
logging:
driver: loki
options:
loki-url: "http://loki:3100/loki/api/v1/push"
loki-retention: "30d"
```
**Create `prometheus.yml` with:**
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'nginx'
static_configs:
- targets: ['nginx:80']
- job_name: 'node'
static_configs:
- targets: ['host.docker.internal:9100']
```
**Start with:**
```bash
docker compose up -d
```
Open `http://localhost:3000` (Grafana, `admin`/`admin`) and add Prometheus as a data source. Import an Nginx dashboard. For logs you go to Explore and select Loki. You immediately see metrics, logs and can later add tracing with Tempo. The Nginx container sends logs to Loki automatically. The entire setup is runnable in under five minutes and shows you live how observability works. You can send requests to `http://localhost:8080` and watch the data in Grafana. Later extend it with Node Exporter for real Linux metrics.
With observability as a firm practice you are ready for the next technologies in the DevOps stack.
## Technology stack for beginners
### Containerisation with Docker
Containerisation with [Docker](https://www.docker.com/){.badge-link-text} forms the first real technology building block that you as a beginner in the DevOps stack absolutely must master. After observability has given you visibility into your system, Docker solves the fundamental problem of environment inconsistency. Instead of installing applications directly on a Linux server, where dependencies, libraries and configurations quickly get mixed up, you pack everything β `code`, `runtime`, `libraries` and `configuration files` β into an isolated unit, the container. Docker uses the Linux kernel features `namespaces` and `cgroups` to separate `processes`, `filesystems` and `resources` strictly, without the overhead of a full [virtual machine](/en/linux-administration/linux-administration-virtualisierung-und-vm-management){.badge-link-text}.
A container starts in a few seconds, uses only a few megabytes and runs identically on every machine with a Docker installation. That is why containerisation has become the basis for reproducible deployments, CI/CD pipelines and scalable environments.
As a Linux administrator you benefit immediately.
You leave behind the classic situation in which an app runs on your test server, but on the production server suddenly causes missing packages or wrong environment variables. With Docker you define the desired state once in a Dockerfile and the image is the same everywhere. You work with images that function like templates, and containers that are started from them. Images are immutable, that is unchangeable, and are stored in a registry such as [Docker Hub](https://hub.docker.com/){.badge-link-text} or a private registry. That makes exchange between developers and admins extremely simple.
You `push` an image and the colleague `pulls` it β done. The isolation also protects the host system. A container cannot simply access files outside its filesystem unless you allow it explicitly with volumes.
The basic concepts are manageable for beginners.
First comes the installation on Ubuntu or another distribution. You add the official Docker repository and install the package. Then you start the Docker daemon, which runs in the background and manages the containers.
The most important commands you learn quickly:
* `docker run` starts a container,
* `docker ps` shows running containers,
* `docker logs` prints the output.
You learn that every container has its own IP in the internal network and that you map ports to the host with `-p`. To create your own images you write a Dockerfile. Each line is a layer. `FROM` defines the base, `RUN` executes commands, `COPY` copies files, `CMD` or `ENTRYPOINT` sets the start command. Multi-stage builds are an advanced trick that you should use early. You build the app in the first stage and copy only the finished artefact into the second, slim stage. That keeps images small and secure.
Docker fits perfectly into the DevOps day-to-day.
In CI/CD pipelines you build the image automatically on every `commit`, tag it with the commit hash and push it into the registry. In the CD phase the server `pulls` the image and starts the container. That replaces manual deploys completely. You integrate Docker directly into your `IaC` with [Terraform](https://www.terraform.io/){.badge-link-text} or [Ansible](https://www.ansible.com/){.badge-link-text}. Containerisation also supports the Lean philosophy, because you keep only what is necessary in the image. Measurement becomes easier, because every container delivers metrics via the Docker daemon or an exporter. Sharing succeeds by putting the Dockerfile into the Git repo and working on it together as a team.
An important aspect for beginners is security.
You never start containers as `root`, but with a dedicated user. You use read-only filesystems where possible and set resource limits with `--cpus` and `--memory`. [Docker Desktop](https://www.docker.com/products/docker-desktop/){.badge-link-text} is practical for local tests, but on servers you work with the plain Docker Engine. You learn to use [Docker Compose](https://docs.docker.com/compose/){.badge-link-text} for several containers, even if that is the next subsection. For the moment it is enough that you manage individual containers cleanly.
**The Docker architecture for beginners illustrated:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DOCKER ARCHITECTURE AND CONTAINER COMPONENTS β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Host system (Linux kernel, namespaces and cgroups) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ β
β β Docker daemon (engine / socket) β β
β β βββ Images βββΊ Immutable templates (rootfs) β β
β β βββ ContainerβββΊ Isolated runtime instances β β
β β βββ Volumes βββΊ Persistent data stores β β
β β βββ Networks βββΊ Virtual bridge networks β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram shows how the kernel is shared and yet everything stays isolated. You see why Docker is so efficient.
Practical implementation always starts with a simple example.
You create a Dockerfile for a static website and start the container. That immediately gives you the feeling of how containerisation works. Later you extend it with environment variables, volumes for persistent data and health checks that feed into [observability](/en/devops/devops-fundamentals-in-modern-software-development#observability){.badge-link-text}. Docker makes the switch from development to production seamless. The developer tests locally with the same image that later runs on the server. That reduces errors and speeds up feedback loops enormously.
π‘ **Tip:** A good hint for beginners is always to use the official Docker documentation and work with --help. The commands are self-explanatory and you learn faster than if you only copy tutorials.
Containerisation also changes your Linux understanding. You learn that the host only needs the daemon and the runtime. Everything else runs isolated. That makes updates and maintenance easier, because you can keep the host minimal. You integrate Docker into your monitoring stack by enabling the Docker exporter. That way you see container metrics directly in Prometheus.
β οΈ **Important:** Watch that you do not see Docker as a cure-all. For very resource-intensive applications or special kernel modules you sometimes still need VMs. Containers are light, but not universal.
You also learn registry usage. Instead of building everything locally, you `push` images into a private registry. That speeds up `deploys` and enables team access. The commands `docker login`, `docker push` and `docker pull` become your daily tools. In the pipeline that runs automatically. As an admin you configure the registry with authentication and scan functions for security.
Scalability is another large benefit. With a simple command you start several instances of a container and put a load balancer in front. That is the entry into real horizontal scaling. You combine Docker with your existing Linux tools such as [nginx](https://www.nginx.com/){.badge-link-text} as a reverse proxy or [systemd](https://linuxcommand.dev/?q=systemd){.badge-link-text} for daemon management.
β **Watch out:** A typical beginner mistake is to pack everything into a huge image. Keep images small, use multi-stage builds and remove unnecessary tools from the final image. That saves storage and improves security.
In daily practice as a Linux administrator Docker replaces many old scripts. Instead of `apt install` on every server you run a `docker run`. That is reproducible and versioned. You can containerise old applications by writing a Dockerfile and building in legacy dependencies. That is the path to bring existing systems into the DevOps world.
Containerisation with Docker is the entry point that shows you how DevOps works in practice. It connects all the previous principles and practices into a coherent stack. You will notice how fast your deployments become and how stable your environments are.
π§ **Practical example:**
A real, immediately understandable example is containerising a simple Nginx website. Create a directory and the file Dockerfile:
```dockerfile
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/
EXPOSE 80
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -f http://localhost || exit 1
USER nginx
```
**Create an `index.html` with the content:**
```html
Hello from Docker!
```
Build the image:
```bash
docker build -t mynginx .
```
**Start the container with best practices:**
```bash
docker run -d \
--name mynginx \
-p 8080:80 \
--restart unless-stopped \
--read-only \
--tmpfs /tmp \
mynginx
```
**Check the running status:**
```bash
docker ps
docker logs mynginx
```
Test the health check:
```bash
curl -f http://localhost:8080
```
The example shows you everything important: Dockerfile syntax, build, run with security flags and health check. You can now tag and push the image:
```bash
docker tag mynginx ghcr.io/youruser/mynginx:latest
docker push ghcr.io/youruser/mynginx:latest
```
On another machine you `pull` it and start it with the same command. The environment is identical. That is the moment when you really feel the value of containerisation. Later extend the example with volumes for data or environment variables for configuration. Try it on your Linux system and watch how easy everything becomes.
With Docker as a solid basis you can take the next step to orchestration and local development with Docker Compose.
### Orchestration and local development
Docker Compose takes the individual containers you already know from Docker and brings them together into a coordinated, multi-container stack. Instead of starting every service individually with long `docker run` commands and linking `ports`, `volumes` or `networks` by hand, you describe the entire application environment in a single YAML file. A single command then starts the complete stack, connects all services automatically over an internal network and takes care of dependencies, volumes and environment variables.
That is the orchestration for local development day-to-day and the direct entry into more complex setups before you later switch to [Kubernetes](https://kubernetes.io/){.badge-link-text}. As a beginner you see immediately how much simpler and more realistic your local development becomes, because you no longer test only a single container, but a complete, production-like environment with frontend, backend, database and cache.
The YAML file `docker-compose.yml` is the central place for your definition. Under the `services` key you list every container. Each service gets a name, the desired `image`, `ports`, `volumes`, `environment variables` and `dependencies`. Docker Compose automatically creates its own network so that the services can reach each other by their service name. You no longer need to remember IP addresses or complicated port mappings.
For beginners that is the largest gain: you can start a complete app locally that looks just like it will later in production. That massively reduces the infamous βit works differently on my machineβ problems and makes tests much more reliable.
Orchestration in Compose goes far beyond simple starting.
You define dependencies with `depends_on` so that the database comes up first before the backend service starts. With `healthcheck` you can check whether a service is really ready. Volumes provide persistent data so that your database is not empty on every restart. Environment variables or `.env` files make configurations flexible for different environments. You can even set restart policies so that containers start again automatically if they crash.
You write all of that declaratively in the YAML file, which you version in `Git`. The team can work on it together, review `pull requests` and reproduce the local development environment exactly.
For your daily Linux administration, Compose is a real time saver.
You test new features locally with a realistic stack, without provisioning extra [VMs](/en/tag/virtualization){.badge-link-text} or cloud resources. Later in the CI/CD pipeline you can use the same `docker-compose.yml` to run integration tests against the complete stack. **That connects seamlessly with your IaC practice:** `Terraform` or `Ansible` can complement the Compose file or orchestrate the production containers. You quickly learn that `Compose` is not only there for development, but also suffices for small production setups or staging environments on a single server.
**The commands are deliberately kept simple:**
* `docker compose up -d` starts the entire stack in the background.
* `docker compose down` cleans everything up again.
* `docker compose logs -f` shows logs of all services in real time.
* `docker compose ps` shows you the status
* `docker compose exec` lets you jump into a running container.
For beginners it is important to understand that Compose uses the Docker daemon and needs no additional runtime. You work with the same images as with `docker run`, just orchestrated.
**A diagram shows the difference clearly:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DOCKER STANDALONE VS. DOCKER COMPOSE β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Single Docker container (standalone): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β docker run -d -p 80:80 nginx β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Docker Compose stack (docker-compose.yml): β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β’ web: Nginx web server / reverse proxy β β
β β β’ api: Node.js / Python backend service β β
β β β’ db: PostgreSQL database (with volume) β β
β β β’ cache: Redis in-memory cache β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ β
β β Command: docker compose up -d (starts whole stack) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram makes visible how Compose turns loose containers into a working stack.
As a beginner you start best with a small project.
You take your existing Nginx application and add a simple database. The YAML file defines both services and their connection. You see immediately how the backend service can reach the database without you having to configure ports or IPs. That is the moment when local development becomes production-like. You can test features as if they were already live, and wire in the observability tools from the previous section directly.
Security in Compose is another important point.
You start services with non-root users, set resource limits and use `read_only` where possible. Secrets you can handle via `secrets` or environment files without putting them in the code. Compose also supports networks with isolation, so that not every service can reach every other. For beginners it is advisable always to consult the official Docker documentation and work with `docker compose --help`. The commands are self-explanatory and you learn faster than if you only copy ready-made examples.
Integration with CI/CD is seamless. In your pipeline you can use `docker compose up` for integration tests and then `down` again. That makes tests reproducible and isolated. In IaC you define the Compose file as part of your infrastructure and let Terraform deploy it on servers. That way everything stays versioned and consistent. You quickly notice how Compose closes the gap between development and operations and supports the culture of collaboration.
π‘ **Tip:** A helpful tip for beginners is always to start with a minimal `docker-compose.yml` and extend it step by step. That way you keep the overview and avoid configuration errors that are hard to debug.
Local development with Compose saves you real time. You start the stack in the morning with one command and have a complete environment. Changes to the code are rebuilt immediately with `docker compose up --build`. You can run several environments in parallel by using different project names or files. That is ideal for feature branches or parallel tests.
β οΈ **Important:** Watch that you do not use Compose for large, highly available production environments. For real orchestration with auto-scaling and self-healing you later need Kubernetes. Compose is perfect for local and small setups, but not for hundreds of nodes.
You also learn advanced features such as `profiles`, with which you start services only when needed, or `extends`, to reuse YAML files. That keeps your configuration clear. In practice you combine Compose with your Linux knowledge: you mount host directories as volumes and use bind mounts for fast code changes during development.
β **Watch out:** A typical beginner mistake is to treat `depends_on` as a guarantee that a service is really ready. It only starts the order. Always use health checks to make sure the database really answers before the app service starts.
Scalability is another strong point. With `docker compose up --scale web=3` you start several instances of a service. That simulates load balancing locally and helps you recognise scaling problems early. You integrate that into your [observability](/en/devops/devops-fundamentals-in-modern-software-development#observability){.badge-link-text} so that metrics and logs of all instances are visible centrally.
In your Linux day-to-day Compose replaces many manual scripts. Instead of chaining several `docker run` commands in a shell script, you have a declarative file that you simply commit. That makes the workflow reproducible and team-capable. You can also use the Compose file in CI/CD to run integration tests against the real stack.
Container orchestration with Compose makes your local development more productive and closer to production. You no longer test only individual parts, but the entire system. That reduces surprises at deploy time and strengthens confidence in your changes. The combination of Docker and Compose is the practical entry that makes all the previous DevOps principles tangible.
π§ **Practical example:**
A complete, immediately understandable example for beginners is a stack with Nginx, a Node.js API and PostgreSQL.
**Create the file `docker-compose.yml`:**
```yaml
version: '3.8'
services:
db:
image: postgres:16-alpine
environment:
POSTGRES_DB: myapp
POSTGRES_USER: user
POSTGRES_PASSWORD: secret
volumes:
- db-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U user"]
interval: 10s
timeout: 5s
retries: 5
api:
build: ./api
depends_on:
db:
condition: service_healthy
environment:
DATABASE_URL: postgres://user:secret@db:5432/myapp
ports:
- "3000:3000"
web:
image: nginx:alpine
volumes:
- ./web:/usr/share/nginx/html
ports:
- "8080:80"
depends_on:
- api
volumes:
db-data:
```
In the `./api` folder you put a simple Dockerfile and a Node.js app. In the `./web` folder there is an index.html.
**Start the stack with:**
```bash
docker compose up -d --build
```
The command builds the API image, starts the database first, waits for the health check and then starts API and web. Open `http://localhost:8080` and `http://localhost:3000`. You see the complete application running.
**Logs of all services with:**
```bash
docker compose logs -f
```
**Scale the API with:**
```bash
docker compose up --scale api=2 -d
```
Stop everything with:
```bash
docker compose down -v
```
The example shows you declarative definition, dependencies, health checks, volumes and scalable services in a single command. The entire environment is local, versioned and reproducible. You can put the file in Git and share it with the team. Later extend it with Redis or further services. Try it on your Linux machine and watch how quickly you bring up a realistic app environment.
With Docker Compose as a solid orchestration foundation you are ready for modern CI/CD tools.
### Observability: monitoring, logging and tracing
With Docker Compose as a solid orchestration foundation you are ready to build the next central building block of your technology stack: Observability.
Observability means that you can understand the inner state of your entire system β from individual containers to the complete stack β at any time, without having to look into every service by hand. It consists of three interlinked pillars: Monitoring for metrics, Logging for events and Tracing for the complete path of a request.
As a beginner you learn here how to build these three elements directly into your Docker environments and thereby not only recognise problems, but also localise their cause precisely. That is the point at which your local development and your later production deployments really become transparent.
**Monitoring** gives you continuous, quantitative data. You collect metrics such as CPU utilisation, memory consumption, network throughput, request latency or error rates. With Docker you can achieve that very simply, because every container provides its own metrics via the Docker daemon or a dedicated exporter. You configure Prometheus as the central metric collector and [Grafana](https://grafana.com/){.badge-link-text} as the visualisation layer. As a Linux administrator you use the Node Exporter on the host to capture kernel metrics, and the Docker exporter to obtain container-specific data.
The metrics are stored in time series so that you not only see the current state, but also trends over hours or days. You define service level objectives and receive alerts only when something really falls out of line. That reduces alarm floods and gives you clear recommended actions.
**Logging** collects all qualitative events. Every container writes structured logs β ideally in JSON format with timestamp, level and correlation ID. Instead of searching for logs locally on the host, you centralise them with [Loki](https://grafana.com/oss/loki/){.badge-link-text} or a similar system. Docker Compose makes that especially simple, because you can configure logging drivers directly in the YAML file. The logs are sent automatically to the central collector. You can then search specifically for certain strings, errors or time ranges.
For beginners it is important to understand that good logging is not just `echo` commands, but structured output with clear levels such as `INFO`, `WARN` or `ERROR`. That enables you to find the context for a metric: when CPU is high, you immediately see in the log which request was responsible.
**Tracing** completes the picture. It follows a single request through all services of your stack. Each component adds a Trace-ID so that you can see the entire path from the frontend through the API to the database. With OpenTelemetry and a backend such as Jaeger or Tempo you get detailed latency figures per service and recognise bottlenecks at a glance. In a Docker Compose environment that is especially valuable, because you already simulate distributed systems locally.
**A slow login suddenly becomes transparent:** you see exactly where the delay arises. Tracing makes microservices or complex stacks debuggable and helps you recognise performance problems early.
The three pillars together produce genuine observability.
You can discover an `anomaly` in Grafana, find the exact point in time in the log and reconstruct the complete path in the trace. Everything happens in a unified interface. **As a beginner you build that step by step:** first `metrics`, then `logs` and finally `tracing`. You integrate it directly into your Compose file so that every new service is automatically observable. That connects seamlessly with the previous steps: Docker delivers the containers, Compose the orchestration and observability the visibility.
In practice as a Linux administrator you configure observability declaratively in your IaC.
New containers start with built-in exporters, logs are centralised via Docker logging drivers and tracing uses sidecar patterns or OpenTelemetry instrumentation. You learn that observability does not only replace reactive firefighting, but enables proactive action. You recognise trends before they become outages, and you can build capacity planning on real data.
**Integration with CI/CD is seamless:** your pipeline can automatically test metrics and logs and ensure that new features are observable.
**The following diagram makes the relationship in the stack clear:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β COMPOSE STACK WITH OBSERVABILITY LAYER β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Application services (Compose stack) β β
β β [ Web (Nginx) ] [ API (Node.js) ] [ DB (Postgresβ β
β ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββ β
β β Logs and metrics β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Observability layer (monitoring and telemetry) β β
β β βββ Metrics: Prometheus (scraping) βββΊ Grafana β β
β β βββ Logs: Promtail βββΊ Loki βββΊ Grafana β β
β β βββ Traces: OpenTelemetry agent βββΊ Jaeger β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram shows how observability sits over the entire container stack and makes all services transparent.
**The benefits for beginners are enormous:** you test locally not only individual parts, but the entire stack with full visibility. That reduces surprises at deploy time and strengthens confidence in your changes. You learn to write queries in Prometheus, build dashboards in Grafana and set alerts sensibly. All of that stays declarative and versioned in Git, so the team can work on it together.
π‘ **Tip:** A helpful tip for beginners is to start with a minimal stack that contains only the most important metrics, logs and a simple trace. That way you keep the overview and avoid configuration errors that are hard to debug.
The security of observability is another important point. You configure access rights on Grafana and Prometheus so that sensitive data is protected. You use TLS for the collectors and store secrets outside the Compose file. That keeps your system secure and compliant.
β οΈ **Important:** Watch that you do not collect too many metrics. Start with the ten most important per service, otherwise the system becomes slow and confusing. Scale the collection only when you really need it.
In daily Linux administration you combine observability with your existing tools. You mount host logs as volumes and use systemd-journald as a source. Docker containers send their output to the collector automatically. That makes your work scalable and less error-prone. The combination with Docker Compose makes the entire stack observable without additional manual steps.
β **Watch out:** A typical beginner mistake is to introduce observability only after the first major outage. Then historical data is missing and you cannot learn what happened before. Build it in from day one, including in local development.
**The three pillars make your system transparent and reliable.** You can analyse incidents in minutes instead of hours. That strengthens the trust of the entire team and supports the blameless culture from the CALMS framework. You see not only whether something is running, but why it is slow or faulty.
Container orchestration with Compose and the direct inclusion of observability make your local development more productive and closer to production. You no longer test only individual parts, but the entire system with full visibility. That reduces surprises at deploy time and strengthens confidence in your changes. The combination of Docker, Compose and observability is the practical entry that makes all the previous DevOps principles tangible.
π§ **Practical example:**
A complete, immediately understandable example is a Docker Compose stack with [Prometheus](https://prometheus.io/){.badge-link-text}, [Grafana](https://grafana.com/){.badge-link-text}, [Loki](https://grafana.com/oss/loki/){.badge-link-text}, [Promtail](https://grafana.com/oss/promtail/){.badge-link-text} and an Nginx service.
**Create the file `docker-compose.yml`:**
```yaml
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
depends_on:
- prometheus
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
command:
- '-config.file=/etc/loki/local-config.yaml'
promtail:
image: grafana/promtail:latest
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
command:
- '-config.file=/etc/promtail/config.yml'
nginx:
image: nginx:alpine
ports:
- "8080:80"
logging:
driver: loki
options:
loki-url: "http://loki:3100/loki/api/v1/push"
loki-retention: "30d"
volumes:
prometheus-data:
```
**Create `prometheus.yml`:**
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'nginx'
static_configs:
- targets: ['nginx:80']
- job_name: 'node-exporter'
static_configs:
- targets: ['host.docker.internal:9100']
```
**Start the stack with:**
```bash
docker compose up -d
```
Open `http://localhost:3000` (Grafana, `admin`/`admin`), add `Prometheus` as a data source and import an Nginx dashboard. For logs you go to Explore and select Loki. You see metrics and logs in real time. The Nginx container sends logs to Loki automatically.
**Test with:**
```bash
curl -f http://localhost:8080
```
The example shows you declarative definition, automatic log forwarding, metric collection and visualisation in a single command. The entire environment is local, versioned and reproducible. You can put the file in `Git` and share it with the team. Later extend it with Jaeger for tracing or Node Exporter for real Linux metrics. Try it on your Linux machine and watch how quickly you bring up a fully observable app environment.
With observability as a firm part of your stack you can take the next step to modern CI/CD tools.
## Your first production DevOps workflow
### Building a complete example pipeline
`Code` β `Build` β `Test` β `Deploy`
Now that you master Docker, Docker Compose and observability, you can take the jump into practice and build your first complete production pipeline. The pipeline takes every code commit automatically through the four classic stages: Code, Build, Test and Deploy. It connects all the previous building blocks into a continuous, automated flow that reaches from your Git push to the running service in staging or production. As a beginner you see here how the theoretical CALMS principles and the technology stack work together concretely.
You do not need an expensive tool or a complex Kubernetes cluster β a [GitHub repository](https://github.com/){.badge-link-text} and [GitHub Actions](https://docs.github.com/en/actions){.badge-link-text} are fully sufficient to test the entire workflow locally and in production.
The first stage is the code trigger.
As soon as you push your commit to the main branch, the pipeline recognises the push and starts automatically. The version-control system `Git` is the starting point. You work with `branches`, `pull requests` and `code reviews`, exactly as you learned from the Culture pillar. The pipeline first checks whether the code is clean. That prevents faulty code from going any further.
For beginners it is important to understand that this trigger ensures the speed and the reproducibility of the entire chain. Every developer in the team sees immediately whether their commit makes the pipeline green or red. That fosters shared responsibility and reduces manual handovers.
In the build stage a runnable artefact is produced from the code.
For a Docker-based application that means the Dockerfile is executed and a new image is built. You tag the image with the commit hash or a semantic version tag, so that you later know exactly which code sits in which container. The build runs on a runner in the cloud or locally on your Linux machine. Docker Compose can already be used here to start and test the entire stack for the build. The build stage is declarative and reproducible β exactly like your IaC definitions. The image is then pushed into a registry such as GitHub Container Registry or Docker Hub. That way it is ready for all subsequent stages.
The test stage is the decisive quality gate. Several layers of tests run here in parallel.
First unit tests, which check the code itself. Then integration tests, which bring up the complete Docker Compose stack and check whether web, API and database work together. Security scans with Trivy or Snyk search for known vulnerabilities in the dependencies and in the image. **Observability is already wired in here:** the pipeline starts the stack with Prometheus and Grafana and checks whether the metrics are green.
You learn that tests are not only βgreen or redβ, but can also include performance and load tests. Every failed test stops the pipeline immediately. That is the βfail fastβ principle in action and protects your production from faulty code.
Only when all tests have passed does the deploy stage come.
Here the image is rolled out onto a target server or into a cloud environment. For beginners a simple SSH deploy onto a Linux server is ideal. You can simulate blue-green deployment by stopping the old container and starting the new one. Or you use Docker Compose on the server to bring the entire stack up again. In more advanced setups a rollout with canary releases is added, in which only a small part of the users sees the new version. The deploy stage can be approved manually or run automatically, depending on the maturity of your team. What matters is that the deploy is always linked with observability: after the rollout the pipeline checks the health checks and metrics before it is marked as successful.
The entire pipeline is defined in a single YAML file and lives in the repository.
It is versioned, reviewed and tested like any other code. That makes the workflow transparent and traceable for the entire team. As a Linux administrator you configure the runners so that they use your existing servers or run in the cloud. You see how all the previous chapters β Culture, Automation, IaC, Docker, Compose and observability β flow together here.
**The flow for beginners shown clearly:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PRODUCTION CI/CD WORKFLOW FLOW β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β 1. Git push βββΊ GitHub / GitLab repo β
β β β
β βΌ β
β 2. Build βββΊ Create and tag Docker image β
β β β
β βΌ β
β 3. Test βββΊ Unit, integration and security tests β
β β β
β βΌ β
β 4. Deploy βββΊ Staging βββΊ Health check βββΊ Prod β
β β β
β βΌ β
β 5. Monitor βββΊ Check metrics and alerts in Grafana β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram shows how linear and at the same time branched the stages are, and where observability sits as the last safeguard.
The pipeline is not static. You can extend it, add new jobs or use parallel stages. For beginners it is advisable first to build a simple version with only three stages and then expand it step by step. That way you keep the overview and really learn to understand the relationships. The pipeline runs on every push, pull request or merge and gives you immediate feedback. That is the core of a productive DevOps workflow.
π‘ **Tip:** Many beginners make the mistake of starting the pipeline too complex. Start with a minimal workflow that only contains build, a simple test and deploy. The quick success motivates you to extend the pipeline later and add observability or security scans.
In practice as a Linux administrator you see how the pipeline replaces your manual deploy scripts. Instead of SSH logins and manual docker run commands, everything runs automatically. You can also use the pipeline for IaC changes: terraform plan and apply become part of the stages. That treats infrastructure and application changes equally and versions them.
β οΈ **Important:** Watch that secrets such as SSH keys or database passwords never land in the repository. Always use GitHub Secrets or similar mechanisms so that the pipeline stays secure and no sensitive data appears in the log.
The complete pipeline connects speed, stability, security and scalability, the four core promises of DevOps that you learned at the beginning. You see live how a commit becomes a running application in minutes. That reduces waiting times and raises quality dramatically.
β **Watch out:** A widespread beginner mistake is to release the deploy stage without a manual approval or health check. That can lead to outages if a test was overlooked. Always start with a manual approval until you have absolute confidence in your tests and observability.
The pipeline is the place where everything comes together. You use `Docker` for the containers, `Compose` for local orchestration, `observability` for monitoring and `IaC` for the infrastructure. Every stage is declarative and versioned. The team works on it together and learns continuously.
π§ **Practical example:**
A real, immediately understandable example is a complete GitHub Actions pipeline for a Docker-based application. In the repository create the folder `.github/workflows` and the file `pipeline.yml` with the following content:
```yaml
name: Complete DevOps Pipeline
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build Docker Image
uses: docker/build-push-action@v5
with:
context: .
push: false
tags: myapp:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Run Trivy Security Scan
uses: aquasecurity/trivy-action@master
with:
image-ref: myapp:latest
format: 'table'
exit-code: '1'
severity: 'CRITICAL,HIGH'
test:
needs: build
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Start Docker Compose Stack for Tests
run: docker compose up -d --build
- name: Run Integration Tests
run: |
sleep 10
curl -f http://localhost:8080 || exit 1
echo "Integration test passed"
- name: Stop Test Stack
if: always()
run: docker compose down
deploy:
needs: test
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- name: Deploy to Production Server
uses: appleboy/ssh-action@v1.0.3
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_KEY }}
script: |
docker pull ghcr.io/${{ github.repository_owner }}/myapp:latest
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp -p 8080:80 --restart unless-stopped ghcr.io/${{ github.repository_owner }}/myapp:latest
echo "Deployment complete - running health check"
sleep 5
curl -f http://localhost:8080 || exit 1
```
Add a simple Dockerfile and a `docker-compose.yml` for tests to your repository. Commit the pipeline file and push to the main branch. The pipeline starts automatically. Under GitHub Actions you see how the stages run one after another:
* `Build` security scan
* `Integration test` with Compose
* `Deploy` via SSH.
The health check at the end ensures that the service is really running. The entire example is set up in under ten minutes and shows you live the flow Code β Build β Test β Deploy. You can store the secrets in the repository settings and test the example on your Linux server. Later extend it with observability checks or blue-green logic. Try it and watch how a simple commit triggers the complete chain.
With this first production workflow you are ready to look at the typical pitfalls.
### Typical pitfalls
After you have successfully built your first pipeline from code through build and test to deploy, pitfalls quickly appear in practice that can stall the entire workflow. As a beginner you learn here to recognise these typical mistakes early and avoid them systematically, so that your pipeline not only works, but also stays robust and maintainable.
The most common trap is missing or incomplete configuration of [health checks](https://docs.docker.com/engine/userguide/healthchecks/){.badge-link-text} in the deploy stage. If you start a new container without checking whether the application is really ready, the pipeline can mark the service as successful even though it is not answering yet. That leads to outages for users even though everything looked green. You avoid that by defining a real health check in every Dockerfile and in the Compose file that checks an internal endpoint and only releases the next stage after a successful result.
Another classic problem arises with state management in IaC components inside the pipeline.
If [Terraform](https://www.terraform.io/){.badge-link-text} or [Ansible](https://www.ansible.com/){.badge-link-text} stores the state locally on the runner, a parallel job or an abort can corrupt the state. You then suddenly see resources that are no longer managed correctly. The solution is a remote backend with locking from the start. In [GitHub Actions](https://docs.github.com/en/actions/using-workflows/storing-workflow-data/using-the-cache-action){.badge-link-text} you configure an S3 backend or [GitHub Cache](https://docs.github.com/en/actions/using-workflows/storing-workflow-data/using-the-cache-action){.badge-link-text} for the state and enable locking so that only one job can run apply at a time. That sounds involved at first, but it is the key to stable infrastructure changes.
π‘ **Tip:** Many beginners underestimate how important the order of services in Docker Compose is. `depends_on` does start the order, but it does not guarantee that the database is really ready when the API starts. You avoid that by using real health checks in the Compose file and in the pipeline, so that the API only starts after a successful `pg_isready`.
A third stumbling block is unsafe handling of secrets.
If you write database passwords or SSH keys directly into the pipeline YAML or let them land in logs, sensitive data is visible to anyone who looks at the repository. That is not only a security risk, it often also violates compliance requirements. You solve that by using [GitHub Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets){.badge-link-text} or comparable mechanisms and injecting the values into the pipeline only as environment variables.
In your Dockerfile and in Compose files you reference these variables without ever hard-coding them. In addition you enable secret scanning in your CI/CD toolchain, which automatically searches for keys committed by accident.
Many teams also stumble over missing or incomplete integration tests in the test stage.
A pure unit test on the code is not enough when the services in Docker Compose have to communicate with each other. The pipeline could be green even though the API cannot reach the database. You avoid that by starting the complete stack with docker compose up in the test stage and running real end-to-end tests. A simple curl against the health endpoint of the web layer is enough as a minimal test; later you extend it with API tests using tools such as Postman or Newman.
The scaling traps are especially treacherous when you later work with several instances. A container runs perfectly locally, but as soon as you start with
```bash
docker compose up --scale api=3
```
`load balancing` or `session affinity` is missing. Requests always land on the same service and you suddenly see error rates. You prevent that by building a simple load balancer such as nginx into the Compose file from the start and configuring session handling. In the pipeline you test the scaling explicitly in a separate stage.
Another problem is missing drift detection in IaC.
After a manual intervention on the server the real state diverges from the code. The pipeline then deploys something that no longer fits. You solve that with `terraform plan` in every pipeline run and an explicit check that the plan is clean. Only with a clean plan may the deploy continue.
The observability trap appears when you collect metrics and logs but do not configure alerts or dashboards for them. The pipeline runs, but you only notice hours later that something went wrong. You avoid that by building a concluding observability check into the deploy stage that tests Prometheus queries against defined thresholds and aborts on deviation.
**The most common pitfalls and their position in the pipeline:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PIPELINE FAILURE POINTS AND COUNTERMEASURES β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Typical failure points in CI/CD pipelines: β
β ββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ β
β β Failure point β Cause and countermeasure β β
β ββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€ β
β β Build β Secrets hardcoded βββΊ Vault β β
β β Test β Weak isolation βββΊ Compose β β
β β Deploy β Missing health check βββΊ curl /h β β
β β Observability β No alert rules βββΊ PagerDuty β β
β ββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram shows at which point which trap most often strikes and where you catch it best.
Avoiding all pitfalls starts with a clear checklist that you review in every `pull request`. You document the pipeline in a `README` and keep it as a living document so that new team members immediately see which checks are active. As a Linux administrator you configure the runners so that they use your existing tools such as Docker and Compose and do not introduce additional dependencies.
In practice you see that most problems do not arise from missing technology, but from missing discipline in the configuration. You avoid them by first testing every new feature-branch pipeline locally with `docker compose test` before it reaches the cloud pipeline. That saves time and frustration.
Through this conscious avoidance of pitfalls the pipeline becomes a reliable tool that brings you real value. You gain speed without losing stability, and you build a system that can grow.
β οΈ **Important:** Pay particular attention that you never release production deploys without a prior plan check and health check. A single overlooked error can take the entire service down and destroy the team's trust in the pipeline.
You also learn that rollback strategies must be part of the pipeline. On a failed deploy the pipeline automatically starts the previous tag or the last stable image. That is implementable in the deploy stage with a simple docker run of the old image and gives you safety on critical changes.
β **Watch out:** A typical beginner mistake is to build the pipeline only for the application code and leave the IaC changes aside. That leads to inconsistencies between code and infrastructure. Integrate terraform plan and apply into the same pipeline from the start so that both stay in sync.
Avoiding these pitfalls makes your pipeline not only functional, but a real production tool that you can operate with a clear conscience. You will see how the error rate drops and how much calmer your day-to-day as a DevOps engineer becomes.
π§ **Practical example:**
A real example of how you avoid the most common pitfall of a missing health check is extending your `docker-compose.yml` and the pipeline. Add the following health check for the API service in the Compose file:
```yaml
api:
build: ./api
depends_on:
db:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
interval: 10s
timeout: 5s
retries: 5
```
**In the pipeline file add an explicit check after the docker run in the deploy stage:**
```bash
docker run -d --name myapp -p 8080:80 --restart unless-stopped ghcr.io/${{ github.repository_owner }}/myapp:latest
sleep 10
if ! curl -f http://localhost:8080/health; then
echo "Health check failed - initiating rollback"
docker stop myapp || true
docker rm myapp || true
# Start the previous container here
exit 1
fi
echo "Health check passed"
```
Run the pipeline locally with `act` or directly on [GitHub](https://github.com){.badge-link-text}. You see that the `deploy` only counts as successful if the service really answers. The example shows you how, with a few lines of code, you avoid the most common trap and at the same time build in an automatic rollback. Test it on your Linux machine, deliberately change the health endpoint to a wrong path and watch how the pipeline aborts and rolls back. That immediately gives you confidence in the robustness of your workflow.
With these avoided pitfalls you lay the foundation for the next steps on your DevOps journey.
### From pipeline to DevOps platform
Once your pipeline runs stably and the typical pitfalls are avoided, it is time to take the next large step: to build a full DevOps platform from the individual pipeline. The pipeline is the engine, the platform is the entire ecosystem that developers and admins can use independently, without you having to intervene by hand every time. As a beginner you learn here how you introduce `GitOps`, extend `IaC` across several environments, centralise `observability` and create `self-service functions` so that the team can work independently.
That is the transition from it works to it runs on its own and grows with you. You build on everything you have learned so far β `Docker`, `Compose`, the `pipeline` and the `CALMS principles` β and turn it into a productive, scalable system.
The first concrete step is GitOps.
Instead of the pipeline executing the deploy via SSH, the desired state is defined in Git as the single source of truth. Tools such as [ArgoCD](https://argo-cd.readthedocs.io/en/stable/){.badge-link-text} or [Flux](https://fluxcd.io/){.badge-link-text} pull the current state from the repository and synchronise it automatically with the clusters or servers. You as a Linux administrator first configure a simple ArgoCD instance on your test server. The application and the infrastructure are stored as YAML manifests in the Git repo.
**Every merge into the main branch triggers an automatic synchronisation.** That makes deploys declarative, auditable and rollback-capable. You avoid manual interventions and see immediately in the ArgoCD interface whether the live state matches the Git state. For beginners you start with a single application that you have already built in your pipeline. You put the deployment YAML next to the Dockerfile in the repo and let ArgoCD do the rest.
The next expansion is extending IaC across several environments.
Your Terraform or Ansible definitions are organised in folders such as `dev`, `staging` and `prod`. With [Terraform Workspaces](https://developer.hashicorp.com/terraform/language/state/workspaces){.badge-link-text} or environment-specific variable files you can use the same code base for different environments. The `pipeline` recognises from the branch or a label which environment to deploy into. That prevents production changes from landing in staging by accident.
You learn how to set up remote-state backends with locking for all environments so that several team members can work in parallel without blocking each other. The platform thereby becomes safer and more traceable.
Observability becomes a central platform component.
Instead of local Grafana instances you build a central observability cluster with [Prometheus](https://prometheus.io/){.badge-link-text}, [Loki](https://grafana.com/oss/loki/){.badge-link-text}, [Tempo](https://grafana.com/oss/tempo/){.badge-link-text} and [Grafana](https://grafana.com/oss/grafana/){.badge-link-text}. Every new application that is deployed via the `pipeline` registers itself automatically through service discovery. You configure standardised dashboards and alerts that apply to all teams.
As a beginner you add labels in your Compose file and in the Kubernetes manifests that Prometheus recognises automatically. That makes the entire platform observable without you having to write new configurations every time. You also integrate tracing with OpenTelemetry so that requests can be followed through the entire platform. The data flows into a unified system that you lock down with roles and access rights.
Self-service is the point that makes the platform truly productive.
Developers should be able to provision new environments themselves without contacting you as the admin every time. For that you build an internal portal or use existing tools such as [Backstage](https://backstage.io/){.badge-link-text} or a simple Git repo template. A developer creates a new branch or a new ticket, the platform automatically generates the necessary manifests, starts the `pipeline` and provisions the environment. As a Linux administrator you define the templates once and release them. That reduces your support tickets massively and fosters the culture of shared responsibility.
**DevSecOps becomes a firm part of the platform. Security scans do not only run in the `pipeline`, but continuously in the background.**
Image scanning, policy as code with OPA or Kyverno and automatic vulnerability management are enabled by default. You configure the platform so that new images may only be deployed if they have no critical vulnerabilities. That happens declaratively and is visible to all teams. You as a beginner start with Trivy in the `pipeline` and later extend it with cluster-wide policies.
**Multi-environment management and canary releases make the platform scalable.**
You define traffic splitting in the manifests so that you can first roll out new versions to 10 % of the users. Observability shows you live whether the new version is stable. On problems the platform switches back automatically. All of that is controlled through [GitOps](https://gitops.tech/){.badge-link-text}, so that you only have to make changes in `Git`. The `pipeline` validates and the platform executes.
The platform grows with your requirements.
You integrate feature flags to steer features without a new deploy. You expand blue-green deployments so that zero-downtime updates are possible. You extend observability with business metrics so that you see not only technical but also user-related data. Everything stays declarative, versioned and automatic.
**From the simple pipeline to a full platform:**
```markdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β EVOLUTION TO A FULL PLATFORM β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β Stage 1: Simple pipeline (Code βββΊ Build βββΊ Deploy) β
β β β
β βΌ β
β Stage 2: Multi-environment (staging + production + IaC) β
β β β
β βΌ β
β Stage 3: GitOps and platform engineering β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β β’ Declarative GitOps control (ArgoCD / Flux) β β
β β β’ Full observability and DevSecOps gates β β
β β β’ Self-service developer portal and canary deploys β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
The diagram makes it clear how many manual steps disappear and how the platform becomes autonomous.
As a beginner you build the platform step by step.
You start with `GitOps` for a single application, then add `IaC` for staging and finally extend with `self-service` and expanded `observability`. **Every step brings measurable benefit:** less manual work, higher security and faster feedback loops. You document the platform in a central `README` and keep the configuration in the `Git repo` so that it is accessible to everyone.
**Integration with your existing Linux servers is simple.** You install [ArgoCD](https://argoproj.github.io/argo-cd/){.badge-link-text} or [Flux](https://fluxcd.io/){.badge-link-text} on a dedicated host and let it manage the Docker containers or later Kubernetes resources. The pipeline remains the entry point, the platform takes over operations. You see how the four promises of DevOps β speed, stability, security and scalability β really take hold at platform level.
The platform becomes the team's central tool. Developers provision environments themselves, admins monitor centrally and everyone works with the same Git-based workflow. That is the point at which DevOps is no longer only a pipeline, but a living, self-steering platform.
π‘ **Tip:** A practical tip for beginners is to build the platform first on a single test server and only then expand it to several environments. That way you learn the relationships at a small scale and avoid complex errors in production.
You also learn how to keep the platform maintainable. Modules in Terraform, reusable ArgoCD applications and standardised observability templates ensure that the platform does not become a monolith. Every new application uses the same templates and becomes automatically observable and secure.
β οΈ **Important:** Watch that you do not build the platform overnight. Plan small, incremental steps and test every extension thoroughly in a separate environment before you make it productive. Too fast an expansion otherwise leads to unstable processes and frustration in the team.
In daily work as a Linux administrator the platform becomes your new day-to-day. You configure new services once in Git and the platform does the rest. You concentrate on strategic topics such as scaling, security and optimisation instead of repetitive deploy tasks. That is the actual added value of a full DevOps platform.
β **Watch out:** A typical beginner mistake is to see the platform only technically and forget the cultural aspects. Without self-service and shared responsibility the platform remains a pure admin tool. Integrate the team from the start into the definition of the templates and processes.
The full DevOps platform is the goal that you have already reached with your first pipeline. Every step β GitOps, expanded IaC, central observability, self-service and DevSecOps β builds directly on what you have already implemented. You will see how the work changes: faster, more stable, more secure and more scalable.
π§ **Practical example:**
A real, immediately understandable example for the entry into a full platform is extending your existing `pipeline` with `GitOps` through a simple `ArgoCD`-like synchronisation on a Linux server. First you install `ArgoCD` locally with `Helm` or the `manifest-based installer`. Then create a `GitOps` repository structure:
```bash
mkdir -p gitops/app
cd gitops/app
cat > deployment.yaml <