--- id: 2024-11-08-kubernetes-container-orchestration slug: kubernetes-container-orchestration title: "Kubernetes: container orchestration from install to advanced cluster practice" excerpt: "Kubernetes from cluster install and practical workloads through monitoring, security, operators and a service mesh." date: "2024-11-08T05:20:43+01:00" updated: "2025-05-09T09:02:58+01:00" author: name: "Erik van Hooven" handle: "evanhooven" category: "devops" tags: ["cloudcomputing", "cloudnative", "container", "devops", "docker", "infrastructure", "k8s", "kubernetes", "microservices", "orchestration"] reading_time: 90 toc: true --- Modern software development has to run and scale complex applications efficiently. Kubernetes (often abbreviated K8s) has become a capable answer to that demand. As an open-source platform it automates management, deployment and scaling of containerised applications. With automatic load balancing, self-healing and flexible resource management, Kubernetes supplies the tools needed to operate modern cloud-native applications. The platform was originally developed at Google and handed to the open-source community in 2014, where it has evolved continuously since.
❗ **Important note:** The material builds on the fundamentals in [DevOps fundamentals: modern software development](/en/devops/devops-fundamentals-in-modern-software-development){.badge-link-text}. Basic DevOps knowledge is assumed: * A basic understanding of container technologies * Experience with Linux systems and the command line * A basic understanding of DevOps practices
## Why Kubernetes? In modern microservice architectures, development teams face complex challenges: * Managing hundreds or thousands of containers * Automatic scaling under load * Self-healing after failures * Rolling updates without downtime * Load balancing between services * Service discovery in dynamic environments Kubernetes solves these challenges through a carefully designed system of abstractions and automation. ## Architecture and components ### Control Plane (Master Node) The control plane is the “brain” of the Kubernetes cluster and consists of several critical components: **API Server (kube-apiserver):** ```markdown ┌─────────────────────────────────────────────────────────────┐ │ API server request processing │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Inbound kubectl or client request │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ 1. Authentication ──► Check identity (TLS/token) │ │ │ │ 2. Authorization ──► Check rights (RBAC) │ │ │ │ 3. Admission Control──► Mutating and validating webh. │ │ │ │ 4. API Server Core ──► Schema validation │ │ │ │ 5. Data storage ──► Persistence in etcd │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` The API server is the central communication hub and does the following: * REST interface for every cluster operation * Authentication and authorisation of requests * Validation of object configurations * Persistence in etcd ```yaml apiVersion: v1 kind: Pod metadata: name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=192.168.1.10 - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt - --enable-admission-plugins=NodeRestriction - --enable-bootstrap-token-auth=true image: k8s.gcr.io/kube-apiserver:v1.24.0 name: kube-apiserver ``` **etcd:** The distributed key-value store holds the entire cluster state: * Highly available data storage * Consistent data holding through Raft consensus * Versioned storage of every Kubernetes object ```yaml apiVersion: v1 kind: Pod metadata: name: etcd namespace: kube-system spec: containers: - command: - etcd - --advertise-client-urls=https://192.168.1.10:2379 - --data-dir=/var/lib/etcd - --initial-cluster-state=new - --initial-cluster-token=etcd-cluster image: k8s.gcr.io/etcd:3.5.3-0 name: etcd ``` **Scheduler (kube-scheduler):** The scheduler is responsible for intelligent placement of pods: * Resource analysis of worker nodes * Consideration of affinity/anti-affinity * Prioritisation based on defined policies ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Scheduling algorithm (kube-scheduler) │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Pod Creation ──► Filtering ──► Scoring ──► Binding │ │ │ │ ▲ │ │ │ │ │ └───────────────── (Retry on Failure) ──┘ │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ │ * Filtering: Excludes nodes without enough resources │ │ * Scoring: Rates remaining nodes against criteria │ │ * Binding: Assigns the pod to the best worker node │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Controller Manager (kube-controller-manager):** The controller manager implements the core logic of the Kubernetes control loop: * Node controller: monitors node state * Replication controller: ensures the desired number of pods is running * Endpoints controller: links services to pods * Service account and token controller: manages access tokens ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Kubernetes reconciliation control loop │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ Desired state (from manifest / etcd) │ │ │ └─────────────────────────┬─────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ Capture current state (via kubelet) │◄──┐ │ │ └─────────────────────────┬─────────────────────────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ │ Compare and run actions (reconciliation) │───┘ │ │ └───────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ```yaml apiVersion: v1 kind: Pod metadata: name: kube-controller-manager namespace: kube-system spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf - --bind-address=127.0.0.1 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --cluster-cidr=10.244.0.0/16 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt image: k8s.gcr.io/kube-controller-manager:v1.24.0 name: kube-controller-manager ``` ### Worker Node components **Kubelet:** The kubelet is the primary node agent: * Pod lifecycle management * Container health checks * Volume mounting * Node status reporting ```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: "0.0.0.0" port: 10250 serializeImagePulls: true evictionHard: memory.available: "100Mi" nodefs.available: "10%" nodefs.inodesFree: "5%" ``` **Container runtime:** The container runtime (for example containerd) is responsible for: * Container execution * Image management * Container isolation ```yaml version = 2 [plugins."io.containerd.grpc.v1.cri"] sandbox_image = "k8s.gcr.io/pause:3.6" [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" ``` **Kube-proxy:** Kube-proxy implements the Kubernetes Service concept: * Service load balancing * iptables rules * Session affinity ```yaml apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration bindAddress: 0.0.0.0 clientConnection: acceptContentTypes: "" burst: 10 contentType: application/vnd.kubernetes.protobuf kubeconfig: /var/lib/kube-proxy/kubeconfig.conf qps: 5 clusterCIDR: 10.244.0.0/16 mode: "ipvs" ``` ## Installation and setup
💡 **Information:** The examples use Ubuntu/Debian as the sample distribution. The core concepts are the same on every Linux system, but package installation and some configuration paths can differ by distribution.
### Prerequisites Before the actual installation, it is important to understand what a Kubernetes cluster requires from hardware and software. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Kubernetes cluster architecture │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Control Plane (Master Node) │ │ │ │ ├── kube-apiserver ├── kube-scheduler │ │ │ │ ├── etcd (key-value store) └── kube-controller-mgr │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ TLS / gRPC connection │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Worker nodes (runtime for workloads) │ │ │ │ ├── Kubelet (node agent) ├── Container runtime │ │ │ │ └── kube-proxy (network) └── Pods / containers │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Hardware requirements:** **Control Plane Node (Master):** * **CPU:** Minimum 2 cores, recommended 4 cores, production 8+ cores * **RAM:** Minimum 2 GB, recommended 4 GB, production 16+ GB * **Disk:** Minimum 50 GB, recommended 100 GB, production 250+ GB (SSD recommended) **Worker Nodes:** * **CPU:** Minimum 1 core, recommended 2 cores, production 4+ cores * **RAM:** Minimum 1 GB, recommended 2 GB, production 8+ GB * **Disk:** Minimum 20 GB, recommended 50 GB, production 100+ GB
💡 **Important notes for planning:** * Plan cluster size according to your workloads * Account for high-availability requirements * Observe network-policy requirements * Plan storage resources carefully
**Software requirements:** **Operating system:** ```bash # Check the Ubuntu version lsb_release -a # Minimum requirements: # - Ubuntu 20.04 LTS or newer # - Linux kernel 3.10 or higher uname -r ``` **System configuration:** ```bash # Check the required kernel modules lsmod | grep -e br_netfilter -e overlay # If they are not loaded, enable the modules: cat < ⚠️ **Important:** After the system update you should check whether a reboot is required and make sure swap is disabled: ```bash # Disable swap sudo swapoff -a # Comment out the swap entry in /etc/fstab sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab # Check whether a reboot is required if [ -f /var/run/reboot-required ]; then echo 'Reboot required!' fi ``` **Container runtime installation:** Kubernetes needs a container runtime. We use containerd, because it is the current standard: ```bash # Add the Docker repository curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null # Install containerd sudo apt update sudo apt install -y containerd.io ``` **Configuration of containerd:** ```bash # Create the default configuration sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml # Enable SystemdCgroup sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml # Restart containerd sudo systemctl restart containerd sudo systemctl enable containerd ```
💡 **Best practices for containerd:** * Configure SystemdCgroup * Set appropriate resource limits * Enable log rotation
### Kubernetes component installation ```bash # Add the Kubernetes repository curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list # Install Kubernetes components sudo apt update sudo apt install -y kubelet kubeadm kubectl # Pin versions to prevent unwanted updates sudo apt-mark hold kubelet kubeadm kubectl ```
⚠️ **Important:** Check the installed versions:
```bash kubectl version --client kubeadm version kubelet --version ```
💡 **Typical pitfalls:** * Forgetting to enable kernel modules * Incorrect network configuration * Insufficient resources
### Cluster initialisation **Control Plane setup:** ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Cluster initialisation and node join │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────┬─────────────────────────┐ │ │ │ Control Plane (Master) │ Worker nodes │ │ │ ├───────────────────────────┼─────────────────────────┤ │ │ │ • kubeadm init │ • kubeadm join │ │ │ │ • CNI plugin installation │ • Token authentication │ │ │ │ • Admin kubeconfig created│ • Register kubelet │ │ │ └───────────────────────────┴─────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Cluster CNI network mesh (Calico / Flannel / Cilium) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ```
⚠️ **Check before initialisation:** * Every prerequisite is met * Enough resources are available * Network ports are reachable * Hostname and DNS are configured correctly
```bash # Initialise the cluster sudo kubeadm init \ --pod-network-cidr=192.168.0.0/16 \ --kubernetes-version=$(kubeadm version -o short) \ --control-plane-endpoint="$(hostname -I | awk '{print $1}')" \ --upload-certs # Set up kubeconfig for the current user mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ```
💡 **Important notes on cluster initialisation:** * Write down the `kubeadm join` command printed at the end of initialisation * Back up certificates and token information * The control plane node is not intended for workloads by default
### Network plugin installation We use Calico as the CNI plugin because of its flexibility and advanced network features: ```bash # Install the Calico operator kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml # Create Calico custom resources kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml ``` **Network policy configuration:** ```yaml # Example of a basic network policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress spec: podSelector: {} policyTypes: - Ingress ```
⚠️ **Typical pitfalls in network configuration:** * Overlapping pod CIDR ranges * Missing network policies * Incorrect MTU settings * DNS problems
**Check the installation:** ```bash # Check cluster status kubectl get nodes kubectl get pods --all-namespaces kubectl cluster-info # Check CNI status kubectl get pods -n calico-system ``` ### Worker node integration ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Worker node integration workflow │ ├─────────────────────────────────────────────────────────────┤ │ │ │ 1. Preparation ──► Kernel modules and swap off │ │ │ │ │ ▼ │ │ 2. Container runtime ──► Install containerd │ │ │ │ │ ▼ │ │ 3. Join and taints ──► kubeadm join with token and CA │ │ │ │ │ ▼ │ │ 4. Validation ──► kubectl get nodes (Ready) │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Worker node preparation:** ```bash # System update and base package installation sudo apt update && sudo apt upgrade -y sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release # Container runtime (containerd) installation curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt update sudo apt install -y containerd.io ```
💡 **Important preparation steps:** * Set a unique hostname * Adjust firewall rules * Check resources * Configure SELinux/AppArmor
**Worker node configuration:** ```bash # Configure containerd sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml sudo systemctl restart containerd sudo systemctl enable containerd # Install Kubernetes components curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt update sudo apt install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl ``` **Worker node join:** ```bash # Generate a token on the control plane node kubeadm token create --print-join-command # Run on the worker node sudo kubeadm join :6443 \ --token \ --discovery-token-ca-cert-hash sha256: ```
💡 **Important notes:** * The join command must be run with root rights * Tokens are valid for 24 hours by default * If a token has expired, create a new one: `kubeadm token create --print-join-command`
**Node labels and taints:** ```bash # Node labelling for specific workloads kubectl label node worker-1 workload=production kubectl label node worker-2 workload=staging # Taint for dedicated workloads kubectl taint nodes worker-1 dedicated=production:NoSchedule ```
💡 **Best practices for node management:** * Implement systematic naming conventions * Use labels for workload segregation * Plan node capacity carefully * Implement node auto-scaling
**Validation and troubleshooting:** ```bash # Check node status kubectl get nodes -o wide # Show node details kubectl describe node worker-1 # System component status kubectl get pods -n kube-system ``` ### Advanced network configuration **Network policies:** ```yaml # Example of a basic network policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress spec: podSelector: {} policyTypes: - Ingress ``` **Service network configuration:** ```bash # Check the service CIDR configuration kubectl cluster-info dump | grep -m 1 service-cluster-ip-range # Check CoreDNS status kubectl get pods -n kube-system -l k8s-app=kube-dns ``` **Load balancing:** ```yaml # Example of a LoadBalancer service apiVersion: v1 kind: Service metadata: name: example-service spec: type: LoadBalancer ports: - port: 80 targetPort: 8080 selector: app: example ``` ### Checking the installation ```bash # Check node status kubectl get nodes -o wide # Check system pods kubectl get pods --all-namespaces # Cluster component status kubectl get componentstatuses # Test network functionality kubectl run test-pod --image=busybox -- sleep 3600 kubectl exec test-pod -- ping -c 3 8.8.8.8 ```
⚠️ **Important:** Document every configuration step you take and store important information such as tokens and certificates safely.
## Practical implementation This section covers the practical use of Kubernetes. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Kubernetes resource categories │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Workloads: Pods, Deployments, StatefulSets, DaemonS. │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ Discovery and routing: Services, Ingress, EndpointSl. │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ Storage: Volumes, PersistentVolumes, StorageClasses │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ Configuration: ConfigMaps, Secrets, ResourceQuotas │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### Deployment strategies Kubernetes offers several strategies for deploying applications.
💡 **Best practices for deployments:** * Always set resource limits * Define readiness/liveness probes * Use labels for better organisation * Plan update strategies carefully
**Rolling updates:** Rolling updates are the default strategy in Kubernetes and allow updates without downtime: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: example-deployment spec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 1 selector: matchLabels: app: example template: metadata: labels: app: example spec: containers: - name: nginx image: nginx:1.21 ports: - containerPort: 80 resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" ```
💡 **Important notes on rolling updates:** * The parameters `maxSurge` and `maxUnavailable` determine update speed * Readiness probes are critical for successful rolling updates * Set resource limits and requests for predictable behaviour
**Blue-green deployments:** Blue-green deployments enable low-risk updates through parallel environments: ```yaml # Blue Deployment apiVersion: apps/v1 kind: Deployment metadata: name: app-blue spec: replicas: 3 selector: matchLabels: app: myapp version: blue template: metadata: labels: app: myapp version: blue spec: containers: - name: myapp image: myapp:1.0 --- # Service for the blue-green switch apiVersion: v1 kind: Service metadata: name: myapp-service spec: selector: app: myapp version: blue # Change to 'green' for the switch ports: - port: 80 targetPort: 8080 ``` **Canary deployments:** Canary deployments allow stepwise testing of new versions with a portion of the traffic: ```yaml # Stable Deployment (90% traffic) apiVersion: apps/v1 kind: Deployment metadata: name: app-stable spec: replicas: 9 selector: matchLabels: app: myapp version: stable template: metadata: labels: app: myapp version: stable spec: containers: - name: myapp image: myapp:1.0 --- # Canary Deployment (10% traffic) apiVersion: apps/v1 kind: Deployment metadata: name: app-canary spec: replicas: 1 selector: matchLabels: app: myapp version: canary template: metadata: labels: app: myapp version: canary spec: containers: - name: myapp image: myapp:2.0 ```
⚠️ **Important for canary deployments:** * Implement thorough monitoring * Plan rollback strategies * Use a service mesh for fine-grained traffic control
### Service and Ingress configuration ```markdown ┌─────────────────────────────────────────────────────────────┐ │ External traffic flow in the cluster │ ├─────────────────────────────────────────────────────────────┤ │ │ │ External user request (HTTPS) │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Ingress controller (Nginx / Traefik / Envoy) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Host / path routing │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Kubernetes Service (ClusterIP / NodePort) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Round-robin load balancing │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Pods (application containers with a local IP) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Service types:**
💡 **Important:** Choose the service type based on your requirements: * **ClusterIP:** Internal * **NodePort:** Directly via node ports * **LoadBalancer:** Cloud-provider integration
```yaml # ClusterIP (default) apiVersion: v1 kind: Service metadata: name: backend-service spec: type: ClusterIP selector: app: backend ports: - port: 80 targetPort: 8080 --- # NodePort apiVersion: v1 kind: Service metadata: name: frontend-service spec: type: NodePort selector: app: frontend ports: - port: 80 targetPort: 80 nodePort: 30080 --- # LoadBalancer apiVersion: v1 kind: Service metadata: name: public-service spec: type: LoadBalancer selector: app: public ports: - port: 80 targetPort: 8080 ``` **Ingress configuration:** Ingress enables layer-7 routing and SSL termination: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: example.com http: paths: - path: /api pathType: Prefix backend: service: name: api-service port: number: 80 - path: / pathType: Prefix backend: service: name: web-service port: number: 80 ```
💡 **Best practices:** * Configure SSL/TLS * Set rate limiting * Implement path-based routing * Use meaningful annotations
**Service mesh integration:** ```yaml # Virtual Service (Istio) apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews-route spec: hosts: - reviews http: - match: - headers: end-user: exact: jason route: - destination: host: reviews subset: v2 - route: - destination: host: reviews subset: v1 ``` ### Storage management **Persistent Volumes:** ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Persistent storage architecture │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ StorageClass (dynamic provisioning / CSI driver) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Provisioning │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ PersistentVolume (PV — real storage resource) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Bind (claim) │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ PersistentVolumeClaim (PVC) ──► Pod volume mount │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: example-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: standard hostPath: path: /mnt/data --- # Persistent Volume Claim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: standard ``` **Storage Classes:** ```yaml # Definition of a Storage Class for SSD storage apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-storage provisioner: kubernetes.io/aws-ebs parameters: type: gp3 iopsPerGB: "10" encrypted: "true" reclaimPolicy: Retain allowVolumeExpansion: true ```
💡 **Important notes on Storage Classes:** * The provisioner depends on your cloud platform or storage solution * Parameters vary by provisioner * The reclaimPolicy determines what happens to the volume after the PVC is deleted
**Volume snapshots:** ```yaml # Volume Snapshot Class apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-hostpath-snapclass driver: hostpath.csi.k8s.io deletionPolicy: Delete --- # Volume Snapshot apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: data-snapshot spec: volumeSnapshotClassName: csi-hostpath-snapclass source: persistentVolumeClaimName: example-pvc ``` **Encrypted storage:** ```yaml # Secret for the encryption key apiVersion: v1 kind: Secret metadata: name: luks-key type: Opaque data: key: --- # Encrypted PersistentVolume apiVersion: v1 kind: PersistentVolume metadata: name: encrypted-pv spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce encryption: secretRef: name: luks-key hostPath: path: /mnt/encrypted-data ```
⚠️ **Important security notes:** * Encryption keys must be managed securely * Implement regular backup strategies * Configure access rights carefully
**Best practices for storage management:** 1. **Capacity planning:** Implement proactive monitoring, plan growth ahead, set sensible quotas. 2. **Backup strategies:** Regular snapshots, off-site backups, disaster recovery tests. 3. **Performance optimisation:** Choose matching storage classes, optimise chunk sizes, implement caching where it makes sense. ## Monitoring and maintenance Monitoring and maintenance of a Kubernetes cluster are decisive for stable operation. ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Monitoring and maintenance overview │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────┬─────────────────────────┐ │ │ │ Monitoring and observability│ Cluster maintenance │ │ │ ├───────────────────────────┼─────────────────────────┤ │ │ │ • Node and pod metrics │ • kubeadm upgrade │ │ │ │ • Prometheus alerts │ • Node drain and cordon │ │ │ │ • Log aggregation (Loki) │ • etcd snapshot backups │ │ │ │ • Tracing (Jaeger) │ • Certificate update │ │ │ └───────────────────────────┴─────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### Monitoring architecture ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Prometheus and Grafana monitoring stack │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Grafana dashboards (visualisation and alert panels) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ PromQL queries │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Prometheus server (metrics scraping and TSDB) │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ ├── Node-Exporter ├── cAdvisor (container) │ │ │ │ └── kube-state-metrics └── Alertmanager (notif.) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### Prometheus installation ```yaml # Create the namespace apiVersion: v1 kind: Namespace metadata: name: monitoring --- # Prometheus Operator installation apiVersion: apps/v1 kind: Deployment metadata: name: prometheus-operator namespace: monitoring spec: replicas: 1 selector: matchLabels: app: prometheus-operator template: metadata: labels: app: prometheus-operator spec: containers: - name: prometheus-operator image: quay.io/prometheus-operator/prometheus-operator:v0.59.1 args: - --kubelet-service=kube-system/kubelet - --config-reloader-image=jimmidyson/configmap-reload:v0.5.0 ```
💡 **Important notes on Prometheus installation:** * Make sure enough resources are available * Configure retention policies according to your requirements * Plan the scaling of storage requirements
### Monitoring configuration **Service monitoring:** ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: app-monitor namespace: monitoring spec: selector: matchLabels: app: myapp endpoints: - port: metrics interval: 15s path: /metrics ``` **Alert rules:** ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: node-alerts namespace: monitoring spec: groups: - name: node.rules rules: - alert: HighCPUUsage expr: node_cpu_usage_percentage > 80 for: 5m labels: severity: warning annotations: description: "CPU usage is above 80% for 5 minutes" ``` ### Alerting and notifications ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kubernetes-alerts namespace: monitoring spec: groups: - name: kubernetes.rules rules: - alert: KubernetesPodCrashLooping expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 5 > 0 for: 15m labels: severity: warning annotations: description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping" summary: "Pod is crash looping" ```
💡 **Important notes on alerting:** * Define sensible thresholds * Avoid alert fatigue from too many notifications * Implement escalation paths
**Resource quotas monitoring:** ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources spec: hard: requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi ``` **Log aggregation:** ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Log aggregation architecture │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Log sources: pods (stdout/stderr) and node systemd logs │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Log collector DaemonSet (Promtail / Fluentd / Vector) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Batch push (JSON/Loki) │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Central log store (Grafana Loki / OpenSearch) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ LogQL / dashboards │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Analysis, alerts and incident response in Grafana │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### Maintenance strategies **Cluster updates:** ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Rolling update workflow for cluster nodes │ ├─────────────────────────────────────────────────────────────┤ │ │ │ 1. Preparation ──► Create etcd snapshot and plan │ │ │ │ │ ▼ │ │ 2. Node cordon ──► kubectl cordon (stop scheduling) │ │ │ │ │ ▼ │ │ 3. Node drain ──► kubectl drain (evacuate) │ │ │ │ │ ▼ │ │ 4. Node upgrade ──► kubeadm upgrade and kubelet restart │ │ │ │ │ ▼ │ │ 5. Node uncordon──► kubectl uncordon (release) │ │ │ └─────────────────────────────────────────────────────────────┘ ``` **Update procedure for the control plane:** ```bash # 1. Back up important components ETCD_BACKUP_DIR="/backup/etcd-$(date +%Y%m%d)" mkdir -p $ETCD_BACKUP_DIR # Create an etcd snapshot kubectl exec -n kube-system etcd-master-1 -- etcdctl snapshot save \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ $ETCD_BACKUP_DIR/etcd-snapshot.db ```
⚠️ **Important notes on the update process:** * Always run updates in a test environment first * Plan sufficient maintenance windows * Keep rollback procedures ready * Document every step carefully
### Backup and recovery ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Backup strategy for Kubernetes │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ 1. etcd data backup (etcdctl snapshot save) │ │ │ │ * Saves cluster state, secrets, CRDs and status │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ 2. Declarative manifests (GitOps / ArgoCD / Helm) │ │ │ │ * Infrastructure as code in the Git repository │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ 3. Persistent data (Velero / CSI volume snapshots) │ │ │ │ * Backs up databases and filesystems on PVs │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ```bash #!/bin/bash # Backup script for critical cluster components BACKUP_DIR="/backup/k8s-$(date +%Y%m%d)" RETENTION_DAYS=30 mkdir -p "${BACKUP_DIR}"/{etcd,manifests,secrets} # etcd backup etcdctl snapshot save "${BACKUP_DIR}/etcd/snapshot.db" # Back up Kubernetes manifests kubectl get all --all-namespaces -o yaml > \ "${BACKUP_DIR}/manifests/all-resources.yaml" # Back up secrets (encrypted) kubectl get secrets --all-namespaces -o yaml | \ gpg --encrypt > "${BACKUP_DIR}/secrets/all-secrets.yaml.gpg" # Clean old backups find /backup -type d -mtime +${RETENTION_DAYS} -exec rm -rf {} + ``` ## Security and best practices The security of a Kubernetes cluster is of decisive importance. ### RBAC (Role-Based Access Control) ```markdown ┌─────────────────────────────────────────────────────────────┐ │ RBAC role and rights structure │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Subjects (users, groups and ServiceAccounts) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Binding via RoleBinding │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Role (namespace-specific: read/write pods) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ Binding via ClusterRoleBinding │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ ClusterRole (cluster-wide: nodes, namespaces, PVs) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ```yaml # Namespace-specific role apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: development name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- # Cluster-wide role apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: pod-reader-global rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] ```
💡 **Important RBAC principles:** * Principle of least privilege * Regular audit reviews * Documentation of every role and permission
### Pod Security Policies ```yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false seLinux: rule: RunAsAny runAsUser: rule: MustRunAsNonRoot fsGroup: rule: RunAsAny volumes: - 'configMap' - 'emptyDir' - 'persistentVolumeClaim' ``` ### Network Policies ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: api-allow spec: podSelector: matchLabels: app: api policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 ``` ### Secrets management ```yaml # Secret definition apiVersion: v1 kind: Secret metadata: name: app-secrets type: Opaque data: db-password: BASE64_ENCODED_PASSWORD api-key: BASE64_ENCODED_API_KEY ```
💡 **Best practices for secrets:** * Encryption at rest * Regular rotation * Access restriction * Audit logging
### Container security ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Container security layered model │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Build phase: minimal base images and Trivy scanning │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ Registry phase: Cosign image signing and admission │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ Runtime phase: non-root user and read-only rootfs │ │ │ ├───────────────────────────────────────────────────────┤ │ │ │ Network phase: strict default-deny NetworkPolicies │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### Cluster hardening ```yaml # Pod Security Policy apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret' - 'downwardAPI' - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: 'MustRunAsNonRoot' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: - min: 1 max: 65535 fsGroup: rule: 'MustRunAs' ranges: - min: 1 max: 65535 ``` ### Audit logging ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: RequestResponse resources: - group: "" resources: ["pods", "services"] - level: Metadata resources: - group: "" resources: ["configmaps"] - level: None users: ["system:kube-proxy"] resources: - group: "" resources: ["endpoints"] ``` ### Automated security checks ```yaml # kube-bench job for automated CIS checks apiVersion: batch/v1 kind: Job metadata: name: kube-bench spec: template: spec: hostPID: true containers: - name: kube-bench image: aquasec/kube-bench:latest command: ["kube-bench", "--benchmark", "cis-1.6"] volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet - name: etc-systemd mountPath: /etc/systemd - name: etc-kubernetes mountPath: /etc/kubernetes restartPolicy: Never volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" ``` ## Advanced concepts ### Custom Resources Custom Resources (CRs) extend the Kubernetes API with user-defined resource types. ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: webapps.mycompany.com spec: group: mycompany.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: image: type: string replicas: type: integer minimum: 1 scope: Namespaced names: plural: webapps singular: webapp kind: WebApp shortNames: - wa ``` ### Operators Operators automate the management of complex applications in Kubernetes. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: mysql-operator spec: replicas: 1 selector: matchLabels: name: mysql-operator template: metadata: labels: name: mysql-operator spec: containers: - name: operator image: mysql-operator:v1.0 ports: - containerPort: 8080 env: - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ``` ### Service Mesh ```markdown ┌─────────────────────────────────────────────────────────────┐ │ Service mesh architecture (Istio / Linkerd) │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Control plane (Istiod: configuration and certificates)│ │ │ └───────────────────────────────────────────────────────┘ │ │ │ mTLS and routing rules │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Data plane: sidecar proxies (Envoy beside every pod) │ │ │ │ ├── Service A (Pod) ◄──► Envoy sidecar proxy │ │ │ │ │ │ Encrypted (mTLS) │ │ │ │ └── Service B (Pod) ◄──► Envoy sidecar proxy │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ## Exercise This exercise deploys a web application with all important Kubernetes concepts. ### Exercise goals * Create a deployment with several replicas * Configure a service and ingress * Implement monitoring * Apply security policies ### Step 1: Create the deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: webapp spec: replicas: 3 selector: matchLabels: app: webapp template: metadata: labels: app: webapp spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" ``` ### Step 2: Configure the service ```yaml apiVersion: v1 kind: Service metadata: name: webapp-service spec: selector: app: webapp ports: - port: 80 targetPort: 80 type: ClusterIP ``` ### Step 3: Set up Ingress ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webapp-ingress spec: rules: - host: webapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: webapp-service port: number: 80 ``` ### Step 4: Enable monitoring ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: webapp-monitor spec: selector: matchLabels: app: webapp endpoints: - port: metrics ``` ### Step 5: Apply a security policy ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: webapp-policy spec: podSelector: matchLabels: app: webapp ingress: - from: - podSelector: matchLabels: access: allowed ``` ### Checking the exercise ```bash # Check deployment status kubectl get deployments kubectl get pods # Check service and ingress kubectl get services kubectl get ingress # Test access curl -H "Host: webapp.example.com" http://CLUSTER-IP ``` ## Command Reference (Cheatsheet) For quick access in day-to-day operations, the following reference table summarises the most important Kubernetes commands for cluster management, fault diagnosis and resource scaling: | Command / syntax | Description and practical use | |---|---| | `kubectl get nodes -o wide` | Lists every cluster node including status, version, kernel and internal IP. | | `kubectl get pods -A` | Shows every pod across all namespaces plus status and restart counters. | | `kubectl describe pod ` | Detailed fault diagnosis with events, container status and exit codes. | | `kubectl logs -f [-c ]` | Streams live logs of a pod or a specific sidecar container. | | `kubectl exec -it -- /bin/sh` | Opens an interactive shell in the container for live inspection. | | `kubectl apply -f ` | Declarative rollout or update of Kubernetes resources. | | `kubectl rollout status deployment/` | Monitors the progress of a rolling update in real time. | | `kubectl rollout undo deployment/` | Performs an immediate rollback to the previous revision. | | `kubectl get services,ingress` | Shows active services, ClusterIPs, NodePorts and Ingress routes. | | `kubectl get pv,pvc` | Overview of PersistentVolumes and their bind status to claims. | | `kubectl top nodes && kubectl top pods` | Shows current CPU and memory use of nodes and pods. | | `kubectl cordon && kubectl drain ` | Blocks a node for new pods and evacuates existing workloads safely. | | `kubectl uncordon ` | Releases a maintained worker node for scheduling again. | | `kubeadm init --pod-network-cidr=192.168.0.0/16` | Initialises the Kubernetes control plane with a CNI network range. | | `kubeadm join :6443 --token --discovery-token-ca-cert-hash ` | Joins a worker node to the existing cluster. | | `kubeadm upgrade plan` | Checks available versions and shows upgrade steps for the control plane. | | `etcdctl snapshot save snapshot.db` | Creates a consistent point-in-time snapshot of etcd state. | ## Further Resources The following official documentation, tutorials and community projects go deeper into the concepts and best practices covered here: | Resource | Description | |---|---| | [Kubernetes documentation](https://kubernetes.io/docs/home/){.badge-link-text} | Official and complete documentation of every K8s component and API. | | [Kubernetes tutorials](https://kubernetes.io/docs/tutorials/){.badge-link-text} | Practice-oriented walkthroughs for deployments, services and scaling. | | [Kubernetes blog](https://kubernetes.io/blog/){.badge-link-text} | Current announcements, release notes and in-depth architecture pieces. | | [Interactive basics tutorial](https://kubernetes.io/docs/tutorials/kubernetes-basics/){.badge-link-text} | Interactive introduction to cluster creation and workload management. | | [Minikube getting started](https://minikube.sigs.k8s.io/docs/start/){.badge-link-text} | Official reference for running single-node clusters locally. | | [Kubernetes training labs](https://kubernetes.io/training/){.badge-link-text} | CNCF training materials and interactive exercise environments. | | [Kubernetes GitHub repository](https://github.com/kubernetes/kubernetes){.badge-link-text} | Source code, issue tracker and technical design proposals (KEPs). | | [Kubernetes community](https://kubernetes.io/community/){.badge-link-text} | Forums, special interest groups (SIGs) and Slack channels of the community. | | [CNCF Kubernetes project](https://www.cncf.io/projects/kubernetes/){.badge-link-text} | Governance, ecosystem overview and graduation status within the CNCF. | | [kubectl cheat sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/){.badge-link-text} | Official reference of every `kubectl` option and JSONPath query. | | [Helm package manager](https://helm.sh/){.badge-link-text} | Package management for Kubernetes for templating and distributing apps. | | [Lens Kubernetes IDE](https://k8slens.dev/){.badge-link-text} | Powerful graphical user interface for cluster management. |
💡 **Note:** The Kubernetes landscape evolves constantly. Always consult the latest documentation.
## Conclusion Kubernetes has become the de-facto standard for container orchestration and provides a robust platform for modern cloud-native applications. The concepts and practices covered here show the versatility and capability of Kubernetes. From basic installation through practical implementation to advanced concepts such as Custom Resources and a service mesh, Kubernetes supplies the tools needed to operate complex container environments. Continuous development of Kubernetes, especially in automation and security, keeps the platform viable. With growing adoption of cloud-native technologies and the rising importance of DevOps practices, Kubernetes will continue to play a central role in modern IT infrastructure. New developments such as GitOps, KubeVirt for VM management and tighter integration of AI-assisted operations will make the platform even more capable. 👉 **Overview:** [All DevOps articles](/en/category/devops){.badge-link-text}