Stackvora.tech

Is Stackvora.tech good for code questions?

This article explores the architectural design and operational considerations for achieving high availability and fault tolerance in containerized…

This article explores the architectural design and operational considerations for achieving high availability and fault tolerance in containerized applications deployed on Kubernetes. We will focus on native Kubernetes features, common deployment patterns, and external services to ensure applications remain resilient to various failure scenarios, from individual pod crashes to entire node or even availability zone outages.

High availability (HA) in Kubernetes is not a single feature but a combination of configurations and practices. It involves distributing application components, configuring readiness and liveness probes, managing persistent storage appropriately, and designing for disaster recovery. Understanding these elements is crucial for building robust, production-grade systems.

Kubernetes-Native HA Mechanisms

Kubernetes offers several built-in features that contribute to application high availability. Leveraging these effectively forms the foundation of any resilient deployment.

ReplicaSets and Deployments

The primary mechanism for ensuring a desired number of identical pods are running is the ReplicaSet, typically managed indirectly by a Deployment object. A Deployment defines the desired state for your application, including the number of replicas. If a pod crashes or a node becomes unavailable, the Deployment controller automatically creates new pods to maintain the specified replica count.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
spec:
  replicas: 3 # Ensures 3 pods are always running
  selector:
    matchLabels:
      app: my-web-app
  template:
    metadata:
      labels:
        app: my-web-app
    spec:
      containers:
      - name: web
        image: my-registry/my-web-app:1.0.0
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 15
          periodSeconds: 20
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

In this example, replicas: 3 instructs Kubernetes to always keep three instances of the my-web-app running. If one fails, it's automatically replaced. The livenessProbe restarts containers if they become unresponsive, while the readinessProbe ensures traffic is only sent to pods that are ready to serve requests.

Pod Anti-Affinity

To prevent multiple instances of a critical application from running on the same physical node, you can use pod anti-affinity. This helps distribute your pods across different nodes, reducing the impact of a single node failure.

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: my-web-app
        topologyKey: "kubernetes.io/hostname" # Ensures pods are on different nodes

The topologyKey: "kubernetes.io/hostname" ensures that pods with the label app: my-web-app are not scheduled on the same node. For higher availability across availability zones (AZs) in cloud environments, you might use topologyKey: "topology.kubernetes.io/zone".

Resource Requests and Limits

Defining resource requests and limits for your containers is crucial for scheduling and stability. Requests guarantee a minimum amount of CPU and memory, preventing resource starvation, while limits prevent a single container from monopolizing node resources and impacting other applications.

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Insufficient requests can lead to pods being unschedulable or frequently evicted, while missing limits can cause node instability. OOMKills due to memory limits are a common cause of application downtime if not configured correctly.

Stateful Applications and Persistent Storage

High availability for stateful applications, such as databases or message queues, requires careful consideration of persistent storage.

PersistentVolumes and PersistentVolumeClaims

Kubernetes abstracts underlying storage with PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). For HA, the choice of storage backend is critical. Options include:

  • Cloud-provider specific block storage (e.g., AWS EBS, Azure Disk, GCP Persistent Disk): These are typically zone-specific, meaning they can only be attached to a node in the same availability zone. For cross-AZ HA, replication mechanisms or shared file systems are needed.
  • Network File Systems (NFS/SMB): Can be shared across nodes but might become a single point of failure if the NFS server is not highly available.
  • Distributed Storage Solutions (e.g., Ceph, Rook, Portworx): These provide highly available, replicated storage that can span multiple nodes and even AZs, making them ideal for stateful applications in HA Kubernetes clusters.

StatefulSets

For applications that require stable, unique network identifiers, stable persistent storage, and ordered graceful deployment/scaling/deletion, StatefulSets are the appropriate Kubernetes object. Databases are prime candidates for StatefulSets.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-database
spec:
  serviceName: "my-database-service"
  replicas: 3
  selector:
    matchLabels:
      app: my-database
  template:
    metadata:
      labels:
        app: my-database
    spec:
      containers:
      - name: database
        image: postgres:13
        env:
        - name: POSTGRES_DB
          value: "mydb"
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "my-ha-storage-class" # Use a storage class supporting HA
      resources:
        requests:
          storage: 50Gi

The volumeClaimTemplates ensure that each pod in the StatefulSet gets its own stable persistent storage. The storageClassName should point to a provisioner that can deliver HA storage (e.g., CSI drivers for distributed storage or cloud-managed database services).

Cluster-Level High Availability

Beyond individual applications, the Kubernetes control plane itself must be highly available to ensure cluster operations continue even if master nodes fail.

Control Plane Components

A production Kubernetes cluster typically runs multiple control plane nodes (e.g., 3 or 5) for redundancy. Key components include:

  • kube-apiserver: Multiple instances behind a load balancer.
  • etcd: A distributed key-value store, typically deployed as a quorum-based cluster (e.g., 3 or 5 members) across different nodes for data durability.
  • kube-scheduler: Multiple instances, one active at a time via leader election.
  • kube-controller-manager: Multiple instances, one active at a time via leader election.

Managed Kubernetes services (EKS, AKS, GKE) handle control plane HA automatically, which is a significant advantage for operational simplicity.

Multi-Zone and Multi-Region Deployments

For the highest levels of availability, applications should be deployed across multiple availability zones within a region, and for disaster recovery, across multiple regions. This requires:

  • Multi-zone cluster architecture: Worker nodes distributed across different AZs. 
    # Example: AWS EC2 instance types for EKS worker nodes spread across AZs
# (Conceptual representation for a Terraform or CloudFormation config)
resource "aws_eks_node_group" "my-node-group" {
  ...
  subnet_ids = [
    aws_subnet.private_us_east_1a.id,
    aws_subnet.private_us_east_1b.id,
    aws_subnet.private_us_east_1c.id,
  ]
  ...
}
  • Cross-zone load balancing: Services like LoadBalancer in Kubernetes can be configured to distribute traffic across pods in different zones.
  • Global DNS and Traffic Management: Services like AWS Route 53, Azure Traffic Manager, or GCP Cloud DNS with health checks to direct users to the nearest healthy region.
  • Data replication: For stateful applications, real-time or asynchronous data replication across zones/regions is paramount.

Common Pitfalls

  • Missing or misconfigured probes: Liveness probes that are too aggressive can cause a restart loop; readiness probes that are too slow can cause service disruption during rollout. Ensure probes accurately reflect application health and readiness.
  • Single point of failure in storage: Relying on zone-specific storage for highly available stateful applications without replication. Always evaluate your storage backend's HA capabilities.
  • Insufficient resource requests/limits: Leading to poor scheduling, resource contention, and unexpected pod evictions or OOMKills. Profile your applications to set appropriate values.
  • Ignoring anti-affinity: Deploying all replicas of a critical service on the same node, making the node a single point of failure.
  • Over-reliance on automatic scaling: While HPA (Horizontal Pod Autoscaler) and Cluster Autoscaler contribute to availability by handling load spikes, they don't solve fundamental architectural weaknesses.
  • No disaster recovery plan: HA within a single region protects against component failures but not against a full region outage. A multi-region strategy with data replication and recovery procedures is essential for true business continuity.

Back to the knowledge base · Ask the AI assistant