Containerization and Kubernetes: Lessons Learned in 2022

2022 was a maturing year for container adoption. Kubernetes became the de facto standard, but organizations also learned that not everything needs Kubernetes. Let’s review the lessons learned.

When to Use Kubernetes (and When Not To)

# Decision framework for container orchestration
def recommend_platform(workload_characteristics: dict) -> str:
    """
    Recommend container platform based on workload characteristics.
    """

    score = {
        "kubernetes": 0,
        "container_apps": 0,
        "app_service": 0,
        "functions": 0
    }

    # Scale requirements
    if workload_characteristics.get("scale_to_thousands"):
        score["kubernetes"] += 3
        score["container_apps"] += 2
    elif workload_characteristics.get("scale_to_hundreds"):
        score["kubernetes"] += 2
        score["container_apps"] += 3
    else:
        score["app_service"] += 2
        score["functions"] += 2

    # Complexity
    if workload_characteristics.get("complex_networking"):
        score["kubernetes"] += 3
    if workload_characteristics.get("service_mesh_needed"):
        score["kubernetes"] += 3
    if workload_characteristics.get("custom_operators"):
        score["kubernetes"] += 3

    # Team expertise
    if workload_characteristics.get("kubernetes_expertise"):
        score["kubernetes"] += 2
    else:
        score["container_apps"] += 2
        score["app_service"] += 2

    # Workload type
    if workload_characteristics.get("event_driven"):
        score["functions"] += 3
        score["container_apps"] += 2
    if workload_characteristics.get("long_running"):
        score["kubernetes"] += 2
        score["container_apps"] += 2
        score["app_service"] += 2

    # Cost sensitivity
    if workload_characteristics.get("cost_sensitive"):
        score["container_apps"] += 2
        score["app_service"] += 2
        score["functions"] += 3

    # Multi-cloud requirement
    if workload_characteristics.get("multi_cloud"):
        score["kubernetes"] += 3

    best_option = max(score, key=score.get)
    return best_option, score

# Example usage
workload = {
    "scale_to_hundreds": True,
    "complex_networking": False,
    "kubernetes_expertise": False,
    "event_driven": True,
    "cost_sensitive": True
}

recommendation, scores = recommend_platform(workload)
# Returns: ("container_apps", {...})

AKS Best Practices from 2022

// Production-ready AKS cluster
resource aks 'Microsoft.ContainerService/managedClusters@2022-11-01' = {
  name: 'aks-${environment}'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${managedIdentity.id}': {}
    }
  }
  properties: {
    dnsPrefix: 'aks-${environment}'
    kubernetesVersion: '1.25.5'

    // Enable Azure AD integration
    aadProfile: {
      managed: true
      enableAzureRBAC: true
      adminGroupObjectIDs: [adminGroupId]
    }

    // Network configuration
    networkProfile: {
      networkPlugin: 'azure'
      networkPolicy: 'calico'
      loadBalancerSku: 'standard'
      outboundType: 'userDefinedRouting'
      serviceCidr: '10.0.0.0/16'
      dnsServiceIP: '10.0.0.10'
    }

    // System node pool
    agentPoolProfiles: [
      {
        name: 'system'
        count: 3
        vmSize: 'Standard_D4s_v3'
        mode: 'System'
        osType: 'Linux'
        osDiskType: 'Ephemeral'
        osDiskSizeGB: 100
        availabilityZones: ['1', '2', '3']
        enableAutoScaling: true
        minCount: 3
        maxCount: 5
        vnetSubnetID: aksSubnet.id
        nodeTaints: ['CriticalAddonsOnly=true:NoSchedule']
        nodeLabels: {
          'node-type': 'system'
        }
      }
    ]

    // Security settings
    apiServerAccessProfile: {
      enablePrivateCluster: true
      privateDNSZone: privateDnsZone.id
    }

    autoUpgradeProfile: {
      upgradeChannel: 'stable'
    }

    disableLocalAccounts: true

    securityProfile: {
      defender: {
        securityMonitoring: {
          enabled: true
        }
        logAnalyticsWorkspaceResourceId: logAnalyticsWorkspace.id
      }
      workloadIdentity: {
        enabled: true
      }
    }

    // Monitoring
    addonProfiles: {
      omsagent: {
        enabled: true
        config: {
          logAnalyticsWorkspaceResourceId: logAnalyticsWorkspace.id
        }
      }
      azureKeyvaultSecretsProvider: {
        enabled: true
        config: {
          enableSecretRotation: 'true'
          rotationPollInterval: '2m'
        }
      }
    }

    oidcIssuerProfile: {
      enabled: true
    }
  }
}

// Workload node pool
resource workloadPool 'Microsoft.ContainerService/managedClusters/agentPools@2022-11-01' = {
  parent: aks
  name: 'workload'
  properties: {
    count: 5
    vmSize: 'Standard_D8s_v3'
    mode: 'User'
    osType: 'Linux'
    osDiskType: 'Ephemeral'
    osDiskSizeGB: 128
    availabilityZones: ['1', '2', '3']
    enableAutoScaling: true
    minCount: 3
    maxCount: 20
    vnetSubnetID: aksSubnet.id
    nodeLabels: {
      'node-type': 'workload'
    }
  }
}

Container Security Lessons

# Pod Security Standards (PSS) - Restricted
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

---
# Secure pod configuration
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  namespace: production
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault

  containers:
    - name: app
      image: myregistry.azurecr.io/app:v1.0.0@sha256:abc123...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

      resources:
        requests:
          memory: "256Mi"
          cpu: "250m"
        limits:
          memory: "512Mi"
          cpu: "500m"

      volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: secrets
          mountPath: /secrets
          readOnly: true

      livenessProbe:
        httpGet:
          path: /health/live
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 10

      readinessProbe:
        httpGet:
          path: /health/ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5

  volumes:
    - name: tmp
      emptyDir: {}
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: azure-keyvault

  serviceAccountName: app-service-account
  automountServiceAccountToken: false

# Container image security scanning
import subprocess
import json
from dataclasses import dataclass
from typing import List

@dataclass
class Vulnerability:
    cve_id: str
    severity: str
    package: str
    fixed_version: str
    description: str

class ImageScanner:
    def __init__(self, registry: str):
        self.registry = registry

    def scan_image(self, image: str, tag: str) -> List[Vulnerability]:
        """Scan container image for vulnerabilities."""

        full_image = f"{self.registry}/{image}:{tag}"

        # Using Trivy for scanning
        result = subprocess.run(
            ["trivy", "image", "--format", "json", full_image],
            capture_output=True,
            text=True
        )

        scan_results = json.loads(result.stdout)
        vulnerabilities = []

        for result in scan_results.get("Results", []):
            for vuln in result.get("Vulnerabilities", []):
                vulnerabilities.append(Vulnerability(
                    cve_id=vuln["VulnerabilityID"],
                    severity=vuln["Severity"],
                    package=vuln["PkgName"],
                    fixed_version=vuln.get("FixedVersion", "N/A"),
                    description=vuln.get("Description", "")
                ))

        return vulnerabilities

    def enforce_policy(self, vulnerabilities: List[Vulnerability],
                       policy: dict) -> bool:
        """Enforce security policy on scan results."""

        critical_count = sum(1 for v in vulnerabilities if v.severity == "CRITICAL")
        high_count = sum(1 for v in vulnerabilities if v.severity == "HIGH")

        if critical_count > policy.get("max_critical", 0):
            return False

        if high_count > policy.get("max_high", 5):
            return False

        return True

GitOps with Flux

# Flux GitOps configuration
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/company/k8s-infrastructure
  ref:
    branch: main
  secretRef:
    name: github-credentials

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: infrastructure
  path: ./clusters/production
  prune: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: api
      namespace: production
  timeout: 5m

---
# Image automation
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: app
  namespace: flux-system
spec:
  image: myregistry.azurecr.io/app
  interval: 5m

---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: app
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: app
  policy:
    semver:
      range: '>=1.0.0'

---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: app
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: infrastructure
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux
        email: flux@company.com
      messageTemplate: 'Update {{.AutomationObject.Name}} to {{.NewImage}}'
    push:
      branch: main
  update:
    path: ./apps/production
    strategy: Setters

Container Apps - The Simpler Alternative

// Azure Container Apps for simpler workloads
resource containerAppEnvironment 'Microsoft.App/managedEnvironments@2022-10-01' = {
  name: 'cae-${environment}'
  location: location
  properties: {
    daprAIInstrumentationKey: appInsights.properties.InstrumentationKey
    vnetConfiguration: {
      infrastructureSubnetId: infrastructureSubnet.id
    }
    workloadProfiles: [
      {
        name: 'Consumption'
        workloadProfileType: 'Consumption'
      }
      {
        name: 'Dedicated'
        workloadProfileType: 'D4'
        minimumCount: 1
        maximumCount: 10
      }
    ]
  }
}

resource containerApp 'Microsoft.App/containerApps@2022-10-01' = {
  name: 'api-${environment}'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${managedIdentity.id}': {}
    }
  }
  properties: {
    managedEnvironmentId: containerAppEnvironment.id
    workloadProfileName: 'Consumption'

    configuration: {
      activeRevisionsMode: 'Multiple'
      ingress: {
        external: true
        targetPort: 8080
        transport: 'http2'
        traffic: [
          {
            latestRevision: true
            weight: 100
          }
        ]
      }
      secrets: [
        {
          name: 'connection-string'
          keyVaultUrl: '${keyVault.properties.vaultUri}secrets/connection-string'
          identity: managedIdentity.id
        }
      ]
      registries: [
        {
          server: containerRegistry.properties.loginServer
          identity: managedIdentity.id
        }
      ]
    }

    template: {
      containers: [
        {
          name: 'api'
          image: '${containerRegistry.properties.loginServer}/api:latest'
          resources: {
            cpu: json('0.5')
            memory: '1Gi'
          }
          env: [
            {
              name: 'ConnectionString'
              secretRef: 'connection-string'
            }
            {
              name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
              value: appInsights.properties.ConnectionString
            }
          ]
          probes: [
            {
              type: 'Liveness'
              httpGet: {
                path: '/health/live'
                port: 8080
              }
              periodSeconds: 10
            }
            {
              type: 'Readiness'
              httpGet: {
                path: '/health/ready'
                port: 8080
              }
              periodSeconds: 5
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'http-scaling'
            http: {
              metadata: {
                concurrentRequests: '100'
              }
            }
          }
        ]
      }
    }
  }
}

Key Lessons from 2022

kubernetes_lessons_2022:
  architecture:
    - "Start simple, add complexity only when needed"
    - "Consider managed alternatives before self-managing"
    - "Multi-cluster is often overkill - one well-designed cluster can do a lot"
    - "Use namespaces effectively for isolation"

  security:
    - "Enable Pod Security Standards from day one"
    - "Use workload identity instead of cluster-level permissions"
    - "Scan images in CI/CD, not just registries"
    - "Network policies are essential, not optional"

  operations:
    - "GitOps reduces operational burden significantly"
    - "Invest in observability early"
    - "Automate cluster upgrades"
    - "Have a disaster recovery plan tested regularly"

  cost:
    - "Right-size node pools based on actual usage"
    - "Use spot nodes for appropriate workloads"
    - "Clean up unused resources automatically"
    - "Monitor cluster efficiency metrics"

  team:
    - "Platform teams should provide golden paths"
    - "Developers don't need to be Kubernetes experts"
    - "Document tribal knowledge"
    - "Invest in training"

Conclusion

2022 taught us that Kubernetes is powerful but not always necessary. Container Apps emerged as a strong alternative for simpler workloads. Security became non-negotiable with supply chain attacks increasing. GitOps matured as the preferred deployment model. As we enter 2023, focus on platform engineering - making Kubernetes accessible to developers without requiring deep expertise.

Resources

• AKS Best Practices
• Azure Container Apps
• Flux CD
• Pod Security Standards