GitOps Practices: Declarative Infrastructure and Application Delivery

GitOps emerged as the operational model for cloud-native applications in 2021. By using Git as the single source of truth, teams achieved faster deployments, better auditability, and more reliable operations.

The GitOps Principles

1. Declarative: Desired state is expressed declaratively
2. Versioned and Immutable: Git stores the canonical desired state
3. Pulled Automatically: Software agents pull the desired state
4. Continuously Reconciled: Agents continuously reconcile actual state

Setting Up Flux for GitOps

# Bootstrap Flux on AKS cluster
flux bootstrap github \
    --owner=myorg \
    --repository=infrastructure \
    --branch=main \
    --path=clusters/production \
    --personal

# clusters/production/flux-system/gotk-components.yaml
# This is auto-generated by bootstrap

# clusters/production/apps/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  validation: client
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: api-gateway
      namespace: production
  timeout: 5m

# clusters/production/infrastructure/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 1h
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/production
  prune: true
  validation: client
  dependsOn:
    - name: cert-manager

Application Deployment with GitOps

# apps/base/api-service/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  labels:
    app: api-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
        - name: api
          image: myregistry.azurecr.io/api-service:v1.2.3
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: ASPNETCORE_ENVIRONMENT
              value: "Production"

# apps/base/api-service/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - hpa.yaml

# apps/production/api-service/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base/api-service
patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
    target:
      kind: Deployment
      name: api-service
images:
  - name: myregistry.azurecr.io/api-service
    newTag: v1.2.3

Image Automation with Flux

# Automatically update images when new versions are available
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: api-service
  namespace: flux-system
spec:
  image: myregistry.azurecr.io/api-service
  interval: 5m
  secretRef:
    name: acr-credentials
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: api-service
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: api-service
  policy:
    semver:
      range: ">=1.0.0"
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: api-service
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux
        email: flux@company.com
      messageTemplate: 'Update {{.AutomationObject.Name}}'
    push:
      branch: main
  update:
    path: ./apps/production
    strategy: Setters

Progressive Delivery with Flagger

# Canary deployment configuration
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: api-service
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  progressDeadlineSeconds: 600
  service:
    port: 80
    targetPort: 8080
    gateways:
      - public-gateway.istio-system.svc.cluster.local
    hosts:
      - api.company.com
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://api-service-canary.production:80/"
---
# Custom metric template
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: request-success-rate
  namespace: production
spec:
  provider:
    type: prometheus
    address: http://prometheus.monitoring:9090
  query: |
    sum(rate(http_requests_total{
      namespace="{{ namespace }}",
      deployment="{{ target }}",
      status!~"5.*"
    }[{{ interval }}])) /
    sum(rate(http_requests_total{
      namespace="{{ namespace }}",
      deployment="{{ target }}"
    }[{{ interval }}])) * 100

Secret Management with Sealed Secrets

# Encrypt secrets for GitOps
kubeseal --controller-namespace kube-system \
    --controller-name sealed-secrets \
    --format yaml \
    < secret.yaml > sealed-secret.yaml

# sealed-secret.yaml (safe to commit to Git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: api-secrets
  namespace: production
spec:
  encryptedData:
    DATABASE_URL: AgBy8BZ1...encrypted...==
    API_KEY: AgBy8BZ2...encrypted...==
  template:
    metadata:
      name: api-secrets
      namespace: production
    type: Opaque

Multi-Cluster GitOps

# clusters/fleet/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - clusters/production-east
  - clusters/production-west
  - clusters/staging

# clusters/production-east/cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: production-east
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: production-east-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AzureCluster
    name: production-east

Drift Detection and Reconciliation

# Monitor GitOps reconciliation status
from kubernetes import client, config
import json

def check_kustomization_status():
    config.load_kube_config()
    api = client.CustomObjectsApi()

    kustomizations = api.list_namespaced_custom_object(
        group="kustomize.toolkit.fluxcd.io",
        version="v1beta2",
        namespace="flux-system",
        plural="kustomizations"
    )

    status_report = []
    for ks in kustomizations['items']:
        name = ks['metadata']['name']
        conditions = ks.get('status', {}).get('conditions', [])

        ready_condition = next(
            (c for c in conditions if c['type'] == 'Ready'),
            None
        )

        status_report.append({
            'name': name,
            'ready': ready_condition['status'] == 'True' if ready_condition else False,
            'message': ready_condition['message'] if ready_condition else 'Unknown',
            'last_applied_revision': ks.get('status', {}).get('lastAppliedRevision', 'N/A')
        })

    return status_report

def force_reconciliation(kustomization_name: str):
    """Force immediate reconciliation"""
    config.load_kube_config()
    api = client.CustomObjectsApi()

    # Annotate to trigger reconciliation
    patch = {
        "metadata": {
            "annotations": {
                "reconcile.fluxcd.io/requestedAt": datetime.utcnow().isoformat()
            }
        }
    }

    api.patch_namespaced_custom_object(
        group="kustomize.toolkit.fluxcd.io",
        version="v1beta2",
        namespace="flux-system",
        plural="kustomizations",
        name=kustomization_name,
        body=patch
    )

GitOps Best Practices

1. Separate Repos: Application code and deployment configs in different repos
2. Environment Promotion: Use branches or directories for environments
3. Automated Image Updates: Let tooling update image tags
4. Seal Your Secrets: Never commit plaintext secrets
5. Monitor Reconciliation: Alert on failed reconciliations

# Alert on GitOps failures
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gitops-alerts
  namespace: monitoring
spec:
  groups:
    - name: gitops
      rules:
        - alert: KustomizationNotReady
          expr: gotk_reconcile_condition{type="Ready",status="False"} == 1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Kustomization {{ $labels.name }} not ready"
            description: "Kustomization has been not ready for more than 10 minutes"

        - alert: GitRepositorySyncFailed
          expr: gotk_reconcile_condition{type="Ready",status="False",kind="GitRepository"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Git repository sync failed"

GitOps in 2021 became the standard for Kubernetes deployments. The combination of declarative configurations, Git as source of truth, and automated reconciliation delivers reliability that manual deployments cannot match.

Resources

• Flux Documentation
• ArgoCD Documentation
• GitOps Principles