June 29, 2021 6 min read

Kubernetes Operators for Azure Resources

Introduction

Kubernetes Operators extend the Kubernetes API to manage custom resources and automate complex operational tasks. For Azure, operators like Azure Service Operator enable provisioning and managing Azure resources directly from Kubernetes. This guide explores building and using operators for Azure integration.

Understanding Operators

Operator Pattern

┌─────────────────────────────────────────────────────────────────┐
│                     Kubernetes Cluster                           │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                    Control Plane                             ││
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      ││
│  │  │ API Server   │  │  Controller  │  │   etcd       │      ││
│  │  │              │◄─┤   Manager    │  │              │      ││
│  │  └──────────────┘  └──────────────┘  └──────────────┘      ││
│  └─────────────────────────────────────────────────────────────┘│
│                                ▲                                 │
│                                │ Watch & Reconcile               │
│  ┌─────────────────────────────┴───────────────────────────────┐│
│  │                    Custom Operator                           ││
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      ││
│  │  │   Custom     │  │  Reconciler  │  │   Azure      │      ││
│  │  │   Resource   │──┤   Logic      │──┤   SDK        │──────┼┼──► Azure
│  │  │  Definitions │  │              │  │              │      ││
│  │  └──────────────┘  └──────────────┘  └──────────────┘      ││
│  └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘

Azure Service Operator

Installation

# Install cert-manager (required)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

# Install Azure Service Operator
helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
helm repo update

helm install aso2 aso2/azure-service-operator \
    --create-namespace \
    --namespace azureserviceoperator-system \
    --set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
    --set azureTenantID=$AZURE_TENANT_ID \
    --set azureClientID=$AZURE_CLIENT_ID \
    --set azureClientSecret=$AZURE_CLIENT_SECRET

Creating Azure Resources

# resourcegroup.yaml
apiVersion: resources.azure.com/v1api20200601
kind: ResourceGroup
metadata:
  name: rg-myapp
  namespace: default
spec:
  location: australiaeast
  tags:
    environment: production
    managed-by: aso
---
# storageaccount.yaml
apiVersion: storage.azure.com/v1api20210401
kind: StorageAccount
metadata:
  name: stmyappprod
  namespace: default
spec:
  location: australiaeast
  owner:
    name: rg-myapp
  kind: StorageV2
  sku:
    name: Standard_LRS
  accessTier: Hot
---
# cosmosdb.yaml
apiVersion: documentdb.azure.com/v1api20210515
kind: DatabaseAccount
metadata:
  name: cosmos-myapp
  namespace: default
spec:
  location: australiaeast
  owner:
    name: rg-myapp
  kind: GlobalDocumentDB
  databaseAccountOfferType: Standard
  locations:
    - locationName: australiaeast
      failoverPriority: 0
  consistencyPolicy:
    defaultConsistencyLevel: Session

SQL Database with Operator

# sql-server.yaml
apiVersion: sql.azure.com/v1api20211101
kind: Server
metadata:
  name: sql-myapp
  namespace: default
spec:
  location: australiaeast
  owner:
    name: rg-myapp
  administratorLogin: sqladmin
  administratorLoginPassword:
    name: sql-admin-password
    key: password
---
# sql-database.yaml
apiVersion: sql.azure.com/v1api20211101
kind: ServersDatabase
metadata:
  name: db-myapp
  namespace: default
spec:
  location: australiaeast
  owner:
    name: sql-myapp
  sku:
    name: S1
    tier: Standard

Building Custom Operators

Operator SDK Setup

# Initialize operator project
operator-sdk init --domain mycompany.com --repo github.com/mycompany/azure-app-operator

# Create API and controller
operator-sdk create api --group apps --version v1alpha1 --kind AzureWebApp --resource --controller

Custom Resource Definition

// api/v1alpha1/azurewebapp_types.go
package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// AzureWebAppSpec defines the desired state
type AzureWebAppSpec struct {
    // ResourceGroup is the Azure resource group name
    ResourceGroup string `json:"resourceGroup"`

    // Location is the Azure region
    Location string `json:"location"`

    // AppServicePlanSku defines the App Service Plan tier
    AppServicePlanSku string `json:"appServicePlanSku,omitempty"`

    // Runtime specifies the runtime stack
    Runtime string `json:"runtime"`

    // Image is the container image to deploy
    Image string `json:"image,omitempty"`

    // EnvironmentVariables for the app
    EnvironmentVariables map[string]string `json:"environmentVariables,omitempty"`

    // Scaling configuration
    Scaling ScalingSpec `json:"scaling,omitempty"`
}

type ScalingSpec struct {
    MinInstances int32 `json:"minInstances"`
    MaxInstances int32 `json:"maxInstances"`
}

// AzureWebAppStatus defines the observed state
type AzureWebAppStatus struct {
    // State of the Azure resource
    State string `json:"state,omitempty"`

    // URL of the deployed app
    URL string `json:"url,omitempty"`

    // ResourceID is the Azure resource ID
    ResourceID string `json:"resourceId,omitempty"`

    // Conditions represent the latest observations
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:printcolumn:name="State",type=string,JSONPath=`.status.state`
//+kubebuilder:printcolumn:name="URL",type=string,JSONPath=`.status.url`

type AzureWebApp struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   AzureWebAppSpec   `json:"spec,omitempty"`
    Status AzureWebAppStatus `json:"status,omitempty"`
}

Controller Implementation

// controllers/azurewebapp_controller.go
package controllers

import (
    "context"
    "fmt"

    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/appservice/armappservice"
    appsv1alpha1 "github.com/mycompany/azure-app-operator/api/v1alpha1"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"
)

type AzureWebAppReconciler struct {
    client.Client
    Scheme         *runtime.Scheme
    SubscriptionID string
}

func (r *AzureWebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // Fetch the AzureWebApp instance
    var webapp appsv1alpha1.AzureWebApp
    if err := r.Get(ctx, req.NamespacedName, &webapp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Handle deletion
    if !webapp.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, &webapp)
    }

    // Add finalizer if not present
    if !containsString(webapp.Finalizers, finalizerName) {
        webapp.Finalizers = append(webapp.Finalizers, finalizerName)
        if err := r.Update(ctx, &webapp); err != nil {
            return ctrl.Result{}, err
        }
    }

    // Reconcile Azure resources
    result, err := r.reconcileAzureResources(ctx, &webapp)
    if err != nil {
        logger.Error(err, "Failed to reconcile Azure resources")
        meta.SetStatusCondition(&webapp.Status.Conditions, metav1.Condition{
            Type:    "Ready",
            Status:  metav1.ConditionFalse,
            Reason:  "ReconcileError",
            Message: err.Error(),
        })
        r.Status().Update(ctx, &webapp)
        return result, err
    }

    return result, nil
}

func (r *AzureWebAppReconciler) reconcileAzureResources(
    ctx context.Context,
    webapp *appsv1alpha1.AzureWebApp,
) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // Create Azure credential
    cred, err := azidentity.NewDefaultAzureCredential(nil)
    if err != nil {
        return ctrl.Result{}, err
    }

    // Create App Service client
    webAppsClient, err := armappservice.NewWebAppsClient(r.SubscriptionID, cred, nil)
    if err != nil {
        return ctrl.Result{}, err
    }

    // Check if web app exists
    existing, err := webAppsClient.Get(ctx, webapp.Spec.ResourceGroup, webapp.Name, nil)
    if err != nil {
        // Create if not exists
        logger.Info("Creating Azure Web App", "name", webapp.Name)
        return r.createWebApp(ctx, webapp, webAppsClient)
    }

    // Update if exists
    logger.Info("Updating Azure Web App", "name", webapp.Name)
    return r.updateWebApp(ctx, webapp, webAppsClient, existing.WebApp)
}

func (r *AzureWebAppReconciler) createWebApp(
    ctx context.Context,
    webapp *appsv1alpha1.AzureWebApp,
    client *armappservice.WebAppsClient,
) (ctrl.Result, error) {
    poller, err := client.BeginCreateOrUpdate(ctx,
        webapp.Spec.ResourceGroup,
        webapp.Name,
        armappservice.Site{
            Location: &webapp.Spec.Location,
            Properties: &armappservice.SiteProperties{
                SiteConfig: &armappservice.SiteConfig{
                    LinuxFxVersion: &webapp.Spec.Runtime,
                },
            },
        },
        nil)

    if err != nil {
        return ctrl.Result{}, err
    }

    result, err := poller.PollUntilDone(ctx, nil)
    if err != nil {
        return ctrl.Result{}, err
    }

    // Update status
    webapp.Status.State = "Running"
    webapp.Status.URL = fmt.Sprintf("https://%s.azurewebsites.net", webapp.Name)
    webapp.Status.ResourceID = *result.ID

    meta.SetStatusCondition(&webapp.Status.Conditions, metav1.Condition{
        Type:    "Ready",
        Status:  metav1.ConditionTrue,
        Reason:  "Created",
        Message: "Azure Web App created successfully",
    })

    return ctrl.Result{}, r.Status().Update(ctx, webapp)
}

func (r *AzureWebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appsv1alpha1.AzureWebApp{}).
        Complete(r)
}

Using Custom Operator

Deploy Custom Resource

# mywebapp.yaml
apiVersion: apps.mycompany.com/v1alpha1
kind: AzureWebApp
metadata:
  name: my-api
  namespace: production
spec:
  resourceGroup: rg-production
  location: australiaeast
  runtime: DOTNETCORE|6.0
  image: myacr.azurecr.io/api:latest
  appServicePlanSku: S1
  environmentVariables:
    ASPNETCORE_ENVIRONMENT: Production
    ConnectionStrings__Database: "@Microsoft.KeyVault(VaultName=kv-prod;SecretName=db-connection)"
  scaling:
    minInstances: 2
    maxInstances: 10

Verify Deployment

# Check custom resource status
kubectl get azurewebapps -n production

# Describe for details
kubectl describe azurewebapp my-api -n production

# View events
kubectl get events -n production --field-selector involvedObject.name=my-api

Crossplane for Azure

Installation

# Install Crossplane
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace

# Install Azure provider
kubectl apply -f - <<EOF
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure
spec:
  package: xpkg.upbound.io/upbound/provider-azure:v0.34.0
EOF

Crossplane Composite Resources

# definition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xazureapps.platform.mycompany.com
spec:
  group: platform.mycompany.com
  names:
    kind: XAzureApp
    plural: xazureapps
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                location:
                  type: string
                environment:
                  type: string
                  enum: [dev, staging, prod]
              required:
                - location
                - environment
---
# composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: azure-app-composition
spec:
  compositeTypeRef:
    apiVersion: platform.mycompany.com/v1alpha1
    kind: XAzureApp
  resources:
    - name: resourcegroup
      base:
        apiVersion: azure.upbound.io/v1beta1
        kind: ResourceGroup
        spec:
          forProvider:
            location: australiaeast
    - name: appserviceplan
      base:
        apiVersion: web.azure.upbound.io/v1beta1
        kind: ServicePlan
        spec:
          forProvider:
            location: australiaeast
            osType: Linux
            skuName: S1

Conclusion

Kubernetes Operators provide a powerful way to manage Azure resources declaratively within Kubernetes. Azure Service Operator offers ready-to-use CRDs for common Azure services, while custom operators enable domain-specific automation. Combined with GitOps workflows, operators enable true infrastructure as code for hybrid Kubernetes-Azure environments.

Introduction

Understanding Operators

Operator Pattern

Azure Service Operator

Installation

Creating Azure Resources

SQL Database with Operator

Building Custom Operators

Operator SDK Setup

Custom Resource Definition

Controller Implementation

Using Custom Operator

Deploy Custom Resource

Verify Deployment

Crossplane for Azure

Installation

Crossplane Composite Resources

Conclusion

References