Integrate Apache Airflow with Kubernetes RBAC and service accounts for secure workflow orchestration

Create dedicated namespace for Airflow

Create a separate namespace to isolate Airflow resources and apply namespace-specific RBAC policies.

kubectl create namespace airflow
kubectl config set-context --current --namespace=airflow

Create Airflow service accounts

Create separate service accounts for the Airflow scheduler/webserver and worker pods to implement role separation.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: airflow-scheduler
  namespace: airflow
  labels:
    app: airflow
    component: scheduler
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: airflow-worker
  namespace: airflow
  labels:
    app: airflow
    component: worker
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: airflow-webserver
  namespace: airflow
  labels:
    app: airflow
    component: webserver

kubectl apply -f airflow-serviceaccounts.yaml

Create RBAC roles for Airflow components

Define specific roles with minimal permissions needed for each Airflow component to function properly.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: airflow
  name: airflow-scheduler-role
rules:
apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list"]
apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
apiGroups: [""]
  resources: ["events"]
  verbs: ["list"]
apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: airflow
  name: airflow-worker-role
rules:
apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: airflow
  name: airflow-webserver-role
rules:
apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "list"]

kubectl apply -f airflow-rbac-roles.yaml

Create role bindings

Bind the service accounts to their respective roles to grant the necessary permissions while maintaining separation of concerns.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: airflow-scheduler-binding
  namespace: airflow
subjects:
kind: ServiceAccount
  name: airflow-scheduler
  namespace: airflow
roleRef:
  kind: Role
  name: airflow-scheduler-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: airflow-worker-binding
  namespace: airflow
subjects:
kind: ServiceAccount
  name: airflow-worker
  namespace: airflow
roleRef:
  kind: Role
  name: airflow-worker-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: airflow-webserver-binding
  namespace: airflow
subjects:
kind: ServiceAccount
  name: airflow-webserver
  namespace: airflow
roleRef:
  kind: Role
  name: airflow-webserver-role
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f airflow-rolebindings.yaml

Create Airflow configuration with RBAC integration

Configure Airflow to use the KubernetesExecutor with the service accounts and RBAC settings you created.

apiVersion: v1
kind: ConfigMap
metadata:
  name: airflow-config
  namespace: airflow
data:
  airflow.cfg: |
    [core]
    executor = KubernetesExecutor
    dags_folder = /opt/airflow/dags
    base_log_folder = /opt/airflow/logs
    remote_logging = False
    load_examples = False
    
    [kubernetes]
    namespace = airflow
    worker_service_account_name = airflow-worker
    image_pull_policy = Always
    delete_worker_pods = True
    delete_worker_pods_on_failure = False
    worker_pods_creation_batch_size = 10
    multi_namespace_mode = False
    in_cluster = True
    cluster_context = default
    config_file = /root/.kube/config
    
    [kubernetes_secrets]
    # Add secret configurations as needed
    
    [webserver]
    base_url = http://localhost:8080
    web_server_port = 8080
    
    [scheduler]
    catchup_by_default = False

kubectl apply -f airflow-config.yaml

Deploy Airflow scheduler with RBAC

Deploy the Airflow scheduler using the scheduler service account and proper RBAC configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-scheduler
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow
      component: scheduler
  template:
    metadata:
      labels:
        app: airflow
        component: scheduler
    spec:
      serviceAccountName: airflow-scheduler
      securityContext:
        runAsUser: 50000
        runAsGroup: 50000
        fsGroup: 50000
      containers:
      - name: scheduler
        image: apache/airflow:2.8.1
        env:
        - name: AIRFLOW__CORE__EXECUTOR
          value: "KubernetesExecutor"
        - name: AIRFLOW__KUBERNETES__NAMESPACE
          value: "airflow"
        - name: AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME
          value: "airflow-worker"
        - name: AIRFLOW__KUBERNETES__IN_CLUSTER
          value: "True"
        command:
        - airflow
        - scheduler
        volumeMounts:
        - name: airflow-config
          mountPath: /opt/airflow/airflow.cfg
          subPath: airflow.cfg
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
      volumes:
      - name: airflow-config
        configMap:
          name: airflow-config

kubectl apply -f airflow-scheduler.yaml

Deploy Airflow webserver with RBAC

Deploy the Airflow webserver using the webserver service account for read-only cluster access.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-webserver
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow
      component: webserver
  template:
    metadata:
      labels:
        app: airflow
        component: webserver
    spec:
      serviceAccountName: airflow-webserver
      securityContext:
        runAsUser: 50000
        runAsGroup: 50000
        fsGroup: 50000
      containers:
      - name: webserver
        image: apache/airflow:2.8.1
        env:
        - name: AIRFLOW__CORE__EXECUTOR
          value: "KubernetesExecutor"
        - name: AIRFLOW__KUBERNETES__NAMESPACE
          value: "airflow"
        command:
        - airflow
        - webserver
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: airflow-config
          mountPath: /opt/airflow/airflow.cfg
          subPath: airflow.cfg
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
      volumes:
      - name: airflow-config
        configMap:
          name: airflow-config
---
apiVersion: v1
kind: Service
metadata:
  name: airflow-webserver
  namespace: airflow
spec:
  selector:
    app: airflow
    component: webserver
  ports:
  - port: 8080
    targetPort: 8080
  type: ClusterIP

kubectl apply -f airflow-webserver.yaml

Create pod security policy for Airflow workers

Define security constraints for Airflow worker pods to prevent privilege escalation and enforce security boundaries.

apiVersion: v1
kind: SecurityContext
metadata:
  name: airflow-worker-security-context
spec:
  runAsUser: 50000
  runAsGroup: 50000
  runAsNonRoot: true
  fsGroup: 50000
  seccompProfile:
    type: RuntimeDefault
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: airflow-network-policy
  namespace: airflow
spec:
  podSelector:
    matchLabels:
      app: airflow
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: airflow
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to: []
    ports:
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 80
    - protocol: UDP
      port: 53

kubectl apply -f airflow-pod-security.yaml

Configure worker pod template with security settings

Create a pod template that Airflow will use for spawning worker pods with proper security contexts and resource limits.

apiVersion: v1
kind: Pod
metadata:
  name: airflow-worker-template
  namespace: airflow
spec:
  serviceAccountName: airflow-worker
  securityContext:
    runAsUser: 50000
    runAsGroup: 50000
    runAsNonRoot: true
    fsGroup: 50000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: base
    image: apache/airflow:2.8.1
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      capabilities:
        drop:
        - ALL
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"
    env:
    - name: AIRFLOW__KUBERNETES__NAMESPACE
      value: airflow
    - name: AIRFLOW__CORE__EXECUTOR
      value: KubernetesExecutor
  restartPolicy: Never

kubectl apply -f worker-pod-template.yaml

Update Airflow configuration to use pod template

Update the Airflow configuration to reference the secure pod template for worker pods.

kubectl patch configmap airflow-config -n airflow --patch='
{
  "data": {
    "airflow.cfg": "[core]\nexecutor = KubernetesExecutor\ndags_folder = /opt/airflow/dags\nbase_log_folder = /opt/airflow/logs\nremote_logging = False\nload_examples = False\n\n[kubernetes]\nnamespace = airflow\nworker_service_account_name = airflow-worker\nimage_pull_policy = Always\ndelete_worker_pods = True\ndelete_worker_pods_on_failure = False\nworker_pods_creation_batch_size = 10\nmulti_namespace_mode = False\nin_cluster = True\ncluster_context = default\nconfig_file = /root/.kube/config\npod_template_file = /opt/airflow/pod-template.yaml\n\n[webserver]\nbase_url = http://localhost:8080\nweb_server_port = 8080\n\n[scheduler]\ncatchup_by_default = False"
  }
}'
kubectl rollout restart deployment airflow-scheduler -n airflow
kubectl rollout restart deployment airflow-webserver -n airflow

Create cluster-wide permissions for cross-namespace access

If your workflows need to interact with resources in other namespaces, create cluster roles with specific permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: airflow-cross-namespace-reader
rules:
apiGroups: [""]
  resources: ["nodes", "namespaces"]
  verbs: ["get", "list"]
apiGroups: ["metrics.k8s.io"]
  resources: ["nodes", "pods"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: airflow-scheduler-cluster-binding
subjects:
kind: ServiceAccount
  name: airflow-scheduler
  namespace: airflow
roleRef:
  kind: ClusterRole
  name: airflow-cross-namespace-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f airflow-cluster-rbac.yaml

Configure resource quotas and limits

Set resource quotas to prevent Airflow from consuming excessive cluster resources and ensure fair resource allocation.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: airflow-resource-quota
  namespace: airflow
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "20"
    persistentvolumeclaims: "4"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: airflow-limit-range
  namespace: airflow
spec:
  limits:
  - type: Container
    default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    max:
      cpu: 2
      memory: 4Gi
    min:
      cpu: 50m
      memory: 64Mi

kubectl apply -f airflow-resource-quota.yaml

Check RBAC permissions

Verify that your service accounts have the correct permissions and can perform the required operations.

kubectl auth can-i create pods --as=system:serviceaccount:airflow:airflow-scheduler -n airflow
kubectl auth can-i get pods --as=system:serviceaccount:airflow:airflow-worker -n airflow
kubectl auth can-i delete pods --as=system:serviceaccount:airflow:airflow-worker -n airflow

Verify Airflow deployment status

Check that all Airflow components are running and properly configured with RBAC.

kubectl get pods -n airflow
kubectl get serviceaccounts -n airflow
kubectl get roles -n airflow
kubectl get rolebindings -n airflow
kubectl logs deployment/airflow-scheduler -n airflow --tail=20

Test worker pod creation

Create a test DAG to verify that worker pods are created with proper RBAC permissions and security contexts.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2024, 1, 1),
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    'test_rbac_dag',
    default_args=default_args,
    description='Test RBAC configuration',
    schedule_interval=None,
    catchup=False
)

test_task = KubernetesPodOperator(
    task_id='test_rbac_task',
    name='test-rbac-pod',
    namespace='airflow',
    image='alpine:latest',
    cmds=['sh', '-c'],
    arguments=['echo "RBAC test successful" && whoami && id'],
    service_account_name='airflow-worker',
    is_delete_operator_pod=True,
    dag=dag
)