Configure Fluentd with Kubernetes DaemonSet

Deploy Fluentd as a DaemonSet on Kubernetes for centralized log collection with multi-format parsing, routing to multiple outputs, and RBAC security. Includes configuration for Elasticsearch, S3, and custom log sources.

Prerequisites

Kubernetes cluster with admin access
kubectl configured
Elasticsearch or log storage backend

What this solves

Fluentd provides centralized log collection for Kubernetes clusters, gathering container logs from all nodes and routing them to storage backends like Elasticsearch or S3. This tutorial shows you how to deploy Fluentd as a DaemonSet with proper RBAC permissions, configure multi-format log parsing, and set up routing to multiple output destinations for production-grade log management.

Step-by-step configuration

Create Fluentd namespace and RBAC configuration

Set up the namespace and service account with cluster-wide permissions to read container logs from all nodes.

apiVersion: v1
kind: Namespace
metadata:
  name: fluentd-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: fluentd-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups: [""]
  resources:
    - pods
    - namespaces
  verbs:
    - get
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: fluentd-system

kubectl apply -f fluentd-rbac.yaml

Create Fluentd configuration with log routing

Configure Fluentd with input sources, filtering, parsing, and multiple output destinations including Elasticsearch and S3.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: fluentd-system
data:
  fluent.conf: |
    
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      
    

    
      @type tail
      @id in_tail_minion
      path /var/log/salt/minion
      pos_file /var/log/fluentd-salt.log.pos
      tag salt
      
    

    
      @type tail
      @id in_tail_startupscript
      path /var/log/startupscript.log
      pos_file /var/log/fluentd-startupscript.log.pos
      tag startupscript
      
    

    
      @type tail
      @id in_tail_docker
      path /var/log/docker.log
      pos_file /var/log/fluentd-docker.log.pos
      tag docker
      
    

    
      @type tail
      @id in_tail_etcd
      path /var/log/etcd.log
      pos_file /var/log/fluentd-etcd.log.pos
      tag etcd
      
    

    
      @type tail
      @id in_tail_kubelet
      multiline_flush_interval 5s
      path /var/log/kubelet.log
      pos_file /var/log/fluentd-kubelet.log.pos
      tag kubelet
      
    

    
      @type tail
      @id in_tail_kube_proxy
      multiline_flush_interval 5s
      path /var/log/kube-proxy.log
      pos_file /var/log/fluentd-kube-proxy.log.pos
      tag kube-proxy
      
    

    
      @type tail
      @id in_tail_kube_apiserver
      multiline_flush_interval 5s
      path /var/log/kube-apiserver.log
      pos_file /var/log/fluentd-kube-apiserver.log.pos
      tag kube-apiserver
      
    

    
      @type tail
      @id in_tail_kube_controller_manager
      multiline_flush_interval 5s
      path /var/log/kube-controller-manager.log
      pos_file /var/log/fluentd-kube-controller-manager.log.pos
      tag kube-controller-manager
      
    

    
      @type tail
      @id in_tail_kube_scheduler
      multiline_flush_interval 5s
      path /var/log/kube-scheduler.log
      pos_file /var/log/fluentd-kube-scheduler.log.pos
      tag kube-scheduler

kubectl apply -f fluentd-configmap.yaml

Create secrets for external services

Configure authentication credentials for Elasticsearch and AWS S3 access.

kubectl create secret generic fluentd-secrets -n fluentd-system \
  --from-literal=FLUENT_ELASTICSEARCH_USER=elastic \
  --from-literal=FLUENT_ELASTICSEARCH_PASSWORD=your-elasticsearch-password \
  --from-literal=AWS_ACCESS_KEY_ID=your-aws-access-key \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-aws-secret-key \
  --from-literal=S3_BUCKET_NAME=your-logs-bucket \
  --from-literal=AWS_REGION=us-west-2

Deploy Fluentd DaemonSet

Create the DaemonSet to run Fluentd on every node with proper resource limits and volume mounts.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: fluentd-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch7-1
        env:
          - name: FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch.logging.svc.cluster.local"
          - name: FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: /var/log/containers/fluent*
          - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
            value: "true"
          - name: FLUENT_ELASTICSEARCH_SSL_VERSION
            value: "TLSv1_2"
          - name: FLUENT_ELASTICSEARCH_USER
            valueFrom:
              secretKeyRef:
                name: fluentd-secrets
                key: FLUENT_ELASTICSEARCH_USER
          - name: FLUENT_ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                name: fluentd-secrets
                key: FLUENT_ELASTICSEARCH_PASSWORD
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: fluentd-secrets
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: fluentd-secrets
                key: AWS_SECRET_ACCESS_KEY
          - name: S3_BUCKET_NAME
            valueFrom:
              secretKeyRef:
                name: fluentd-secrets
                key: S3_BUCKET_NAME
          - name: AWS_REGION
            valueFrom:
              secretKeyRef:
                name: fluentd-secrets
                key: AWS_REGION
        resources:
          limits:
            memory: 512Mi
            cpu: 200m
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc/fluent.conf
          subPath: fluent.conf
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: runlogjournal
          mountPath: /run/log/journal
          readOnly: true
        - name: dmesg
          mountPath: /var/log/dmesg
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: runlogjournal
        hostPath:
          path: /run/log/journal
      - name: dmesg
        hostPath:
          path: /var/log/dmesg

kubectl apply -f fluentd-daemonset.yaml

Configure custom application log parsing

Add additional parsing rules for specific application formats like JSON logs and multiline stack traces.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-custom-parsing
  namespace: fluentd-system
data:
  custom-parsing.conf: |

kubectl apply -f fluentd-custom-parsing.yaml

Update DaemonSet to include custom parsing

Add the custom parsing configuration to the DaemonSet volume mounts.

kubectl patch daemonset fluentd -n fluentd-system --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/volumeMounts/-",
    "value": {
      "name": "fluentd-custom-parsing",
      "mountPath": "/fluentd/etc/conf.d/custom-parsing.conf",
      "subPath": "custom-parsing.conf"
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
      "name": "fluentd-custom-parsing",
      "configMap": {
        "name": "fluentd-custom-parsing"
      }
    }
  }
]'

Configure log monitoring and alerting

Set up monitoring for Fluentd performance and log flow health.

apiVersion: v1
kind: Service
metadata:
  name: fluentd-metrics
  namespace: fluentd-system
  labels:
    k8s-app: fluentd-logging
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "24231"
spec:
  ports:
  - name: prometheus
    port: 24231
    protocol: TCP
    targetPort: 24231
  selector:
    k8s-app: fluentd-logging
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: fluentd-metrics
  namespace: fluentd-system
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  endpoints:
  - port: prometheus
    interval: 30s
    path: /metrics

kubectl apply -f fluentd-monitoring.yaml

Configure output routing strategies

Set up log routing by namespace

Configure different output destinations based on Kubernetes namespace and application labels.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-routing
  namespace: fluentd-system
data:
  routing.conf: |

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash

set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Default values
NAMESPACE="fluentd-system"
ELASTICSEARCH_HOST="${1:-elasticsearch.logging.svc.cluster.local}"
S3_BUCKET="${2:-}"

# Usage function
usage() {
    echo "Usage: $0 [elasticsearch-host] [s3-bucket]"
    echo "  elasticsearch-host: Elasticsearch service host (default: elasticsearch.logging.svc.cluster.local)"
    echo "  s3-bucket: Optional S3 bucket for log storage"
    exit 1
}

# Error handling
cleanup() {
    echo -e "${RED}[ERROR] Installation failed. Cleaning up...${NC}"
    kubectl delete namespace "$NAMESPACE" --ignore-not-found=true 2>/dev/null || true
}

trap cleanup ERR

# Check if running as root or with sudo
if [[ $EUID -eq 0 ]]; then
    SUDO=""
else
    SUDO="sudo"
    if ! command -v sudo &> /dev/null; then
        echo -e "${RED}[ERROR] This script requires root privileges or sudo${NC}"
        exit 1
    fi
fi

# Auto-detect distribution
echo -e "${YELLOW}[1/8] Detecting system distribution...${NC}"
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_INSTALL="apt install -y"
            PKG_UPDATE="apt update"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora) 
            PKG_MGR="dnf"
            PKG_INSTALL="dnf install -y"
            PKG_UPDATE="dnf check-update || true"
            ;;
        amzn) 
            PKG_MGR="yum"
            PKG_INSTALL="yum install -y"
            PKG_UPDATE="yum check-update || true"
            ;;
        *) 
            echo -e "${RED}[ERROR] Unsupported distribution: $ID${NC}"
            exit 1
            ;;
    esac
else
    echo -e "${RED}[ERROR] Cannot detect distribution${NC}"
    exit 1
fi

echo -e "${GREEN}Detected: $PRETTY_NAME ($PKG_MGR)${NC}"

# Check prerequisites
echo -e "${YELLOW}[2/8] Checking prerequisites...${NC}"
if ! command -v kubectl &> /dev/null; then
    echo -e "${RED}[ERROR] kubectl is not installed${NC}"
    exit 1
fi

if ! kubectl cluster-info &> /dev/null; then
    echo -e "${RED}[ERROR] Cannot connect to Kubernetes cluster${NC}"
    exit 1
fi

# Install required packages
echo -e "${YELLOW}[3/8] Installing required packages...${NC}"
$SUDO $PKG_UPDATE
$SUDO $PKG_INSTALL curl

# Create namespace and RBAC
echo -e "${YELLOW}[4/8] Creating Fluentd namespace and RBAC...${NC}"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: $NAMESPACE
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: $NAMESPACE
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups: [""]
  resources:
    - pods
    - namespaces
    - nodes
    - nodes/proxy
  verbs:
    - get
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: $NAMESPACE
EOF

# Create Fluentd ConfigMap
echo -e "${YELLOW}[5/8] Creating Fluentd configuration...${NC}"
S3_CONFIG=""
if [[ -n "$S3_BUCKET" ]]; then
    S3_CONFIG='
    <match kubernetes.var.log.containers.**>
      @type copy
      <store>
        @type elasticsearch
        host '"$ELASTICSEARCH_HOST"'
        port 9200
        logstash_format true
        logstash_prefix fluentd
        logstash_dateformat %Y%m%d
        include_tag_key true
        type_name access_log
        tag_key @log_name
        flush_interval 1s
      </store>
      <store>
        @type s3
        s3_bucket '"$S3_BUCKET"'
        s3_region us-west-2
        path logs/
        s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
        time_slice_format %Y%m%d%H
        <buffer time>
          @type file
          path /var/log/fluent/s3
          timekey 3600
          timekey_wait 10m
          chunk_limit_size 256m
        </buffer>
        <format>
          @type json
        </format>
      </store>
    </match>'
else
    S3_CONFIG='
    <match kubernetes.var.log.containers.**>
      @type elasticsearch
      host '"$ELASTICSEARCH_HOST"'
      port 9200
      logstash_format true
      logstash_prefix fluentd
      logstash_dateformat %Y%m%d
      include_tag_key true
      type_name access_log
      tag_key @log_name
      flush_interval 1s
    </match>'
fi

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: $NAMESPACE
data:
  fluent.conf: |
    <source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type cri
      </parse>
    </source>
    
    <source>
      @type tail
      @id in_tail_kubelet
      path /var/log/kubelet.log
      pos_file /var/log/fluentd-kubelet.log.pos
      tag kubelet
      <parse>
        @type kubernetes
      </parse>
    </source>
    
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>
    
    $S3_CONFIG
    
    <match **>
      @type stdout
    </match>
EOF

# Create Fluentd DaemonSet
echo -e "${YELLOW}[6/8] Creating Fluentd DaemonSet...${NC}"
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: $NAMESPACE
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        env:
        - name: FLUENT_CONF
          value: "fluent.conf"
        - name: FLUENTD_SYSTEMD_CONF
          value: "disable"
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluentd-config
          mountPath: /fluentd/etc
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsNonRoot: false
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentd-config
        configMap:
          name: fluentd-config
EOF

# Wait for DaemonSet rollout
echo -e "${YELLOW}[7/8] Waiting for Fluentd DaemonSet to be ready...${NC}"
kubectl rollout status daemonset/fluentd -n "$NAMESPACE" --timeout=300s

# Verify installation
echo -e "${YELLOW}[8/8] Verifying Fluentd installation...${NC}"
sleep 10

RUNNING_PODS=$(kubectl get pods -n "$NAMESPACE" -l k8s-app=fluentd-logging --field-selector=status.phase=Running --no-headers | wc -l)
TOTAL_NODES=$(kubectl get nodes --no-headers | wc -l)

if [[ $RUNNING_PODS -eq $TOTAL_NODES ]]; then
    echo -e "${GREEN}✓ Fluentd is running on all $TOTAL_NODES nodes${NC}"
else
    echo -e "${YELLOW}⚠ Fluentd is running on $RUNNING_PODS out of $TOTAL_NODES nodes${NC}"
fi

# Check logs for errors
if kubectl logs -n "$NAMESPACE" -l k8s-app=fluentd-logging --tail=50 | grep -q "ERROR\|FATAL"; then
    echo -e "${YELLOW}⚠ Found errors in Fluentd logs. Check with: kubectl logs -n $NAMESPACE -l k8s-app=fluentd-logging${NC}"
else
    echo -e "${GREEN}✓ No critical errors found in Fluentd logs${NC}"
fi

echo -e "${GREEN}[SUCCESS] Fluentd installation completed!${NC}"
echo ""
echo "Next steps:"
echo "1. Monitor logs: kubectl logs -n $NAMESPACE -l k8s-app=fluentd-logging -f"
echo "2. Check status: kubectl get pods -n $NAMESPACE"
echo "3. View configuration: kubectl get configmap fluentd-config -n $NAMESPACE -o yaml"

Review the script before running. Execute with: bash install.sh

#fluentd #kubernetes #daemonset #log-collection #elasticsearch

Configure Fluentd with Kubernetes DaemonSet and log routing for centralized collection