Set up container runtime security with Falco and Sysdig for threat detection

Update system packages

Start by updating your package manager to ensure you get the latest versions of required dependencies.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget gnupg software-properties-common

sudo dnf update -y
sudo dnf install -y curl wget gnupg2 yum-utils

Install kernel headers and build tools

Falco requires kernel headers and build tools to compile its eBPF programs and kernel modules for system call monitoring.

sudo apt install -y linux-headers-$(uname -r) build-essential dkms
sudo apt install -y clang llvm

sudo dnf install -y kernel-devel-$(uname -r) gcc make
sudo dnf install -y clang llvm

Add Falco repository

Add the official Falco repository to install the latest stable version with automatic updates.

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" | sudo tee /etc/apt/sources.list.d/falcosecurity.list
sudo apt update

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | sudo gpg --import
cat > /etc/yum.repos.d/falcosecurity.repo << EOF
[falcosecurity]
name=Falco Security Repository
baseurl=https://download.falco.org/packages/rpm
enabled=1
gpgcheck=1
gpgkey=https://falco.org/repo/falcosecurity-packages.asc
EOF

Install Falco

Install Falco with the default configuration that includes comprehensive container security rules.

sudo apt install -y falco

sudo dnf install -y falco

Configure Falco main settings

Configure Falco's main settings for JSON output, logging, and alert destinations.

rules_file:
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco_rules.local.yaml
  - /etc/falco/k8s_audit_rules.yaml
  - /etc/falco/rules.d

time_format_iso_8601: true
json_output: true
json_include_output_property: true
json_include_tags_property: true

log_stderr: true
log_syslog: true
log_level: info

priority: debug
buffered_outputs: false

metadata_download:
  max_mb: 100
  chunk_wait_us: 1000
  watch_freq_sec: 1

syscall_event_drops:
  threshold: 0.1
  actions:
    - log
    - alert
  rate: 0.03333
  max_burst: 1

base_syscalls:
  custom_set: []
  repair: false

modern_bpf:
  cpus_for_each_syscall_buffer: 2

outputs:
  rate: 1
  max_burst: 1000

Create custom security rules

Create local rules for container-specific threats like privilege escalation, suspicious network activity, and file access violations.

- rule: Suspicious Container Network Activity
  desc: Detect suspicious network connections from containers
  condition: >
    spawned_process and container and
    ((proc.name=wget or proc.name=curl) and
     (proc.args contains "http://" and not proc.args contains "https://"))
  output: >
    Suspicious network activity in container
    (user=%user.name command=%proc.cmdline container_id=%container.id
     container_name=%container.name image=%container.image.repository)
  priority: WARNING
  tags: [container, network, malware]

rule: Container Privilege Escalation Attempt
  desc: Detect attempts to escalate privileges in containers
  condition: >
    spawned_process and container and
    ((proc.name=sudo or proc.name=su) or
     (proc.args contains "chmod +s" or proc.args contains "chmod 4755"))
  output: >
    Privilege escalation attempt in container
    (user=%user.name command=%proc.cmdline container_id=%container.id
     container_name=%container.name image=%container.image.repository)
  priority: HIGH
  tags: [container, privilege_escalation, security]

rule: Sensitive File Access in Container
  desc: Monitor access to sensitive files from containers
  condition: >
    open_read and container and
    (fd.name startswith "/etc/passwd" or
     fd.name startswith "/etc/shadow" or
     fd.name startswith "/root/.ssh" or
     fd.name startswith "/home/*/.ssh")
  output: >
    Sensitive file accessed in container
    (user=%user.name file=%fd.name container_id=%container.id
     container_name=%container.name image=%container.image.repository)
  priority: WARNING
  tags: [container, filesystem, credentials]

rule: Container Exec into Production Namespace
  desc: Detect kubectl exec into production namespaces
  condition: >
    k8s_audit and ka.verb=create and ka.uri.path contains "/exec" and
    (ka.target.namespace=production or
     ka.target.namespace=prod or
     ka.target.namespace=live)
  output: >
    Kubectl exec into production namespace
    (user=%ka.user.name namespace=%ka.target.namespace
     pod=%ka.target.pod verb=%ka.verb uri=%ka.uri.path)
  priority: HIGH
  tags: [k8s, exec, production]

Configure Falco alerting

Set up webhook notifications to send alerts to external systems like Slack, PagerDuty, or SIEM platforms.

# Add to the existing falco.yaml file
http_output:
  enabled: true
  url: "http://localhost:2801/"
  user_agent: "falcosecurity/falco"

program_output:
  enabled: true
  keep_alive: false
  program: "jq '{text: .output}' | curl -X POST -H 'Content-type: application/json' --data @- https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

file_output:
  enabled: true
  keep_alive: false
  filename: "/var/log/falco/events.log"

Install Sysdig Agent

Install the Sysdig Agent to complement Falco with detailed container performance metrics and additional security visibility.

curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key YOUR_SYSDIG_ACCESS_KEY --collector collector.sysdigcloud.com --collector_port 6443 --secure true

Configure Sysdig Agent for Kubernetes

Configure the Sysdig Agent with Kubernetes metadata collection and container runtime monitoring.

customerid: YOUR_SYSDIG_ACCESS_KEY
tags: environment:production,team:security

k8s_cluster_name: production-cluster
k8s_node_name: 
k8s_pod_name: 
k8s_namespace_name: 

container_engines:
  - docker
  - containerd
  - cri-o

secure:
  enabled: true
  
collector: collector.sysdigcloud.com
collector_port: 6443
ssl: true

metrics_filter:
  - include: "*"
  - exclude: "host.cpu.used.percent"

event_forwarder:
  enabled: true
  config:
    - filter: "evt.type=execve"
      outputs:
        - syslog

prometheus:
  enabled: true
  histograms: true
  interval: 10

log:
  console_priority: info
  file_priority: info
  event_priority: information

Deploy Falco on Kubernetes

Deploy Falco as a DaemonSet in Kubernetes to monitor all nodes for container runtime security events.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

falco:
  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml
    - /etc/falco/k8s_audit_rules.yaml
  
  json_output: true
  log_stderr: true
  log_syslog: true
  log_level: info
  
  priority: debug
  
  http_output:
    enabled: true
    url: "http://falco-exporter:9376/events"
  
  grpc:
    enabled: true
    bind_address: "0.0.0.0:5060"
    threadiness: 8
  
  grpc_output:
    enabled: true

driver:
  kind: ebpf
  
resources:
  limits:
    cpu: 1000m
    memory: 1024Mi
  requests:
    cpu: 100m
    memory: 512Mi

tolerationen:
effect: NoSchedule
  operator: Exists
effect: NoExecute
  operator: Exists

serviceAccount:
  create: true
  name: falco

rbac:
  create: true

falcoctl:
  artifact:
    install:
      enabled: true
    follow:
      enabled: true

kubectl create namespace falco
helm install falco falcosecurity/falco -f falco-values.yaml -n falco

Deploy Sysdig Agent on Kubernetes

Deploy the Sysdig Agent as a DaemonSet to collect container metrics and security events from all nodes.

clusterName: "production-cluster"
sysdig:
  accessKey: "YOUR_SYSDIG_ACCESS_KEY"
  
image:
  tag: latest
  
resources:
  limits:
    cpu: 2000m
    memory: 1536Mi
  requests:
    cpu: 600m
    memory: 512Mi

nodeSelector: {}
tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists

secure:
  enabled: true
  
prometheus:
  file: true
  yaml:
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']
    - job_name: 'kubernetes-nodes'
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):(\d+)'
        target_label: __address__
        replacement: '${1}:9100'
        
daemonset:
  updateStrategy:
    type: RollingUpdate
    
serviceAccount:
  create: true
  name: sysdig-agent

rbac:
  create: true

helm repo add sysdig https://charts.sysdig.com
helm repo update
kubectl create namespace sysdig-agent
helm install sysdig-agent sysdig/sysdig-deploy -f sysdig-values.yaml -n sysdig-agent

Configure log aggregation

Set up centralized logging to collect and analyze Falco alerts and Sysdig metrics in your SIEM or monitoring platform.

template(name="falcoTemplate" type="string" string="%timestamp:::date-rfc3339% %hostname% falco: %msg%\n")
if $programname == 'falco' then {
    action(type="omfile" file="/var/log/falco/falco.log" template="falcoTemplate")
    action(type="omfwd" target="203.0.113.10" port="514" protocol="udp" template="falcoTemplate")
    stop
}

sudo mkdir -p /var/log/falco
sudo systemctl restart rsyslog

Enable and start services

Start Falco and enable it to run automatically on system boot for continuous runtime security monitoring.

sudo systemctl enable --now falco
sudo systemctl enable --now dragent
sudo systemctl status falco
sudo systemctl status dragent

Create behavioral analysis rules

Configure advanced behavioral analysis to detect anomalous patterns in container behavior over time.

- rule: Unusual Process in Container
  desc: Detect processes that rarely run in specific container images
  condition: >
    spawned_process and container and
    (proc.name in (nc, ncat, netcat, wget, curl, python, python3, perl, ruby, php) and
     container.image.repository in (nginx, apache, httpd, mysql, postgres, redis) and
     not proc.pname in (sh, bash, dash))
  output: >
    Unusual process spawned in container
    (user=%user.name process=%proc.name parent=%proc.pname
     container=%container.name image=%container.image.repository)
  priority: WARNING
  tags: [container, process, anomaly]

rule: Container Running with Excessive Privileges
  desc: Detect containers running with dangerous capabilities
  condition: >
    container and
    (container.privileged=true or
     container.capability.sys_admin or
     container.capability.net_admin or
     container.capability.sys_ptrace)
  output: >
    Container running with excessive privileges
    (container=%container.name image=%container.image.repository
     privileged=%container.privileged capabilities=%container.capabilities)
  priority: HIGH
  tags: [container, privileges, security]

rule: Cryptomining Activity
  desc: Detect potential cryptocurrency mining activity
  condition: >
    spawned_process and
    (proc.name in (xmrig, cpuminer, ccminer, cgminer, bfgminer, sgminer) or
     proc.cmdline contains stratum or
     proc.cmdline contains "mine" or
     proc.cmdline contains "pool")
  output: >
    Potential cryptomining activity detected
    (user=%user.name command=%proc.cmdline container=%container.name)
  priority: CRITICAL
  tags: [malware, cryptomining, container]

Configure network monitoring

Set up network-based threat detection for suspicious container communications and data exfiltration attempts.

- rule: Outbound Connection to Suspicious Port
  desc: Detect outbound connections to commonly malicious ports
  condition: >
    outbound and
    (fd.sport in (4444, 5555, 6666, 7777, 8888, 9999) or
     fd.dport in (4444, 5555, 6666, 7777, 8888, 9999, 1337, 31337))
  output: >
    Suspicious outbound connection
    (user=%user.name command=%proc.cmdline connection=%fd.name
     container=%container.name)
  priority: HIGH
  tags: [network, malware, container]

rule: DNS Tunneling Attempt
  desc: Detect potential DNS tunneling based on query patterns
  condition: >
    spawned_process and
    (proc.name=dig or proc.name=nslookup or proc.name=host) and
    (proc.args contains "txt" or proc.args contains "AAAA") and
    proc.args rregex "[a-f0-9]{32,}"
  output: >
    Potential DNS tunneling detected
    (user=%user.name command=%proc.cmdline container=%container.name)
  priority: HIGH
  tags: [network, dns, tunneling, container]

rule: Container Network Namespace Manipulation
  desc: Detect attempts to manipulate network namespaces
  condition: >
    spawned_process and container and
    (proc.name=ip and proc.args contains "netns" or
     proc.name=nsenter and proc.args contains "-n")
  output: >
    Network namespace manipulation in container
    (user=%user.name command=%proc.cmdline container=%container.name)
  priority: WARNING
  tags: [container, network, namespace]

Set up integration with monitoring stack

Configure Falco to integrate with Prometheus and Grafana for comprehensive security monitoring dashboards. This links your runtime security with your existing monitoring infrastructure covered in our Prometheus alerting tutorial.

# Add Prometheus metrics endpoint
metrics:
  enabled: true
  interval: 15s
  output_rule: true
  rules_counters_enabled: true
  resource_utilization_enabled: true
  state_counters_enabled: true
  kernel_event_counters_enabled: true
  libbpf_stats_enabled: true
  convert_memory_to_mb: true
  include_empty_values: false

webserver:
  enabled: true
  listen_port: 8765
  k8s_healthz_endpoint: "/healthz"
  ssl_enabled: false
  ssl_certificate: "/etc/falco/certs/server.pem"

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: falco
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: falco-rules
  namespace: falco
spec:
  groups:
  - name: falco
    rules:
    - alert: FalcoAlert
      expr: increase(falco_events_total[5m]) > 0
      for: 0s
      labels:
        severity: warning
      annotations:
        summary: "Falco security event detected"
        description: "{{ $labels.rule }} triggered on {{ $labels.hostname }}"
    
    - alert: FalcoHighSeverityAlert
      expr: increase(falco_events_total{priority=~"Critical|High"}[5m]) > 0
      for: 0s
      labels:
        severity: critical
      annotations:
        summary: "High severity Falco alert"
        description: "Critical security event: {{ $labels.rule }}"

kubectl apply -f prometheus-falco.yaml