Configure Cilium BGP peering with MetalLB integration for Kubernetes load balancing

Verify cluster readiness

Ensure your Kubernetes cluster is running and accessible with proper RBAC permissions.

kubectl get nodes -o wide
kubectl cluster-info
kubectl get pods -A

Install required networking tools

Install FRRouting and BGP utilities for testing BGP connectivity and routing verification.

sudo apt update
sudo apt install -y frr frr-pythontools net-tools tcpdump

sudo dnf update -y
sudo dnf install -y frr net-tools tcpdump

Configure BGP router simulator

Set up FRRouting to simulate an upstream BGP router for testing the peering configuration.

bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=no
vrrpd=no
pathd=no

Start and enable FRRouting

Enable FRRouting service and verify BGP daemon is running.

sudo systemctl enable frr
sudo systemctl start frr
sudo systemctl status frr
vtysh -c "show version"

Install Cilium CLI

Download and install the latest Cilium CLI for cluster management and configuration.

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}

Install Cilium with BGP control plane

Deploy Cilium CNI with BGP control plane enabled for routing advertisement.

cilium install \
  --set bgpControlPlane.enabled=true \
  --set k8s.requireIPv4PodCIDR=true \
  --set tunnel=disabled \
  --set ipam.mode=kubernetes \
  --set enableIPv4Masquerade=true \
  --set enableIPv6Masquerade=false

Verify Cilium installation

Check that all Cilium components are running and BGP control plane is active.

cilium status --wait
kubectl get pods -n kube-system -l k8s-app=cilium
cilium config view | grep -i bgp

Create BGP peering policy

Configure Cilium BGP peering with upstream routers using CiliumBGPPeeringPolicy CRD.

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: bgp-peering-policy
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  virtualRouters:
  - localASN: 65000
    exportPodCIDR: true
    neighbors:
    - peerAddress: 203.0.113.1/32
      peerASN: 65001
      gracefulRestart:
        enabled: true
        restartTimeSeconds: 120
      families:
      - afi: ipv4
        safi: unicast
        advertisements:
          matchLabels:
            advertise: bgp
    serviceSelector:
      matchExpressions:
      - key: somekey
        operator: NotIn
        values: ['never-used-value']

Apply BGP peering configuration

Deploy the BGP peering policy to establish routing between Cilium and upstream routers.

kubectl apply -f cilium-bgp-policy.yaml
kubectl get ciliumBGPpeeringpolicy
kubectl describe ciliumBGPpeeringpolicy bgp-peering-policy

Install MetalLB operator

Deploy MetalLB using the official manifests for load balancer functionality.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml
kubectl wait --namespace metallb-system \
  --for=condition=ready pod \
  --selector=app=metallb \
  --timeout=90s

Configure MetalLB IP address pool

Define the IP address range that MetalLB will assign to LoadBalancer services.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: production-pool
  namespace: metallb-system
spec:
  addresses:
  - 203.0.113.100-203.0.113.150
  autoAssign: true
  avoidBuggyIPs: true

Create BGP advertisement configuration

Configure MetalLB to advertise LoadBalancer IPs via BGP to upstream routers.

apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: production-bgp-adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - production-pool
  peers:
  - 203.0.113.1
  communities:
  - 65000:100
  localPref: 100
  aggregationLength: 32
  aggregationLengthV6: 128

Configure MetalLB BGP peer

Establish BGP peering between MetalLB speakers and upstream routers.

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: upstream-router
  namespace: metallb-system
spec:
  myASN: 65000
  peerASN: 65001
  peerAddress: 203.0.113.1
  sourceAddress: 203.0.113.10
  routerID: 203.0.113.10
  holdTime: 90s
  keepaliveTime: 30s
  password: "bgp-secret-password"
  bfdProfile: default

Apply MetalLB configurations

Deploy all MetalLB configurations and verify the BGP peering establishment.

kubectl apply -f metallb-ippool.yaml
kubectl apply -f metallb-bgp-advertisement.yaml
kubectl apply -f metallb-bgp-peer.yaml
kubectl get ipaddresspool,bgpadvertisement,bgppeer -n metallb-system

Create shared BGP configuration

Configure both Cilium and MetalLB to peer with the same upstream router for route consistency.

apiVersion: v1
kind: ConfigMap
metadata:
  name: bgp-config
  namespace: kube-system
data:
  upstream-asn: "65001"
  local-asn: "65000"
  router-id: "203.0.113.10"
  peer-address: "203.0.113.1"
---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: unified-bgp-policy
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  virtualRouters:
  - localASN: 65000
    exportPodCIDR: true
    serviceSelector:
      matchLabels:
        io.cilium/bgp-announce: "true"
    neighbors:
    - peerAddress: 203.0.113.1/32
      peerASN: 65001
      connectRetryTimeSeconds: 120
      holdTimeSeconds: 90
      keepAliveTimeSeconds: 30
      gracefulRestart:
        enabled: true
        restartTimeSeconds: 120

Update Cilium BGP advertisement selectors

Configure Cilium to advertise specific service types through BGP peering.

kubectl apply -f shared-bgp-config.yaml
kubectl label nodes --all bgp-policy=enabled
kubectl get ciliumBGPpeeringpolicy unified-bgp-policy -o yaml

Configure upstream BGP router

Set up the upstream router to accept BGP peering from both Cilium and MetalLB.

sudo vtysh -c '
configure terminal
router bgp 65001
bgp router-id 203.0.113.1
neighbor 203.0.113.10 remote-as 65000
neighbor 203.0.113.10 description "Kubernetes-Cluster"
address-family ipv4 unicast
 neighbor 203.0.113.10 activate
 neighbor 203.0.113.10 soft-reconfiguration inbound
exit-address-family
exit
exit
write memory'

Verify BGP session establishment

Check that BGP peering is active between all components and routes are being exchanged.

kubectl exec -n kube-system ds/cilium -- cilium bgp peers
kubectl logs -n metallb-system -l component=speaker --tail=20
sudo vtysh -c "show ip bgp summary"
sudo vtysh -c "show ip bgp neighbors"

Deploy test LoadBalancer service

Create a sample application with LoadBalancer service to test the BGP routing configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: test-nginx
  template:
    metadata:
      labels:
        app: test-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx-lb
  labels:
    io.cilium/bgp-announce: "true"
spec:
  type: LoadBalancer
  selector:
    app: test-nginx
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  loadBalancerSourceRanges:
  - 0.0.0.0/0

Apply test deployment

Deploy the test service and verify that it receives an external IP from the MetalLB pool.

kubectl apply -f test-loadbalancer.yaml
kubectl get service test-nginx-lb -w
kubectl get endpoints test-nginx-lb
kubectl describe service test-nginx-lb

Test external connectivity

Verify that the LoadBalancer service is accessible from external networks via the advertised routes.

EXTERNAL_IP=$(kubectl get service test-nginx-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "External IP: $EXTERNAL_IP"
curl -I http://$EXTERNAL_IP
ping -c 4 $EXTERNAL_IP

Verify BGP route advertisement

Check that both Cilium and MetalLB are correctly advertising the LoadBalancer IP via BGP.

sudo vtysh -c "show ip bgp"
sudo vtysh -c "show ip route bgp"
kubectl exec -n kube-system ds/cilium -- cilium bgp routes available
kubectl logs -n metallb-system -l component=speaker | grep -i "announcing"

Configure BGP communities and route policies

Set up BGP communities for fine-grained route control and traffic engineering.

apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: advanced-bgp-adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - production-pool
  communities:
  - 65000:100  # Production traffic
  - 65000:200  # High priority
  localPref: 200
  peers:
  - 203.0.113.1
  - 203.0.113.2

Enable BGP graceful restart

Configure graceful restart to maintain connectivity during BGP session interruptions.

kubectl patch ciliumBGPpeeringpolicy unified-bgp-policy --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/virtualRouters/0/neighbors/0/gracefulRestart/enabled",
    "value": true
  },
  {
    "op": "add",
    "path": "/spec/virtualRouters/0/neighbors/0/gracefulRestart/restartTimeSeconds",
    "value": 120
  }
]'

Configure BFD for fast failover

Enable Bidirectional Forwarding Detection for sub-second failure detection.

apiVersion: metallb.io/v1beta1
kind: BFDProfile
metadata:
  name: fast-detection
  namespace: metallb-system
spec:
  receiveInterval: 300
  transmitInterval: 300
  detectMultiplier: 3
  echoMode: false
  passiveMode: false
  minimumTtl: 254