Install and configure Filebeat 8.15 for efficient log shipping to ELK stack

Add Elastic repository and GPG key

Install the official Elastic repository to get Filebeat 8.15 and security updates.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
sudo tee /etc/yum.repos.d/elastic.repo << 'EOF'
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

Install Filebeat 8.15

Install Filebeat from the official repository to get the latest stable version.

sudo apt install -y filebeat=8.15.*

sudo dnf install -y --enablerepo=elasticsearch filebeat-8.15.*

Configure basic Filebeat settings

Set up the main configuration file with your Elasticsearch connection details and basic logging paths.

# Basic Filebeat configuration
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/syslog
    - /var/log/auth.log
  fields:
    logtype: system
    environment: production
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  fields:
    logtype: nginx
    environment: production
  fields_under_root: true
  multiline.pattern: '^\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after

# Output configuration
output.elasticsearch:
  hosts: ["203.0.113.10:9200", "203.0.113.11:9200"]
  protocol: "https"
  username: "elastic"
  password: "your-elasticsearch-password"
  index: "filebeat-%{+yyyy.MM.dd}"
  
# Processor configuration
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "ecs.version"]

# Logging configuration
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0600

Configure SSL/TLS security

Set up secure connections to Elasticsearch with certificate verification and encryption.

# Add this to your elasticsearch output section
output.elasticsearch:
  hosts: ["203.0.113.10:9200", "203.0.113.11:9200"]
  protocol: "https"
  username: "elastic"
  password: "your-elasticsearch-password"
  
  # SSL/TLS configuration
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
  ssl.certificate: "/etc/filebeat/filebeat.crt"
  ssl.key: "/etc/filebeat/filebeat.key"
  ssl.verification_mode: "strict"
  
  # Connection tuning
  worker: 2
  bulk_max_size: 1024
  timeout: 60s
  compression_level: 1

Set up log collection modules

Enable and configure Filebeat modules for common services like Nginx, Apache, and system logs.

# Enable system module
sudo filebeat modules enable system

# Enable nginx module
sudo filebeat modules enable nginx

# Enable apache module (if needed)
sudo filebeat modules enable apache

# List enabled modules
sudo filebeat modules list

Configure system module

Customize the system module to collect auth logs, syslog, and other system events.

# Module: system
# Docs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html

- module: system
  # Syslog
  syslog:
    enabled: true
    var.paths: ["/var/log/syslog", "/var/log/messages"]
    
  # Authorization logs
  auth:
    enabled: true
    var.paths: ["/var/log/auth.log", "/var/log/secure"]
    
  # Convert timezone
  processors:
    - add_locale:
        format: offset
    - timestamp:
        field: '@timestamp'
        layouts:
          - '2006-01-02T15:04:05.000Z'
          - '2006-01-02T15:04:05Z'
        test:
          - '2023-05-15T14:30:45.123Z'

Configure Nginx module

Set up the Nginx module to parse access and error logs with proper field mapping.

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
    
  # Error logs
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]
    
  # Pipeline for custom log format
  access:
    var.pipeline: filebeat-8.15.0-nginx-access-custom

Set up performance optimization

Configure Filebeat for high-throughput environments with proper queue settings and resource limits.

# Queue settings for high throughput
queue.mem:
  events: 8192
  flush.min_events: 1024
  flush.timeout: 1s

# Resource limits
filebeat.max_procs: 2

# Registry settings for large file tracking
filebeat.registry:
  path: /var/lib/filebeat/registry
  file_permissions: 0600
  flush: 1s

# Harvester settings
filebeat.inputs:
- type: log
  close_inactive: 5m
  close_removed: true
  close_renamed: false
  clean_inactive: 72h
  ignore_older: 24h
  scan_frequency: 10s
  harvester_buffer_size: 16384
  max_bytes: 10485760  # 10MB
  
# Monitor file system changes
filebeat.shutdown_timeout: 5s

Configure monitoring and metrics

Enable internal metrics collection and monitoring endpoints for observability.

# Internal metrics
monitoring:
  enabled: true
  elasticsearch:
    hosts: ["203.0.113.10:9200", "203.0.113.11:9200"]
    protocol: "https"
    username: "elastic"
    password: "your-elasticsearch-password"
    ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
    
# HTTP endpoint for metrics
http:
  enabled: true
  host: "127.0.0.1"
  port: 5066
  
# Metrics collection
metrics.period: 30s

Set proper file permissions

Secure the Filebeat configuration and certificate files with appropriate ownership and permissions.

# Set ownership for Filebeat user
sudo chown -R root:root /etc/filebeat/
sudo chown -R filebeat:filebeat /var/lib/filebeat/
sudo chown -R filebeat:filebeat /var/log/filebeat/

# Set secure permissions
sudo chmod 600 /etc/filebeat/filebeat.yml
sudo chmod 600 /etc/filebeat/*.key
sudo chmod 644 /etc/filebeat/*.crt
sudo chmod 755 /etc/filebeat/
sudo chmod 755 /var/lib/filebeat/
sudo chmod 755 /var/log/filebeat/

Test configuration and setup index template

Validate your configuration and set up the Elasticsearch index template for optimal field mapping.

# Test configuration syntax
sudo filebeat test config

# Test Elasticsearch connection
sudo filebeat test output

# Setup index template and pipelines
sudo filebeat setup --index-management --pipelines

# Load Kibana dashboards (optional)
sudo filebeat setup --dashboards

Enable and start Filebeat service

Start Filebeat and enable it to run automatically on system boot.

# Enable and start Filebeat
sudo systemctl enable filebeat
sudo systemctl start filebeat

# Check service status
sudo systemctl status filebeat

# Check initial logs
sudo journalctl -u filebeat -f --no-pager

Create performance monitoring script

Monitor key Filebeat metrics like harvested events, processing rates, and error counts.

#!/bin/bash
# Filebeat performance monitoring script

API_URL="http://127.0.0.1:5066/stats"
LOG_FILE="/var/log/filebeat-monitor.log"

# Get metrics
METRICS=$(curl -s $API_URL)

# Extract key values
EVENTS_TOTAL=$(echo $METRICS | jq -r '.filebeat.events.total // 0')
HARVESTER_RUNNING=$(echo $METRICS | jq -r '.filebeat.harvester.running // 0')
OUTPUT_EVENTS=$(echo $METRICS | jq -r '.libbeat.output.events.total // 0')
OUTPUT_ERRORS=$(echo $METRICS | jq -r '.libbeat.output.events.failed // 0')
MEMORY_USAGE=$(echo $METRICS | jq -r '.beat.memstats.memory_total // 0')

# Log metrics with timestamp
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
echo "[$TIMESTAMP] Events: $EVENTS_TOTAL, Harvesters: $HARVESTER_RUNNING, Output: $OUTPUT_EVENTS, Errors: $OUTPUT_ERRORS, Memory: $MEMORY_USAGE" >> $LOG_FILE

# Alert on high error rate
if [ "$OUTPUT_ERRORS" -gt 100 ]; then
    echo "[$TIMESTAMP] ALERT: High error count: $OUTPUT_ERRORS" >> $LOG_FILE
fi

# Make script executable
sudo chmod 755 /usr/local/bin/filebeat-monitor.sh

# Add to cron for regular monitoring
sudo crontab -e
# Add this line:
*/5 * * * * /usr/local/bin/filebeat-monitor.sh

Set up log rotation for monitoring

Configure logrotate to manage Filebeat log files and prevent disk space issues.

/var/log/filebeat/filebeat {
    daily
    rotate 7
    copytruncate
    delaycompress
    compress
    notifempty
    missingok
    postrotate
        /bin/systemctl reload filebeat > /dev/null 2>&1 || true
    endscript
}

/var/log/filebeat-monitor.log {
    daily
    rotate 14
    compress
    delaycompress
    copytruncate
    notifempty
    missingok
}

Debug connection issues

Enable detailed logging to troubleshoot connectivity or parsing problems.

# Enable debug logging temporarily
sudo filebeat -e -d "*" -c /etc/filebeat/filebeat.yml

# Check specific module debugging
sudo filebeat -e -d "harvester,publish" -c /etc/filebeat/filebeat.yml

# Test specific log file parsing
sudo filebeat test inputs

Optimize for high-volume logging

Tune Filebeat settings for environments with high log volumes or strict performance requirements.

# High-volume optimization
queue.mem:
  events: 16384
  flush.min_events: 2048
  flush.timeout: 2s

output.elasticsearch:
  worker: 4
  bulk_max_size: 2048
  compression_level: 3
  template.settings:
    index.refresh_interval: "30s"
    index.number_of_shards: 2
    index.number_of_replicas: 0

# Reduce harvester overhead
filebeat.inputs:
- type: log
  scan_frequency: 30s
  harvester_buffer_size: 32768
  max_bytes: 52428800  # 50MB