Configure automated index lifecycle management (ILM) policies in Elasticsearch 8 to optimize storage, performance, and data retention. Learn to set up hot-warm-cold architectures with automatic rollover and deletion based on age and size criteria.
Prerequisites
- Elasticsearch 8.x installed and running
- Administrative access to Elasticsearch cluster
- Basic understanding of Elasticsearch indices and aliases
What this solves
Elasticsearch index lifecycle management (ILM) automates the management of indices as they age, moving them through different phases to optimize performance and storage costs. Without ILM, indices grow indefinitely, consuming disk space and degrading search performance as data accumulates over time.
This tutorial shows you how to configure ILM policies that automatically roll over indices based on size or age, move older data to less expensive storage tiers, and delete indices after specified retention periods.
Step-by-step configuration
Verify Elasticsearch installation and cluster health
Ensure Elasticsearch 8 is running and accessible before configuring ILM policies.
curl -X GET "localhost:9200/_cluster/health?pretty" -u elastic:your_password
curl -X GET "localhost:9200/_cat/indices?v" -u elastic:your_passwordCreate a basic ILM policy for log data
Define an ILM policy that manages log indices through hot, warm, and delete phases based on age and size thresholds.
curl -X PUT "localhost:9200/_ilm/policy/logs-policy?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "10gb",
"max_age": "7d",
"max_docs": 10000000
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"set_priority": {
"priority": 50
},
"allocate": {
"number_of_replicas": 0
},
"shrink": {
"number_of_shards": 1
},
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}'Create an index template with ILM policy
Configure an index template that automatically applies the ILM policy to new indices matching the pattern.
curl -X PUT "localhost:9200/_index_template/logs-template?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"index.lifecycle.name": "logs-policy",
"index.lifecycle.rollover_alias": "logs"
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "text"
},
"level": {
"type": "keyword"
},
"service": {
"type": "keyword"
}
}
}
},
"composed_of": [],
"priority": 200
}'Create initial index and alias
Create the first index in the series with the proper alias configuration for rollover functionality.
curl -X PUT "localhost:9200/logs-000001?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"aliases": {
"logs": {
"is_write_index": true
}
}
}'Configure a retention policy for metrics data
Create a separate ILM policy optimized for metrics data with different retention and rollover settings.
curl -X PUT "localhost:9200/_ilm/policy/metrics-policy?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "5gb",
"max_age": "1d",
"max_docs": 5000000
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "1d",
"actions": {
"set_priority": {
"priority": 50
},
"allocate": {
"number_of_replicas": 0,
"include": {},
"exclude": {},
"require": {
"data_tier": "warm"
}
},
"shrink": {
"number_of_shards": 1
},
"forcemerge": {
"max_num_segments": 1
}
}
},
"cold": {
"min_age": "7d",
"actions": {
"set_priority": {
"priority": 0
},
"allocate": {
"number_of_replicas": 0,
"require": {
"data_tier": "cold"
}
}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}'Create metrics index template
Configure the index template for metrics data with appropriate mappings and ILM settings.
curl -X PUT "localhost:9200/_index_template/metrics-template?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"index_patterns": ["metrics-*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "metrics-policy",
"index.lifecycle.rollover_alias": "metrics",
"index.mapping.total_fields.limit": 2000
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"metric_name": {
"type": "keyword"
},
"value": {
"type": "double"
},
"host": {
"type": "keyword"
},
"environment": {
"type": "keyword"
}
}
}
},
"priority": 200
}'Initialize metrics index and alias
Create the initial metrics index with the write alias for the ILM rollover process.
curl -X PUT "localhost:9200/metrics-000001?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"aliases": {
"metrics": {
"is_write_index": true
}
}
}'Configure data tiers for hot-warm-cold architecture
Configure node attributes for data tier allocation if using dedicated hardware tiers.
# Hot tier node configuration
node.roles: ["master", "data_hot", "ingest"]
node.attr.data_tier: hot
Warm tier node configuration (on separate nodes)
node.roles: ["data_warm"]
node.attr.data_tier: warm
Cold tier node configuration (on separate nodes)
node.roles: ["data_cold"]
node.attr.data_tier: cold
Restart Elasticsearch to apply configuration changes
Restart the Elasticsearch service to load the updated configuration settings.
sudo systemctl restart elasticsearch
sudo systemctl status elasticsearchTest ILM policy with sample data
Add sample data to verify that the ILM policies are working correctly with automatic rollover.
# Add sample log data
for i in {1..1000}; do
curl -X POST "localhost:9200/logs/_doc?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"@timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%S.000Z)'",
"message": "Sample log message '${i}'",
"level": "INFO",
"service": "web-server"
}'
done
Add sample metrics data
for i in {1..1000}; do
curl -X POST "localhost:9200/metrics/_doc?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"@timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%S.000Z)'",
"metric_name": "cpu_usage",
"value": '$((RANDOM % 100))',
"host": "server-'$((i % 5 + 1))',
"environment": "production"
}'
doneNote: For production environments, consider using bulk API operations for better performance when indexing large amounts of data. The ELK Stack tutorial covers efficient data ingestion patterns.
Monitor and optimize ILM performance
Monitor ILM policy execution
Use Elasticsearch APIs to monitor the status and execution of ILM policies across your indices.
# Check ILM policy status
curl -X GET "localhost:9200/_ilm/status?pretty" -u elastic:your_password
View all ILM policies
curl -X GET "localhost:9200/_ilm/policy?pretty" -u elastic:your_password
Check specific index ILM status
curl -X GET "localhost:9200/logs-*/_ilm/explain?pretty" -u elastic:your_password
View index lifecycle step details
curl -X GET "localhost:9200/_cat/indices?v&h=index,status,pri,rep,docs.count,store.size,creation.date.string" -u elastic:your_passwordSet up ILM monitoring alerts
Configure monitoring to alert on ILM policy failures or unexpected behavior.
# Check for indices with ILM errors
curl -X GET "localhost:9200/_ilm/explain?filter_path=indices..step_info.type,indices..failed_step" -u elastic:your_password
Monitor data tier allocation
curl -X GET "localhost:9200/_cat/nodeattrs?v&h=node,attr,value" -u elastic:your_password
Check cluster disk usage
curl -X GET "localhost:9200/_cat/nodes?v&h=name,disk.used_percent,heap.percent,ram.percent" -u elastic:your_passwordOptimize ILM policy settings
Fine-tune ILM policies based on your data patterns and performance requirements.
# Update existing policy with optimized settings
curl -X PUT "localhost:9200/_ilm/policy/logs-policy?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "15gb",
"max_age": "1d",
"max_docs": 15000000
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "3d",
"actions": {
"readonly": {},
"set_priority": {
"priority": 50
},
"allocate": {
"number_of_replicas": 0
},
"shrink": {
"number_of_shards": 1
},
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "60d",
"actions": {
"delete": {}
}
}
}
}
}'Configure ILM polling interval
Adjust how frequently Elasticsearch checks and executes ILM policy actions.
# Set ILM polling interval (default is 10 minutes)
curl -X PUT "localhost:9200/_cluster/settings?pretty" -u elastic:your_password -H 'Content-Type: application/json' -d'
{
"persistent": {
"indices.lifecycle.poll_interval": "5m"
}
}'Verify your setup
# Check ILM is enabled and running
curl -X GET "localhost:9200/_ilm/status?pretty" -u elastic:your_password
Verify policies are created
curl -X GET "localhost:9200/_ilm/policy/logs-policy,metrics-policy?pretty" -u elastic:your_password
Check index templates
curl -X GET "localhost:9200/_index_template/logs-template,metrics-template?pretty" -u elastic:your_password
View current indices and their lifecycle status
curl -X GET "localhost:9200/_cat/indices/logs-,metrics-?v&s=creation.date" -u elastic:your_password
Check alias configuration
curl -X GET "localhost:9200/_alias/logs,metrics?pretty" -u elastic:your_passwordCommon issues
| Symptom | Cause | Fix |
|---|---|---|
| ILM not executing actions | ILM service disabled or polling interval too high | Enable ILM: PUT _ilm/start and check polling interval |
| Rollover not happening | Missing write alias or incorrect alias configuration | Verify alias with is_write_index: true is set |
| Indices stuck in phase | Resource constraints or allocation failures | Check cluster resources and allocation requirements |
| Authentication errors | Missing or incorrect credentials | Verify user has manage_ilm and manage_index_templates privileges |
| Data not moving to warm tier | Node attributes not configured correctly | Ensure nodes have proper data_tier attributes and roles |
| Shrink action failing | Index has multiple replicas or is not read-only | Set replicas to 0 and make index read-only before shrink |
Warning: Test ILM policies thoroughly in non-production environments before applying to production data. Incorrect delete phases can result in permanent data loss.
Next steps
#elasticsearch
#ilm
#index-lifecycle-management
#data-retention
#hot-warm-cold