Configure Apache Cassandra cluster with multi-node setup and performance optimization

Update system packages and install Java

Cassandra requires Java 8 or 11. Install OpenJDK 11 which provides optimal performance for Cassandra workloads.

sudo apt update && sudo apt upgrade -y
sudo apt install -y openjdk-11-jdk python3-pip wget gnupg

sudo dnf update -y
sudo dnf install -y java-11-openjdk-devel python3-pip wget gnupg2

Add Apache Cassandra repository

Add the official Apache Cassandra repository to install the latest stable version with security updates.

wget -q -O - https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -
echo "deb https://downloads.apache.org/cassandra/debian 41x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
sudo apt update

sudo rpm --import https://downloads.apache.org/cassandra/KEYS
sudo tee /etc/yum.repos.d/cassandra.repo << EOF
[cassandra]
name=Apache Cassandra
baseurl=https://downloads.apache.org/cassandra/redhat/41x/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://downloads.apache.org/cassandra/KEYS
EOF

Install Apache Cassandra

Install Cassandra and its dependencies. This creates the cassandra user and sets up the service configuration.

sudo apt install -y cassandra

sudo dnf install -y cassandra

Configure system limits for Cassandra

Cassandra requires increased file descriptor and process limits for optimal performance. Configure systemd and PAM limits.

cassandra soft memlock unlimited
cassandra hard memlock unlimited
cassandra soft nofile 100000
cassandra hard nofile 100000
cassandra soft nproc 32768
cassandra hard nproc 32768
cassandra soft as unlimited
cassandra hard as unlimited

For more details on configuring system resource limits, see our guide on configuring Linux system resource limits.

Configure Cassandra cluster settings

Configure the main Cassandra configuration file with cluster name, seed nodes, and network settings. This example configures a three-node cluster.

cluster_name: 'ProductionCluster'
num_tokens: 256
allocate_tokens_for_keyspace: system_auth
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000

seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "203.0.113.10,203.0.113.11,203.0.113.12"

listen_address: 203.0.113.10
broadcast_address: 203.0.113.10
rpc_address: 0.0.0.0
broadcast_rpc_address: 203.0.113.10

endpoint_snitch: GossipingPropertyFileSnitch

data_file_directories:
    - /var/lib/cassandra/data

commitlog_directory: /var/lib/cassandra/commitlog
saved_caches_directory: /var/lib/cassandra/saved_caches

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32

concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32

disk_optimization_strategy: ssd
memtable_allocation_type: heap_buffers

index_summary_capacity_in_mb: 100
index_summary_resize_interval_in_minutes: 60

trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

storage_port: 7000
ssl_storage_port: 7001
native_transport_port: 9042
rpc_port: 9160

start_native_transport: true
native_transport_max_threads: 128
native_transport_max_frame_size_in_mb: 256

thrift_framed_transport_size_in_mb: 15

incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true

column_index_size_in_kb: 64
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50

compaction_throughput_mb_per_sec: 16
compaction_large_partition_warning_threshold_mb: 100

read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000

internode_compression: dc
inter_dc_tcp_nodelay: false

stream_throughput_outbound_megabits_per_sec: 200
inter_dc_stream_throughput_outbound_megabits_per_sec: 200

Configure datacenter and rack information

Set up datacenter and rack information for proper replica placement and network topology awareness.

dc=dc1
rack=rack1
prefer_local=true

Optimize JVM heap settings

Configure JVM heap size and garbage collection settings. Set heap to 25-50% of system RAM, maximum 31GB to avoid compressed OOP overhead.

-Xms8G
-Xmx8G
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:G1MaxNewSizePercent=75
-XX:G1NewSizePercent=40
-XX:MaxGCPauseMillis=100
-XX:+UseStringDeduplication
-XX:+UseTLAB
-XX:+ResizeTLAB
-XX:+UseNUMA
-XX:+PerfDisableSharedMem
-XX:+AlwaysPreTouch
-XX:+UseBiasedLocking
-XX:+OptimizeStringConcat
-XX:+UseCondCardMark
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.rmi.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=203.0.113.10

Set up data directory permissions

Ensure Cassandra has proper ownership of its data directories. Never use chmod 777 as it creates security vulnerabilities.

sudo chown -R cassandra:cassandra /var/lib/cassandra
sudo chown -R cassandra:cassandra /var/log/cassandra
sudo chmod 755 /var/lib/cassandra
sudo chmod 755 /var/log/cassandra

Configure firewall rules

Open necessary ports for Cassandra cluster communication. Port 7000 for inter-node communication, 9042 for client connections.

sudo ufw allow from 203.0.113.0/24 to any port 7000
sudo ufw allow from 203.0.113.0/24 to any port 7001
sudo ufw allow from 203.0.113.0/24 to any port 9042
sudo ufw allow from 203.0.113.0/24 to any port 7199

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="7000" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="7001" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="9042" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="7199" accept'
sudo firewall-cmd --reload

Start Cassandra service

Enable and start the Cassandra service. The service will automatically join the cluster using the seed node configuration.

sudo systemctl enable cassandra
sudo systemctl start cassandra
sudo systemctl status cassandra

Repeat configuration on additional nodes

Repeat steps 1-9 on each additional cluster node, changing only the listen_address, broadcast_address, and broadcast_rpc_address to match each node's IP address. Keep the seed list identical across all nodes.

Configure authentication and create admin user

Change the default cassandra user password and create administrative users for production security.

cqlsh -u cassandra -p cassandra
ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3};
CREATE ROLE admin WITH PASSWORD = 'SecurePassword123!' AND LOGIN = true AND SUPERUSER = true;
ALTER ROLE cassandra WITH PASSWORD = 'NewCassandraPassword123!';
EXIT;

Configure replication strategy and consistency

Set up keyspaces with appropriate replication strategies for your data. NetworkTopologyStrategy provides better performance and availability.

cqlsh -u admin -p 'SecurePassword123!'
CREATE KEYSPACE production WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'dc1': 3
} AND durable_writes = true;

CREATE TABLE production.users (
  user_id UUID PRIMARY KEY,
  username TEXT,
  email TEXT,
  created_at TIMESTAMP
) WITH compaction = {'class': 'SizeTieredCompactionStrategy'};

EXIT;

Optimize compaction settings

Configure compaction strategies based on your read/write patterns. STCS for write-heavy, LCS for read-heavy workloads.

cqlsh -u admin -p 'SecurePassword123!'
ALTER TABLE production.users WITH compaction = {
  'class': 'LeveledCompactionStrategy',
  'sstable_size_in_mb': 160
};
EXIT;

Configure kernel parameters for performance

Optimize kernel settings for better I/O performance and network handling. These settings improve throughput for database workloads.

vm.max_map_count = 1048575
vm.swappiness = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 65536 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

sudo sysctl -p /etc/sysctl.d/99-cassandra.conf

For comprehensive kernel optimization, see our guide on optimizing Linux system performance.

Set up monitoring and alerting

Configure JMX monitoring and integrate with monitoring systems for production observability.

nodetool sjk mxb -b :7199 -mc
pip3 install cassandra-driver
sudo mkdir -p /opt/cassandra-monitoring
sudo chown cassandra:cassandra /opt/cassandra-monitoring