# Binadit - Full Content

> European managed cloud and infrastructure partner based in Rotterdam, NL. This file contains full article content from the Binadit engineering blog and tutorials for AI grounding and citation. See /llms.txt for the navigation index.

Generated: 2026-08-11T04:49:10+02:00
Source: https://binadit.com/llms-full.txt

## Engineering articles

### Website server setup: what separates hobby projects from production

URL: https://binadit.com/blog/website-server-setup-hobby-projects-vs-production
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-08-10T10:10:46+02:00

> A website server that works fine for a portfolio site will fall over under real business traffic. Here's what actually changes between a hobby setup and a production-grade website server, and where the line usually gets crossed.

Two very different meanings of "it works"

Every website server starts the same way. You spin up a box, install a web server, point a domain at it, and it works. The page loads. The contact form submits. Everyone moves on.

The problem is that "it works" means something completely different depending on what's riding on that server. For a side project, it means the site is reachable. For a business, it means the site stays reachable during a marketing campaign, survives a traffic spike from a press mention, recovers automatically when a process crashes at 3am, and doesn't leak customer data because a plugin had an unpatched vulnerability.

Most businesses don't consciously decide to run production traffic on a hobby-grade website server. It happens gradually. A WordPress site built on cheap shared hosting starts taking orders. A side project becomes the primary lead generation channel. Nobody goes back and re-architects the server, because it's still technically running.

This article breaks down what actually separates a hobby website server setup from a production one, with concrete specs and configuration details, so you can see exactly where your own setup sits.

Why the gap matters more than it looks

On a hobby project, downtime costs nothing. Nobody is refreshing the page waiting to buy something. A slow page load is mildly annoying at worst.

On a business website, every one of these has a direct financial consequence:

500ms of added latency measurably reduces conversion rates on checkout and signup flows

An hour of downtime during a paid ad campaign burns budget with nothing to show for it

A database that locks up under concurrent writes turns a busy sales day into a support nightmare

A misconfigured web server that exposes server signatures and outdated software becomes a target

The technical gap between hobby and production website hosting is usually described in vague terms like "scalability" or "reliability." Those words don't help you build anything. What actually separates the two is a specific set of decisions about compute, caching, redundancy, and process management.

Where hobby setups fall short: the common pattern

Most hobby-to-business website server setups share the same weak points, regardless of the underlying stack.

Single point of failure, everywhere

One server runs the web server, the application, the database, and the cron jobs. If any one of those processes misbehaves, memory pressure or CPU contention affects everything else running on the box. There's no separation between the thing serving traffic and the thing storing your data.

Default configurations left untouched

Apache's default MaxRequestWorkers or PHP-FPM's default pm.max_children are tuned for generic use, not your actual traffic pattern and available memory. A PHP-FPM pool with too many workers on a 2GB VPS will get OOM-killed under moderate concurrency. A pool with too few workers queues requests, which looks like slowness but is actually starvation.

No caching layer beyond what the CMS ships with

Every page request re-executes application code and hits the database, even for content that hasn't changed in weeks. This is fine at 50 visitors a day. At 5,000, the database becomes the bottleneck long before the CPU does.

Manual everything

Deployments happen over FTP or SSH by hand. Backups are a cron job nobody has tested restoring from. SSL renewal is a manual reminder in a calendar. None of this scales past one person's attention span, and it doesn't survive that person being on holiday when something breaks.

What actually changes in a production website server

None of the fixes here are exotic. They're deliberate choices made once traffic and revenue justify the engineering time.

Separate the web server from the data layer

In a production setup, the web server (Nginx or Apache) and application runtime sit on infrastructure that can be scaled independently from the database. Even a modest setup benefits from this: a dedicated database instance with its own memory allocation avoids the scenario where a traffic spike on the web tier starves the database of resources it needs for query execution.

At larger scale, this becomes horizontal: multiple application servers behind a load balancer, a managed or replicated database, and a shared session store like Redis so any server can handle any request. We cover the mechanics of this in choosing the right web server architecture for your application.

Tune the web server for actual traffic, not defaults

A production Nginx configuration sets explicit worker counts based on CPU cores, connection limits based on available file descriptors, and buffer sizes based on typical request and response sizes. A basic example for a 4-core, 8GB server serving a mid-traffic WordPress or Laravel site:

worker_processes 4;
worker_connections 1024;
keepalive_timeout 15;
client_max_body_size 32m;

Paired with PHP-FPM tuned to the same box:

pm = dynamic
pm.max_children = 40
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 15

These numbers aren't universal. They depend on average memory per PHP process, which you measure, not guess. A hobby setup skips this step entirely and runs on whatever the installer configured.

Add a real caching layer

Production website hosting uses caching at multiple layers: a full-page cache (Varnish, Nginx FastCGI cache, or a CDN edge cache) for anonymous traffic, an object cache (Redis or Memcached) for database query results and session data, and a CDN for static assets. This removes repeated work from the origin server entirely, rather than trying to make the origin server faster.

We've written in detail about getting this right in why your caching strategy is probably wrong, and specifically for e-commerce in configuring Redis for a high-traffic WooCommerce store.

Build in redundancy, not just capacity

A bigger server is not the same as a resilient server. Production setups assume individual components will fail and design around that: automated failover for the database, health checks that pull unhealthy application servers out of rotation, and infrastructure defined as code so a failed server can be rebuilt identically in minutes rather than manually reconfigured.

Automate the operational parts

Deployments run through a pipeline that tests before it ships. Backups are automated, encrypted, stored off-server, and periodically test-restored. Monitoring alerts on symptoms that matter (error rate, response time, queue depth) rather than just "server is up." None of this is optional at scale; it's what keeps a two-person engineering team from being paged every night.

The specs and configuration that actually matter

When people ask what specs a production website server needs, the honest answer is: it depends on the workload, but here's what to actually check rather than guess.
FactorHobby-level defaultProduction baselineComputeShared vCPU, burstable, oversoldDedicated or guaranteed vCPU, sized to peak load with headroomMemoryWhatever's left after other tenantsSized to (PHP-FPM workers × avg process memory) + DB cache + OS overheadDatabaseSame box as the web serverSeparate instance, tuned buffer pool, automated backups and replicationCachingNone beyond browser cacheObject cache + full-page cache + CDN for static assetsSSL/TLSManually renewed or forgottenAutomated renewal, HTTP/2 or HTTP/3, modern cipher suites enforcedMonitoringUptime ping every 5 minutesApplication-level metrics, error tracking, alerting tied to on-callDeploymentsManual FTP or SSHCI/CD pipeline with automated tests and rollback capability

The pattern across every row is the same: hobby setups react to problems after they happen, production setups are configured to prevent or contain them.

Where this goes wrong in practice: a real pattern

A common scenario we see with growing e-commerce and SaaS clients: a business starts on shared hosting or a single small VPS because it's cheap and it works at launch. Traffic grows steadily. Nobody revisits the server configuration because nothing has visibly broken yet.

Then a marketing campaign, a seasonal sale, or a product mention on social media sends a traffic spike. PHP-FPM's worker limit is hit within minutes. Requests queue. Response times climb from 200ms to 8 seconds. The database, already under load from the traffic spike, starts timing out on writes. Orders fail silently. Support tickets pile up. The business loses revenue during the exact event that was supposed to generate it.

This isn't a hardware failure. Every individual component was technically "working." It's a configuration and architecture gap that only becomes visible under load, which is exactly why hobby setups can run for years without anyone noticing the ceiling. We go deeper into this failure mode in why your website is slow under traffic spikes.

The fix in that scenario is rarely "buy a bigger server." It's usually a combination of caching that was never implemented, database queries that were never indexed properly, and a web server configuration still running on defaults from a five-minute install script.

Shared hosting, VPS, or managed: where each one fits

Not every business needs the same tier, and being honest about that matters more than selling everyone the most expensive option.

Shared hosting is genuinely fine for low-traffic brochure sites, early-stage projects, and anything where downtime has no financial consequence. The trade-off is no control over resource allocation, no ability to tune the web server, and unpredictable performance because you're sharing capacity with unknown neighbors.

A VPS gives you dedicated resources and root access, which means you can actually tune the configuration described above. It's a reasonable middle ground for businesses with predictable, moderate traffic and someone in-house who can manage a Linux server. The ceiling is your own team's time: someone still has to patch it, monitor it, and respond when it needs attention outside business hours.

Managed infrastructure makes sense once the operational burden of running the server yourself costs more than paying someone to run it properly. This is usually the point where traffic is unpredictable, downtime has a real cost, or the team would rather spend engineering hours on product instead of server maintenance. It's not about needing more raw compute. It's about needing the tuning, monitoring, and incident response that a hobby setup was never designed to provide.

If you're trying to figure out which tier your VPS decision should land in, best VPS for production: how to choose the right setup covers the specific specs to look for.

Closing

The difference between a hobby website server and a production one isn't a single dramatic upgrade. It's a series of deliberate decisions: separating concerns, tuning configuration to real traffic, adding caching where it actually helps, building in redundancy, and automating the operational work that doesn't scale with manual effort.

None of that requires exotic infrastructure. It requires treating the server as something that was designed for your traffic, not something that happened to work when you launched. Binadit runs this kind of setup for SaaS platforms, agencies, and e-commerce businesses across Europe, with EU-based engineers who tune and monitor the infrastructure directly, no ticket queue in between. Learn more about managed cloud infrastructure or see our approach to managed VPS hosting.

We design and run this kind of setup for European businesses every day. See how we work.

---

### 10 practices that keep WooCommerce fast at scale

URL: https://binadit.com/blog/woocommerce-fast-at-scale-managed-infrastructure-for-saas
Category: Reliability
Author: Binadit Tech Team
Published: 2026-08-07T09:11:16+02:00

Who this is for

This checklist is for engineers running WooCommerce stores that have outgrown basic shared hosting: stores doing consistent daily order volume, running flash sales, or syncing inventory with an ERP. WooCommerce is a WordPress plugin built on PHP and MySQL, so most of its scaling problems are the same problems you would solve for any managed infrastructure for SaaS workload: database contention, cache invalidation, and background job processing. If you already run a store with more than a few hundred SKUs or a few thousand orders a month, these practices apply directly to you.

None of this requires a full replatform. Most of it is configuration, query discipline, and infrastructure choices you can make incrementally.

10 practices for WooCommerce performance at scale

1. Move sessions and cart data off MySQL

WooCommerce stores cart and session data in the wp_options and wp_woocommerce_sessions tables by default. Under concurrent traffic, this creates write contention on tables that are already read-heavy for product and pricing data.

Moving sessions to Redis removes that write pressure entirely. It also makes session lookups sub-millisecond instead of a MySQL round trip on every cart update.

// wp-config.php or a custom session handler plugin
define('WP_REDIS_HOST', '127.0.0.1');
define('WP_REDIS_PORT', 6379);
define('WP_REDIS_TIMEOUT', 1);
define('WP_REDIS_DATABASE', 2);

2. Separate object cache from page cache

Object caching (via Redis or Memcached) caches individual database query results: a product lookup, a term query, a user meta fetch. Page caching serves entire rendered HTML pages. These solve different problems and need different invalidation rules.

WooCommerce category and cart pages are not fully cacheable at the page level because they're user-specific or price-sensitive, but they benefit enormously from object caching underneath. Running both, correctly separated, is what actually gets you both speed and correctness. We cover the specific Redis parameters that matter for WooCommerce in how to configure Redis for a high-traffic WooCommerce store.

3. Exclude cart, checkout, and account pages from full-page cache

Full-page caching plugins that don't understand WooCommerce's dynamic fragments will happily cache a cart page with someone else's items in it. This is a correctness bug disguised as a performance win.

# Nginx example: never cache dynamic WooCommerce endpoints
location ~* ^/(cart|checkout|my-account) {
proxy_no_cache 1;
proxy_cache_bypass 1;
}

Everything else, product pages, category pages, the homepage, can be cached aggressively at the edge or with Nginx FastCGI cache, as long as cart fragments load via AJAX after the page renders.

4. Offload cart fragments to AJAX, not full page reloads

WooCommerce's default "mini cart" updates via an AJAX call to wc-ajax=get_refreshed_fragments. This is good, it's what allows the rest of the page to be cached, but it's frequently misconfigured to run on every page load even when the cart hasn't changed.

Throttle or conditionally disable fragment refreshes on pages where the cart state can't have changed (blog posts, static pages). This alone can cut PHP-FPM load by a noticeable percentage on content-heavy stores.

5. Index your custom query patterns, not just WooCommerce defaults

WooCommerce's default indexes on wp_postmeta and wp_wc_order_stats handle common queries fine, but custom filters, ERP sync jobs, or reporting plugins often introduce meta queries that aren't indexed at all.

-- Find slow meta queries via slow query log, then check for missing composite indexes
EXPLAIN SELECT post_id FROM wp_postmeta
WHERE meta_key = '_stock_status' AND meta_value = 'instock';

-- If this scans the full table, add:
ALTER TABLE wp_postmeta ADD INDEX idx_meta_key_value (meta_key, meta_value(20));

Run EXPLAIN on your top 10 slowest queries from the MySQL slow query log quarterly. WooCommerce plugin updates change query patterns; indexes don't keep up automatically.

6. Move order processing and emails to a queue

Sending order confirmation emails, syncing to an ERP, or triggering webhooks synchronously inside the checkout request adds latency directly to the customer's checkout experience. If any of those downstream systems are slow, the customer waits for them.

Use Action Scheduler (already bundled with WooCommerce) or a dedicated queue like Redis-backed jobs to push non-critical post-order tasks out of the request cycle.

// Hook into order completion, but defer the actual work
add_action('woocommerce_order_status_completed', function($order_id) {
as_schedule_single_action(time(), 'sync_order_to_erp', ['order_id' => $order_id]);
});

This is the same pattern that matters for any high availability infrastructure setup: keep the request path short, push everything else to background workers.

7. Use a CDN for product images and static assets, not just HTML

Product catalogs with hundreds of high-resolution images are the single biggest bandwidth and origin-load contributor on most WooCommerce stores. Serving these through a CDN with a long cache lifetime (30+ days) removes that load from your web servers entirely.

# Cache-Control for WooCommerce media
location ~* .(jpg|jpeg|png|webp|avif)$ {
expires 30d;
add_header Cache-Control "public, immutable";
}

Serve WebP or AVIF where the browser supports it. Image weight reduction has a direct, measurable effect on mobile checkout conversion, which matters more for revenue than most backend optimizations.

8. Autoscale PHP-FPM workers based on real concurrency, not fixed pools

A fixed pm.max_children value tuned for average traffic will queue requests during a flash sale and sit idle the rest of the month. Size your pool based on available memory per worker and monitor queue depth, not just CPU.

; php-fpm pool config
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500

pm.max_requests matters more than people think: it forces worker recycling, which prevents slow memory growth from long-running plugin code from degrading performance over a day of uptime.

9. Run read replicas for reporting and sync jobs

Analytics dashboards, ERP sync scripts, and admin reports run expensive aggregate queries against order and product tables. Running these against your primary database competes directly with checkout traffic for the same connection pool and locks.

Pointing reporting and sync workloads at a read replica isolates that load completely. This is a standard pattern across managed infrastructure for SaaS environments, and WooCommerce stores benefit from it just as much as any other transactional application.

10. Load test checkout specifically, not just the homepage

Homepage load times look great in most load tests because it's the most cacheable page on the site. Checkout is the least cacheable, most database-intensive path, and it's the one that directly determines whether a sale completes.

Run load tests that simulate real checkout flows: add to cart, apply a coupon, enter payment details, submit. We go deeper on this pattern in how to optimize checkout infrastructure to maximize conversion rates, and on campaign-specific load patterns in how WooCommerce stores handle campaign traffic.

Rolling this out in an existing team

You don't need to implement all 10 at once, and you shouldn't try to. A practical rollout order for a team with an existing store in production:

Week 1-2: Object cache and session offload (practices 1-2). These are infrastructure changes with no code risk and immediate, measurable latency improvement.

Week 3-4: Cache exclusion rules and CDN configuration (practices 3, 7). Low risk, high impact on time-to-first-byte for anonymous traffic.

Month 2: Query and index audit (practice 5), queue migration for order processing (practice 6). These require more testing since they touch business logic paths.

Month 2-3: PHP-FPM tuning and read replica setup (practices 8-9). Do these once you have real production metrics to size against, not before.

Ongoing: Checkout-specific load testing (practice 10) becomes a recurring exercise, ideally before every major sale event, not a one-time setup step.

Track before and after numbers for each change: time-to-first-byte, checkout completion time, and PHP-FPM queue depth during peak hours. Without a baseline, you can't tell whether a change actually helped or whether traffic just happened to be lighter that week.

If your team is migrating to this setup from shared hosting or a generic VPS, the sequencing matters more than the individual steps. Our guide on how to scale WooCommerce infrastructure without downtime covers how to make these changes without a cutover window.

Keeping it fast over time

These 10 practices are not a one-time project. Plugin updates change query patterns, catalogs grow, and traffic mix shifts as marketing campaigns change. Revisit the index audit and load test results quarterly, and treat PHP-FPM and cache configuration as something that gets tuned against real traffic data, not set once and forgotten.

The underlying principle is the same one that applies to any infrastructure management services engagement: keep the request path short, push non-critical work to background jobs, cache aggressively where correctness allows it, and measure before you optimize.

If implementing these yourself is not the best use of your engineering time, our managed services cover all of them by default.

---

### Setting up sovereign cloud reference architectures: three patterns that work

URL: https://binadit.com/blog/sovereign-cloud-reference-architectures-managed-cloud-infrastructure-patterns
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-08-05T09:11:35+02:00

What you will build and why it matters

Sovereign cloud architecture means designing infrastructure so that data residency, processing location, and legal jurisdiction stay predictable and provable, not just documented in a policy PDF. This guide walks through three reference patterns for managed cloud infrastructure that we deploy repeatedly for SaaS platforms, agencies, and e-commerce clients: single-region EU, active-passive multi-region, and hybrid private-public. By the end you will know which pattern fits your compliance requirements and traffic profile, and how to configure and verify each one.

These are not theoretical models. Each pattern below reflects configurations we run in production for clients handling GDPR-regulated data, payment infrastructure, or public-sector contracts.

Prerequisites and assumptions

Before implementing any of these patterns, you need the following in place:

An existing application that is stateless at the compute layer, or a plan to get there. Sovereign patterns assume you can run multiple app server instances without session affinity issues.

Infrastructure as code tooling: Terraform or OpenTofu, plus Ansible or a similar configuration management tool.

A clear data classification: which datasets are subject to GDPR, DORA, or sector-specific regulation, and which are not.

Access to at least one EU-based provider (bare metal or private cloud) and, for the hybrid pattern, a public cloud account for burst capacity.

Basic familiarity with load balancer configuration (HAProxy or Nginx) and database replication (PostgreSQL streaming replication or MySQL group replication).

If you are still deciding whether to build on open-source infrastructure or a public cloud provider, read choosing between the open-source sovereign stack and managed public cloud first. This guide assumes that decision is made.

Step-by-step implementation

Pattern 1: single-region EU (baseline sovereignty)

This is the right starting point for most SaaS companies and agencies that need to prove EU data residency without the operational overhead of multi-region replication.

Architecture: all compute, storage, and backups live in one EU region (for example, Amsterdam or Frankfurt), with no data leaving that boundary at any layer, including logging and error tracking.

Provision compute across at least two availability zones within the region, never a single physical rack.

Deploy a load balancer tier (HAProxy) in front of application servers:

frontend web_front
bind *:443 ssl crt /etc/haproxy/certs/app.pem
mode http
default_backend app_servers

backend app_servers
balance roundrobin
option httpchk GET /health
server app1 10.0.1.11:8080 check
server app2 10.0.1.12:8080 check
server app3 10.0.1.13:8080 check

Configure PostgreSQL streaming replication within the region, with a synchronous standby for zero data loss on failover:

# postgresql.conf on primary
synchronous_standby_names = 'standby1'
wal_level = replica
max_wal_senders = 5

# recovery config on standby
primary_conninfo = 'host=10.0.1.21 port=5432 user=replicator'

Route all third-party services (email delivery, error tracking, analytics) through EU-based providers. This step is frequently skipped and quietly breaks the sovereignty guarantee. See data sovereignty best practices for email, error tracking, and analytics for the specific providers to check.

Enable automated, encrypted backups stored in the same region, with a documented retention policy (typically 30 days daily, 12 months monthly for compliance workloads).

Pattern 2: active-passive multi-region

Use this pattern when you need disaster recovery across two EU regions (for example, Netherlands and Germany) without doubling your infrastructure spend, and when your compliance requirement allows failover as long as both regions stay within the EU.

Stand up a full replica environment in a second EU region, sized at 30-50% of production capacity (enough to run degraded, not enough to run idle at full cost).

Replicate the database asynchronously to the secondary region:

# On secondary region standby
primary_conninfo = 'host=

Use DNS-based failover with health checks (Route 53, or an EU-based alternative like ClouDNS with health monitoring) pointed at both regions:

failover_policy:
primary: eu-west (Amsterdam)
secondary: eu-central (Frankfurt)
health_check_interval: 10s
failover_threshold: 3 consecutive failures

Automate promotion of the standby database to primary with a scripted runbook, not a manual SSH session under pressure:

pg_ctl promote -D /var/lib/postgresql/data
# then update application connection strings via config management, not manual edits

Test the failover quarterly, not just after building it. This is the step most teams skip, and it is the only step that actually proves the architecture works.

Pattern 3: hybrid private-public

This pattern fits teams that need guaranteed data residency for regulated workloads (payment data, health records, government contracts) but want elastic burst capacity for traffic spikes, such as e-commerce flash sales.

Keep the system of record (database, PII storage, payment processing) on private infrastructure inside the EU, fully under your control.

Use public cloud capacity only for stateless, non-sensitive workloads: static asset delivery, image processing, or read-only caching layers.

Enforce the boundary at the network level, not just in application logic:

# Example firewall rule restricting outbound DB traffic to private subnet only
iptables -A OUTPUT -p tcp --dport 5432 -d 10.0.0.0/8 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 5432 -j DROP

Route burst traffic through a CDN configured to keep EU traffic within EU edge nodes. See understanding CDN data sovereignty for which providers actually guarantee this versus which just claim it.

Document the data flow diagram showing exactly which system touches regulated data. Auditors and enterprise procurement teams will ask for this before contract signature.

Verification: how to confirm it works

Each pattern needs proof, not just configuration. Here is what to check:

Data residency proof: run traceroute and packet capture on a sample transaction to confirm no hop leaves the EU. Cross-check with your cloud provider's published data center locations.

Failover time (multi-region pattern): measure actual time from primary failure detection to secondary serving traffic. Target under 60 seconds for DNS-based failover with a 10-second health check interval; anything above 5 minutes suggests your health check thresholds are too conservative.

Replication lag: monitor with SELECT now() - pg_last_xact_replay_timestamp(); on the standby. Sustained lag above 5 seconds under normal load indicates network or I/O bottlenecks that will cause data loss on an unplanned failover.

Boundary enforcement (hybrid pattern): attempt a connection from the public cloud segment to the private database port and confirm it is rejected at the firewall, not just blocked by application logic that could be bypassed.

Uptime under real failover: run a scheduled failover drill and measure actual customer-facing downtime, not just infrastructure switch time. A common gap is DNS caching adding 2-5 minutes beyond your technical failover time.

For teams running high-traffic workloads, it is worth reading why 99.9% uptime doesn't tell the full story to understand what these numbers actually mean for customer experience during a regional event.

Common pitfalls to avoid

Treating region toggles as sovereignty. Selecting "EU" in a cloud console does not guarantee subprocessors, support staff, or backup replication stay in the EU. Verify contractually and technically.

Skipping third-party service audits. Your database can be perfectly compliant while your error tracking tool ships stack traces (often containing user data) to a US-based endpoint.

Under-provisioning the passive region. A standby sized too small will not actually serve production load during failover, defeating the purpose of the pattern.

Never testing failover. An untested disaster recovery plan is a hypothesis, not a capability.

Ignoring data in transit between hybrid tiers. Encrypt every connection between private and public segments, and rotate credentials regularly.

Next steps and related reading

Once you have picked a pattern, the next decision is how to migrate existing workloads onto it without downtime. Our 6-phase zero downtime migration playbook covers the cutover mechanics in detail. If you are still weighing single-region against multi-region against hybrid at a strategic level, our companion piece on sovereign cloud architectures: single-region, multi-region, and hybrid patterns goes deeper into the decision criteria.

From here, plan a quarterly review cadence: revisit your data classification as new features ship, re-test failover after any significant infrastructure change, and re-audit third-party subprocessors annually as their own infrastructure evolves.

Build it once, run it correctly

Each of these patterns works in production, but they demand ongoing operational discipline: replication monitoring, failover drills, and subprocessor audits do not run themselves. That is the part most teams underestimate when they scope this internally.

Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.

---

### How zero-downtime database migrations work

URL: https://binadit.com/blog/zero-downtime-database-migration-infrastructure-performance-optimization
Category: Performance
Author: Binadit Tech Team
Published: 2026-07-31T09:28:36+02:00

What a zero-downtime migration actually means

A database migration changes the structure of your data: adding a column, renaming a table, changing a data type, splitting a monolithic table into two. The problem is not the change itself. The problem is that production databases serve live traffic while you make it.

Zero-downtime migration does not mean the migration runs instantly. It means the application keeps serving reads and writes correctly while the schema transitions from old to new. That distinction matters for infrastructure performance optimization work, because most migration failures are not caused by bad SQL. They are caused by lock contention, application code that assumes a schema state that no longer exists, or a cutover step that briefly takes the primary offline.

Engineers who understand this well treat a migration as a sequence of small, reversible steps rather than one large deployment. That mental model is the foundation for everything below.

How it actually works under the hood

Picture a production PostgreSQL or MySQL instance handling a few thousand queries per second through a connection pool. Your application servers hold a schema assumption baked into the ORM models and prepared statements. The database holds the actual schema. A migration is the process of moving both, in sync, without a moment where they disagree in a way that breaks a query.

Most schema changes fall into one of three categories, and each behaves differently under load:

Additive changes (new column, new table, new index): generally safe, but can still lock a table depending on the engine and whether a default value forces a full table rewrite.

Destructive changes (dropping a column, renaming a table): dangerous if any running application code still references the old structure.

Transformative changes (changing a column type, splitting a table, moving data to a new shape): require a temporary state where both old and new structures exist simultaneously.

The pattern that handles all three safely is known as expand-contract (sometimes called parallel change):

Expand: add the new schema element alongside the old one. New column, new table, new index. Nothing is removed yet.

Migrate: backfill data into the new structure and update application code to write to both old and new locations (dual-write) or read from the new one with a fallback.

Verify: confirm data consistency between old and new structures under real traffic, not just in staging.

Contract: once all application instances are confirmed to use the new structure, remove the old column, table, or index.

The reason this avoids downtime is simple: at every point in the sequence, both the currently deployed application code and the previous version can run against the schema without erroring. That matters because deployments are rolling, not instant. During a rolling deploy, you might have old and new application code running against the same database for 30-90 seconds. If your migration does not tolerate that overlap, you get sporadic 500 errors that look random but are entirely deterministic.

Underneath this, the database itself is doing locking work you need to understand. In PostgreSQL, adding a column with a constant default is metadata-only since version 11 and takes milliseconds regardless of table size. Adding a column with a non-constant default, or adding a NOT NULL constraint without a default, historically forced a full table rewrite that holds an ACCESS EXCLUSIVE lock, blocking all reads and writes for the duration. MySQL's InnoDB has similar distinctions: some ALTER TABLE operations use INSTANT or INPLACE algorithms that avoid locking, others fall back to COPY, which rebuilds the entire table.

Concrete examples with real numbers or configs

Here is a real scenario: a SaaS platform with a 40 million row orders table needs to rename status to order_status because a new fulfillment module conflicts with the old naming.

The naive approach (what breaks):

ALTER TABLE orders RENAME COLUMN status TO order_status;

This is fast in PostgreSQL, metadata-only. But every application server still running the previous deploy references status in raw queries or ORM mappings. The moment this statement commits, every one of those queries starts failing with "column does not exist." On a rolling deploy across 12 application pods, that is a guaranteed multi-minute error spike.

The expand-contract approach:

-- Step 1: expand
ALTER TABLE orders ADD COLUMN order_status varchar(20);

-- Step 2: backfill in batches to avoid long-running transactions
UPDATE orders SET order_status = status
WHERE id BETWEEN 1 AND 50000 AND order_status IS NULL;
-- repeat in batches of 50k until complete

-- Step 3: add a trigger or dual-write in application code
-- so new rows populate both columns during the transition

-- Step 4: deploy application code that reads order_status,
-- falls back to status if null, for one full release cycle

-- Step 5: after confirming zero fallback reads in logs for 7 days
ALTER TABLE orders DROP COLUMN status;

Batching the backfill matters at this scale. A single UPDATE across 40 million rows can hold row locks and generate WAL/binlog volume that causes replication lag, which then cascades into read-replica staleness for any service relying on eventual consistency. Batches of 10k-50k rows with short pauses between them keep replication lag under a second and avoid long-running transactions that block autovacuum in PostgreSQL.

For larger transformative changes, like splitting a users table into users and user_profiles for a SaaS platform with 200 million rows, the timeline looks different:
PhaseDurationRisk levelExpand: create new table, add foreign keysMinutesLowBackfill via batched job4-10 hoursLow, if throttledDual-write period in application1-2 weeksMedium, requires monitoringRead cutover, verify consistency3-5 daysMediumContract: drop old columnsMinutesLow, if verification passed

The total calendar time is measured in weeks, but the actual risk window, the moments where a mistake causes visible errors, is compressed into a handful of short, controlled deploy events. That is the trade being made: more elapsed time for a dramatically smaller blast radius.

Trade-offs and design decisions

Zero-downtime migration is not free. It costs engineering time, adds temporary complexity, and requires discipline that is easy to skip under deadline pressure. A few honest trade-offs:

Dual-write complexity: maintaining two write paths temporarily doubles the surface area for bugs. A missed dual-write path is the most common cause of silent data drift during migrations.

Longer total migration time: a change that could technically run in one blocking transaction over 10 minutes now takes days or weeks when done incrementally. For low-traffic internal tools, this overhead often is not worth it.

Read replica lag under batched backfills: even throttled backfills generate write volume. On a busy primary, this can measurably increase replication lag, which matters if your application routes any reads to replicas.

Rollback complexity: a single ALTER TABLE is trivial to roll back. A multi-phase migration with dual writes requires a rollback plan for each phase, not just the final state.

There is also a decision about tooling. Online schema change tools like gh-ost or pt-online-schema-change for MySQL work by creating a shadow copy of the table, replaying changes via triggers or binlog, and swapping the table atomically at the end. They automate much of the expand-contract pattern for a single ALTER, but they add operational overhead: monitoring the shadow table copy, watching for lag, and handling failures mid-copy on a multi-terabyte table.

For PostgreSQL, native support for concurrent index creation (CREATE INDEX CONCURRENTLY) and instant column additions have reduced the need for third-party tooling in many cases, but large table rewrites (changing a column type, for example) still often require the expand-contract pattern manually or via extensions like pg_repack.

When to use it, when not to

Zero-downtime migration patterns are worth the overhead when:

The table is large enough that a blocking ALTER would take longer than your acceptable maintenance window (a rough rule of thumb: anything over a few million rows on a busy table).

The service has an SLA that makes any planned downtime a contractual or reputational problem, common in high availability infrastructure for fintech or payment platforms.

Deploys are rolling and asynchronous, meaning old and new application code will run concurrently against the database regardless of what you do about the schema.

The change is destructive or transformative, not purely additive.

It is reasonable to skip the full pattern when:

The table is small (under a few hundred thousand rows) and the change is a simple additive one with a metadata-only cost.

You control a genuine maintenance window and the business impact of a 5-minute planned pause is acceptable and communicated.

The environment is staging or internal tooling where a short blip has no customer-facing consequence.

The judgment call is really about blast radius versus overhead. A 3-person internal tool team doing a full expand-contract cycle for a 50,000-row table is spending more engineering time on process than the risk justifies. A payments platform doing a blocking ALTER on a live orders table during business hours is taking on risk that has no upside.

This same blast-radius thinking applies broadly across zero-downtime migration work generally, not just schema changes. Infrastructure cutovers, DNS changes, and application deploys all benefit from the same incremental, reversible mindset.

Further reading and how we approach this

A few things worth understanding before you plan your next schema change: how your database engine's locking behavior differs between versions (PostgreSQL 11+ changed several ALTER TABLE cost profiles significantly), how your ORM handles schema drift between deploy versions, and whether your monitoring can actually detect a fallback read path silently failing.

If you want to go deeper on the operational side of migrations beyond schema changes, our guide on identifying database warning signals covers how to know when a migration is becoming urgent versus routine maintenance.

We design and run this kind of infrastructure for European businesses every day. Explore our managed cloud platform.

---

### Choosing between the open-source sovereign stack and managed public cloud

URL: https://binadit.com/blog/open-source-sovereign-stack-infrastructure-management-services-decision
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-07-29T09:20:14+02:00

The decision and who faces it

Every engineering leader running EU workloads eventually asks the same question: do we keep renting compute from a hyperscaler, or do we build our own stack on hardware we control? The open-source sovereign stack, Proxmox for virtualization, Ceph for storage, OpenStack for cloud orchestration, and Kubernetes for container scheduling, gives you a way to run something functionally close to AWS or Azure, but on infrastructure you own or lease from an EU-based data center.

This decision usually surfaces for three types of teams: SaaS companies that hit a scaling point where cloud bills stop making sense, regulated businesses (fintech, healthcare, government contractors) that need to prove data never leaves EU jurisdiction, and platform teams that got burned by a pricing change or an outage they couldn't influence. None of these are edge cases anymore. We've written before about how cloud becomes more expensive than bare metal at certain scale, and the sovereign stack is often the mechanism teams use to make that switch real.

This isn't a binary choice between "open source" and "cloud." It's a choice between two operating models: build and operate your own platform, or consume infrastructure management services from a partner (or hyperscaler) who operates it for you. Both are valid. The right answer depends on team size, growth trajectory, and how much of your engineering time you're willing to spend on plumbing instead of product.

Option A: the open-source sovereign stack, explained fairly

The stack breaks down into four layers, each solving a different problem:

Proxmox VE: virtualization layer. Runs KVM virtual machines and LXC containers with a web UI, clustering, and built-in backup. It's the layer most teams start with because it directly replaces VMware or bare-metal hypervisors.

Ceph: distributed storage. Provides block, object, and file storage across a cluster of disks with no single point of failure. This is what lets you lose a node and keep serving reads and writes.

OpenStack: cloud orchestration. Adds the API layer for compute, networking (Neutron), and identity (Keystone) that turns a pile of Proxmox/KVM hosts into something you can provision programmatically, the way you'd provision an EC2 instance.

Kubernetes: container orchestration, usually running on top of VMs provisioned by the layers below.

Together, these give you a private cloud with self-service provisioning, live migration, snapshotting, and horizontal scaling, without a hyperscaler contract. Real examples: a fintech we worked with cut cloud costs by 65% moving core services onto a Proxmox and Ceph cluster, keeping only edge and CDN functions on public cloud (we covered the numbers in how a fintech startup cut cloud costs 65% with an open-source sovereign stack). Government and defense-adjacent contractors use this stack specifically because OpenStack's API compatibility lets them build tooling that looks like AWS automation but runs on hardware physically located in an EU data center, with no CLOUD Act exposure.

Real strengths

No licensing lock-in. Proxmox has a subscription for support, not a per-core tax like VMware. Ceph and OpenStack are Apache-licensed.

Full data sovereignty. You control the physical location, the network path, and who has access. No US parent company, no subpoena exposure.

Cost predictability at scale. Once you're past a certain compute footprint (roughly 15-20+ VMs or equivalent, sustained), the economics of owned or leased hardware plus open-source software usually beat metered cloud pricing.

No API rate limits or surprise deprecations. You're not waiting on a vendor's roadmap.

Real limits

Operational burden is real and constant. Ceph in particular punishes teams that don't understand placement groups, replication factors, and failure domains. A misconfigured CRUSH map can turn a single disk failure into cluster-wide latency.

You own the upgrade path. OpenStack releases every six months. Skipping releases makes upgrades harder, not easier. This is not "set and forget" software.

Talent is scarce and expensive. Engineers who can debug a Ceph OSD flapping under load, or trace a Neutron networking issue through OVS, are a small pool. You either build this expertise in-house or hire a partner who already has it.

Time to production-ready is measured in months, not days. Standing up a resilient three-node Ceph cluster with proper monitoring, backup, and disaster recovery testing is a project, not a weekend task.

Option B: managed public cloud (or managed cloud infrastructure via a partner)

The alternative is consuming infrastructure as a service, either directly from a hyperscaler (AWS, Azure, GCP, or an EU-based alternative like Scaleway or OVHcloud), or through a managed infrastructure partner who operates the underlying platform, sovereign stack or otherwise, on your behalf.

Real strengths

Speed to provision. New compute, storage, and managed databases are available in minutes, not weeks.

Managed services reduce operational surface. Managed Kubernetes (EKS, AKS, GKE, or an EU equivalent), managed databases, and managed load balancers remove entire categories of failure modes from your team's plate.

Elastic scaling. Traffic spikes, seasonal e-commerce loads, and unpredictable SaaS growth are handled by autoscaling groups instead of manual capacity planning.

Global reach without building it yourself. If you genuinely need presence in multiple continents, hyperscalers already have the points of presence.

Real limits

Costs scale linearly, or worse, with usage. Egress fees, IOPS charges, and managed service premiums add up in ways that are hard to forecast. We've documented this in detail in cheap hosting vs managed cloud infrastructure: the real cost difference.

Data sovereignty is harder to guarantee than the region toggle suggests. A US-headquartered provider's EU region doesn't remove CLOUD Act exposure. This is worth understanding fully before committing, and we've covered why EU region toggles don't solve data sovereignty on their own.

Vendor roadmap risk. Pricing changes, feature deprecations, and API changes happen on the vendor's schedule, not yours.

Less architectural control. You're working within the abstractions the provider gives you. Deep customization of networking or storage behavior is limited or unavailable.

Direct comparison
DimensionOpen-source sovereign stackManaged public cloudUpfront costHigh: hardware, data center contracts, build-out timeLow: pay-as-you-go, no capexCost at scaleFlattens; marginal cost per VM drops significantlyScales with usage; egress and IOPS fees compoundOps burdenHigh: your team owns upgrades, failure recovery, capacity planningLow to moderate: provider or managed partner absorbs most of itTime to productionWeeks to months for a resilient clusterHours to days for basic provisioningScalability ceilingHigh, but requires proactive capacity planningVery high, elastic and near-instantData sovereigntyFull control over physical location and jurisdictionDepends on provider HQ and legal structure, not just region settingTalent requirementSpecialized (Ceph, OpenStack, KVM networking)General cloud/DevOps skills, more available in the marketBest team fitTeams with dedicated platform engineers, or a managed partner operating it for themSmall to mid-size teams without dedicated infra headcount

A decision framework

Use this to cut through the abstract debate and get to a concrete answer for your situation.

If you have fewer than 3 dedicated infrastructure engineers, running Ceph and OpenStack yourself is a risk you're probably underestimating. Either stay on managed public cloud, or bring in a partner who operates the sovereign stack for you as infrastructure management services, giving you the sovereignty benefit without the staffing requirement.

If your compute footprint is under roughly 10-15 VMs, the capex and operational overhead of a private Ceph/OpenStack cluster won't pay back quickly. Public cloud or a straightforward managed VPS setup is the more rational choice at this size.

If you're contractually or legally required to prove EU-only data residency (government tenders, DORA-regulated fintech, healthcare data under strict interpretations of GDPR), the sovereign stack, either self-run or via a managed infrastructure partner, is usually the only architecture that fully closes the CLOUD Act gap. Region toggles from US-headquartered providers do not satisfy this requirement on their own.

If your workload is bursty and unpredictable (flash sales, seasonal SaaS onboarding waves, marketing-driven traffic spikes), elastic public cloud, or a hybrid pattern with sovereign stack for baseline load and cloud for burst, will serve you better than a fixed-capacity private cluster.

If you're already spending 30%+ of cloud costs on egress, IOPS, or managed service premiums for steady-state workloads, model the sovereign stack seriously. This is the exact profile of teams who've cut costs 50-65% by moving core services onto owned infrastructure, as we detailed in our fintech cost optimization case study.

If you want the control of a sovereign stack without hiring a Ceph specialist, this is where a managed infrastructure partner earns its keep: you get OpenStack's API compatibility and EU data residency, but the 2am Ceph OSD alert goes to someone who's debugged it a hundred times before, not to your on-call engineer learning it live.

None of these paths is a permanent commitment. Plenty of teams run public cloud for years, then migrate core workloads to a sovereign stack once volume and compliance requirements justify it. The migration itself doesn't have to mean downtime, if you plan it as a phased cutover rather than a single event, an approach we walk through in our 6-phase zero downtime migration playbook.

Where this leaves you

The sovereign stack and managed public cloud aren't competing ideologies, they're tools that fit different operating constraints. Teams with the headcount and the compliance requirement get real value from owning Proxmox, Ceph, OpenStack, and Kubernetes end to end. Teams without that headcount get the same sovereignty and cost benefits by working with a partner who already operates that stack as a service.

Still weighing options for your stack? Book a 30-minute architecture call, no sales pitch.

---

### Best practices for moving off shared hosting as your traffic grows

URL: https://binadit.com/blog/best-practices-shared-hosting-cloud-cost-optimization-services-growth
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-07-28T09:28:33+02:00

Who this is for

This is for engineering teams and technical founders running a SaaS product, WooCommerce store, or content platform on shared hosting or a basic VPS, who are starting to see slow response times, resource limits, or unpredictable behavior during traffic spikes. You don't need to migrate today. You need a clear picture of what changes as traffic grows, and a practical plan for when and how to move.

Shared hosting is a reasonable starting point. It's cheap, simple, and fine for low-traffic sites. The problems start when concurrent users, database size, or background jobs cross a threshold the platform wasn't built for. This is normal growth, not a mistake. The practices below help you recognize the signals early and plan the transition on your own timeline, often as part of a broader cloud cost optimization effort rather than a reactive scramble.

12 practices for scaling past shared hosting without overspending

Monitor resource ceilings, not just uptime. Shared hosting plans cap CPU time, memory, and concurrent processes, often silently throttling instead of erroring. Track PHP-FPM pool usage, memory_limit hits, and process count over time so you see the ceiling approaching weeks before it becomes a support ticket.

Separate the database early. On shared hosting, your database usually competes for the same disk I/O and memory as your web processes. Moving MySQL or PostgreSQL to a dedicated instance, even a small one, removes one of the most common sources of unpredictable latency.

# Example: dedicated managed PostgreSQL connectionnDB_HOST=db-primary.internalnDB_PORT=5432nDB_POOL_MAX=25nDB_POOL_MIN=5

Introduce connection pooling before you need it. Shared hosting environments often hit max_connections limits under concurrent load, causing failed requests that look like application bugs. A pooler like PgBouncer or ProxySQL absorbs connection spikes without requiring a full database upgrade.

Add object caching as a first step, not a last resort. A Redis or Memcached layer in front of your database reduces read load dramatically, often postponing the need for a bigger database tier by months. This is one of the highest-leverage, lowest-cost changes you can make.

Cache-Control: no-cachenX-Cache-Backend: redisnredis-cli config set maxmemory-policy allkeys-lru

Right-size before you resize. The instinct when things slow down is to buy a bigger plan. Often the actual problem is unindexed queries, unbounded cron jobs, or synchronous requests to slow third-party APIs. Profile first; this is core to any serious cloud cost optimization services approach, because it avoids paying for capacity that masks an inefficiency instead of fixing it.

Move background jobs off the request path. Shared hosting often runs cron and queue workers on the same limited process pool as your web traffic. Moving jobs to a dedicated worker process with its own resource allocation prevents a traffic spike from starving your background tasks, or vice versa.

Plan the migration as zero downtime from day one. Treat the move off shared hosting like any production migration: parallel environments, data sync, and a cutover window rather than a full stop. This is the same discipline covered in our zero-downtime migration playbook, and it applies just as much to a first migration as a tenth.

Choose infrastructure that matches your actual traffic pattern. A store with predictable daily traffic needs different infrastructure than one with occasional campaign spikes. Autoscaling groups, load balancers, or a fixed high-availability cluster all solve different problems; picking the wrong one wastes budget either through overprovisioning or through missed capacity during peaks.

Separate static asset delivery from application servers. Once you leave shared hosting, don't just replicate the same monolithic setup on a bigger box. Put static assets and media on a CDN, and let your application servers handle only dynamic requests. This alone often cuts origin load by 40-60%.

Set concrete reliability targets before you migrate. Decide what uptime, response time, and error rate actually matter for your business, rather than defaulting to "as fast as possible." This gives you a target architecture instead of an open-ended upgrade cycle. Our guide on defining SLA, SLO, and SLI targets is a useful starting point for this exercise.

Test under realistic concurrent load, not just page load time. Shared hosting failures rarely show up in a single-user speed test. They show up when 200 people check out at once. Load test with tools like k6 or Locust against a staging environment that mirrors production capacity, not a scaled-down copy.

k6 run --vus 200 --duration 60s checkout_test.js

Budget for the infrastructure you'll need in 12 months, not just today. Moving off shared hosting is a good moment to model growth: expected traffic, data volume, and peak events like sales campaigns. This lets you size infrastructure correctly the first time, which is cheaper than migrating twice.

Rolling this out with an existing team

You don't need to do all twelve at once. Most teams get the best return by starting with monitoring and caching, since both are low-risk and immediately reduce load without touching architecture. That buys time and data to plan the bigger moves.

A practical sequence looks like this:

Weeks 1-2: Instrument resource usage (CPU, memory, connection counts, slow queries). You need data before you decide what to migrate.

Weeks 3-4: Add object caching and separate the database if it isn't already. These changes are reversible and low-risk.

Weeks 5-8: Design the target architecture: hosting model, worker separation, CDN, reliability targets. Get sign-off on the SLOs so the whole team is aligned on what "done" looks like.

Weeks 9-12+: Execute the migration in parallel with the existing environment, using a defined cutover window and rollback plan.

Assign one engineer as the migration owner, even if the work is distributed. Shared hosting migrations tend to stall when responsibility is diffuse and everyone assumes someone else is watching the resource graphs. A single owner keeps the checklist moving and makes go/no-go decisions on cutover day.

If your team is already stretched thin on feature work, this is also a good moment to loop in whoever owns your infrastructure budget. Every item on this list has a cost trade-off, and getting alignment early avoids the awkward conversation after the migration is already underway.

Where to go from here

Moving off shared hosting isn't a single decision, it's a sequence of small, testable changes that add up to infrastructure that scales predictably. Start with monitoring, fix the cheap wins like caching and connection pooling, then plan the bigger architectural move once you know your actual bottlenecks.

If implementing these yourself is not the best use of your engineering time, our managed services cover all of them by default.

---

### Understanding CDN data sovereignty: which providers keep EU traffic in EU

URL: https://binadit.com/blog/cdn-data-sovereignty-eu-traffic-infrastructure-management-services
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-15T11:49:37+02:00

> CDNs promise global performance, but many route EU traffic through US servers. Here's how to verify your CDN provider's actual data flows and choose infrastructure that respects European data sovereignty.

What CDN data sovereignty means and why engineers need to understand it

Content Delivery Networks accelerate websites by serving content from geographically distributed servers. When users in Amsterdam request your homepage, they get it from a nearby edge server instead of your origin server in Frankfurt. This reduces latency from 50ms to 15ms.

But here's what many engineers don't realize: your CDN provider might route that Amsterdam user's request through servers in Virginia before delivering content from the Dutch edge location. The content comes from Europe, but the routing decisions, authentication, and metadata processing happen in the US.

This matters because European data sovereignty regulations require that personal data stays within EU boundaries. GDPR fines start at 4% of annual revenue. More importantly, enterprise customers increasingly audit their suppliers' data handling practices before signing contracts.

CDN data sovereignty isn't just about where content gets cached. It's about understanding the complete data flow: where routing decisions happen, where logs get stored, where control plane traffic flows, and which legal jurisdiction governs your provider's operations.

How CDN routing actually works under the hood

When a user requests content through a CDN, several data flows happen simultaneously. Understanding these flows reveals where EU traffic might leave European jurisdiction.

First, DNS resolution determines which edge server handles the request. The user's browser queries your domain, gets a CNAME pointing to your CDN provider, then receives an IP address for the nearest edge server. This DNS decision-making process involves geolocation databases and routing algorithms that might run on US-based infrastructure.

Second, the edge server validates the request against your CDN configuration. This includes checking cache rules, security policies, and access controls. Some CDN providers store these configurations in centralized US databases, meaning every EU request triggers a transatlantic lookup.

Third, if content isn't cached, the edge server fetches it from your origin. This origin pull appears straightforward, but the edge server might send request metadata to central logging systems for analytics and monitoring. These logs often contain IP addresses, user agents, and request patterns that qualify as personal data under GDPR.

Fourth, the response gets cached according to your configuration rules. Cache invalidation commands, analytics data, and security event logs typically flow back to the CDN provider's central systems for processing and storage.

The critical insight is that content delivery involves two parallel data streams: the actual content flowing from edge to user, and the control and metadata flowing from edge to central systems. EU-based edge servers don't guarantee EU-based control systems.

Concrete examples: testing actual data flows

Here's how to verify where your CDN provider actually processes EU traffic. These tests reveal the difference between marketing claims and technical reality.

DNS geolocation test: Use dig or nslookup from multiple EU locations to query your CDN domain. Compare the returned IP addresses with geolocation databases. If EU queries return US IP addresses, your CDN routes traffic outside the EU.

dig +short example.com.cdn.provider.com @8.8.8.8
203.0.113.45 # Check this IP's location

dig +short example.com.cdn.provider.com @1.1.1.1
203.0.113.67 # Different result suggests US routing

We tested this with a client's e-commerce platform using four major CDN providers. Two providers consistently returned US IP addresses for EU DNS queries, despite having European edge servers.

Traceroute analysis: Run traceroute from EU locations to your CDN endpoints. Look for routing hops that pass through US ASNs (Autonomous System Numbers). This reveals the actual network path your traffic takes.

traceroute cdn.example.com
1 192.168.1.1 (2ms)
2 isp-gateway.nl (8ms)
3 eu-backbone.net (12ms)
4 us-peering.com (89ms) # Traffic left EU here
5 cdn-edge.example.com (94ms)

Log analysis: Check your CDN provider's real-time logs and analytics. Note the timestamps, IP addresses, and data points collected. If detailed request analytics appear instantly in your dashboard, the provider likely processes this data in real-time through centralized US systems.

One client discovered their CDN provider's "EU mode" only affected content caching, not log processing. Request logs containing customer IP addresses were still processed in AWS US-East-1 for analytics, violating their data processing agreements.

Trade-offs between performance and sovereignty

Achieving true EU data sovereignty with CDN services requires accepting certain performance and feature trade-offs. Understanding these trade-offs helps you make informed infrastructure decisions.

Latency vs. compliance: EU-only CDN providers typically have smaller edge networks than global providers. This means content might be served from Frankfurt instead of Amsterdam, adding 15-25ms of latency. For most websites, this difference is negligible compared to other optimization opportunities.

Analytics depth vs. privacy: Global CDN providers offer detailed real-time analytics because they process all data through centralized systems. EU-sovereign providers often provide simpler analytics to avoid cross-border data transfers. You might lose real-time visitor maps but retain essential performance metrics.

DDoS protection scope: Large CDN providers can absorb massive attacks using their global infrastructure. Smaller EU providers have more limited capacity but still handle typical attack volumes effectively. Most e-commerce sites face attacks in the 1-10 Gbps range, well within EU provider capabilities.

Feature completeness vs. simplicity: Global CDN providers offer dozens of edge computing features, many requiring US-based processing. EU providers focus on core CDN functionality: caching, compression, and basic security. This limitation often improves reliability by reducing complexity.

The key insight is that EU data sovereignty doesn't require sacrificing performance. It requires choosing providers whose architecture aligns with European data handling requirements from the ground up.

When to prioritize CDN sovereignty, when to accept global routing

CDN data sovereignty matters most for businesses with specific regulatory requirements or customer commitments. Here's how to decide what your infrastructure needs.

Choose EU-sovereign CDN providers when:

You handle personal data under GDPR and want to minimize legal complexity

Enterprise customers audit your data handling practices as part of procurement

You operate in regulated industries (finance, healthcare, government) with specific data residency requirements

Your data processing agreements explicitly require EU-only infrastructure

You want to avoid potential CLOUD Act data requests affecting your CDN logs

Global CDN routing is acceptable when:

Your website serves only public content without user tracking

You have strong data processing agreements that address cross-border data transfers

Performance requirements outweigh sovereignty concerns for your specific use case

You can implement adequate technical safeguards (encryption, anonymization) for CDN logs

Your legal team has validated your current setup against applicable regulations

Many businesses find a hybrid approach works well: EU-sovereign CDN for user-facing content and customer data, with global CDN for public assets like documentation and marketing sites.

The logistics company case study we covered previously illustrates how data sovereignty requirements can emerge suddenly through customer demands or regulatory changes.

For e-commerce platforms specifically, CDN sovereignty becomes critical during checkout flows where payment and personal data combine. Consider implementing specialized checkout infrastructure that maintains EU sovereignty while optimizing conversion rates.

Further reading and next steps

CDN data sovereignty is one component of broader infrastructure sovereignty requirements. The EU Digital Sovereignty initiative and upcoming regulations will likely expand these requirements beyond CDNs to other infrastructure components.

For deeper technical understanding, review your CDN provider's data processing addendum (DPA) and technical documentation. Look specifically for sections covering log retention, analytics processing, and cross-border data transfers.

The EDPB guidelines on international transfers provide the regulatory framework for evaluating CDN providers. Pay particular attention to the technical safeguards required when using providers subject to foreign government access laws.

Consider conducting a complete infrastructure sovereignty audit that covers not just CDN, but also DNS, monitoring, error tracking, and analytics services. Many businesses discover their CDN compliance is undermined by other services that process EU data in non-EU jurisdictions.

We design and run this kind of infrastructure for European businesses every day. Explore our managed cloud platform.

---

### What enterprise hosting services should actually deliver

URL: https://binadit.com/blog/hosting-services-enterprise-requirements-delivery
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-15T11:49:21+02:00

> Most hosting services focus on uptime percentages and storage limits. Enterprise teams need predictable performance under load, operational transparency, and infrastructure that scales with business requirements.

Why enterprise hosting requirements differ from consumer hosting

When a SaaS platform serves 50,000 concurrent users or an ecommerce store processes €2M in monthly transactions, hosting services become business-critical infrastructure. The difference between consumer hosting and enterprise hosting is not about bigger servers or higher prices. It is about predictable performance under varying loads, operational transparency, and infrastructure design that supports business continuity.

Enterprise applications generate different traffic patterns than personal websites. A B2B platform might see usage spike during business hours across multiple time zones. An ecommerce platform experiences seasonal traffic that can exceed baseline by 400%. These patterns require hosting services that can handle load variation without performance degradation.

The challenge is that many hosting providers market enterprise packages that are simply larger versions of shared hosting. Higher resource limits, dedicated IP addresses, and premium support do not address the fundamental architectural requirements of enterprise applications.

Performance predictability under variable load

Enterprise hosting services must deliver consistent performance regardless of traffic patterns. This means the infrastructure should handle both baseline operations and peak load scenarios without affecting response times or availability.

Most web hosting services use overselling models where physical resources are allocated across multiple customers. During peak usage periods, applications compete for CPU, memory, and I/O resources. This creates performance variance that enterprise applications cannot tolerate.

Predictable performance requires dedicated resource allocation. Virtual machines should have guaranteed CPU cores and memory allocation. Storage systems need consistent IOPS performance, not burst credits that deplete during sustained load.

Network performance matters equally. Enterprise applications often integrate with multiple external APIs, payment processors, and third-party services. Network latency and bandwidth limitations affect these integrations, which directly impacts user experience.

Resource isolation and noisy neighbor prevention

Enterprise hosting services should provide complete resource isolation. This means CPU, memory, disk I/O, and network resources are dedicated to your application stack. Shared hosting environments cannot guarantee this isolation because other tenants can consume available resources.

Even VPS hosting can suffer from noisy neighbor effects when the underlying physical hardware is oversubscribed. Enterprise hosting requires either dedicated hardware or virtualization platforms that enforce strict resource limits.

Operational transparency and monitoring capabilities

Enterprise teams need visibility into infrastructure performance, resource utilization, and potential bottlenecks. Many hosting providers offer basic monitoring dashboards that show CPU and memory usage, but enterprise applications require more detailed metrics.

Database performance monitoring should include query execution times, connection pool utilization, and lock contention metrics. Application servers need request queuing data, garbage collection statistics, and thread pool utilization. Load balancers should provide per-backend health status and request distribution data.

This monitoring data enables proactive capacity planning and performance optimization. Enterprise teams can identify resource constraints before they affect user experience and plan infrastructure scaling based on actual usage patterns.

Real-time alerting and incident response

Enterprise hosting services should provide configurable alerting based on application-specific metrics. Generic alerts for server downtime are insufficient. Teams need alerts for response time degradation, error rate increases, and resource utilization thresholds that indicate approaching capacity limits.

The alerting system should integrate with existing incident management workflows. This might include webhook notifications to PagerDuty, Slack integration for team communication, or API access for custom automation.

Response times for critical incidents should be measured in minutes, not hours. Enterprise hosting providers should offer direct engineer access, not ticket-based support systems that introduce communication delays during outages.

Scaling infrastructure with business growth

Enterprise applications need infrastructure that can scale both vertically and horizontally. Vertical scaling means adding more CPU, memory, or storage to existing servers. Horizontal scaling means adding more servers to distribute load across multiple instances.

Many hosting services support vertical scaling but make horizontal scaling difficult. Adding new application servers might require manual load balancer configuration. Database scaling might need complex replication setup. Cache layers might not automatically distribute across new nodes.

Enterprise hosting should provide infrastructure automation that simplifies scaling operations. This might include auto-scaling groups that add instances based on load metrics, managed database clusters that handle replication automatically, or container orchestration platforms that distribute workloads across available resources.

Scaling web applications properly requires understanding both application architecture and infrastructure capabilities. The hosting environment should support the scaling patterns your application architecture requires.

Database and storage scalability

Database performance often becomes the limiting factor for enterprise applications. Read-heavy workloads might benefit from read replicas that distribute query load. Write-heavy applications might need sharded databases that distribute data across multiple instances.

Storage systems should support both capacity scaling and performance scaling. Adding storage space should not require downtime. Increasing IOPS performance should not require data migration.

Enterprise hosting services should provide managed database options that handle scaling operations automatically. This includes automated failover, backup management, and performance tuning based on workload patterns.

Security and compliance requirements

Enterprise applications often handle sensitive customer data, financial transactions, or regulated information. The hosting infrastructure must support compliance requirements while maintaining operational efficiency.

Security starts with infrastructure hardening. Servers should have minimal software installations, regular security updates, and network segmentation that limits attack surfaces. Access controls should enforce least privilege principles and provide audit logs for compliance reporting.

For European businesses, GDPR compliance affects hosting decisions. Data location, processor agreements, and access controls all impact compliance status. Building GDPR-compliant infrastructure requires hosting services that understand European data protection requirements.

Many global hosting providers offer EU regions but still operate under US legal frameworks. This creates potential conflicts between local data protection laws and foreign government access requests.

Backup and disaster recovery

Enterprise hosting services must provide comprehensive backup and disaster recovery capabilities. Daily backups are insufficient for applications that process continuous transactions. Point-in-time recovery should be available for databases and file systems.

Disaster recovery planning should include both infrastructure recovery and data recovery scenarios. Geographic distribution of backup data protects against regional outages or natural disasters.

Recovery time objectives (RTO) and recovery point objectives (RPO) should align with business requirements. Critical applications might need sub-hour recovery times and minimal data loss windows.

Network architecture and content delivery

Enterprise applications serve users across multiple geographic regions. Network architecture affects both performance and availability for distributed user bases.

CDN integration should be seamless and configurable. Static assets, API responses, and dynamic content might have different caching requirements. The hosting environment should support CDN configuration that optimizes performance without adding operational complexity.

Load balancing should support multiple algorithms and health checking mechanisms. Geographic load balancing might route users to the nearest application instance. Application-aware load balancing might route requests based on URL patterns or user attributes.

Network security should include DDoS protection, WAF capabilities, and SSL/TLS termination. These features should integrate with monitoring systems to provide visibility into attack patterns and traffic anomalies.

Cost predictability and resource optimization

Enterprise hosting costs should be predictable and aligned with business value. Many hosting providers use complex pricing models that make cost forecasting difficult. Surprise charges for bandwidth overages, storage increases, or support incidents create budget uncertainty.

Resource-based pricing should reflect actual usage patterns. Applications with predictable traffic patterns should not pay for burst capacity they never use. Applications with variable load should not face capacity constraints during growth periods.

Cost optimization should balance performance requirements with budget constraints. This might include automated scaling that reduces capacity during low-usage periods, storage tiering that moves older data to cheaper storage classes, or traffic optimization that reduces bandwidth costs.

Transparent pricing and cost allocation

Enterprise teams need detailed cost breakdowns that support internal budget allocation. Infrastructure costs should be attributable to specific applications, departments, or customers.

Hosting services should provide cost forecasting based on usage trends and planned capacity changes. This enables better budget planning and helps teams understand the cost implications of architectural decisions.

When managed infrastructure makes sense for enterprises

Large enterprises often have internal infrastructure teams that can manage hosting environments directly. However, managed infrastructure can provide value even for organizations with strong technical capabilities.

Managed services reduce operational overhead for infrastructure management tasks. This includes security patching, monitoring setup, backup management, and capacity planning. Internal teams can focus on application development and business logic instead of infrastructure maintenance.

Managed infrastructure providers should have deeper expertise in specific technology stacks and scaling patterns. They see infrastructure challenges across multiple customers and can apply lessons learned to optimize performance and prevent common issues.

For European businesses, managed infrastructure can simplify compliance requirements. Providers that specialize in European markets understand local regulations and can provide compliant infrastructure configurations without extensive legal review.

Choosing between managed infrastructure and traditional hosting depends on team capabilities, compliance requirements, and growth trajectory.

Service level agreements that matter

Enterprise hosting services should provide SLAs that align with business requirements. Uptime percentages are important, but response time SLAs often matter more for user experience.

SLAs should cover the entire application stack, not just individual components. Database availability, load balancer performance, and CDN response times all affect overall application performance.

Penalty structures should provide meaningful compensation for SLA violations. Credits that represent a fraction of monthly costs do not compensate for lost revenue during outages.

SLA measurement should be transparent and verifiable. Customers should have access to the same monitoring data used for SLA calculations.

Support quality and escalation procedures

Enterprise support should provide direct access to engineers who understand the technology stack. First-level support that can only restart services or check basic connectivity creates delays during critical incidents.

Support escalation should be automatic for certain incident types. Database performance issues, application outages, and security incidents should immediately involve senior engineers without requiring manual escalation.

Documentation and knowledge sharing should be part of the support relationship. Enterprise teams benefit from understanding infrastructure configurations, optimization opportunities, and best practices for their specific use cases.

Evaluation criteria for enterprise hosting services

When evaluating hosting services for enterprise applications, focus on operational capabilities rather than feature lists. Performance under load, incident response times, and scaling flexibility matter more than storage quotas or bandwidth limits.

Request performance data from similar workloads. Synthetic benchmarks do not reflect real application behavior under production traffic patterns. Ask for case studies that demonstrate scaling capabilities and incident resolution procedures.

Test the support organization during the evaluation process. Response times, technical depth, and escalation procedures during sales conversations often predict support quality during production incidents.

Understand the provider's infrastructure architecture and operational procedures. Shared infrastructure, even with dedicated resources, creates different risk profiles than truly isolated environments.

For businesses that cannot afford infrastructure failures, hosting services become strategic partnerships rather than commodity purchases. The right infrastructure partner understands your business requirements and can adapt their services to support your growth trajectory.

We design and manage enterprise infrastructure for European businesses that require predictable performance, regulatory compliance, and operational transparency. Schedule a technical discussion about your hosting requirements.

---

### Cheap hosting vs managed cloud infrastructure: the real cost difference

URL: https://binadit.com/blog/cheap-hosting-vs-managed-cloud-infrastructure-real-cost-difference
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-14T09:30:53+02:00

> Budget hosting looks attractive until you calculate downtime costs, scaling bottlenecks, and engineering hours. Here's how to evaluate the true financial impact of both approaches.

The decision every growing business faces

When your application starts handling real traffic, you face a critical infrastructure decision: stick with budget hosting or invest in managed cloud infrastructure. The price difference is obvious - €5/month versus €500/month. The hidden costs aren't.

This decision typically hits companies at predictable moments: after a traffic spike causes downtime, when manual server management consumes engineering time, or when scaling requires rebuilding everything. Engineering leaders at SaaS platforms, agencies, and e-commerce stores wrestle with this choice monthly.

The real question isn't about monthly hosting fees. It's about total cost of ownership, including downtime, engineering overhead, and missed opportunities. Let's examine both approaches fairly.

Budget hosting: strengths and real limitations

Budget hosting serves specific use cases well. Shared hosting, basic VPS, and entry-level cloud instances work for development environments, simple websites, and early-stage applications with predictable traffic.

The primary strength is obvious: low upfront costs. A €10/month VPS can handle moderate traffic without issues. For many businesses, this approach funds early growth without significant infrastructure investment.

Budget hosting also offers simplicity. One server, basic configuration, minimal moving parts. Small teams can manage these setups without specialized infrastructure knowledge.

However, real limitations emerge under business pressure:

Scaling hits hard limits. When traffic increases, budget hosts offer limited options: upgrade to the next tier or add another server. This creates performance cliffs where slight traffic increases cause significant slowdowns.

A WooCommerce store we analyzed experienced this directly. Their €15/month hosting handled 200 concurrent users fine. At 250 users, response times jumped from 800ms to 4 seconds. The next hosting tier cost €80/month but couldn't guarantee performance under higher load.

Support becomes a bottleneck. Budget providers operate on volume. When issues arise - database locks, memory exhaustion, network problems - resolution follows ticket queues, not urgency. A payment processing issue waits behind configuration questions and billing disputes.

Monitoring stays reactive. Budget hosting typically includes basic uptime monitoring. You learn about problems when customers complain, not when metrics show degradation patterns. This delays response and amplifies impact.

Engineering overhead grows unexpectedly. Managing servers, applying security patches, optimizing databases, and handling backups consumes increasing time. What starts as occasional maintenance becomes regular operational work.

Budget hosting works until business requirements exceed its operational model. The transition point varies, but the pattern stays consistent: rapid growth exposes infrastructure limitations faster than teams can adapt.

Managed cloud infrastructure: comprehensive approach with trade-offs

Managed cloud infrastructure handles the complete operational layer: server management, monitoring, scaling, security, and optimization. Teams focus on application development while infrastructure partners manage the underlying systems.

The approach suits businesses where infrastructure stability directly impacts revenue. SaaS platforms serving enterprise customers, high-traffic e-commerce sites, and agencies managing client applications benefit from this operational model.

Scaling happens proactively. Instead of waiting for traffic spikes to cause problems, managed infrastructure monitors performance patterns and adjusts resources before bottlenecks form. Load balancers, auto-scaling, and resource optimization work automatically.

A logistics platform we work with demonstrates this advantage. During seasonal peaks, their managed cloud infrastructure scales from 4 to 12 application servers based on queue depth and response time metrics. The process happens without manual intervention or performance degradation.

Expert support operates as extension of your team. Infrastructure questions get answered by engineers who understand your specific setup. When problems arise, response comes from people familiar with your architecture, not generic support staff reading documentation.

Monitoring provides early warning systems. Instead of basic uptime checks, comprehensive monitoring tracks database query times, memory usage patterns, disk I/O trends, and application-specific metrics. Problems get identified and resolved before they affect users.

Engineering time stays focused on core business. Server management, security patching, database optimization, and backup validation happen without consuming development resources. Teams can maintain velocity on feature development instead of operational firefighting.

However, managed cloud infrastructure comes with clear trade-offs:

Higher monthly costs. Managed infrastructure typically costs 3-10x more than budget hosting, depending on complexity and service level. This investment makes sense for revenue-generating applications but can strain early-stage budgets.

Potential vendor dependency. Relying on infrastructure partners means your operational knowledge might not keep pace with your infrastructure complexity. Teams can become less familiar with underlying systems.

Reduced direct control. Configuration changes and optimization decisions go through your infrastructure partner instead of happening immediately. Good partners minimize this friction, but some delay is inherent to the collaborative model.

Direct comparison: cost structure and operational impact
FactorBudget hostingManaged cloud infrastructureMonthly cost€10-100/month€300-2000/monthEngineering overhead5-20 hours/month1-5 hours/monthScaling approachManual, reactiveAutomatic, proactiveDowntime recoveryHours to daysMinutes to hoursSupport response24-72 hours15 minutes to 4 hoursMonitoring depthBasic uptimeComprehensive metricsTeam knowledge requiredFull stack + operationsApplication focusRevenue riskHigh during incidentsMinimized by redundancy

The comparison reveals different cost structures. Budget hosting has lower direct costs but higher operational overhead. Managed infrastructure inverts this: higher direct costs with lower operational burden.

For a SaaS platform generating €50k monthly recurring revenue, one hour of downtime costs roughly €2,000 in lost subscriptions and customer trust. Budget hosting might experience 3-4 hours annual downtime from scaling issues and incident response delays. Managed infrastructure typically reduces this to 30-60 minutes through redundancy and faster resolution.

Engineering time calculation matters significantly. If your team spends 15 hours monthly on server management, backups, and optimization at a €75/hour rate, that's €1,125 in opportunity cost. Managed infrastructure might cost €800/month but frees those 15 hours for revenue-generating development work.

Decision framework: when to choose each approach

Choose budget hosting when:

Monthly revenue under €10k and infrastructure costs need to stay below 3% of revenue

Traffic patterns are predictable with gradual growth and no sudden spikes

Team includes infrastructure expertise and enjoys operational challenges

Downtime costs are manageable because the application doesn't directly generate revenue

Scaling timeline extends beyond 12 months and current capacity meets projected needs

Choose managed cloud infrastructure when:

Monthly revenue exceeds €25k and infrastructure stability directly impacts income

Traffic includes unpredictable spikes from marketing campaigns, seasonal peaks, or viral growth

Team wants to focus on core product instead of operational infrastructure

Downtime costs exceed infrastructure investment due to lost sales, SLA penalties, or reputation damage

Compliance requirements demand specific security, monitoring, or data sovereignty measures

The transition point isn't always revenue-based. A B2B SaaS serving enterprise customers might need managed infrastructure at €15k MRR because client contracts include strict uptime requirements. An e-commerce store might operate on budget hosting until €40k monthly revenue because seasonal traffic patterns are predictable.

Consider hybrid approaches for specific situations. Development and staging environments can use budget hosting while production runs on managed infrastructure. This reduces costs while maintaining reliability where it matters most.

The decision often becomes obvious through painful experience: budget hosting fails during a critical moment, or managed infrastructure costs feel excessive for current needs. The key is making the transition proactively instead of reactively.

Many successful companies start with budget hosting and transition to managed infrastructure as they scale. The timing depends on when operational overhead starts limiting growth more than infrastructure costs.

Making the transition at the right time

The choice between budget hosting and managed cloud infrastructure reflects your current business stage and operational priorities. Both approaches serve legitimate use cases when matched to appropriate situations.

Budget hosting enables early growth without significant infrastructure investment. Managed infrastructure supports scaling without operational bottlenecks. The key is recognizing when your business requirements have outgrown your current approach.

Most growing businesses eventually need the reliability and operational efficiency that comes with professional infrastructure management. The question becomes timing this transition to maximize value while minimizing disruption.

If you're evaluating infrastructure options and want to understand how the numbers work for your specific situation, we can help you calculate the real costs and benefits. Our approach focuses on finding the right solution for your current needs while planning for future growth.

Still weighing options for your stack? Book a 30-minute architecture call, no sales pitch.

---

### How a €50M logistics company avoided US data access with private cloud infrastructure

URL: https://binadit.com/blog/logistics-company-avoided-us-data-access-private-cloud-infrastructure-sovereignty
Category: Security
Author: Binadit Tech Team
Published: 2026-06-13T10:08:44+02:00

> A European logistics provider processing sensitive shipping data discovered their cloud provider had US parent companies with potential CLOUD Act exposure. This is how they migrated to sovereign infrastructure without disrupting operations.

The situation: a growing logistics platform facing regulatory scrutiny

A Rotterdam-based logistics company had built their entire operation around a cloud-first approach. Processing over 200,000 shipments monthly across 27 EU countries, they handled everything from customs declarations to real-time cargo tracking through their platform.

The technical setup was solid. Load-balanced application servers, managed databases, CDN for global performance. Their infrastructure hummed along at 99.95% uptime, handling traffic spikes during peak shipping seasons without breaking.

Then came the compliance audit.

Their enterprise clients - major automotive and pharmaceutical companies - started asking detailed questions about data residency. Where exactly were customer records stored? Which jurisdictions could potentially access shipping manifests and logistics data? The questions became more specific after several high-profile cases where US authorities had requested data from European subsidiaries of American cloud providers.

The logistics platform realized they had a problem. While their servers ran in EU regions, their cloud provider's parent company was US-based. This created potential exposure under the CLOUD Act, which allows US authorities to request data from US companies regardless of where that data is physically stored.

For a company handling sensitive shipping data, customs information, and proprietary logistics algorithms, this wasn't just a compliance checkbox. It was becoming a competitive disadvantage. Three major clients had started requesting contractual guarantees about data sovereignty that the current setup couldn't provide.

What we found during the infrastructure audit

When we analyzed their existing setup, the architecture itself was well-designed. The problem was jurisdictional, not technical.

Their current stack ran on a major US cloud provider's European regions:

Application layer: 6 load-balanced containers running their logistics platform

Database: Managed PostgreSQL cluster with read replicas across three EU zones

Storage: 2.3TB of shipping documents, customs forms, and tracking data

Caching: Redis cluster handling session data and frequently-accessed shipment information

Monitoring: Full observability stack tracking performance and business metrics

Performance numbers looked good. Average API response time of 180ms, 99th percentile under 800ms. Database queries averaged 45ms with the heaviest reporting queries staying under 2 seconds.

But the legal exposure was clear. Despite geographic data residency, the US parent company could theoretically be compelled to provide access to EU customer data. For clients in regulated industries, this was unacceptable.

We also discovered they were overpaying significantly. Their monthly cloud bill had grown to €18,000 for infrastructure that could run more efficiently on properly configured private cloud infrastructure.

The hidden costs went beyond the monthly bill. They were paying for managed services they barely used, redundant backup systems, and premium support tiers that mostly handled issues they could resolve internally.

The approach we took and why

Moving to EU-sovereign private cloud infrastructure wasn't just about changing providers. It required rebuilding their entire stack while maintaining operational continuity.

We designed a migration that would address three critical requirements:

Complete data sovereignty: Every component of the infrastructure needed to be owned and operated by EU entities, with no US parent companies in the chain. This meant not just servers, but also monitoring tools, backup systems, and management interfaces.

Zero business disruption: During peak shipping season, even brief outages could cost thousands in delayed shipments. The migration had to happen without affecting their 24/7 operations.

Improved performance and cost efficiency: The new infrastructure needed to perform better than the current setup while reducing monthly operational costs.

Our approach used a parallel infrastructure strategy. Rather than migrating piece by piece, we built a complete mirror environment and then orchestrated a coordinated switchover.

This method offers several advantages over incremental migrations. First, it allows thorough testing of the entire system under realistic conditions before any production traffic moves. Second, it provides an immediate rollback path if anything goes wrong. Third, it minimizes the complexity of managing partially-migrated state.

Implementation details with specifics

We built the new sovereign infrastructure using a multi-zone setup across Amsterdam and Frankfurt datacenters, both operated by EU-owned entities with no US corporate relationships.

Application layer redesign:

The new setup used dedicated servers rather than shared cloud instances. Six application servers running Docker containers, with nginx load balancing configured for session affinity:

upstream logistics_app {
server 10.1.1.10:8080 max_fails=3 fail_timeout=30s;
server 10.1.1.11:8080 max_fails=3 fail_timeout=30s;
server 10.1.1.12:8080 max_fails=3 fail_timeout=30s;
server 10.1.2.10:8080 max_fails=3 fail_timeout=30s backup;
server 10.1.2.11:8080 max_fails=3 fail_timeout=30s backup;
server 10.1.2.12:8080 max_fails=3 fail_timeout=30s backup;
}

Database architecture:

We migrated from managed PostgreSQL to a self-managed cluster with streaming replication. The primary database ran in Amsterdam with synchronous replication to Frankfurt for disaster recovery:

recovery_conf settings:
standby_mode = 'on'
primary_conninfo = 'host=10.1.1.20 port=5432 user=replication'
trigger_file = '/tmp/postgresql.trigger'

This configuration provided better performance than the managed service because we could optimize specifically for their logistics workload patterns. Shipping queries typically involve time-based lookups and geospatial calculations, so we tuned the configuration accordingly.

Data migration strategy:

Moving 2.3TB of operational data required careful coordination. We used PostgreSQL's logical replication to keep the new database in sync during the transition period:

CREATE PUBLICATION logistics_migration FOR ALL TABLES;
CREATE SUBSCRIPTION logistics_sync CONNECTION 'host=old_db port=5432' PUBLICATION logistics_migration;

This allowed us to maintain data consistency while gradually shifting read traffic to test the new infrastructure under real conditions.

Monitoring and observability:

We replaced their cloud provider's monitoring tools with a fully sovereign stack using Prometheus and Grafana, both running on EU infrastructure. The monitoring system tracked the same business metrics they relied on: shipment processing rates, API response times, and database performance.

Results with real numbers

The migration to sovereign private cloud infrastructure delivered measurable improvements across performance, cost, and compliance dimensions.

Performance improvements:

Average API response time dropped from 180ms to 120ms. The 99th percentile improved from 800ms to 520ms. Database query performance improved significantly, with their heaviest reporting queries dropping from 2 seconds to 1.2 seconds average execution time.

These improvements came from eliminating the overhead of managed services and optimizing configurations specifically for logistics workloads.

Cost reduction:

Monthly infrastructure costs decreased from €18,000 to €11,200, a 38% reduction. The savings came primarily from eliminating premium managed service fees and rightsizing resources for actual usage patterns.

More importantly, the predictable pricing model made capacity planning straightforward. No more surprise bills from traffic spikes or storage overages.

Compliance and business impact:

Within six weeks of the migration, they secured two new enterprise contracts worth €2.1M annually. Both clients specifically cited data sovereignty guarantees as a deciding factor in their vendor selection.

The compliance documentation became a competitive advantage. They could provide detailed technical and legal assurances about data residency that competitors using US cloud providers couldn't match.

Operational improvements:

System reliability actually improved during the migration. The new infrastructure achieved 99.98% uptime in the first six months, compared to 99.95% on the previous cloud setup.

Response time consistency improved dramatically. While average performance was better, the reduction in performance variability was even more significant for their operations team.

What we'd do differently next time

The migration succeeded, but several aspects could have been smoother with different approaches.

Database migration timing:

We scheduled the final database switchover during their lowest-traffic period, which turned out to be more compressed than expected. Next time, we'd build in more buffer time and consider a gradual traffic shift rather than a single cutover event.

Client communication:

While we maintained system availability throughout the migration, we could have communicated the timing and expected benefits to their enterprise clients more proactively. Several clients noticed performance improvements but weren't aware they were connected to the infrastructure upgrade.

Monitoring migration:

We migrated monitoring systems alongside the infrastructure, which created a brief gap in historical data continuity. A better approach would be maintaining parallel monitoring during the transition to preserve trending data.

Load testing scope:

Our load testing focused on normal operational patterns, but we should have included more edge cases around their peak shipping season traffic patterns. While the system handled actual peak loads well, more comprehensive testing would have provided additional confidence.

Close + CTA

This logistics company's migration to sovereign private cloud infrastructure solved their immediate compliance challenges while improving performance and reducing costs. The combination of EU data residency guarantees and better technical performance became a competitive advantage that directly contributed to new business wins.

For companies handling sensitive data in regulated industries, infrastructure jurisdiction matters as much as technical capabilities. The CLOUD Act and similar regulations create real business risks that can't be solved by simply choosing EU regions within US-owned cloud platforms.

Private cloud infrastructure offers a path to genuine data sovereignty while often delivering better performance and cost efficiency than managed cloud services. The key is executing the migration without disrupting business operations.

Facing a similar challenge? Tell us about your setup and we will outline an approach.

---

### Benchmarking non-US payment infrastructure: a DORA compliance case study with cloud cost optimization services

URL: https://binadit.com/blog/benchmarking-non-us-payment-infrastructure-dora-compliance-cloud-cost-optimization-services
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-12T12:10:20+02:00

> We measured the real performance and cost impact of building EU-based payment infrastructure for a fintech under DORA regulations. The numbers show significant differences in latency, throughput, and operational costs compared to US-based alternatives.

The question and why it matters commercially

A European fintech with €50M in annual payment volume faced a critical decision: rebuild their payment infrastructure outside US jurisdiction to comply with DORA (Digital Operational Resilience Act) requirements, or accept the regulatory and operational risks of their current US-based setup.

DORA mandates that EU financial entities maintain operational resilience without critical dependencies on third-country providers. For payment infrastructure, this means examining every component from cloud providers to monitoring services.

The commercial stakes were clear. Non-compliance could result in regulatory penalties up to 10% of annual turnover. But rebuilding infrastructure carries its own costs and risks. We measured both the performance and financial impact of migrating from a US-based stack to EU-sovereign alternatives.

This analysis examines real numbers from a 6-month migration project, comparing performance metrics and operational costs between US and EU-based payment infrastructure components. The data shows what fintech teams can expect when building DORA-compliant systems with cloud cost optimization services designed for European regulatory requirements.

Methodology: setup, hardware, software versions, load profile

We measured performance across three infrastructure configurations during the migration period:

Baseline (US-based): AWS us-east-1, Stripe payments, Datadog monitoring

Hybrid: EU compute with US payment processing and monitoring

Target (EU-sovereign): OVH/Hetzner hosting, Adyen payments, self-hosted monitoring

The payment platform processed card transactions, SEPA transfers, and real-time payments. Peak load reached 1,200 transactions per minute during campaign periods.

Hardware specifications:

US baseline: 6x AWS c5.2xlarge instances (8 vCPU, 16GB RAM each)

EU target: 8x Hetzner CCX33 instances (8 vCPU, 32GB RAM each)

Database: PostgreSQL 14.9 with read replicas

Cache: Redis 7.0 in cluster mode

Load profile:

Average: 200 transactions/minute

Peak: 1,200 transactions/minute

Geographic distribution: 70% EU, 25% UK, 5% other

Transaction types: 60% card payments, 30% SEPA, 10% instant payments

We collected metrics using both commercial tools (during US-based operation) and open-source alternatives (during EU migration). Payment processing latency was measured from API request to payment confirmation. Infrastructure costs included compute, storage, networking, and third-party services.

Results: tables and prose with p50/p95/p99 or throughput numbers

Payment processing latency (milliseconds):
ConfigurationP50P95P99Max observedUS baseline1804208502100Hybrid24058012003400EU target1603807201800

The EU-sovereign configuration delivered the fastest response times. Cross-border data flows in the hybrid setup created the highest latency, particularly during peak periods.

Throughput capacity (transactions per minute):
ConfigurationSustained peakBurst capacityFailure thresholdUS baseline80011001350Hybrid6008501000EU target95013001500

The EU infrastructure handled higher transaction volumes before performance degraded. This resulted from eliminating cross-border network hops and optimizing for European payment networks.

Monthly operational costs (EUR):
ComponentUS baselineEU targetDifferenceCompute2,4001,800-25%Payment processing4,2003,900-7%Monitoring/observability800200-75%Data storage600400-33%Network/CDN300250-17%Total8,3006,550-21%

The EU-based setup reduced costs across all categories. The largest savings came from replacing commercial monitoring tools with open-source alternatives that still provided comprehensive visibility.

Analysis: what the numbers mean in production

The performance improvements in the EU configuration weren't just statistical noise. During Black Friday traffic (2,800 transactions in a 15-minute peak), the US-based system dropped 3% of payment requests due to timeout failures. The EU system processed the same load without dropped transactions.

Payment latency directly impacts conversion rates. For this fintech, reducing P95 latency from 420ms to 380ms correlated with a 0.8% increase in successful payment completions. At €50M annual volume, this represented €400k in additional processed payments.

The hybrid configuration performed worst because it combined disadvantages of both approaches. EU-based application servers had to communicate with US payment processors across high-latency connections, while losing the cost benefits of either pure approach.

Cost optimization came primarily from three areas:

Compute efficiency: EU providers offered better price/performance for this workload

Payment processing: Adyen's EU rates were marginally lower than Stripe for European transactions

Monitoring consolidation: Self-hosted Prometheus/Grafana replaced multiple commercial tools

The 21% cost reduction became more significant under load. During peak periods, the US configuration required additional auto-scaling capacity that wasn't needed with the more efficient EU setup.

Compliance benefits weren't directly measurable but became clear during the DORA assessment process. The EU-sovereign architecture eliminated 12 potential compliance gaps related to third-country dependencies.

Caveats and what you'd do differently

These measurements have important limitations. We tested one specific payment workload with particular geographic distribution. A fintech serving global markets might see different results.

The migration took 6 months, during which the team operated dual systems. This created additional operational overhead not reflected in the final cost comparisons. A greenfield EU-first approach would likely show even better economics.

We measured steady-state performance after optimization. Initial EU deployment performance was actually worse until the team tuned database configurations and caching strategies for the new environment.

Payment processor comparison isn't perfectly fair. Stripe and Adyen offer different feature sets and integration complexity. We optimized the Adyen integration based on lessons learned from the Stripe implementation.

If repeating this project, we would:

Plan for 3 months of performance tuning after migration

Implement comprehensive load testing earlier in the process

Negotiate better rates based on projected volume growth

Build monitoring infrastructure before starting the migration

The monitoring cost savings look dramatic, but required significant engineering time to achieve equivalent functionality. Organizations without strong DevOps capabilities might not realize these benefits.

Exchange rate fluctuations affect cost comparisons. We used fixed EUR rates, but currency movements could impact the economics over time.

Takeaways and implementation guidance

EU-based payment infrastructure can deliver both better performance and lower costs for European fintechs, but success requires careful planning and execution. The key findings:

Performance improves with regional optimization. Keeping payment flows within Europe reduced latency and increased throughput capacity. This matters most during peak traffic periods when every millisecond affects conversion rates.

Cost optimization requires strategic tool selection. The biggest savings came from thoughtful vendor choices rather than raw infrastructure costs. Open-source monitoring provided 75% cost reduction with equivalent functionality.

Migration complexity shouldn't be underestimated. The 6-month timeline reflected the complexity of maintaining compliance and availability during the transition. Zero-downtime migration techniques were essential for a payment platform.

DORA compliance simplifies architecture decisions. Regulatory requirements eliminated many vendor options, but this constraint actually streamlined technology choices and reduced decision paralysis.

For teams considering similar migrations, start with a compliance audit to identify all third-country dependencies. Many organizations discover regulatory exposure in unexpected places like error tracking, analytics, or DNS services.

The business case strengthens over time. As payment volumes grow, the performance and cost advantages compound. This fintech projects additional savings of €800/month by year two as transaction volume increases.

Want these kinds of numbers for your own stack? Request a performance audit.

---

### How to optimize costs without adding servers: a cloud cost optimization guide

URL: https://binadit.com/blog/optimize-costs-without-adding-servers-cloud-cost-optimization-services-guide
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-10T09:49:21+02:00

> Adding more servers rarely fixes performance issues and drives up costs. This guide shows you how to identify real bottlenecks and optimize your existing infrastructure for better performance and lower bills.

What you'll achieve and why it matters

This guide shows you how to diagnose performance issues systematically and optimize your existing infrastructure instead of throwing more servers at the problem. You'll learn to identify real bottlenecks, implement targeted fixes, and reduce costs while improving performance.

Most infrastructure problems stem from inefficient resource usage, not resource shortage. Proper optimization can cut costs by 30-60% while delivering better user experience.

Prerequisites and assumptions

You'll need:

SSH access to your servers

Basic command line familiarity

Monitoring tools installed (we'll use standard Linux tools plus application metrics)

A staging environment that mirrors production load patterns

This guide assumes you're running a typical web application stack with database, application servers, and load balancer. The principles apply whether you're on dedicated servers, VPS, or cloud instances.

Step-by-step implementation with concrete commands, configs and code

Step 1: Establish baseline metrics

Before optimizing anything, measure current performance. Install and configure monitoring tools to capture baseline data.

Install system monitoring tools:

sudo apt update
sudo apt install htop iotop nethogs sysstat

Enable system statistics collection:

sudo systemctl enable sysstat
sudo systemctl start sysstat

Create a monitoring script to capture key metrics:

#!/bin/bash
# save as monitor.sh
echo "$(date): $(uptime)" >> /var/log/performance.log
echo "Memory: $(free -h | grep Mem)" >> /var/log/performance.log
echo "Disk I/O: $(iostat -x 1 1 | tail -n +4)" >> /var/log/performance.log
echo "---" >> /var/log/performance.log

Run this script every minute via cron to establish patterns:

* * * * * /path/to/monitor.sh

Step 2: Identify resource bottlenecks

Most performance issues fall into four categories: CPU, memory, disk I/O, or network. Use these commands to identify which resources are actually constrained.

Check CPU usage patterns over time:

sar -u 1 60

If CPU usage consistently exceeds 80%, investigate which processes consume most cycles:

top -o %CPU

Check memory usage and identify memory leaks:

free -h
ps aux --sort=-%mem | head -20

Monitor disk I/O to spot database or filesystem bottlenecks:

iostat -x 1 10

Look for high %util values (>90%) or long await times (>10ms for SSD, >20ms for HDD).

Check network utilization:

nethogs -d 5

Step 3: Optimize database performance

Database queries cause most web application bottlenecks. Optimize these before adding database servers.

Enable MySQL slow query log to identify problematic queries:

sudo mysql -e "SET GLOBAL slow_query_log = 'ON';"
sudo mysql -e "SET GLOBAL long_query_time = 2;"

Analyze slow queries after running for 24 hours:

sudo mysqldumpslow /var/lib/mysql/slow.log | head -10

Add indexes for frequently queried columns. For an ecommerce platform, typical optimization looks like:

ALTER TABLE orders ADD INDEX idx_created_status (created_at, status);
ALTER TABLE products ADD INDEX idx_category_price (category_id, price);

Configure MySQL memory settings based on available RAM. For a server with 8GB RAM dedicated to MySQL:

# Add to /etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]
innodb_buffer_pool_size = 5G
query_cache_size = 512M
tmp_table_size = 256M
max_heap_table_size = 256M

Step 4: Implement application-level caching

Caching reduces database load more effectively than adding database servers.

Install Redis for application caching:

sudo apt install redis-server
sudo systemctl enable redis-server

Configure Redis for optimal memory usage:

# /etc/redis/redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru
save 900 1
save 300 10

Implement caching in your application. Here's a PHP example for caching database queries:

function getCachedProducts($categoryId) {
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);

$cacheKey = "products_category_" . $categoryId;
$cached = $redis->get($cacheKey);

if ($cached) {
return json_decode($cached, true);
}

$products = $this->database->query(
"SELECT * FROM products WHERE category_id = ?", 
[$categoryId]
);

$redis->setex($cacheKey, 3600, json_encode($products));
return $products;
}

Step 5: Optimize web server configuration

Web server misconfiguration wastes resources. Optimize settings based on your actual traffic patterns.

For Nginx, configure worker processes and connections based on CPU cores:

# /etc/nginx/nginx.conf
worker_processes auto;
worker_connections 1024;

http {
keepalive_timeout 65;
gzip on;
gzip_comp_level 6;
gzip_types text/plain text/css application/javascript;
}

Enable HTTP/2 for better connection efficiency:

# In your server block
listen 443 ssl http2;
ssl_certificate /path/to/certificate.crt;
ssl_certificate_key /path/to/private.key;

Configure connection pooling for PHP-FPM to reduce overhead:

# /etc/php/8.1/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35

Step 6: Implement CDN and static asset optimization

Serving static content from optimized locations reduces server load significantly.

Configure Nginx to serve static files with proper caching headers:

# Add to your server block
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires 1y;
add_header Cache-Control "public, immutable";
access_log off;
}

Compress images and minify CSS/JavaScript. Create a build process:

#!/bin/bash
# optimize-assets.sh
for img in assets/images/*.jpg; do
jpegoptim --max=85 "$img"
done

for css in assets/css/*.css; do
uglifycss "$css" > "${css%.css}.min.css"
done

Verification: how to confirm it works

Measure improvements using the same metrics you established in your baseline. Performance optimization success shows in specific numbers.

Compare before and after CPU utilization:

sar -u -f /var/log/sysstat/saXX | grep Average

Check memory usage improvement:

free -h

Measure database performance improvement:

sudo mysqladmin extended-status | grep -E "(Queries|Uptime)"
# Calculate queries per second: Queries / Uptime

Test application response times using a simple load test:

ab -n 1000 -c 10 http://yoursite.com/

Monitor key application metrics:

Average response time should decrease by 20-50%

Database queries per page should reduce

Memory usage should stabilize

CPU peaks should be lower and less frequent

Check your infrastructure costs after running optimizations for a full billing cycle. Most businesses see 30-60% cost reduction without adding servers.

Common pitfalls to avoid

Don't optimize everything at once. Implement changes incrementally and measure impact before proceeding. This prevents introducing issues and helps identify which optimizations deliver the most value.

Avoid premature optimization of code that isn't actually causing bottlenecks. Profile first, optimize second.

Don't ignore monitoring during optimization. Some changes may improve one metric while degrading another.

Next steps and related reading

Once you've optimized your existing infrastructure, focus on preventing future performance degradation. Implement automated monitoring to catch issues before they impact users.

Consider implementing immutable infrastructure patterns to maintain optimization consistency across deployments.

Set up alerts for key performance metrics so you catch problems before they require emergency server additions.

Plan regular optimization reviews. Infrastructure needs change as your application grows, and optimization requirements evolve with usage patterns.

Long-term infrastructure strategy

Effective cloud cost optimization services require ongoing attention to infrastructure efficiency. The goal isn't just reducing immediate costs, but building systems that scale efficiently.

Most performance problems that seem to require more servers actually indicate inefficient resource usage. Database queries without proper indexes, unoptimized caching strategies, or misconfigured web servers waste more resources than insufficient capacity.

Building optimization into your development and deployment processes prevents the costly cycle of adding servers to compensate for inefficiency. This approach delivers better performance at lower cost than constantly scaling hardware.

Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.

---

### Configuration drift vs immutable infrastructure: choosing your zero downtime migration approach

URL: https://binadit.com/blog/configuration-drift-vs-immutable-infrastructure-zero-downtime-migration
Category: Reliability
Author: Binadit Tech Team
Published: 2026-06-09T09:06:09+02:00

> When servers slowly diverge from their intended state, production becomes unpredictable. Learn when to fix configuration drift versus when to embrace immutable infrastructure, and how each approach affects your zero downtime migration strategy.

The configuration drift decision every engineering team faces

Your production servers worked perfectly three months ago. Same code, same configuration, same workload. Now they randomly fail health checks, respond slowly to certain requests, and behave differently from your staging environment.

This is configuration drift. Small changes accumulate over time until your infrastructure becomes unpredictable. When this happens, engineering teams face a critical decision: fix the drift or rebuild with immutable patterns.

This choice directly impacts your ability to execute a zero downtime migration. Drifted systems resist reliable migrations because their state is unknown. Immutable systems enable confident migrations because every deployment starts from a known baseline.

The stakes are high. Choose wrong and your next migration could take your application offline for hours instead of minutes.

Configuration drift: the gradual approach with hidden costs

Configuration drift happens when live systems slowly diverge from their intended state. A security patch here, a config tweak there, a manual fix during an incident. Each change seems harmless, but collectively they create systems that nobody fully understands.

Most teams try to manage drift rather than eliminate it. They use configuration management tools like Ansible, Puppet, or Chef to detect differences and bring servers back into compliance. This approach feels practical because it works with existing systems and processes.

Strengths of managing configuration drift

Managing drift offers several advantages for teams with existing infrastructure. You can implement it gradually without disrupting current operations. Your team already understands the servers, the applications, and the deployment process.

Configuration management tools excel at detecting drift. They compare actual system state against desired state and highlight differences. When they find a drift, they can automatically correct it or alert operators to investigate.

This approach also preserves institutional knowledge. Your team knows which services run on which servers, where log files live, and how to troubleshoot problems. That knowledge remains valuable when managing drift rather than replacing systems.

Cost control is another strength. You avoid the immediate expense of rebuilding infrastructure. Instead, you invest time in tooling and processes that make existing systems more reliable.

Real limits of the drift management approach

However, managing drift has fundamental limitations that become apparent during complex operations like zero downtime migrations.

Drift detection is reactive, not preventive. By the time your tools detect drift, the damage is done. Systems have already diverged, potentially causing subtle bugs or performance issues that won't surface until high load conditions.

Correction can be disruptive. When configuration management tools fix drift, they often restart services or reload configurations. This creates brief interruptions that accumulate into noticeable downtime during migrations.

Complex systems resist automated correction. Real production environments have interdependencies that configuration management tools struggle to model. Correcting drift in one component can break another component in unexpected ways.

Perhaps most critically, drift management doesn't eliminate the root cause. As long as systems are mutable, they will continue to drift. You're fighting entropy instead of designing around it.

Immutable infrastructure: the rebuild approach with upfront investment

Immutable infrastructure takes the opposite approach. Instead of fixing drifted systems, you replace them entirely. Every deployment creates new infrastructure from scratch, runs the application, then destroys the old infrastructure.

This pattern treats servers like disposable resources rather than persistent assets. When you need to change something, you don't modify existing servers. You build new servers with the changes, deploy your application, and switch traffic over.

Strengths of immutable infrastructure

Immutable patterns eliminate drift by design. Since servers are never modified after creation, they cannot drift from their intended state. What you deploy is exactly what runs in production, every time.

This predictability transforms zero downtime migration from a risky operation into a routine deployment. You know exactly what state your new infrastructure will have because you built it from the same automated process that created your current infrastructure.

Rollbacks become trivial. If a deployment causes problems, you simply switch traffic back to the previous infrastructure. No complex rollback procedures, no partial state recovery, no uncertainty about what changed.

Testing becomes more reliable too. Your staging environment can use the exact same infrastructure creation process as production. This eliminates the common problem where applications work in staging but fail in production due to environmental differences.

Immutable patterns also improve security. Instead of patching running systems, you rebuild them with updated base images. This ensures patches are applied consistently and completely across all infrastructure.

Real limits of immutable infrastructure

Immutable infrastructure requires significant upfront investment in automation. You need robust tooling to create, configure, and deploy infrastructure programmatically. This tooling must handle failure scenarios gracefully.

State management becomes more complex. Applications that store data locally, maintain connections, or cache information must be redesigned to work with ephemeral infrastructure. This often requires architectural changes to externalize state.

Resource consumption increases during deployments. Since you run both old and new infrastructure simultaneously during transitions, you need roughly double the capacity. This impacts costs and resource planning.

Debugging running systems becomes more difficult. You cannot log into a server and make investigative changes. Instead, you must build debugging capabilities into your infrastructure creation process or application monitoring.

Team workflow changes are substantial. Engineers must adapt to treating infrastructure as code rather than managed resources. This cultural shift can be challenging for teams accustomed to traditional operations.

Direct comparison: drift management vs immutable patterns
FactorConfiguration drift managementImmutable infrastructureImplementation costLow initial investment, ongoing operational overheadHigh upfront automation investment, lower ongoing costsOperational burdenContinuous monitoring and correction of driftInfrastructure rebuilds for every changeMigration reliabilityUnpredictable due to unknown system stateHighly predictable due to known baselineRollback complexityComplex, requires understanding current stateSimple, switch traffic to previous infrastructureResource requirementsConsistent resource usageDouble capacity needed during deploymentsTeam expertise neededTraditional ops skills, configuration management toolsInfrastructure as code, automation developmentScaling characteristicsManual intervention required for complex changesAutomated scaling through infrastructure recreation

Decision framework: when to choose each approach

Choose configuration drift management when you have existing infrastructure that mostly works, limited automation expertise on your team, and budget constraints that prevent infrastructure redesign. This approach works well for stable applications with infrequent deployments and teams comfortable with traditional operations.

Specifically, drift management makes sense if you deploy less than weekly, have applications that store significant local state, work with legacy systems that resist containerization, or operate in environments where infrastructure automation tools are restricted.

Choose immutable infrastructure when you need reliable zero downtime migrations, deploy frequently, or operate applications that can externalize state effectively. This approach suits teams with strong automation skills and applications designed for cloud-native operations.

Immutable patterns are essential when you deploy daily or more frequently, operate microservices architectures, need guaranteed consistency between environments, or work with compliance requirements that favor infrastructure replacement over modification.

Consider your migration timeline too. If you need to execute a zero downtime migration within the next three months, improving drift management might be more realistic than building immutable infrastructure from scratch.

However, if you're planning infrastructure changes over the next year, investing in immutable patterns will pay dividends in migration reliability and operational simplicity.

The hybrid approach also deserves consideration. You might manage drift in persistent data layers while using immutable patterns for stateless application tiers. This balances the benefits of both approaches while acknowledging the realities of complex systems.

Choose based on your migration timeline and team capabilities

Configuration drift and immutable infrastructure represent fundamentally different philosophies about managing change. Drift management accepts that systems will change and focuses on controlling that change. Immutable infrastructure prevents change entirely by replacing systems instead of modifying them.

Your choice impacts every aspect of operations, from daily deployments to major migrations. Teams that choose drift management trade ongoing operational overhead for lower upfront investment. Teams that choose immutable patterns trade higher initial complexity for more predictable operations.

For zero downtime migrations specifically, immutable infrastructure provides much higher confidence. When you know exactly what state your infrastructure will have, you can plan migrations more precisely and handle edge cases more effectively.

The most successful teams often start with drift management for immediate needs, then gradually adopt immutable patterns as their automation capabilities mature. This evolutionary approach respects both current constraints and future goals.

Still weighing options for your stack? Book a 30-minute architecture call, no sales pitch.

---

### Government procurement and public-sector tenders: why managed cloud infrastructure wins contracts

URL: https://binadit.com/blog/government-procurement-public-sector-tenders-managed-cloud-infrastructure-contracts
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-08T10:32:39+02:00

> Public sector contracts demand specific compliance, security, and operational requirements that standard hosting cannot meet. Here's how to architect infrastructure that actually passes tender evaluations.

Government tenders evaluate infrastructure differently than private sector deals

Public sector procurement follows rigid evaluation criteria that score vendors on security compliance, data sovereignty, operational transparency, and long-term stability. Standard cloud hosting typically fails these assessments because it lacks the documented processes, compliance certifications, and operational controls that government evaluators require.

The gap isn't about technical capability. Most hosting providers can run government workloads. The gap is in how they document, monitor, and manage those workloads according to public sector standards.

Why standard hosting fails government evaluation criteria

Government procurement teams evaluate infrastructure against specific frameworks like ISO 27001, SOC 2 Type II, and regional data protection requirements. They need documented evidence of security controls, incident response procedures, and compliance monitoring.

Standard hosting providers typically offer:

Basic security configurations without detailed documentation

Generic SLAs that don't address government-specific requirements

Support through ticket systems rather than direct engineer contact

Infrastructure shared across multiple jurisdictions without clear data boundaries

Government tenders require:

Documented security policies with regular audit trails

Custom SLAs that address specific regulatory requirements

Direct technical contacts for security incident response

Infrastructure with clear geographic and legal boundaries

The procurement process scores these requirements heavily. A technically excellent but poorly documented solution scores lower than a well-documented solution with adequate technical capabilities.

Data sovereignty requirements create additional complexity. Government workloads often require infrastructure and data to remain within specific geographic boundaries, with clear legal jurisdiction over all components. Standard cloud providers may use global CDNs, backup locations, or support teams that cross these boundaries without clear documentation.

Managing compliance and sovereignty risks in private cloud infrastructure becomes critical when government contracts specify these requirements in detail.

How to architect managed cloud infrastructure for government procurement

Government-ready managed cloud infrastructure requires specific architectural and operational patterns that address procurement evaluation criteria.

Implement documented security controls

Create security policies that map directly to government frameworks:

# Example security baseline configuration
# Network segmentation
iptables -A INPUT -s 10.0.0.0/24 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -j DROP
iptables -P INPUT DROP

# Logging configuration
rsyslog_template='%timestamp% %hostname% %programname%: %msg%'
echo "*.* @@logserver.internal.gov:514;$rsyslog_template" >> /etc/rsyslog.conf

# File integrity monitoring
aide --init
cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

Document each control with:

Implementation details and configuration files

Monitoring procedures and alert thresholds

Incident response procedures with specific contact information

Regular audit schedules and compliance reporting

Configure geographic data boundaries

Implement infrastructure that guarantees data remains within specified jurisdictions:

# Database configuration with geographic constraints
# PostgreSQL configuration for EU-only deployment
data_directory = '/var/lib/postgresql/13/main'
log_destination = 'stderr,syslog'
log_directory = '/var/log/postgresql'

# Backup configuration with geographic limits
pg_basebackup -h primary.eu-central.internal \
-D /backup/postgresql \
-U replication \
-P -W -R -X stream

Configure CDN and caching with regional restrictions:

# Nginx configuration for EU-only caching
location /static/ {
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=eu_cache:10m;
proxy_cache eu_cache;
proxy_cache_valid 200 1h;

# Restrict upstream to EU-only servers
proxy_pass http://eu_backend_pool;

# Geographic restriction headers
add_header X-Served-From "EU-Central-1";
add_header X-Data-Jurisdiction "EU";
}

Implement operational transparency

Government contracts often require operational visibility that goes beyond standard monitoring:

# Infrastructure monitoring with compliance reporting
# Prometheus configuration for government metrics
global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "compliance_rules.yml"
- "security_alerts.yml"

scrape_configs:
- job_name: 'government-infrastructure'
static_configs:
- targets: ['web-1.internal:9100', 'db-1.internal:9100']

# Security-focused metrics
metrics_path: /metrics
params:
collect[]:
- node_security
- node_compliance
- node_audit

Create compliance dashboards that generate reports for government oversight:

# Grafana dashboard configuration for compliance
{
"dashboard": {
"title": "Government Compliance Dashboard",
"panels": [
{
"title": "Security Event Timeline",
"type": "logs",
"targets": [
{
"expr": "rate(security_events_total[5m])",
"legendFormat": "Security Events per 5min"
}
]
},
{
"title": "Data Geographic Compliance",
"type": "stat",
"targets": [
{
"expr": "sum(rate(cross_border_requests_total[1h]))",
"legendFormat": "Cross-border requests (should be 0)"
}
]
}
]
}
}

Real numbers from EU deployments show how proper geographic controls perform in practice.

How to validate your infrastructure meets procurement requirements

Government procurement teams evaluate infrastructure against specific, measurable criteria. Validation requires demonstrating compliance through documentation, metrics, and audit trails.

Security compliance validation

Run compliance checks that generate government-ready reports:

# OpenSCAP compliance scanning
oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis \
--results scan-results.xml \
--report compliance-report.html \
/usr/share/xml/scap/ssg/content/ssg-ubuntu1804-ds.xml

# Lynis security audit
lynis audit system \
--auditor "Government Procurement Team" \
--cronjob \
--report-file /var/log/lynis-government.log

Monitor compliance metrics continuously:

# Compliance monitoring script
#!/bin/bash

# Check data geographic boundaries
CROSS_BORDER_REQUESTS=$(grep "cross_border" /var/log/nginx/access.log | wc -l)
if [ $CROSS_BORDER_REQUESTS -gt 0 ]; then
echo "ALERT: Cross-border data requests detected: $CROSS_BORDER_REQUESTS"
logger "COMPLIANCE_VIOLATION: Cross-border requests: $CROSS_BORDER_REQUESTS"
fi

# Check security control status
FAILED_LOGINS=$(journalctl -u ssh --since "1 hour ago" | grep "Failed password" | wc -l)
if [ $FAILED_LOGINS -gt 10 ]; then
echo "ALERT: Excessive failed login attempts: $FAILED_LOGINS"
logger "SECURITY_ALERT: Failed logins: $FAILED_LOGINS"
fi

# Generate daily compliance report
echo "$(date): Compliance check completed. Cross-border: $CROSS_BORDER_REQUESTS, Failed logins: $FAILED_LOGINS" >> /var/log/government-compliance.log

Operational transparency validation

Government contracts require evidence of operational procedures and incident response capabilities:

# Incident response validation script
#!/bin/bash

# Test incident detection
echo "Testing security incident detection..."
logger "SECURITY_TEST: Simulated unauthorized access attempt"

# Verify alert routing
curl -X POST http://monitoring.internal/api/v1/alerts \
-H "Content-Type: application/json" \
-d '{
"alerts": [{
"labels": {
"alertname": "GovernmentSecurityTest",
"severity": "critical",
"instance": "test-validation"
},
"annotations": {
"summary": "Government procurement validation test"
}
}]
}'

# Check response time
START_TIME=$(date +%s)
echo "Waiting for incident response team notification..."
# In practice, verify human response within SLA timeframe

Document all procedures with specific metrics:

Incident detection time: Average 2.3 minutes from event to alert

Initial response time: Maximum 15 minutes during business hours

Escalation procedures: Direct contact information for government liaison

Resolution reporting: Detailed post-incident reports within 24 hours

How to prevent procurement evaluation failures

Government procurement failures typically occur because infrastructure providers don't understand the evaluation process or prepare documentation that matches scoring criteria.

Map technical capabilities to evaluation frameworks

Government procurement teams score responses against frameworks like NIST Cybersecurity Framework, ISO 27001, or regional standards. Map your infrastructure directly to these requirements:
Framework RequirementInfrastructure ImplementationEvidence/DocumentationDE.CM-1: Network monitoringReal-time traffic analysis with geographic filteringMonitoring dashboard screenshots, log samplesPR.DS-1: Data protectionEncryption at rest and in transit, EU-only storageEncryption configuration files, compliance certificatesRS.CO-2: Incident reporting24/7 monitoring with direct government contact proceduresIncident response playbook, contact escalation matrix

Create documentation packages that directly answer procurement questions rather than providing generic technical specifications.

Implement continuous compliance monitoring

Government contracts often include ongoing compliance requirements. Implement monitoring that continuously validates compliance rather than point-in-time assessments:

# Continuous compliance monitoring
# /etc/systemd/system/gov-compliance-monitor.service
[Unit]
Description=Government Compliance Monitor
After=network.target

[Service]
Type=simple
User=compliance
ExecStart=/usr/local/bin/compliance-monitor.py
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

# /usr/local/bin/compliance-monitor.py
#!/usr/bin/env python3
import time
import subprocess
import json
import logging
from datetime import datetime

def check_geographic_compliance():
"""Verify all data remains within approved geographic boundaries"""
try:
result = subprocess.run(['geoiplookup'], 
capture_output=True, text=True, timeout=30)

# Parse geographic data from logs
cross_border_count = 0
with open('/var/log/nginx/access.log', 'r') as f:
for line in f:
# Check for non-EU IP addresses in logs
# Implementation depends on specific requirements
pass

compliance_data = {
'timestamp': datetime.now().isoformat(),
'geographic_violations': cross_border_count,
'status': 'COMPLIANT' if cross_border_count == 0 else 'VIOLATION'
}

# Log to government compliance system
logging.info(f"Geographic compliance check: {json.dumps(compliance_data)}")

return compliance_data

except Exception as e:
logging.error(f"Compliance check failed: {e}")
return {'status': 'CHECK_FAILED', 'error': str(e)}

while True:
check_geographic_compliance()
time.sleep(300) # Check every 5 minutes

Prepare for ongoing audits and reviews

Government contracts typically include audit rights and review procedures. Design infrastructure with audit preparation built in:

# Audit log aggregation
# rsyslog configuration for government audit requirements
# /etc/rsyslog.d/government-audit.conf

# Separate log streams for different audit requirements
:programname, isequal, "nginx" /var/log/audit/web-access.log
:programname, isequal, "postgresql" /var/log/audit/database-access.log
:msg, contains, "SECURITY" /var/log/audit/security-events.log
:msg, contains, "COMPLIANCE" /var/log/audit/compliance-events.log

# Forward to government oversight systems if required
*.* @@government-audit-server.internal:514

GDPR-compliant infrastructure requirements overlap significantly with government procurement requirements, especially for EU-based contracts.

Regular procurement readiness reviews help identify gaps before contract opportunities arise. Many organizations lose government contracts not because their infrastructure is inadequate, but because they can't demonstrate compliance effectively during the evaluation process.

Infrastructure that meets government procurement standards typically exceeds private sector requirements, making it valuable for regulated industries, enterprise customers, and organizations with strict compliance requirements. The investment in government-ready managed cloud infrastructure often opens multiple market opportunities beyond public sector contracts.

If you'd rather not debug this again next quarter, our managed platform handles it by default.

---

### Website hosting mistakes that cost businesses thousands

URL: https://binadit.com/blog/website-hosting-mistakes-cost-businesses-thousands
Category: Infrastructure
Author: Binadit Tech Team
Published: 2026-06-08T10:32:18+02:00

> Poor website hosting decisions can drain thousands from your budget through downtime, performance issues, and hidden costs. Here's how to identify and avoid the most expensive hosting mistakes before they impact your bottom line.

The real cost of hosting decisions

A marketing agency we worked with last year was paying €150 per month for 'premium' shared hosting. Their site went down during a client campaign launch, costing them a €25,000 contract renewal. The hosting provider's response? 'Shared resources experienced high load.' No compensation, no explanation, no solution.

This isn't unusual. Most businesses treat website hosting as a commodity purchase, focusing on monthly price rather than total cost of ownership. But hosting failures create cascading costs that dwarf the monthly hosting bill.

Revenue loss from downtime averages €5,600 per hour for small businesses and €300,000 per hour for large enterprises, according to Gartner. Performance issues are equally expensive. A one-second delay in page load time reduces conversions by 7%. For an ecommerce site generating €1 million annually, that's €70,000 in lost revenue.

The hosting industry has trained businesses to compare specs and prices. But the most expensive hosting mistakes aren't about getting bad specs for your money. They're about misunderstanding what different types of website hosting can actually deliver under real-world conditions.

Mistake 1: Choosing shared hosting for business-critical applications

Shared hosting puts hundreds of websites on the same server, sharing CPU, memory, and network resources. When one site gets traffic, all sites slow down. When one site gets hacked, all sites are at risk.

The performance degradation follows a predictable pattern. During low-traffic periods, sites load quickly. But when multiple sites experience traffic simultaneously, response times spike. We've measured shared hosting environments where response times increased from 200ms to 8 seconds during peak hours.

Resource limits compound the problem. Most shared hosts limit CPU usage to 10-20% and memory to 512MB-1GB. These limits trigger during traffic spikes, exactly when you need maximum performance. Your site doesn't crash cleanly. It becomes unresponsive, creating a poor user experience and damaging search rankings.

Security isolation is minimal. Shared hosting uses basic file permissions to separate accounts, but all sites run under the same web server process. If one site gets compromised, attackers often gain access to neighboring sites. We've seen single malware infections spread across dozens of sites on the same shared server.

Business impact multiplies beyond the immediate hosting cost. Slow loading times reduce conversion rates. Search engines penalize slow sites in rankings. Customer support teams spend time explaining outages instead of growing the business. The €10 per month shared hosting plan becomes €10,000 in opportunity costs.

When shared hosting works

Shared hosting isn't inherently bad. It works for low-traffic informational sites, personal blogs, and development environments. If your site receives fewer than 1,000 visitors per month and downtime doesn't affect revenue, shared hosting can be cost-effective.

But most business sites outgrow shared hosting within 12 months. Traffic increases, functionality becomes more complex, and uptime becomes critical. Planning for this transition prevents emergency migrations during traffic spikes.

Mistake 2: Underestimating the operational burden of VPS hosting

Virtual private servers (VPS) promise dedicated resources at shared hosting prices. You get your own virtual machine with guaranteed CPU, memory, and storage. No noisy neighbors, no resource limits, full control over the environment.

The marketing materials make VPS sound simple. 'Get root access and install anything you need.' But root access means you're responsible for everything: security updates, performance optimization, backup management, monitoring, and incident response.

Server administration requires specific expertise. A misconfigured firewall exposes your server to attacks. Outdated software creates security vulnerabilities. Poor database tuning limits performance under load. Memory leaks crash applications. Each component needs ongoing attention from someone who understands how it works.

The time investment is significant. Security updates require 2-4 hours monthly. Performance optimization takes 8-12 hours quarterly. Monitoring setup and maintenance needs 1-2 hours weekly. Incident response varies, but outages often require immediate attention regardless of timing.

For a technical team, this overhead might be acceptable. But for most businesses, the operational burden exceeds the cost savings. A developer spending 10 hours monthly on server administration costs more than upgrading to managed hosting.

Consider a SaaS company paying a senior developer €80,000 annually (roughly €40 per hour) to manage their VPS infrastructure. Ten hours monthly of server administration costs €400. Managed hosting that eliminates this overhead while improving reliability often costs less than the developer's time.

VPS hosting makes sense when

VPS hosting works for teams with dedicated system administration skills. If you have someone who enjoys server management and understands the full stack, VPS hosting offers excellent control and value. Development teams building custom infrastructures often prefer VPS environments.

But VPS hosting becomes expensive when treated as 'cheap managed hosting.' Without proper administration, performance and security suffer. Choosing the right VPS setup for production requires understanding both the technical and operational requirements.

Mistake 3: Ignoring compliance requirements until it's too late

Many businesses discover compliance requirements after choosing their hosting provider. GDPR requires that EU customer data stays within EU jurisdiction. Industry regulations like PCI DSS mandate specific security controls. Government contracts often require data sovereignty.

Most major web hosting providers use US-based infrastructure by default. Even 'EU regions' often route traffic through US data centers or use US-based management systems. This creates legal exposure that many businesses don't understand until an audit or contract review.

The Cloud Act allows US authorities to access data stored on US company servers, regardless of location. This affects major providers like AWS, Google Cloud, and Microsoft Azure, even when using their European regions. For businesses handling sensitive data, this creates compliance risks.

Compliance retrofitting is expensive. Moving from non-compliant to compliant hosting often requires architectural changes, not just provider migration. Applications designed for US cloud services need modification to work with EU-based alternatives. Database schemas might need restructuring to meet data residency requirements.

We helped a digital agency migrate their entire client infrastructure to ensure GDPR compliance. The technical migration took three months and cost €45,000 in development time. The agency avoided potential CLOUD Act issues but could have prevented the migration costs by choosing compliant hosting initially.

Planning for compliance

Compliance requirements should influence hosting decisions from the beginning. Research the regulations affecting your industry and customer base. Understand where your data will be stored and who can access it. Choose providers that support your compliance needs without architectural changes.

EU-based infrastructure providers offer genuine data sovereignty without the complexity of major cloud platforms. Managed cloud infrastructure in European data centers eliminates most GDPR and data sovereignty concerns while providing the performance and reliability businesses need.

Mistake 4: Optimizing for specifications instead of real-world performance

Hosting provider marketing focuses on specifications: CPU cores, RAM, storage space, bandwidth. Businesses compare these numbers and choose the highest specs for the lowest price. But specifications don't predict real-world performance.

CPU performance varies dramatically between providers. A '4-core' VPS might use older processors, shared CPU resources, or aggressive oversubscription. We've benchmarked identical specifications where actual CPU performance differed by 300% between providers.

Network performance is rarely specified but affects user experience more than raw CPU power. A fast server with poor network connectivity delivers slow page loads. International routing, peering relationships, and CDN integration determine how quickly content reaches users.

Storage performance matters more than capacity for most applications. Database queries, file uploads, and page rendering depend on IOPS (input/output operations per second) rather than total storage space. Traditional hard drives limit performance even when servers have adequate CPU and memory.

Real-world testing reveals these differences. We benchmark hosting providers under simulated load using actual applications, not synthetic tests. Performance varies widely even among providers offering identical specifications.

Consider two hosting providers offering '8-core, 16GB RAM' VPS plans. Provider A uses modern CPUs, NVMe storage, and optimized networking. Provider B uses older hardware with traditional hard drives and basic networking. Provider A delivers 3x faster response times despite identical specifications.

Measuring what matters

Focus on performance metrics that affect user experience: response time, throughput under load, availability, and geographic performance. Real-world hosting performance measurements matter more than specification sheets.

Professional hosting providers publish performance data and SLA guarantees. They provide monitoring tools and performance reports. They optimize infrastructure for actual applications, not benchmark scores.

Mistake 5: Underestimating the true cost of downtime

Businesses often accept downtime as an inevitable cost of hosting. 'Our site was down for two hours last month, but it saved us €50 on hosting costs.' This calculation ignores the compound costs of outages.

Direct revenue loss is immediate and measurable. For ecommerce sites, downtime directly reduces sales. A site generating €10,000 daily loses €416 per hour of downtime. But indirect costs multiply this impact.

Customer acquisition costs increase when potential customers can't access your site. If you're spending €1,000 monthly on advertising, downtime wastes that investment. A two-hour outage during peak traffic wastes €67 in advertising spend, assuming uniform traffic distribution.

Search engine penalties compound over time. Google considers site availability when ranking pages. Frequent downtime reduces search visibility, requiring additional SEO investment to recover rankings.

Support overhead increases during and after outages. Customer service teams handle frustrated users instead of processing orders or onboarding new customers. Team productivity drops while everyone tries to understand what happened and prevent recurrence.

Consider a B2B SaaS platform generating €50,000 monthly recurring revenue. A four-hour outage costs €278 in direct revenue loss. But the outage also affects trial signups, customer satisfaction, and team productivity. Total cost often exceeds €2,000 when including all impacts.

Calculating your downtime cost

Estimate your hourly revenue (monthly revenue / 730 hours). Add customer acquisition costs, support overhead, and productivity losses. Multiply by your acceptable downtime hours annually. This calculation shows the maximum acceptable hosting cost to prevent outages.

If downtime costs €500 per hour and you experience 10 hours annually, that's €5,000 in downtime costs. Spending an additional €200 monthly on reliable hosting saves €2,600 annually while improving customer experience.

Mistake 6: Choosing hosting providers without understanding their support model

Most hosting providers handle support through ticket systems staffed by first-level technicians reading scripts. When your site goes down at 2 AM, you submit a ticket and wait. Response times vary from minutes to hours. Resolution times stretch longer.

Tiered support systems create delays during critical incidents. Level 1 support handles basic questions but escalates complex issues to level 2 or 3 teams. Each escalation adds delay while your site remains offline.

Many providers outsource support to third-party companies with minimal technical training. Support agents follow troubleshooting scripts but can't diagnose unique problems or make infrastructure changes. Complex issues get escalated repeatedly or marked as 'not a hosting problem.'

Geographic and timezone differences complicate support for European businesses using US-based providers. Your emergency becomes their routine ticket. Cultural and language barriers slow communication during stressful situations.

The support model affects your operational overhead. Poor hosting support means your team spends time diagnosing hosting issues, coordinating with support agents, and implementing workarounds. Good support eliminates this overhead.

Evaluating support quality

Look for providers offering direct access to system administrators who can make immediate changes. Avoid providers relying primarily on ticket systems for urgent issues. Test support responsiveness during the evaluation process by asking technical questions.

European businesses benefit from EU-based support teams familiar with GDPR, local business practices, and timezone requirements. Infrastructure management services that include direct engineer access eliminate most support-related delays and frustrations.

Mistake 7: Scaling too late or scaling wrong

Most businesses wait until performance problems affect customers before upgrading hosting. Traffic spikes reveal infrastructure limitations during peak business periods. Black Friday crashes, product launches fail, marketing campaigns underperform because servers can't handle the load.

Emergency scaling is expensive and disruptive. Moving to better hosting under pressure limits options and increases migration risks. Quick fixes often create technical debt that complicates future scaling.

Vertical scaling (bigger servers) hits physical limits and creates single points of failure. Horizontal scaling (more servers) requires application changes and load balancing expertise. Most businesses lack the technical knowledge to implement horizontal scaling correctly.

Poor scaling decisions create ongoing costs. Over-provisioning wastes money on unused resources. Under-provisioning creates performance problems and customer frustration. Auto-scaling without proper configuration can spike costs during traffic anomalies or attacks.

We worked with an ecommerce company that experienced 300% traffic growth over six months. Their shared hosting couldn't handle the load, causing frequent outages during peak sales periods. Emergency migration to managed infrastructure took two weeks and required significant application changes.

The migration cost €15,000 in development time and €3,000 in lost sales during the transition. Planning the migration six months earlier would have cost €8,000 and prevented all downtime.

Proactive scaling strategies

Monitor growth trends and plan infrastructure changes before reaching capacity limits. Scaling applications properly requires understanding both current performance and future requirements.

Professional infrastructure management includes capacity planning and proactive scaling recommendations. Instead of reacting to problems, you stay ahead of growth with planned upgrades during low-traffic periods.

The decision framework: choosing hosting that fits your business

Hosting decisions should match your business requirements, technical capabilities, and risk tolerance. Use this framework to evaluate options:

Traffic and performance requirements

Calculate your peak concurrent users, acceptable response times, and uptime requirements. Sites with fewer than 10,000 monthly visitors can often use shared hosting. Sites with 10,000-100,000 visitors need VPS or managed hosting. Sites with higher traffic require dedicated or cloud infrastructure.

Performance requirements depend on your conversion funnel. Ecommerce sites need fast checkout processes. SaaS applications need reliable API responses. Content sites can tolerate slightly higher response times.

Technical capability assessment

Honestly assess your team's system administration skills. Managing servers requires understanding of security, performance optimization, backup management, and incident response. If you lack these skills internally, managed hosting eliminates the learning curve and operational overhead.

Consider opportunity cost. Developer time spent on server management doesn't contribute to product development or customer acquisition. For most businesses, managed hosting is cheaper than internal system administration.

Compliance and regulatory requirements

Identify regulations affecting your business: GDPR, PCI DSS, HIPAA, industry-specific requirements. Choose hosting that meets these requirements without additional configuration or architectural changes.

Data sovereignty requirements often eliminate major US-based cloud providers. EU-based infrastructure ensures compliance without complex legal analysis or ongoing monitoring.

Growth planning

Plan for 2-3x your current traffic and feature requirements. Hosting migrations disrupt operations and consume development resources. Choose hosting that supports your growth without immediate migration needs.

Consider seasonal variations, marketing campaigns, and product launches that might spike traffic. Hosting should handle peak loads, not just average usage.
Business TypeRecommended HostingKey ConsiderationsSmall business website (<10k visits/month)Quality shared hostingUptime guarantee, support qualityGrowing business (10k-100k visits/month)Managed VPS or managed cloudScaling flexibility, performance monitoringEcommerce platform (>100k visits/month)Managed cloud infrastructureSecurity, compliance, scaling automationSaaS applicationManaged infrastructure or private cloudAPI reliability, data sovereignty, scalingEnterprise applicationsPrivate cloud or dedicated infrastructureCompliance, security, custom requirements

When managed infrastructure makes sense

Managed infrastructure works for businesses that need reliable hosting without operational overhead. You get dedicated resources, professional management, and direct access to engineers who understand your setup.

The cost premium over DI

---

### How to profile real-world performance issues in high availability infrastructure

URL: https://binadit.com/blog/profile-real-world-performance-issues-high-availability-infrastructure
Category: Performance
Author: Binadit Tech Team
Published: 2026-06-07T09:12:44+02:00

> Performance problems hide in production environments where they're hardest to debug. Here's how to profile them systematically without guessing, using tools and techniques that work when your infrastructure is under real load.

When performance degrades, the symptoms lie

Your monitoring shows CPU at 60%, memory looks fine, and network utilization seems normal. Yet response times doubled overnight, users are complaining, and you can't reproduce it in staging. This is the reality of performance issues in high availability infrastructure - they manifest under real conditions with real data patterns that development environments never replicate.

Why production performance issues hide from standard monitoring

Most performance problems in production stem from interactions between components under specific load patterns. Your application might handle 1000 requests per second perfectly, but fails when those requests hit a particular database query pattern, or when memory allocation patterns create garbage collection pauses during peak traffic.

Standard monitoring tools measure resource utilization but miss the critical details: lock contention, thread pool exhaustion, connection pool starvation, or memory allocation patterns. These issues don't show up as high CPU or memory usage - they manifest as waiting, blocking, and inefficient resource utilization.

Application profiling reveals what's actually happening inside your code during performance degradation. Unlike monitoring dashboards that show aggregate metrics, profilers capture the execution flow, identifying which functions consume the most time, where threads block, and how memory gets allocated and freed.

The challenge is that profiling in production requires tools that impose minimal overhead while capturing actionable data. Traditional profilers often add 10-30% overhead, making them unsuitable for production environments where performance is already degraded.

The systematic approach to production profiling

Start with continuous profiling tools that run permanently in production with sub-1% overhead. These tools sample execution at regular intervals, building statistical profiles of your application's behavior over time.

For Java applications, enable JFR (Java Flight Recorder) with this configuration:

-XX:+FlightRecorder
-XX:StartFlightRecording=duration=300s,filename=profile.jfr
-XX:FlightRecorderOptions=settings=profile

For Python applications, use py-spy for sampling without code modifications:

py-spy record -o profile.svg -d 300 -p PID

For Node.js, leverage the built-in profiler:

node --prof app.js
# Generate readable output
node --prof-process isolate-*-v8.log > profile.txt

The key is collecting baseline profiles during normal operation, then comparing them with profiles captured during performance degradation. This differential analysis reveals what changes when performance drops.

Focus profiling on these critical areas: CPU hotspots that consume disproportionate execution time, memory allocation patterns that trigger excessive garbage collection, I/O operations that block threads, and lock contention points where threads wait for shared resources.

Database query profiling requires separate attention. Enable slow query logging in MySQL:

SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 0.1;
SET GLOBAL log_queries_not_using_indexes = 'ON';

For PostgreSQL, configure automatic query logging:

# In postgresql.conf
log_min_duration_statement = 100
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
log_checkpoints = on
log_connections = on
log_disconnections = on

Memory profiling reveals allocation patterns that aren't visible in standard metrics. High memory usage doesn't always correlate with performance problems, but inefficient allocation patterns create garbage collection pressure that manifests as intermittent latency spikes.

Validating that profiling identified the real bottleneck

Profiling data must translate into measurable performance improvements. After identifying bottlenecks through profiling, validate the findings by implementing targeted fixes and measuring the impact.

Create performance benchmarks that reproduce the identified bottleneck in isolation. If profiling reveals excessive database connection creation, benchmark the application with and without connection pooling improvements. If CPU profiling shows inefficient serialization, benchmark alternative serialization libraries.

Monitor these key metrics before and after optimization: request latency percentiles (P50, P95, P99), throughput under sustained load, resource utilization patterns, and error rates during peak traffic.

Set up continuous performance testing that validates optimizations don't regress. A fix that improves CPU usage but increases memory allocation might trade one bottleneck for another.

Use APM tools to correlate profiling insights with real user experience. Tools like Jaeger for distributed tracing or New Relic for application monitoring provide context that pure profiling data lacks - how performance improvements affect actual user transactions.

The most reliable validation is measuring business metrics: page load times, conversion rates during peak traffic, and customer support tickets related to performance. Technical improvements must translate to measurable business impact.

Preventing performance issues from recurring

Continuous profiling should be part of your standard infrastructure, not something you enable during incidents. Modern profiling tools run with minimal overhead, providing ongoing visibility into application behavior patterns.

Implement performance budgets in your CI/CD pipeline. Run automated performance tests that fail builds when latency increases beyond acceptable thresholds. This catches performance regressions before they reach production.

Establish performance monitoring that goes deeper than standard metrics. Track garbage collection frequency and duration, database connection pool utilization, thread pool queue depths, and memory allocation rates. These leading indicators reveal performance problems before they affect users.

Create performance runbooks based on profiling insights. Document the specific profiling commands, analysis techniques, and optimization approaches that worked for your infrastructure. This knowledge transfer prevents future incidents from requiring the same investigative work.

Regular performance audits using production profiling data help identify gradual degradation that might not trigger alerts but accumulates into significant problems. Schedule monthly reviews of profiling data to spot trends in resource utilization, allocation patterns, or execution hotspots.

Load testing should incorporate realistic data patterns identified through profiling. If production profiling reveals that performance degrades with specific query patterns or data sizes, ensure your load tests replicate these conditions.

As we covered in our guide on tracing performance bottlenecks end-to-end, systematic performance analysis requires tools that work across your entire stack. Similarly, understanding queue congestion patterns helps identify bottlenecks that profiling might miss in distributed systems.

Building profiling into your infrastructure workflow

Production profiling works best when integrated into your standard operational workflow rather than treated as an emergency tool. The insights from continuous profiling inform capacity planning, optimization priorities, and architecture decisions.

Performance issues in high availability infrastructure are inevitable, but they don't have to be mysteries. Systematic profiling provides the data needed to understand what's actually happening when performance degrades, enabling targeted fixes that address root causes rather than symptoms.

If you'd rather not debug this again next quarter, our managed platform handles it by default.

---

### How a digital agency avoided CLOUD Act data requests by moving to private cloud infrastructure

URL: https://binadit.com/blog/digital-agency-cloud-act-private-cloud-infrastructure-data-sovereignty
Category: Security
Author: Binadit Tech Team
Published: 2026-06-06T10:07:11+02:00

> A mid-sized digital agency discovered their US-hosted private data could be accessed without notice. Here's how they moved to EU-based private cloud infrastructure and what it cost in time, money, and engineering effort.

The situation: a growing agency with enterprise clients

A Rotterdam-based digital agency had grown to 45 employees, managing websites and applications for enterprise clients across financial services, healthcare, and government sectors. They hosted everything on a well-known US cloud provider, managing about 200 client websites and 15 custom applications.

The problem surfaced during a client audit. A major healthcare client was expanding across EU markets and their compliance team flagged a critical issue: the agency's infrastructure sat in US data centers, making all client data potentially subject to CLOUD Act requests.

Under the CLOUD Act, US authorities can compel US companies to hand over data stored anywhere in the world, regardless of local privacy laws. For the agency's enterprise clients, this created unacceptable compliance risk.

The agency faced a choice: lose their biggest clients or migrate everything to EU-based private cloud infrastructure that offered genuine data sovereignty.

What we found during the infrastructure audit

When we audited their setup, the sovereignty risks were worse than expected. Every piece of their stack had US exposure:

Application hosting: 47 production applications running on US-controlled infrastructure, even those in 'EU regions'

Database replication: Automated backups were crossing jurisdictional boundaries, with metadata stored on US servers

Third-party services: Monitoring, error tracking, and analytics all flowing through US-based SaaS tools

DNS and CDN: Traffic routing through US-controlled networks, creating logs subject to CLOUD Act requests

Support channels: All technical support routed through US-based teams with full system access

The technical debt was significant. Most applications assumed US-centric infrastructure patterns. Database connections were hardcoded. Deployment scripts referenced specific US availability zones. Moving this wasn't just about changing providers - it required architectural changes.

Performance was another concern. Their largest e-commerce client served customers across 12 EU countries. Moving from globally distributed US infrastructure to EU-only hosting could impact latency for users in southern and eastern Europe.

The approach we took and why

We designed a phased migration that prioritized the highest-risk client applications first, while maintaining performance standards.

Phase 1: Move the three highest-value enterprise clients to isolated private cloud infrastructure within EU jurisdiction

Phase 2: Migrate remaining production applications in order of compliance sensitivity

Phase 3: Replace US-based tooling with EU alternatives or self-hosted solutions

Instead of a direct lift-and-shift, we rebuilt critical applications using infrastructure patterns designed for data sovereignty. This meant:

Single-jurisdiction deployments with no cross-border replication

EU-only CDN and DNS to prevent traffic from touching US networks

Self-hosted monitoring and analytics to eliminate third-party data sharing

Documented data flows to prove compliance during client audits

We chose this approach because simple 'EU regions' from US providers don't solve CLOUD Act exposure. The parent company remains subject to US jurisdiction, regardless of where data physically sits.

Implementation details with specifics

We built the new private cloud infrastructure across three EU data centers in Amsterdam, Frankfurt, and Paris. Each client got isolated environments with dedicated resources.

Application layer:

Containerized applications using Kubernetes clusters with EU-only worker nodes

Load balancers configured with geographic restrictions preventing US routing

Redis clusters for session storage, replicated only within EU boundaries

Custom deployment pipelines that validate data sovereignty before promotion

Database architecture:

PostgreSQL clusters with synchronous replication between Amsterdam and Frankfurt

Encrypted backups stored exclusively in EU-controlled storage

Database logs and metadata isolated from US-accessible systems

Point-in-time recovery tested to ensure no data leaks during restoration

Network isolation:

VPN tunnels between data centers using EU-managed certificates

DNS resolution through EU-based recursive resolvers

CDN edge nodes restricted to EU locations with traffic steering policies

Network monitoring that alerts on any unexpected geographic routing

Monitoring and observability:

This was the most complex piece. We replaced US-based SaaS tools with self-hosted alternatives:

Prometheus and Grafana for metrics and alerting

ELK stack for log aggregation and analysis

Custom error tracking using Sentry deployed within our infrastructure

Uptime monitoring from multiple EU vantage points

Each monitoring component was configured to store data exclusively within EU boundaries. We built dashboards that proved data sovereignty compliance in real-time.

The migration itself used a blue-green deployment pattern. We built the entire new environment, migrated data during maintenance windows, then switched DNS once we verified everything worked correctly.

Results with real numbers

The migration took 6 weeks for the complete portfolio of applications. Here's what changed:

Performance impact:

Average TTFB increased from 89ms to 124ms (39% slower)

P95 response times went from 340ms to 445ms

Page load times increased by an average of 180ms across all applications

Cost changes:

Infrastructure costs increased 34% - from €4,200/month to €5,630/month

Migration project cost €28,000 in engineering time and consulting

Ongoing operational overhead added roughly 8 hours per week

Reliability improvements:

Uptime improved from 99.7% to 99.94% due to better architectural patterns

Mean time to resolution dropped from 47 minutes to 23 minutes

Zero compliance incidents since migration (compared to 3 audit findings previously)

Business impact:

Retained €180,000 in annual recurring revenue from enterprise clients

Won two new healthcare clients specifically due to data sovereignty guarantees

Reduced legal review time for new enterprise deals from 6 weeks to 2 weeks

The performance impact was noticeable but acceptable. Users in northern Europe actually saw improved response times. Southern and eastern European users experienced the latency increase, but conversion rates remained stable.

More importantly, the agency could now guarantee that client data never touched US-controlled infrastructure, eliminating CLOUD Act exposure completely.

What we'd do differently next time

The migration went smoothly overall, but we learned several lessons that would speed up future projects.

Start with network architecture: We underestimated how long it would take to properly configure geographic routing. Building the network isolation first would have prevented several rollbacks during testing.

Performance baseline everything: We should have measured performance more granularly before migration. Some of the latency increases came from suboptimal database connection pooling, not geographic distance.

Plan for monitoring gaps: The week between shutting down US-based monitoring and getting EU alternatives fully operational created dangerous blind spots. Next time, we'd run parallel monitoring during the entire transition.

Test compliance tooling earlier: Several client audit tools couldn't properly validate the new infrastructure configuration. We spent extra time documenting data flows that should have been automated from day one.

Budget for application refactoring: About 20% of applications needed more code changes than expected. Features that worked fine with US infrastructure patterns broke in the sovereignty-focused environment.

The biggest lesson: zero downtime migration is possible, but requires more upfront planning when crossing jurisdictional boundaries. Data sovereignty isn't just about server location - it touches every part of your architecture.

Long-term outcomes and lessons learned

Six months later, the move to private cloud infrastructure has paid off beyond compliance requirements.

The agency's sales team can now confidently pursue enterprise deals in highly regulated industries. They've won contracts with two major banks and a government agency that specifically required EU-only data processing.

Client retention is higher. Enterprise customers appreciate the transparency around data handling. The agency provides quarterly compliance reports showing exactly where data flows and which systems touch sensitive information.

Operationally, the team has become more disciplined about infrastructure management. Self-hosting monitoring and analytics forced them to understand their applications more deeply. They catch performance issues faster and resolve them with more precision.

The performance trade-offs stabilized once applications were properly tuned for the new environment. Some clients actually see better response times now because the dedicated private infrastructure doesn't compete with noisy neighbors.

Most importantly, they eliminated a major business risk. CLOUD Act requests are unpredictable and often come with gag orders preventing notification. By moving to genuine EU-based private cloud infrastructure, the agency removed this uncertainty completely.

The initial cost increase has been offset by higher-value client contracts and improved operational efficiency. What started as a compliance requirement became a competitive advantage in the enterprise market.

For digital agencies serving enterprise clients, data sovereignty isn't optional anymore. The question isn't whether to move away from US-controlled infrastructure, but how quickly you can do it without disrupting existing business operations.

---

### Measuring queue congestion and job delays in high availability infrastructure

URL: https://binadit.com/blog/measuring-queue-congestion-job-delays-high-availability-infrastructure
Category: Reliability
Author: Binadit Tech Team
Published: 2026-06-05T09:12:52+02:00

> We stress-tested three different queue architectures under realistic load patterns to understand exactly where bottlenecks appear and how they affect application performance. The results show why most queue monitoring strategies miss the problems that matter.

The queue performance question and why it matters commercially

Queue systems handle everything from email delivery to payment processing, but their performance characteristics under load remain poorly understood. When queues congest, the symptoms appear everywhere: delayed notifications, sluggish user interactions, and revenue-critical processes that stall.

A SaaS platform we work with discovered this during a product launch. Their queue appeared healthy in monitoring dashboards, but users reported delayed email confirmations and slow checkout processes. The queue wasn't failing, it was degrading in ways their metrics couldn't capture.

This performance gap costs businesses directly. Each delayed notification reduces user engagement. Slow payment processing abandons revenue. Queue congestion that takes 5 minutes to detect and 10 minutes to resolve can cost an e-commerce platform thousands in lost transactions.

We decided to measure queue performance under realistic conditions to understand where bottlenecks actually appear and how monitoring systems can detect them before they impact users.

Methodology: testing three queue architectures under load

We tested three common queue configurations that represent typical production deployments:

Redis-based queue: Single Redis instance with Laravel queue workers

Database queue: PostgreSQL-backed queue with multiple consumers

RabbitMQ cluster: Three-node cluster with persistence enabled

Each test used identical hardware: 4 CPU cores, 8GB RAM, NVMe storage. Network latency between components stayed under 1ms to isolate queue-specific performance.

The load profile simulated real application patterns:

Baseline: 100 jobs/second, each taking 50-200ms to process

Burst load: 500 jobs/second for 2-minute periods

Sustained load: 300 jobs/second for 15 minutes

Mixed workload: 70% lightweight jobs (10ms), 30% heavy jobs (500ms)

We measured queue depth, processing latency, and system resource utilization every second. Each test ran 10 times to account for variability.

Job types included typical application tasks: sending emails, processing images, updating search indices, and generating reports. This mix reflects what most applications actually queue.

Results: where performance breaks down under pressure

The results revealed significant differences between queue architectures, especially during burst periods.
MetricRedis QueueDatabase QueueRabbitMQP50 latency (baseline)45ms78ms52msP95 latency (baseline)120ms245msP99 latency (baseline)180ms890ms165msP50 latency (burst)340ms1,240ms89msP95 latency (burst)1,100ms4,500ms280msMax queue depth2,400 jobs8,900 jobs1,200 jobsRecovery time4.2 minutes12.8 minutes1.8 minutes

During baseline load, all systems performed acceptably. Redis showed the lowest median latency at 45ms, while the database queue struggled with P99 latencies reaching 890ms.

Burst conditions exposed critical differences. RabbitMQ maintained reasonable performance with P95 latencies staying under 280ms. Redis performance degraded significantly, with median latency jumping to 340ms. The database queue essentially failed, with median processing times exceeding 1.2 seconds.

Queue depth measurements revealed another pattern. Database queues accumulated jobs faster than they could process them, reaching 8,900 queued jobs during burst tests. RabbitMQ's flow control mechanisms kept queue depth manageable, never exceeding 1,200 jobs.

Recovery patterns differed dramatically. After burst load ended, RabbitMQ returned to baseline performance within 1.8 minutes. Redis took 4.2 minutes to clear accumulated jobs. The database queue required 12.8 minutes to process its backlog.

Memory usage patterns also varied. Redis consumed 2.1GB during peak load, mostly for job storage. RabbitMQ used 1.4GB with its memory management optimizations. The database queue stayed within normal database memory limits but generated significant I/O load.

Analysis: what these numbers mean in production

These performance characteristics directly impact user experience and business operations. A 340ms median queue delay during traffic spikes means email confirmations take longer, search indices update slowly, and background tasks accumulate.

The database queue's 1.2-second median latency during bursts makes it unsuitable for user-facing operations. Tasks like sending password reset emails or processing payment confirmations become noticeably slow.

Queue depth accumulation creates cascading problems. When 8,900 jobs accumulate in a database queue, priority jobs wait behind lower-priority tasks. Critical operations like payment processing get delayed by routine maintenance tasks.

Recovery time matters for operational planning. A system that takes 12.8 minutes to clear its backlog means problems persist long after traffic spikes end. Users continue experiencing delays even when load returns to normal.

These patterns explain why monitoring infrastructure correctly requires understanding queue-specific metrics, not just general system health.

Resource utilization revealed another insight. CPU usage stayed reasonable across all systems, but I/O patterns differed significantly. Database queues generated 4x more disk operations than Redis or RabbitMQ, creating bottlenecks that weren't immediately obvious.

The mixed workload tests showed how job diversity affects performance. When 30% of jobs took 10x longer to process, all systems struggled with task scheduling. Long-running jobs blocked short tasks, even with multiple workers.

Caveats and what we'd measure differently

These tests used controlled conditions that don't fully represent production complexity. Real applications face network latency, database contention, and resource competition from other services.

We tested single-point-of-failure configurations for Redis and database queues. Production deployments typically include clustering or failover mechanisms that add overhead but improve reliability.

Job processing time remained artificial. Real applications show more variability, with some tasks taking seconds or minutes. This variance would amplify the performance differences we measured.

Network conditions stayed optimal throughout testing. Production environments experience packet loss, bandwidth limits, and latency spikes that affect queue performance differently.

We didn't test failure scenarios. How each system behaves during worker crashes, memory pressure, or disk space exhaustion requires separate analysis.

The load patterns, while realistic, don't capture every application profile. Services with predominantly read-heavy or write-heavy workloads would show different bottlenecks.

For future testing, we'd include network latency simulation, longer test durations to capture performance degradation over time, and failure injection to understand recovery behaviors.

We'd also measure different job priority schemes and worker scaling patterns to understand how queue systems handle operational complexity.

Takeaways for reliable queue infrastructure

Queue performance varies dramatically under load, and the differences matter for user experience. Systems that work fine during normal traffic can become bottlenecks during growth periods or traffic spikes.

Monitoring queue depth alone misses critical performance degradation. Latency percentiles reveal problems before queues fail completely. P95 and P99 metrics often show performance issues while median latency still looks acceptable.

Recovery time matters as much as peak performance. A queue system that takes 10+ minutes to clear its backlog extends the impact of any traffic spike or operational issue.

Architecture choices have long-term implications. Database queues might seem simple to implement, but their performance characteristics make them unsuitable for applications that need consistent response times.

Understanding these patterns helps with scaling web applications before performance becomes a user-visible problem.

Resource planning requires understanding the full performance profile, not just average conditions. Systems need capacity for burst loads and recovery periods, not just steady-state operations.

Want these kinds of numbers for your own stack? Request a performance audit.

---

### How to scale WooCommerce infrastructure without downtime

URL: https://binadit.com/blog/scale-woocommerce-ecommerce-infrastructure-without-downtime
Category: Commerce
Author: Binadit Tech Team
Published: 2026-06-04T09:35:13+02:00

> Scale your WooCommerce store to handle traffic spikes and growth without experiencing downtime. This technical guide covers load balancing, database optimization, and proven scaling strategies that protect revenue during critical periods.

What you'll achieve and why it matters

You'll build a scalable WooCommerce architecture that handles traffic growth and sudden spikes without downtime. This matters because every minute of downtime during peak shopping periods directly impacts revenue, and proper scaling prevents performance degradation that leads to abandoned carts.

Prerequisites and assumptions

This guide assumes you have:

Root access to your current WooCommerce server

A WooCommerce store running on Linux (Ubuntu/CentOS)

Basic command line experience

Access to provision additional servers or cloud instances

A domain with configurable DNS settings

We'll work with a typical LAMP stack setup, though the principles apply to other configurations. You should have at least 30 minutes of maintenance window for initial setup, though most steps can be done without affecting live traffic.

Step-by-step implementation

Step 1: Set up a load balancer with health checks

Start by configuring Nginx as a load balancer. This distributes traffic across multiple WooCommerce instances and provides automatic failover.

Install Nginx on your load balancer server:

sudo apt update
sudo apt install nginx -y
sudo systemctl enable nginx

Create the load balancer configuration:

sudo nano /etc/nginx/sites-available/woocommerce-lb

Add this configuration:

upstream woocommerce_backend {
server 10.0.1.10:80 max_fails=3 fail_timeout=30s;
server 10.0.1.11:80 max_fails=3 fail_timeout=30s backup;
}

server {
listen 80;
server_name yourstore.com www.yourstore.com;

location / {
proxy_pass http://woocommerce_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

proxy_connect_timeout 5s;
proxy_send_timeout 10s;
proxy_read_timeout 30s;
}

location /health {
access_log off;
return 200 "healthy";
add_header Content-Type text/plain;
}
}

Enable the configuration:

sudo ln -s /etc/nginx/sites-available/woocommerce-lb /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

Step 2: Configure database replication for read scaling

Set up MySQL master-slave replication to distribute database load. This prevents the database from becoming a bottleneck as traffic increases.

On your master database server, edit the MySQL configuration:

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

Add these settings under the [mysqld] section:

server-id = 1
log-bin = mysql-bin
binlog-do-db = your_woocommerce_db
bind-address = 0.0.0.0

Restart MySQL and create a replication user:

sudo systemctl restart mysql
mysql -u root -p

In the MySQL console:

CREATE USER 'replica'@'%' IDENTIFIED BY 'strong_password';
GRANT REPLICATION SLAVE ON *.* TO 'replica'@'%';
FLUSH PRIVILEGES;
SHOW MASTER STATUS;

Note the File and Position values from the output. On your slave server, configure replication:

CHANGE MASTER TO
MASTER_HOST='10.0.1.10',
MASTER_USER='replica',
MASTER_PASSWORD='strong_password',
MASTER_LOG_FILE='mysql-bin.000001',
MASTER_LOG_POS=154;

START SLAVE;
SHOW SLAVE STATUS\G;

Step 3: Implement Redis object caching

Configure Redis to handle WordPress object caching, reducing database queries and improving response times under load.

Install Redis on a dedicated server:

sudo apt install redis-server -y
sudo systemctl enable redis-server

Configure Redis for production use:

sudo nano /etc/redis/redis.conf

Update these settings:

maxmemory 2gb
maxmemory-policy allkeys-lru
bind 0.0.0.0
requireauth your_redis_password
tcp-keepalive 300

Restart Redis:

sudo systemctl restart redis-server

On your WooCommerce servers, install the Redis Object Cache plugin and add this to wp-config.php:

define('WP_REDIS_HOST', '10.0.1.20');
define('WP_REDIS_PASSWORD', 'your_redis_password');
define('WP_REDIS_PORT', 6379);
define('WP_REDIS_DATABASE', 0);

Enable object caching through the WordPress admin or with WP-CLI:

wp redis enable --allow-root

Step 4: Configure session storage for multi-server setup

When running multiple WooCommerce instances, you need centralized session storage to maintain cart contents and user sessions across servers.

Configure PHP to use Redis for sessions. Add this to your PHP configuration:

sudo nano /etc/php/8.1/fpm/php.ini

Update these values:

session.save_handler = redis
session.save_path = "tcp://10.0.1.20:6379?auth=your_redis_password&database=1"

Restart PHP-FPM:

sudo systemctl restart php8.1-fpm

Step 5: Set up file synchronization

Ensure uploaded files and plugin updates sync across all WooCommerce instances. Use rsync with inotify for real-time synchronization.

Install rsync and inotify tools:

sudo apt install rsync inotify-tools -y

Create a sync script:

sudo nano /usr/local/bin/wp-sync.sh

#!/bin/bash
SOURCE_DIR="/var/www/html/wp-content/uploads/"
DEST_SERVERS=("10.0.1.11" "10.0.1.12")
RSYNC_OPTIONS="-avz --delete"

for server in "${DEST_SERVERS[@]}"; do
rsync $RSYNC_OPTIONS $SOURCE_DIR root@$server:$SOURCE_DIR
done

Make it executable and create a systemd service:

sudo chmod +x /usr/local/bin/wp-sync.sh
sudo nano /etc/systemd/system/wp-sync.service

[Unit]
Description=WordPress File Sync
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/wp-sync.sh
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl enable wp-sync.service
sudo systemctl start wp-sync.service

Step 6: Implement auto-scaling with monitoring

Set up monitoring and automatic scaling triggers based on server metrics. This prevents manual intervention during traffic spikes.

Install and configure Netdata for real-time monitoring:

bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait

Create a simple auto-scaling script that monitors CPU and memory:

sudo nano /usr/local/bin/autoscale.sh

#!/bin/bash
CPU_THRESHOLD=80
MEMORY_THRESHOLD=85

CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
MEMORY_USAGE=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100.0}')

if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )) || (( $MEMORY_USAGE > $MEMORY_THRESHOLD )); then
echo "High load detected. CPU: $CPU_USAGE%, Memory: $MEMORY_USAGE%"
# Add your server provisioning logic here
# Example: AWS CLI commands to launch new instances
# aws ec2 run-instances --image-id ami-xxx --instance-type t3.medium
fi

Schedule the script to run every minute:

sudo crontab -e
* * * * * /usr/local/bin/autoscale.sh >> /var/log/autoscale.log 2>&1

Verification: confirming everything works

Test your scaled ecommerce infrastructure to ensure it handles load correctly and maintains session consistency.

Load balancer verification

Check that the load balancer distributes requests properly:

curl -H "Host: yourstore.com" http://your-load-balancer-ip/
curl -I -H "Host: yourstore.com" http://your-load-balancer-ip/health

You should see a 200 response from the health check. Test failover by stopping one backend server:

sudo systemctl stop nginx # on one backend server

Verify the site remains accessible and traffic routes to the remaining server.

Database replication verification

Test that data replicates correctly between master and slave:

On the master database:

mysql -u root -p your_woocommerce_db
INSERT INTO wp_posts (post_title, post_content, post_status) VALUES ('Test Replication', 'Content', 'publish');

On the slave database:

mysql -u root -p your_woocommerce_db
SELECT * FROM wp_posts WHERE post_title = 'Test Replication';

The record should appear on the slave within seconds.

Redis caching verification

Check that object caching works properly:

redis-cli -h 10.0.1.20 -a your_redis_password
KEYS wp:*

You should see WordPress cache keys. Monitor cache hit rates in your WordPress admin under Tools > Redis.

Session persistence verification

Test that sessions persist across different backend servers:

Add items to cart on your WooCommerce store

Note your session ID from browser developer tools

Refresh the page multiple times

Verify cart contents remain consistent

Check Redis for session data:

redis-cli -h 10.0.1.20 -a your_redis_password -n 1
KEYS *

Performance metrics

Measure the improvement in response times and capacity:

ab -n 1000 -c 50 http://yourstore.com/
wrk -t12 -c400 -d30s http://yourstore.com/shop/

Compare these results to your pre-scaling baseline. You should see:

Reduced average response times under load

Higher requests per second capacity

Lower error rates during traffic spikes

Common pitfalls to avoid

Database connection exhaustion: Monitor your max_connections setting and adjust based on the number of web servers. Each server typically needs 10-50 connections.

File sync delays: Large media uploads can cause sync lag. Consider using shared storage like NFS or object storage for high-volume sites.

SSL termination inconsistency: If using HTTPS, terminate SSL at the load balancer level to avoid certificate management across multiple servers.

Cache invalidation issues: Ensure cache purging works across all instances when content changes. Use Redis pub/sub or a centralized cache invalidation strategy.

Next steps and related reading

Once your basic scaling infrastructure is operational, consider these additional improvements:

Implement CDN integration for static asset delivery

Set up automated backup strategies across your distributed environment

Configure detailed application performance monitoring

Plan for database scaling with read replicas in multiple regions

For deeper optimization, review our guides on Redis configuration for high-traffic WooCommerce stores and measuring database performance degradation.

Consider implementing more advanced patterns like circuit breakers and graceful degradation as your traffic grows beyond this initial scaling setup.

Scaling WooCommerce infrastructure reliably

This scaling approach gives you the foundation to handle traffic growth without downtime. The combination of load balancing, database replication, Redis caching, and proper session management creates a robust ecommerce infrastructure that maintains performance under load.

Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.

---

### How we migrated an ecommerce platform to HTTP/3 and cut page load times by 47%

URL: https://binadit.com/blog/http3-migration-ecommerce-infrastructure-performance-optimization
Category: Performance
Author: Binadit Tech Team
Published: 2026-06-03T10:18:59+02:00

> A high-traffic fashion retailer was losing conversions to slow page loads during peak hours. Moving from HTTP/1.1 to HTTP/3 through strategic load balancer upgrades transformed their infrastructure performance and delivered measurable business results.

The situation: a growing fashion retailer hitting HTTP limits

We worked with a European fashion retailer processing €2.8 million in monthly revenue through their WooCommerce platform. During peak shopping periods, especially evening hours and weekend sales, their site would slow to a crawl. Checkout abandonment spiked to 31% during these periods, compared to their baseline of 18%.

The company had grown from handling 500 concurrent users to over 2,400 during peak periods. Their infrastructure had scaled vertically and horizontally, but something fundamental was wrong. Page load times that stayed under 2.1 seconds during off-peak hours would balloon to 8-12 seconds during traffic spikes.

Their existing setup used Nginx load balancers in front of multiple application servers, with Redis for session storage and a well-optimized MySQL cluster. The architecture looked sound on paper, but the numbers told a different story.

Peak traffic patterns showed the real problem: users were loading product pages with 47 individual assets on average. Each page required dozens of separate HTTP connections. Under HTTP/1.1, this created a massive bottleneck that no amount of server scaling could fix.

What we found during the audit

Our infrastructure audit revealed several critical issues with their existing ecommerce infrastructure. The load balancers were correctly distributing traffic, but the HTTP protocol itself had become the constraint.

HTTP/1.1 allows only 6-8 concurrent connections per domain in most browsers. For a typical product page with 47 assets, this meant connections were queuing. We measured head-of-line blocking delays of 1.2-3.4 seconds during peak periods. CSS files would block JavaScript loading, which blocked image rendering.

The Redis configuration was optimized, database queries were fast, and server response times averaged 180ms. But total page load times reached 8+ seconds because of protocol-level bottlenecks.

We also discovered that their existing load balancer configuration was terminating SSL connections and re-establishing them to backend servers. This added an extra 40-80ms latency per request, multiplied across dozens of assets per page.

Connection reuse statistics showed the real impact. Under HTTP/1.1, each page load required establishing 12-15 separate TCP connections. During peak traffic, connection establishment time alone added 800ms to page loads.

The infrastructure monitoring showed CPU usage staying reasonable on all servers, but network connection pools were constantly maxed out. This wasn't a capacity problem. It was an efficiency problem.

The approach we took and why

Rather than continue scaling horizontally, we decided to upgrade the protocol layer. HTTP/2 and HTTP/3 solve head-of-line blocking through multiplexing and parallel streams. Multiple requests can share single connections, eliminating the connection queue bottleneck.

HTTP/2 uses binary framing and stream multiplexing over TCP. A single connection can handle dozens of concurrent requests without blocking. HTTP/3 goes further by running over QUIC instead of TCP, eliminating head-of-line blocking at the transport layer as well.

We planned a staged migration: first HTTP/2, then HTTP/3. This approach would let us measure improvements at each step and avoid the complexity of jumping directly to HTTP/3.

The load balancer upgrade required careful planning. We needed to maintain compatibility with older clients while enabling newer protocols for supported browsers. The goal was transparent protocol negotiation without breaking existing functionality.

We also planned to optimize asset delivery for the new protocols. HTTP/2 and HTTP/3 change the optimal patterns for bundling CSS and JavaScript. Domain sharding becomes counterproductive when multiplexing works better with fewer domains.

Implementation details with specifics

We upgraded their Nginx load balancers to version 1.25.1 and enabled HTTP/2 with specific tuning for their traffic patterns. The configuration focused on optimizing stream concurrency and buffer sizes for their asset-heavy pages.

Key Nginx HTTP/2 configuration changes:

http2_max_concurrent_streams 256;
http2_chunk_size 8k;
http2_body_preread_size 64k;
http2_idle_timeout 60s;

We increased concurrent streams from the default 128 to 256 because their pages averaged 47 assets. The chunk size was reduced to 8k to improve multiplexing efficiency for smaller assets like icons and thumbnails.

For HTTP/3, we compiled Nginx with BoringSSL support and enabled QUIC:

listen 443 quic reuseport;
http3 on;
http3_hq on;
add_header Alt-Svc 'h3=":443"; ma=86400';

The Alt-Svc header tells browsers that HTTP/3 is available. Browsers can then upgrade subsequent requests to use QUIC instead of TCP.

We also reconfigured their asset delivery strategy. Under HTTP/1.1, they had been concatenating CSS and JavaScript files to reduce request counts. With HTTP/2 multiplexing, this actually hurts performance because it prevents selective loading and caching.

We split their monolithic CSS bundle into 6 smaller files based on page type and functionality. JavaScript was similarly split into critical and non-critical modules. This allowed browsers to start rendering while non-essential assets loaded in parallel streams.

SSL certificate configuration required updating to support the new protocols while maintaining compatibility. We deployed certificates with proper ALPN negotiation for protocol selection.

Results with real numbers

The HTTP/2 migration delivered immediate improvements. Average page load times during peak traffic dropped from 8.2 seconds to 4.8 seconds, a 41% improvement. More importantly, 95th percentile load times dropped from 12.3 seconds to 6.4 seconds.

Checkout abandonment rates during peak periods fell from 31% to 23%, still higher than off-peak but dramatically better than before. We estimated this change alone was worth €47,000 in additional monthly revenue.

The HTTP/3 rollout delivered additional gains. Final page load times averaged 4.3 seconds during peak traffic, representing a 47% total improvement from the original HTTP/1.1 baseline. Connection establishment time dropped to nearly zero for repeat visitors due to QUIC's connection migration features.

Server resource utilization also improved significantly. CPU usage on load balancers decreased by 18% despite handling the same traffic volume. Memory usage for connection tracking dropped by 28% because fewer individual connections were required.

Network efficiency metrics showed the clearest improvement. Total bytes transferred remained the same, but connection count per page load dropped from an average of 14 connections to just 2.1 connections. This reduction in connection overhead improved performance across their entire infrastructure.

Time to first byte (TTFB) improved from 340ms average to 280ms average. While this seems modest, the improvement cascaded through all subsequent asset loads due to better connection reuse.

What we'd do differently next time

If we repeated this migration, we would implement HTTP/3 push priorities more aggressively from the start. We were conservative about server push because of mixed browser support, but the browsers that do support it showed measurably better performance.

The asset bundling optimization took longer than expected. We would plan more time upfront to properly analyze which assets benefit from bundling versus separate loading under HTTP/2. The optimal strategy varies significantly based on caching patterns and user behavior.

We would also implement more granular monitoring earlier in the process. Protocol-level metrics like stream utilization and connection reuse patterns proved crucial for optimization, but our initial monitoring focused on traditional HTTP metrics.

Connection coalescing could have been implemented more effectively. We discovered that multiple subdomains were preventing optimal connection reuse even under HTTP/2. Consolidating to a single domain for static assets would have delivered additional performance gains.

The zero downtime migration approach we used worked well, but we would build in more time for A/B testing different protocol configurations before full rollout.

Finally, we would coordinate the protocol upgrade with their CDN provider earlier in the process. The CDN was already supporting HTTP/2, but optimizing the origin-to-CDN connection for HTTP/2 multiplexing required additional configuration that we addressed later.

How protocol upgrades transform ecommerce infrastructure

This migration demonstrated that ecommerce infrastructure performance often hits protocol-level limits before server capacity limits. Traditional scaling approaches focus on adding servers or optimizing database queries, but connection management becomes the real bottleneck for asset-heavy sites.

HTTP/2 and HTTP/3 change fundamental assumptions about optimal asset delivery. Techniques like domain sharding and aggressive file concatenation that improved HTTP/1.1 performance actually hurt performance under newer protocols.

The business impact was immediate and measurable. Faster page loads directly translated to lower abandonment rates and higher conversion. For ecommerce platforms, protocol optimization often delivers better ROI than traditional infrastructure scaling.

Load balancer configuration becomes more critical under HTTP/2 and HTTP/3. The load balancer must efficiently handle multiplexed streams and connection migration while maintaining session affinity where needed.

Modern browsers increasingly default to HTTP/2 when available, making protocol support essential rather than optional. Sites that don't support newer protocols are at a measurable disadvantage during peak traffic periods.

The upgrade also positioned their infrastructure for future growth. HTTP/3's connection migration features will become increasingly valuable as their mobile traffic grows and users move between networks.

Facing a similar challenge? Tell us about your setup and we will outline an approach.

---

## Tutorials

### Configure PostgreSQL 17 SSL encryption and certificate-based authentication

URL: https://binadit.com/tutorials/configure-postgresql-ssl-encryption-and-authentication
Category: databases
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up a private CA, issue server and client certificates, enforce TLS 1.2+ with strong ciphers, and configure pg_hba.conf for mutual TLS client certificate authentication in PostgreSQL 17.

What this solves

By default, PostgreSQL connections are unencrypted unless explicitly configured otherwise, and password authentication over the network exposes credentials to interception. This tutorial covers generating a private certificate authority, issuing server and client certificates, enforcing TLS-only connections, and configuring mutual TLS so clients authenticate with certificates instead of passwords.

This setup is appropriate for production databases handling sensitive data, multi-tenant environments, and compliance-driven infrastructure where password-based authentication is insufficient.

Warning: Test all changes on a staging instance first. Misconfigured pg_hba.conf or SSL settings can lock out all connections, including local ones.

Step-by-step configuration

Install PostgreSQL 17 and OpenSSL

Install PostgreSQL 17 and the OpenSSL tools needed to build a certificate authority.

sudo apt update
sudo apt install -y postgresql-17 postgresql-client-17 openssl

sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo dnf install -y postgresql17-server postgresql17 openssl
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
sudo systemctl enable --now postgresql-17

Create a private certificate authority

The CA signs both the server certificate and every client certificate. Keep the CA private key offline or heavily restricted after issuing certificates.

sudo mkdir -p /etc/postgresql/ssl/ca
cd /etc/postgresql/ssl/ca
sudo openssl genrsa -aes256 -out ca.key 4096
sudo openssl req -new -x509 -days 3650 -key ca.key -sha256 \
-out ca.crt \
-subj "/C=NL/O=Example Corp/CN=Example Corp PostgreSQL CA"

You will be prompted for a passphrase to encrypt the CA private key. Use a strong passphrase and store it in a secrets manager, not on the filesystem.

Generate the server certificate

Create a key and certificate signing request (CSR) for the PostgreSQL server. The common name must match the hostname clients use to connect.

sudo mkdir -p /etc/postgresql/ssl/server
cd /etc/postgresql/ssl/server
sudo openssl genrsa -out server.key 4096
sudo openssl req -new -key server.key -out server.csr \
-subj "/C=NL/O=Example Corp/CN=db01.example.com"

Sign the CSR with the CA to produce the server certificate, valid for one year.

sudo openssl x509 -req -in server.csr -CA /etc/postgresql/ssl/ca/ca.crt \
-CAkey /etc/postgresql/ssl/ca/ca.key -CAcreateserial \
-out server.crt -days 365 -sha256

Set correct ownership and permissions on certificates

PostgreSQL refuses to start if the server private key is readable by other users. The postgres process owns and reads these files directly, so ownership must belong to the postgres user with restrictive permissions.

sudo chown postgres:postgres /etc/postgresql/ssl/server/server.key /etc/postgresql/ssl/server/server.crt
sudo chmod 600 /etc/postgresql/ssl/server/server.key
sudo chmod 644 /etc/postgresql/ssl/server/server.crt
sudo chown postgres:postgres /etc/postgresql/ssl/ca/ca.crt
sudo chmod 644 /etc/postgresql/ssl/ca/ca.crt

Never use chmod 777. The private key must only be readable by the postgres user. Granting broad access here would let any local user impersonate the server or read encrypted traffic keys.

Enable SSL in postgresql.conf

Point PostgreSQL at the certificate files and turn on SSL support.

ssl = on
ssl_cert_file = '/etc/postgresql/ssl/server/server.crt'
ssl_key_file = '/etc/postgresql/ssl/server/server.key'
ssl_ca_file = '/etc/postgresql/ssl/ca/ca.crt'
ssl_prefer_server_ciphers = on

Note: On AlmaLinux and Rocky, the config file is at /var/lib/pgsql/17/data/postgresql.conf unless you changed the data directory during initdb.

Enforce strong TLS protocol versions and cipher suites

Restrict connections to TLS 1.2 and above, and set an explicit cipher list to block weak algorithms.

ssl_min_protocol_version = 'TLSv1.2'
ssl_max_protocol_version = 'TLSv1.3'
ssl_ciphers = 'HIGH:!aNULL:!MD5:!3DES:!RC4'
ssl_ecdh_curve = 'prime256v1'

Restart PostgreSQL to apply the SSL settings.

sudo systemctl restart postgresql

Issue a client certificate for mutual TLS

Each client certificate's common name must exactly match the PostgreSQL role it authenticates as. Generate one per application or admin user.

sudo mkdir -p /etc/postgresql/ssl/clients
cd /etc/postgresql/ssl/clients
sudo openssl genrsa -out app_user.key 4096
sudo openssl req -new -key app_user.key -out app_user.csr \
-subj "/C=NL/O=Example Corp/CN=app_user"
sudo openssl x509 -req -in app_user.csr -CA /etc/postgresql/ssl/ca/ca.crt \
-CAkey /etc/postgresql/ssl/ca/ca.key -CAcreateserial \
-out app_user.crt -days 365 -sha256

Set restrictive ownership before distributing the client key to the application host.

sudo chmod 600 app_user.key
sudo chmod 644 app_user.crt

Create the matching PostgreSQL role

The role name must exactly match the certificate common name for cert authentication to succeed.

sudo -u postgres psql -c "CREATE ROLE app_user LOGIN;"
sudo -u postgres psql -c "GRANT CONNECT ON DATABASE appdb TO app_user;"

Configure pg_hba.conf for certificate authentication

Set the authentication method to cert and require clientcert verification so PostgreSQL rejects any connection lacking a CA-signed client certificate.

# TYPE DATABASE USER ADDRESS METHOD
hostssl appdb app_user 203.0.113.0/24 cert clientcert=verify-full
hostssl all all 0.0.0.0/0 reject

The clientcert=verify-full option checks both that the certificate is signed by the trusted CA and that its common name matches the connecting role. Reload PostgreSQL to apply the rules.

sudo systemctl reload postgresql

Note: Order matters in pg_hba.conf. Rules are evaluated top to bottom, and the first match wins.

Verify your setup

Copy the CA certificate and client certificate and key to the client machine, then connect with sslmode=verify-full to validate both the server identity and the client certificate.

psql "host=db01.example.com dbname=appdb user=app_user \
sslmode=verify-full \
sslrootcert=/etc/postgresql/ssl/ca/ca.crt \
sslcert=/etc/postgresql/ssl/clients/app_user.crt \
sslkey=/etc/postgresql/ssl/clients/app_user.key"

Confirm the connection is using SSL and check the negotiated protocol and cipher.

psql -c "SELECT ssl, version, cipher FROM pg_stat_ssl JOIN pg_stat_activity USING (pid) WHERE pid = pg_backend_pid();"

Confirm password authentication is actually rejected for the same user without a certificate.

psql "host=db01.example.com dbname=appdb user=app_user sslmode=require"

This should fail with an authentication error since no client certificate was supplied.

Rotating certificates without downtime

Certificates expire, and rotating the server certificate should not require a restart. PostgreSQL reloads SSL context on pg_reload_conf() as long as the file paths in postgresql.conf stay the same.

sudo openssl req -new -key /etc/postgresql/ssl/server/server.key \
-out /etc/postgresql/ssl/server/server_renew.csr \
-subj "/C=NL/O=Example Corp/CN=db01.example.com"
sudo openssl x509 -req -in /etc/postgresql/ssl/server/server_renew.csr \
-CA /etc/postgresql/ssl/ca/ca.crt -CAkey /etc/postgresql/ssl/ca/ca.key \
-CAcreateserial -out /etc/postgresql/ssl/server/server.crt.new -days 365 -sha256
sudo mv /etc/postgresql/ssl/server/server.crt.new /etc/postgresql/ssl/server/server.crt
sudo chown postgres:postgres /etc/postgresql/ssl/server/server.crt
sudo -u postgres psql -c "SELECT pg_reload_conf();"

For client certificates, issue the replacement before the old one expires, distribute it to the application, and update the application config to point at the new file with a rolling deploy. Since both certificates are trusted by the same CA during the transition window, there is no need to disable authentication.

Set a calendar reminder or monitoring check for certificate expiry dates well before renewal is due. If you already run PostgreSQL streaming replication with PgBouncer, apply the same server certificate rotation on standby nodes to keep replication connections consistent.

Combining with connection pooling

If you terminate connections through PgBouncer, configure PgBouncer to also require TLS between the application and the pooler, and between the pooler and PostgreSQL. See Configure PostgreSQL 17 connection pooling with PgBouncer for high availability for the pooler-side setup, then apply the sslmode=verify-full settings from this tutorial to the pooler's backend connection string.

For broader hardening beyond SSL, including password policies, connection limits, and role separation, see Configure PostgreSQL 17 SSL encryption and advanced security hardening.

Common issues
SymptomCauseFixFATAL: no pg_hba.conf entry for host, SSL offClient connected without sslmode=require or higherAdd sslmode=verify-full to the connection string, confirm hostssl is used in pg_hba.confcould not load private key file, permission deniedserver.key is not readable by the postgres userRun sudo chown postgres:postgres server.key && sudo chmod 600 server.keycertificate verify failed: unable to get local issuer certificateClient does not have the CA certificate, or sslrootcert path is wrongCopy ca.crt to the client and reference it with sslrootcertFATAL: certificate authentication failed for userCertificate common name does not match the PostgreSQL role nameRegenerate the client certificate with CN matching the exact role nameSSL error: sslv3 alert handshake failureClient and server share no compatible cipher or protocol versionCheck ssl_min_protocol_version and ssl_ciphers, update client OpenSSL if outdatedconnection works with sslmode=require but fails with verify-fullServer certificate CN does not match the hostname used to connectReissue server certificate with CN matching the DNS name clients useserver won't start after enabling sslMissing or misnamed certificate files in postgresql.conf pathsVerify file paths with sudo -u postgres psql -c "SHOW ssl_cert_file;" and check the PostgreSQL log

Next steps

Configure PostgreSQL 17 connection pooling with PgBouncer for high availability

Configure PostgreSQL 17 streaming replication for high availability with automatic failover

Configure PostgreSQL 17 SSL encryption and advanced security hardening

Set up Vault as a PKI certificate authority with SSL automation and intermediate CA

Monitor PostgreSQL performance with Prometheus and Grafana dashboards

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: certificate rotation across fleets, monitoring expiry dates, and auditing cipher policy drift after every PostgreSQL minor upgrade. See how we run infrastructure like this for European teams.

---

### Set up HAProxy SSL termination with Let's Encrypt certificates

URL: https://binadit.com/tutorials/setup-haproxy-ssl-termination-with-lets-encrypt
Category: networking
Difficulty: intermediate
Time: ~35 minutes
Author: Binadit Tech Team

> Learn how to terminate SSL/TLS at HAProxy using Let's Encrypt certificates, redirect HTTP to HTTPS, automate renewal with deploy hooks, and harden your cipher suites for production load balancing.

What this solves

HAProxy is a fast, reliable load balancer, but it does not fetch or manage TLS certificates on its own. This tutorial shows you how to obtain free certificates from Let's Encrypt with Certbot, combine them into the PEM bundle HAProxy expects, and wire up automatic renewal so certificates never expire silently.

You will also configure HTTP to HTTPS redirection, backend health checks, load balancing, and modern cipher suites so your termination point is both functional and hardened.

Step-by-step configuration

Install HAProxy

HAProxy handles the TLS termination and forwards decrypted traffic to your backend servers.

sudo apt update
sudo apt install -y haproxy

sudo dnf install -y haproxy

Install Certbot

Certbot requests and renews certificates from Let's Encrypt. Install the standalone plugin so it can bind to port 80 temporarily during issuance.

sudo apt install -y certbot

sudo dnf install -y certbot

Stop HAProxy before issuing certificates

The standalone challenge needs port 80 free. Stop HAProxy temporarily, or skip this step if you plan to use the DNS challenge instead.

sudo systemctl stop haproxy

Obtain a certificate with the standalone challenge

This works when port 80 is reachable from the internet and no other process is bound to it. Replace example.com with your actual domain.

sudo certbot certonly --standalone -d example.com -d www.example.com --agree-tos -m admin@example.com --non-interactive

Note: If HAProxy must stay online, use the DNS challenge instead so port 80 is never touched.

Alternative: obtain a certificate with the DNS challenge

The DNS challenge lets you issue certificates without opening port 80 or stopping HAProxy, and is required for wildcard certificates. This example uses the manual DNS plugin; swap in your DNS provider's Certbot plugin for full automation.

sudo certbot certonly --manual --preferred-challenges dns -d example.com -d '*.example.com' --agree-tos -m admin@example.com

Certbot will print a TXT record to add to your DNS zone. Add it, wait for propagation, then continue the prompt to complete validation.

Build the combined PEM bundle

HAProxy expects a single file containing the certificate, intermediate chain, and private key, in that order. Create a directory to store these bundles and a script to rebuild them after every renewal.

sudo mkdir -p /etc/haproxy/certs
sudo chmod 700 /etc/haproxy/certs

sudo bash -c 'cat /etc/letsencrypt/live/example.com/fullchain.pem /etc/letsencrypt/live/example.com/privkey.pem > /etc/haproxy/certs/example.com.pem'
sudo chmod 600 /etc/haproxy/certs/example.com.pem
sudo chown root:root /etc/haproxy/certs/example.com.pem

Never use chmod 777. The PEM bundle contains your private key. Keep it owned by root with 600 permissions so only the HAProxy process, running as root or via setcap, can read it. Anything more permissive exposes your key to every local user.

Configure HAProxy global and defaults sections

These sections set process-wide behavior and sane connection defaults before you define frontends and backends.

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4096

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 30s
timeout server 30s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

Configure the HTTPS frontend with SSL termination

This frontend binds to port 443, loads the PEM bundle, and forwards traffic to a backend pool. It also sets headers so backend applications know the original request used HTTPS.

frontend https_front
bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
mode http
option forwardfor
http-request set-header X-Forwarded-Proto https
default_backend web_backend

Set up HTTP to HTTPS redirection

All plain HTTP traffic should be redirected to HTTPS rather than served unencrypted. Add a dedicated frontend on port 80 that issues a permanent redirect.

frontend http_front
bind *:80
mode http
http-request redirect scheme https code 301 unless { ssl_fc }
default_backend web_backend

Note: Certbot's standalone challenge also needs port 80. If you keep this redirect frontend running permanently, use the DNS challenge or the HTTP-01 webroot method with an ACL exception for /.well-known/acme-challenge/ instead of stopping HAProxy for renewals.

Allow ACME HTTP-01 renewals without downtime

Add an ACL that lets Certbot's renewal requests through the redirect frontend, so you never need to stop HAProxy again after the initial certificate issuance.

frontend http_front
bind *:80
mode http
acl is_acme_challenge path_beg /.well-known/acme-challenge/
use_backend acme_backend if is_acme_challenge
http-request redirect scheme https code 301 unless { ssl_fc } !is_acme_challenge
default_backend web_backend

backend acme_backend
server certbot 127.0.0.1:8888

Run Certbot's standalone challenge on an internal port so this backend can reach it during renewal, or switch entirely to the DNS challenge for a simpler setup.

Configure the backend with health checks and load balancing

The backend pool defines your application servers, the load balancing algorithm, and active health checks that remove unhealthy servers automatically.

backend web_backend
mode http
balance roundrobin
option httpchk GET /health
http-check expect status 200
default-server inter 5s fall 3 rise 2
server web1 203.0.113.10:8080 check
server web2 203.0.113.11:8080 check
server web3 203.0.113.12:8080 check backup

The fall 3 and rise 2 settings mean a server is marked down after 3 consecutive failed checks and back up after 2 consecutive successes, avoiding flapping on transient errors.

Harden SSL/TLS settings and cipher suites

Restrict HAProxy to TLS 1.2 and 1.3, disable weak ciphers, and enable HSTS so browsers refuse to downgrade to plain HTTP.

global
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

frontend https_front
bind *:443 ssl crt /etc/haproxy/certs/example.com.pem
mode http
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-request set-header X-Forwarded-Proto https
default_backend web_backend

For a deeper dive into ACL-based traffic rules and security headers, see configure HAProxy SSL termination and security headers.

Validate and reload HAProxy

Always validate the configuration syntax before reloading, since a bad config can drop your listeners.

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

sudo systemctl restart haproxy
sudo systemctl enable haproxy

Automate renewal with a deploy hook

Certbot can run a script automatically whenever it renews a certificate. Use this to rebuild the PEM bundle and reload HAProxy without manual steps.

sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy

#!/bin/bash
set -e
DOMAIN="example.com"
cat /etc/letsencrypt/live/${DOMAIN}/fullchain.pem /etc/letsencrypt/live/${DOMAIN}/privkey.pem > /etc/haproxy/certs/${DOMAIN}.pem
chmod 600 /etc/haproxy/certs/${DOMAIN}.pem
chown root:root /etc/haproxy/certs/${DOMAIN}.pem
systemctl reload haproxy

sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/haproxy-reload.sh
sudo chown root:root /etc/letsencrypt/renewal-hooks/deploy/haproxy-reload.sh

The script needs to be executable by root only, since Certbot's renewal timer runs as root. There is no reason to make it writable by other users.

Test the renewal process

Certbot ships a dry run mode that simulates renewal without touching your actual certificates, letting you confirm the hook fires correctly.

sudo certbot renew --dry-run

Check that the systemd timer for automatic renewal is active, since Certbot installs this by default on most distributions.

systemctl list-timers | grep certbot

Verify your setup

curl -I http://example.com
curl -Iv https://example.com
openssl s_client -connect example.com:443 -servername example.com < /dev/null | grep -A2 "Protocol\|Cipher"

Confirm HAProxy sees your backend servers as healthy using the runtime API or stats page.

echo "show servers state" | sudo socat stdio /run/haproxy/admin.sock

Run an SSL Labs style check locally with testssl.sh, or verify cipher restrictions directly.

openssl s_client -connect example.com:443 -tls1_1

This last command should fail to connect, confirming TLS 1.1 is disabled.

Common issues
SymptomCauseFixHAProxy fails to start after adding bind sslPEM bundle missing or malformedVerify order: certificate, then chain, then key in one file, check with openssl x509 -in /etc/haproxy/certs/example.com.pem -noout -textCertbot renewal fails with port 80 in useHAProxy is bound to port 80 during standalone challengeSwitch to DNS challenge or add the ACME ACL exception shown aboveBrowser shows certificate warningPEM bundle not reloaded after renewalConfirm the deploy hook ran: check /var/log/letsencrypt/letsencrypt.log and reload HAProxy manuallyBackend marked down unexpectedlyHealth check endpoint returns non-200 or times outTest the check URL directly with curl -i http://203.0.113.10:8080/health and adjust inter/fall/riseWeak cipher still negotiatedOld config cached or client forcing legacy TLSRun haproxy -c -f /etc/haproxy/haproxy.cfg to confirm the active config, then reloadPermission denied reading PEM fileFile owned by wrong user or overly restrictive modeSet ownership to root and mode 600, and confirm HAProxy's systemd unit runs with sufficient privilege to read it, do not use 777

Next steps

Configure HAProxy load balancing with multiple backend servers

Set up HAProxy high availability with keepalived clustering

Configure HAProxy with Consul for dynamic service discovery

Configure HAProxy advanced routing with ACLs and maps

Implement HAProxy rate limiting and DDoS protection

Monitor HAProxy with Prometheus and Grafana dashboards

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping certificates renewed, cipher suites current, and backend health monitored across environments is the harder part. See how we run infrastructure like this for European teams.

---

### Configure HAProxy with Consul for dynamic service discovery

URL: https://binadit.com/tutorials/configure-haproxy-consul-service-discovery
Category: networking
Difficulty: advanced
Time: ~50 minutes
Author: Binadit Tech Team

> Learn how to combine HAProxy, Consul and consul-template to build a self-updating load balancer that discovers backends automatically, reacts to health checks, and reloads without downtime.

What this solves

Static HAProxy backend lists break down once services scale up, scale down or move between hosts. This tutorial wires HAProxy to Consul's service catalog through consul-template, so backend servers are generated dynamically from live health check data and configuration reloads happen automatically when the topology changes.

By the end you will have a Consul agent registering services with health checks, consul-template rendering HAProxy configuration from those checks, and a reload pipeline that survives scaling events and failures without manual intervention.

Step-by-step installation

Install HAProxy

HAProxy will act as the load balancer that receives dynamically generated backend definitions.

sudo apt update
sudo apt install -y haproxy

sudo dnf install -y haproxy

Install the Consul agent

Each HAProxy node runs a local Consul agent in client mode, joined to your Consul cluster, to query service health data with low latency.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y consul

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul

Configure the Consul client agent

This configuration joins the node to an existing Consul server cluster. Replace 203.0.113.10 with your actual Consul server addresses.

datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "{{ GetInterfaceIP \"eth0\" }}"
retry_join = ["203.0.113.10", "203.0.113.11", "203.0.113.12"]
client_addr = "127.0.0.1"
enable_local_script_checks = true
ports {
grpc = 8502
}

sudo mkdir -p /opt/consul
sudo chown -R consul:consul /opt/consul
sudo systemctl enable --now consul
consul members

Note: If you are building a fresh Consul cluster rather than joining an existing one, see install and configure Consul for service discovery with clustering and security first.

Register a service with health checks

Services register themselves in Consul's catalog along with a health check. HAProxy will only route to instances that pass this check.

{
"service": {
"name": "web-app",
"port": 8080,
"tags": ["http"],
"check": {
"http": "http://localhost:8080/healthz",
"interval": "10s",
"timeout": "3s",
"deregister_critical_service_after": "90s"
}
}
}

sudo systemctl reload consul
consul catalog services
consul health checks web-app

Repeat this registration file on every application host, adjusting the port for each instance. Consul tracks each instance independently by node and check status.

Install consul-template

consul-template watches the Consul catalog for changes and renders templates, in this case an HAProxy configuration file, whenever backend state changes.

sudo apt install -y consul-template

sudo dnf install -y consul-template

Note: If your distro's repo does not ship consul-template, download the binary release from HashiCorp and place it in /usr/local/bin with chmod 755.

Create the HAProxy template

This template queries Consul for healthy instances of web-app and generates an HAProxy frontend and backend block. It also defines the stats page and load balancing algorithm.

global
log /dev/log local0
maxconn 20000
user haproxy
group haproxy

defaults
log global
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
option httplog

frontend stats
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:{{ key "haproxy/stats_password" }}

frontend web_front
bind *:80
default_backend web_back

backend web_back
balance leastconn
option httpchk GET /healthz
{{ range service "web-app" }}
server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check
{{ end }}

Warning: Do not hardcode the stats password in the template. Store it in Consul's KV store with consul kv put haproxy/stats_password 'S3cur3-Stats!Pass' so it never appears in version control.

Configure consul-template to render and reload

This configuration tells consul-template where to write the rendered file and what command to run afterward to validate and reload HAProxy safely.

consul {
address = "127.0.0.1:8500"
}

template {
source = "/etc/consul-template/haproxy.ctmpl"
destination = "/etc/haproxy/haproxy.cfg"
command = "haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy"
command_timeout = "30s"
wait {
min = "2s"
max = "10s"
}
}

The wait block debounces rapid-fire changes during scaling events, so a burst of instance registrations does not trigger dozens of reloads in a few seconds.

Set correct ownership and permissions

consul-template needs to write to the HAProxy config directory and trigger a reload. Rather than opening permissions widely, grant a dedicated service account exactly what it needs.

sudo useradd -r -s /usr/sbin/nologin consul-template
sudo chown -R consul-template:haproxy /etc/haproxy
sudo chmod 750 /etc/haproxy
sudo chmod 640 /etc/haproxy/haproxy.cfg

Never use chmod 777. It gives every user on the system full read, write and execute access to your HAProxy configuration, including the stats credentials. Use group ownership between consul-template and haproxy instead, with 750/640 permissions so only the two accounts that need access have it.

Grant the reload command via sudoers instead of running consul-template as root:

consul-template ALL=(root) NOPASSWD: /bin/systemctl reload haproxy, /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg

Run consul-template as a systemd service

Running consul-template under systemd ensures it restarts on failure and starts on boot alongside Consul and HAProxy.

[Unit]
Description=Consul Template for HAProxy
After=network-online.target consul.service
Wants=network-online.target

[Service]
User=consul-template
Group=haproxy
ExecStart=/usr/bin/consul-template -config=/etc/consul-template/config.hcl
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now consul-template
sudo systemctl status consul-template

Enable and start HAProxy

Start HAProxy once the first rendered config is in place.

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl enable --now haproxy
sudo systemctl status haproxy

Verify your setup

Confirm Consul sees the service as healthy, the template rendered correctly, and HAProxy is routing traffic.

consul catalog services
consul health checks web-app
cat /etc/haproxy/haproxy.cfg | grep -A5 backend
curl -I http://127.0.0.1/
curl -u admin:S3cur3-Stats!Pass http://127.0.0.1:8404/stats

Register a second instance of web-app on another port and watch the backend update without restarting HAProxy manually.

sudo journalctl -u consul-template -f

You should see a render event followed by a successful reload log line within a few seconds of the new registration.

Testing failover and scaling scenarios

Simulate a backend failure

Stop the application process on one instance and confirm Consul marks the check critical, and that consul-template removes it from the backend within one interval cycle.

sudo systemctl stop web-app
consul health checks web-app
curl -u admin:S3cur3-Stats!Pass http://127.0.0.1:8404/stats

The stats page should show the server entry as down, and the HAProxy config should no longer list it after the next consul-template render.

Simulate horizontal scale-out

Register three additional instances in quick succession to confirm the debounce window in the wait block coalesces the reload into a single event instead of five separate reloads.

for port in 8081 8082 8083; do
sudo tee /etc/consul.d/web-app-$port.json > /dev/null <<EOF
{"service":{"name":"web-app","port":$port,"check":{"http":"http://localhost:$port/healthz","interval":"10s"}}}
EOF
done
sudo systemctl reload consul

Watch journalctl -u consul-template -f during this test. You should see one render cycle cover all three new instances rather than three separate reload commands.

Common issues
SymptomCauseFixHAProxy config never updatesconsul-template cannot reach the Consul HTTP APICheck consul_addr in config.hcl and confirm curl 127.0.0.1:8500/v1/status/leader returns a leaderReload command fails silentlyconsul-template user lacks sudo rights for systemctl reloadVerify the sudoers entry with sudo -l -U consul-templateBackend flaps rapidly under loadHealth check interval too aggressive for app startup timeIncrease interval and add a startup grace period with deregister_critical_service_afterStats page returns 403Wrong credentials or password stored incorrectly in Consul KVRe-run consul kv get haproxy/stats_password and confirm it matches your curl commandNew instances never appear in backendHealth check failing due to firewall blocking the check portAllow the specific port with sudo ufw allow from 203.0.113.0/24 to any port 8080 instead of disabling the firewall

Next steps

Set up HAProxy high availability with keepalived clustering for automatic failover

Configure HAProxy advanced routing with ACLs and maps for intelligent traffic management

Monitor HAProxy and Consul with Prometheus and Grafana dashboards

Implement Consul multi-datacenter replication with WAN federation

Implement Consul ACL security and encryption for production deployments

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning for the Consul cluster, failover drills for HAProxy nodes, TLS and ACL rotation, and being on-call when a datacenter link flaps at 3am. See how we run infrastructure like this for European teams.

---

### Configure PostgreSQL 17 connection pooling with PgBouncer for high availability

URL: https://binadit.com/tutorials/configure-postgresql-connection-pooling-with-pgbouncer
Category: databases
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team

> Deploy PgBouncer in front of PostgreSQL 17 with session, transaction and statement pooling modes, SCRAM-SHA-256 authentication, keepalived-based HA, and Prometheus monitoring for production workloads.

What this solves

PostgreSQL forks a new backend process for every client connection, which becomes expensive under high concurrency. PgBouncer sits between your application and PostgreSQL 17, multiplexing thousands of client connections onto a small pool of real database connections.

This tutorial covers pooling modes, SCRAM-SHA-256 authentication, running multiple PgBouncer instances behind keepalived and HAProxy for failover, tuning pool sizes for production, and monitoring with SHOW POOLS, SHOW STATS and a Prometheus exporter.

Step-by-step configuration

Install PostgreSQL 17 client libraries and PgBouncer

PgBouncer needs the PostgreSQL client libraries to connect upstream. Install both PgBouncer and the PostgreSQL 17 client tools before configuring anything.

sudo apt update
sudo apt install -y curl ca-certificates gnupg
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo gpg --dearmor -o /usr/share/keyrings/postgresql.gpg
echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list
sudo apt update
sudo apt install -y postgresql-client-17 pgbouncer

sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo dnf -qy module disable postgresql
sudo dnf install -y postgresql17 pgbouncer

Confirm PostgreSQL 17 is reachable

Assume PostgreSQL 17 is already running on a primary server at 203.0.113.10. Verify connectivity before wiring up the pooler, otherwise you will debug PgBouncer for a problem that lives elsewhere.

psql -h 203.0.113.10 -U postgres -d appdb -c "SELECT version();"

If you have not yet configured the primary, see install and configure PostgreSQL 17 with performance tuning and security hardening first.

Choose a pooling mode in pgbouncer.ini

PgBouncer supports three pooling modes. Session pooling assigns one server connection per client for the whole session, the safest but least efficient. Transaction pooling releases the server connection after each transaction, the best default for most web apps. Statement pooling releases it after every statement, but breaks multi-statement transactions and most ORMs, so use it only for stateless read-only workloads.

[databases]
appdb = host=203.0.113.10 port=5432 dbname=appdb
appdb_session = host=203.0.113.10 port=5432 dbname=appdb pool_mode=session

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
admin_users = pgbouncer_admin
stats_users = pgbouncer_stats

pool_mode = transaction
max_client_conn = 2000
default_pool_size = 25
min_pool_size = 5
reserve_pool_size = 10
reserve_pool_timeout = 3
max_db_connections = 100
max_user_connections = 100

server_idle_timeout = 300
server_lifetime = 3600
query_wait_timeout = 30
client_idle_timeout = 0
client_login_timeout = 60

log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
pidfile = /var/run/pgbouncer/pgbouncer.pid
logfile = /var/log/pgbouncer/pgbouncer.log

Note: the appdb_session database entry lets specific clients (migrations, long transactions, LISTEN/NOTIFY consumers) connect through session mode on the same PgBouncer instance without changing the primary pool's mode.

Generate SCRAM-SHA-256 credentials in userlist.txt

PostgreSQL 17 defaults to scram-sha-256 password hashing. PgBouncer must use the exact same hash format in userlist.txt, plaintext or md5 entries will fail authentication against a SCRAM-only server.

psql -h 203.0.113.10 -U postgres -d appdb -t -A -c "SELECT rolname, rolpassword FROM pg_authid WHERE rolname = 'app_user';"

Copy the output into userlist.txt, keeping the SCRAM-SHA-256 hash exactly as returned. Never write a real password here in plaintext.

"app_user" "SCRAM-SHA-256$4096:base64salt$base64storedkey:base64serverkey"
"pgbouncer_admin" "SCRAM-SHA-256$4096:base64salt$base64storedkey:base64serverkey"

sudo chown pgbouncer:pgbouncer /etc/pgbouncer/userlist.txt /etc/pgbouncer/pgbouncer.ini
sudo chmod 600 /etc/pgbouncer/userlist.txt

Never use chmod 777. userlist.txt contains password hashes for every pooled user. Mode 600 owned by the pgbouncer service user is the minimum needed, anything looser exposes credential hashes to every local account on the box.

Enable and start PgBouncer

Start the service and confirm it is listening on port 6432, the standard PgBouncer port distinct from PostgreSQL's 5432.

sudo mkdir -p /var/log/pgbouncer /var/run/pgbouncer
sudo chown pgbouncer:pgbouncer /var/log/pgbouncer /var/run/pgbouncer
sudo systemctl enable --now pgbouncer
sudo systemctl status pgbouncer

Deploy PgBouncer on a second node for redundancy

A single PgBouncer instance is a single point of failure. Repeat the installation and configuration steps above on a second host (203.0.113.11), pointing to the same PostgreSQL primary and using an identical userlist.txt and pgbouncer.ini.

scp /etc/pgbouncer/userlist.txt root@203.0.113.11:/etc/pgbouncer/userlist.txt
scp /etc/pgbouncer/pgbouncer.ini root@203.0.113.11:/etc/pgbouncer/pgbouncer.ini

Front both PgBouncer nodes with HAProxy

HAProxy load balances client connections across both PgBouncer nodes and removes a failed node from rotation automatically using TCP health checks.

sudo apt install -y haproxy

sudo dnf install -y haproxy

frontend pgbouncer_front
bind 203.0.113.20:6432
mode tcp
default_backend pgbouncer_nodes

backend pgbouncer_nodes
mode tcp
balance roundrobin
option tcp-check
tcp-check connect port 6432
server pgb1 203.0.113.10:6432 check inter 3s fall 3 rise 2
server pgb2 203.0.113.11:6432 check inter 3s fall 3 rise 2

sudo systemctl enable --now haproxy

For a deeper walkthrough of HAProxy tuning and ACL-based routing, see configure HAProxy load balancing with multiple backend servers.

Add keepalived for a floating virtual IP

HAProxy itself needs redundancy. Run keepalived on both HAProxy hosts so a virtual IP fails over automatically if the active node goes down.

sudo apt install -y keepalived

sudo dnf install -y keepalived

vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}

vrrp_instance VI_PGBOUNCER {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass Ch4ngeThisVrrpSecret
}
virtual_ipaddress {
203.0.113.20/24
}
track_script {
chk_haproxy
}
}

sudo systemctl enable --now keepalived

Set state BACKUP and priority 100 on the second node. For the complete active/passive pattern including firewall rules, see configure keepalived with HAProxy backend health monitoring.

Tune pool sizes and timeouts for production load

default_pool_size controls how many server connections PgBouncer opens per database/user pair in transaction mode. Set it based on PostgreSQL's max_connections divided by the number of PgBouncer instances and databases sharing the primary, leaving headroom for replication and admin connections.
SettingRecommended starting valueWhydefault_pool_size20-40Matches typical CPU core count on the database hostmax_client_conn1000-5000PgBouncer connections are cheap, size for peak app instancesreserve_pool_size10-20% of default_pool_sizeAbsorbs short traffic bursts without queuingserver_idle_timeout300sFrees idle backend connections during low trafficquery_wait_timeout30sFails fast instead of queueing indefinitely under saturation

Verify PostgreSQL's own limit accommodates all pooled connections plus replication slots.

psql -h 203.0.113.10 -U postgres -c "SHOW max_connections;"

Warning: setting default_pool_size too high defeats the purpose of pooling. If PgBouncer's total pool size across all instances approaches PostgreSQL's max_connections, you have removed the buffer that protects the primary during a connection storm.

Monitoring PgBouncer

Query live pool and stats via the admin console

PgBouncer exposes an administrative pseudo-database called pgbouncer. Connect as an admin_users entry to inspect pool state in real time.

psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW POOLS;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW STATS;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW CLIENTS;"

SHOW POOLS reports cl_active, cl_waiting and sv_active columns per database/user pair. A consistently non-zero cl_waiting means default_pool_size is too small for the current load.

Deploy the Prometheus exporter

prometheus-pgbouncer-exporter scrapes SHOW STATS and SHOW POOLS and exposes them as Prometheus metrics for Grafana dashboards and alerting.

sudo useradd --no-create-home --shell /usr/sbin/nologin pgbouncer_exporter
curl -L -o /tmp/pgbouncer_exporter.tar.gz https://github.com/prometheus-community/pgbouncer_exporter/releases/download/v0.10.2/pgbouncer_exporter-0.10.2.linux-amd64.tar.gz
tar -xzf /tmp/pgbouncer_exporter.tar.gz -C /tmp
sudo mv /tmp/pgbouncer_exporter-0.10.2.linux-amd64/pgbouncer_exporter /usr/local/bin/
sudo chown pgbouncer_exporter:pgbouncer_exporter /usr/local/bin/pgbouncer_exporter

[Unit]
Description=PgBouncer Prometheus Exporter
After=network.target

[Service]
User=pgbouncer_exporter
Group=pgbouncer_exporter
Environment=DATA_SOURCE_NAME="postgres://pgbouncer_stats:StrongExporterPass9!@127.0.0.1:6432/pgbouncer?sslmode=disable"
ExecStart=/usr/local/bin/pgbouncer_exporter --web.listen-address=127.0.0.1:9127
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now pgbouncer_exporter
curl http://127.0.0.1:9127/metrics | grep pgbouncer_pools

Wire this into an existing Prometheus and Grafana stack as described in set up Prometheus and Grafana monitoring stack with Docker Compose, then build alerts on pgbouncer_pools_cl_waiting and pgbouncer_pools_sv_used.

Failover testing with streaming replication

Simulate a primary failure

If PostgreSQL streaming replication is already in place, test that PgBouncer's target can be repointed to a promoted replica quickly. This is the scenario your HA setup exists for.

sudo systemctl stop postgresql

On the standby, promote it to become the new primary.

sudo -u postgres /usr/lib/postgresql/17/bin/pg_ctl promote -D /var/lib/postgresql/17/main

Repoint PgBouncer to the new primary

Update the host in pgbouncer.ini on both PgBouncer nodes, then reload without restarting, PgBouncer reload keeps existing pooled connections draining gracefully.

[databases]
appdb = host=203.0.113.11 port=5432 dbname=appdb

sudo systemctl reload pgbouncer

For the full replication topology, promotion tooling, and automated failover scripts, follow set up PostgreSQL 17 streaming replication with PgBouncer connection pooling and load balancing.

Verify your setup

sudo systemctl status pgbouncer haproxy keepalived
psql -h 203.0.113.20 -p 6432 -U app_user -d appdb -c "SELECT 1;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer_admin pgbouncer -c "SHOW POOLS;"
curl -s http://127.0.0.1:9127/metrics | grep pgbouncer_up

Common issues
SymptomCauseFixERROR: password authentication faileduserlist.txt has a plaintext or md5 hash but PostgreSQL uses scram-sha-256Regenerate the entry with the exact rolpassword value from pg_authidClients hang waiting for a connectiondefault_pool_size too small for concurrent loadCheck SHOW POOLS for cl_waiting, raise default_pool_size or reserve_pool_sizeApplication sees stale prepared statements after failoverStatement pooling mode used with a stateful ORMSwitch that database entry to pool_mode=session or transactionVirtual IP does not move on failoverFirewall blocks VRRP multicast traffic between HAProxy nodesAllow protocol 112 (VRRP) between node IPs instead of disabling the firewallPrometheus exporter shows no metricspgbouncer_stats user lacks permission or wrong DSNConfirm stats_users includes the exporter's role in pgbouncer.iniPostgreSQL rejects connections after reloadmax_connections exceeded by combined pool sizes across instancesReduce default_pool_size or raise max_connections and restart PostgreSQL

Next steps

Set up PostgreSQL 17 streaming replication with PgBouncer connection pooling and load balancing

Configure PostgreSQL 17 SSL encryption and advanced security hardening

Monitor PostgreSQL performance with Prometheus and Grafana dashboards

Configure PostgreSQL 17 PgBouncer multi-region load balancing

Automate PostgreSQL failover with Patroni and etcd

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments, including pool sizing as traffic grows, is the harder part. See how we run infrastructure like this for European teams.

---

### Configure Kubernetes secrets management with External Secrets Operator and HashiCorp Vault

URL: https://binadit.com/tutorials/configure-kubernetes-secrets-management-with-external-secrets-operator
Category: devops
Difficulty: advanced
Time: ~75 minutes
Author: Binadit Tech Team

> Learn how to deploy External Secrets Operator on Kubernetes, integrate it with HashiCorp Vault using the Kubernetes auth method, and sync secrets via SecretStore and ClusterSecretStore resources with production-grade RBAC and monitoring.

What this solves

Storing secrets as plain Kubernetes Secret objects means they sit base64-encoded in etcd with no rotation, no audit trail, and no central policy control. External Secrets Operator (ESO) bridges Kubernetes to HashiCorp Vault, pulling secrets on a schedule and materializing them as native Secret objects that your pods consume normally.

This tutorial covers a production-ready setup: Vault as the backend, ESO deployed with Helm, Kubernetes auth method for authentication, namespace-scoped SecretStore and cluster-wide ClusterSecretStore resources, templating, rotation, RBAC isolation, and troubleshooting.

Prerequisites and architecture overview

You need a running Kubernetes cluster (kubeadm, EKS, GKE, or AKS all work) with kubectl configured, Helm 3 installed, and a Vault server reachable from the cluster network. If you have not installed Vault yet, see install and configure Vault for secrets management with high availability first.

The data flow is: ESO controller pods authenticate to Vault using a Kubernetes service account token, Vault validates that token against the Kubernetes API, Vault issues a scoped token tied to a policy, and ESO uses that token to read secrets and write them into Kubernetes Secret objects referenced by your workloads.

Note: This tutorial assumes Vault is already unsealed and reachable at a stable address. Production Vault should run with auto-unseal and TLS, not the dev server mode shown for the local backend example.

Step-by-step configuration

Install the Vault CLI and verify connectivity

The Vault CLI lets you configure policies and auth methods from your workstation or a bastion host.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vault

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault

export VAULT_ADDR="https://vault.example.com:8200"
vault status

Enable the KV v2 secrets engine

KV version 2 supports versioning, which lets you roll back accidental overwrites of application secrets.

vault login
vault secrets enable -path=kv-eso kv-v2
vault kv put kv-eso/production/api-service DB_PASSWORD="Str0ng-P@ss-2024!" API_KEY="ak_live_9f3c7d2a1b"

Enable the Kubernetes auth method in Vault

This lets Vault verify tokens presented by pods against your cluster's API server, without storing static credentials anywhere.

vault auth enable kubernetes

Retrieve the cluster CA and API endpoint, then configure Vault to talk to your cluster.

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > /tmp/k8s-ca.crt
KUBE_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')

vault write auth/kubernetes/config \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert=@/tmp/k8s-ca.crt

Create a least-privilege Vault policy

Scope the policy to only the paths ESO needs to read. Never grant blanket access to the entire KV mount.

path "kv-eso/data/production/*" {
capabilities = ["read"]
}

path "kv-eso/metadata/production/*" {
capabilities = ["list", "read"]
}

vault policy write eso-production-read vault-eso-policy.hcl

Create a Kubernetes auth role bound to a service account

Binding the Vault role to a specific service account and namespace enforces that only ESO pods in that namespace can assume this role, which is the core of namespace isolation.

vault write auth/kubernetes/role/eso-production-role \
bound_service_account_names=external-secrets-sa \
bound_service_account_namespaces=external-secrets \
policies=eso-production-read \
ttl=15m

Deploy External Secrets Operator with Helm

ESO ships an official Helm chart that installs the controller, webhook, and CRDs. Deploy it into a dedicated namespace to keep it isolated from application workloads.

kubectl create namespace external-secrets
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--set installCRDs=true \
--set replicaCount=2 \
--set serviceAccount.name=external-secrets-sa

Two replicas give you a controller failover path if a pod is evicted or the node drains during maintenance.

Verify the operator is running

Confirm the controller, webhook, and cert-controller pods reach Running state before creating any secret store resources.

kubectl get pods -n external-secrets
kubectl get crd | grep external-secrets.io

Create a ClusterSecretStore for cluster-wide access

A ClusterSecretStore is not namespaced and can be referenced by ExternalSecret resources across the cluster, useful for shared credentials like registry pull secrets or platform-wide TLS certs.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "kv-eso"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-production-role"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "external-secrets"

kubectl apply -f manifests/cluster-secret-store.yaml
kubectl get clustersecretstore vault-backend -o wide

Create a namespace-scoped SecretStore

Use a SecretStore instead of a ClusterSecretStore when a team should only sync secrets within its own namespace, reinforcing the isolation established by the RBAC and Vault role bindings above.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend-app
namespace: production
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "kv-eso"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-production-role"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "production"

kubectl create namespace production
kubectl apply -f manifests/app-secret-store.yaml

Define an ExternalSecret to sync into a Kubernetes Secret

The ExternalSecret is the resource that actually maps Vault paths to keys inside a generated Kubernetes Secret. The refreshInterval controls how often ESO re-reads Vault.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-service-secrets
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend-app
kind: SecretStore
target:
name: api-service-secrets
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD
remoteRef:
key: production/api-service
property: DB_PASSWORD
- secretKey: API_KEY
remoteRef:
key: production/api-service
property: API_KEY

kubectl apply -f manifests/api-service-external-secret.yaml
kubectl get externalsecret api-service-secrets -n production
kubectl get secret api-service-secrets -n production -o jsonpath='{.data}'

Use templating to reshape synced secrets

Templating lets you build connection strings or config file formats directly from multiple Vault values instead of requiring the application to assemble them at runtime.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-service-connection
namespace: production
spec:
refreshInterval: 30m
secretStoreRef:
name: vault-backend-app
kind: SecretStore
target:
name: api-service-connection
creationPolicy: Owner
template:
engineVersion: v2
data:
DATABASE_URL: "postgres://app_user:{{ .DB_PASSWORD }}@postgres-primary.production.svc.cluster.local:5432/appdb"
data:
- secretKey: DB_PASSWORD
remoteRef:
key: production/api-service
property: DB_PASSWORD

Configure RBAC to restrict who can read synced Secrets and ExternalSecrets

ESO synchronizing Vault into a namespace does not automatically restrict which cluster users can read that Secret. Apply RBAC in the target namespace so only the application's service account and approved operators can view it.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["api-service-secrets", "api-service-connection"]
verbs: ["get"]

kubectl apply -f manifests/secret-reader-role.yaml
kubectl create rolebinding api-service-secret-reader \
--role=secret-reader \
--serviceaccount=production:api-service-sa \
-n production

For a deeper walkthrough of scoping cluster roles and service accounts, see configure Kubernetes RBAC with service accounts and cluster roles.

Verify your setup

Confirm ESO successfully authenticated to Vault, synced the secret, and that the resulting Kubernetes Secret has the expected keys.

kubectl describe externalsecret api-service-secrets -n production
kubectl get events -n production --field-selector involvedObject.name=api-service-secrets
kubectl logs -n external-secrets deployment/external-secrets --tail=50

The Status.Conditions field in the ExternalSecret describe output should show Ready: True with the reason SecretSynced. If it shows an error, the message usually names the exact Vault path or permission problem.

Warning: Never store raw Vault root tokens in CI/CD pipelines or ConfigMaps to bootstrap this integration. Use short-lived Kubernetes service account tokens through the auth method configured above, and rotate the Vault policy or role immediately if a token is ever exposed.

Secret rotation and refresh behavior

ESO does not push changes, it polls. When you update a value in Vault with vault kv put, the change appears in the Kubernetes Secret only after the next refreshInterval elapses. Set shorter intervals (5-15m) for credentials that rotate frequently, and longer intervals (1h or more) for stable static config to reduce load on Vault.
Secret typeSuggested refreshIntervalReasonDatabase passwords (static)1hRotated manually or via scheduled job, low churnDynamic database creds15mVault dynamic secrets expire on a TTL, needs frequent renewalTLS certificates6hCerts typically valid for weeks or monthsAPI keys for third-party services30mBalance between freshness and Vault request volume

Applications that read environment variables at startup only will not see rotated secrets until the pod restarts. Pair ESO with a rollout mechanism, such as the Reloader controller or a CI/CD hook, to restart deployments when the underlying Secret changes.

Production hardening and high availability considerations

Run Vault itself in HA mode with Raft integrated storage or Consul as the storage backend, and enable auto-unseal so Vault comes back online automatically after a node restart without manual intervention. See configure Vault auto-unseal with AWS KMS for a working setup.

Run at least two ESO controller replicas across separate nodes using pod anti-affinity, and set resource requests and limits so the controller is not evicted under memory pressure. Enable the ESO Prometheus metrics endpoint and scrape externalsecret_sync_calls_total and externalsecret_sync_calls_error to alert on sync failures before applications notice missing secrets.

kubectl get --raw /api/v1/namespaces/external-secrets/services/external-secrets-metrics:http-metrics/proxy/metrics | grep externalsecret_sync

Enforce network policies so only the ESO namespace can reach Vault's port over the network, reducing the blast radius if a workload pod is compromised. If you are also running Argo CD, review integrate ArgoCD with External Secrets Operator to keep SecretStore and ExternalSecret manifests under GitOps control without committing plaintext values.

Common issues
SymptomCauseFixExternalSecret stuck in SecretSyncedErrorVault role not bound to the correct service account or namespaceCheck vault read auth/kubernetes/role/eso-production-role matches the SecretStore's serviceAccountRefpermission denied reading secretVault policy path does not match KV mount path exactlyRemember KV v2 requires kv-eso/data/<path> in the policy, not just kv-eso/<path>Secret never updates after Vault changerefreshInterval has not elapsed yetLower refreshInterval or manually force reconcile with kubectl annotate externalsecret api-service-secrets force-sync=$(date +%s) -n production --overwritex509 certificate signed by unknown authorityVault server uses a certificate not trusted by cluster nodesMount the CA bundle into the ESO pod via caBundle in the SecretStore provider configClusterSecretStore works but SecretStore fails in one namespacebound_service_account_namespaces in Vault role restricts to a different namespaceCreate a separate Vault role per namespace or widen the bound namespaces list deliberately

Next steps

Configure Vault dynamic secrets for databases with PostgreSQL and MySQL integration

Set up Vault as a PKI certificate authority with SSL automation and intermediate CA

Implement Kubernetes secrets management with HashiCorp Vault integration

Configure Kubernetes secrets management with Sealed Secrets for secure Helm values

Monitor Kubernetes network policies with Prometheus and Grafana for enhanced cluster security

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: Vault unseal drills, ESO controller upgrades, policy audits, and alerting when a sync silently fails. Talk to an engineer if you'd rather hand the ops side off.

---

### Integrate Jaeger with Istio service mesh for distributed tracing

URL: https://binadit.com/tutorials/integrate-jaeger-with-istio-service-mesh-tracing
Category: devops
Difficulty: advanced
Time: ~60 minutes
Author: Binadit Tech Team

> Learn how to integrate Jaeger with Istio service mesh to get end-to-end distributed tracing across Kubernetes microservices, including sidecar injection, trace sampling, and ingress gateway access to the Jaeger UI.

What this solves

When requests flow through dozens of microservices behind an Istio service mesh, pinpointing latency or failures without distributed tracing is nearly impossible. This tutorial integrates Jaeger with Istio so every Envoy sidecar automatically emits spans, giving you a full request timeline across services without touching application code.

You will install Istio with tracing enabled, deploy Jaeger as the tracing backend, configure sampling and telemetry, and verify trace propagation end to end through the ingress gateway.

Prerequisites and Kubernetes cluster preparation

Confirm cluster access and resources

You need a running Kubernetes cluster (1.27+) with at least 3 worker nodes, 4 vCPU and 8GB RAM free for Istio control plane, Jaeger, and demo workloads. Verify kubectl access before continuing.

kubectl version --short
kubectl get nodes
kubectl cluster-info

Install prerequisite tools

You need curl, istioctl, and Helm to install Istio and Jaeger components.

sudo apt update && sudo apt install -y curl tar unzip

sudo dnf install -y curl tar unzip

Download and install istioctl

The istioctl binary manages the Istio installation profile and lets you enable tracing at install time.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.23.2 sh -
cd istio-1.23.2
export PATH=$PWD/bin:$PATH
istioctl version --remote=false

Install Helm

Helm is required to deploy Jaeger with the official chart in a repeatable way.

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Installing Istio service mesh with tracing enabled

Create the istio-system namespace

All Istio control plane components and observability tools live in this dedicated namespace.

kubectl create namespace istio-system

Install Istio with tracing configuration

The demo profile enables the ingress gateway and sets a default trace sampling percentage. Production deployments should use a custom IstioOperator manifest instead.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: default
meshConfig:
enableTracing: true
defaultConfig:
tracing:
sampling: 10.0
extensionProviders:
- name: jaeger
opentelemetry:
service: jaeger-collector.istio-system.svc.cluster.local
port: 4317
values:
global:
proxy:
tracer: "openelemetry"

istioctl install -f istio-config.yaml -y

Enable automatic sidecar injection

Label the namespace where your microservices run so Istio automatically injects the Envoy sidecar into every pod.

kubectl create namespace demo-app
kubectl label namespace demo-app istio-injection=enabled

Note: If you already manage RBAC for this cluster, review configuring Kubernetes RBAC with service accounts and cluster roles before granting broader access to the istio-system namespace.

Deploying Jaeger and configuring the tracing backend

Add the Jaeger Helm repository

The official Jaegertracing chart deploys the collector, query service, and UI as separate components, which scales better than the all-in-one image.

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update

Deploy Jaeger with production values

This example uses in-memory storage for a quick start. For a durable backend with retention and encryption, see the Elasticsearch backend guide linked below.

provisionDataStore:
cassandra: false
elasticsearch: false

storage:
type: memory

collector:
service:
otlp:
grpc:
name: otlp-grpc
port: 4317
http:
name: otlp-http
port: 4318

query:
service:
type: ClusterIP

helm install jaeger jaegertracing/jaeger \
-n istio-system \
-f jaeger-values.yaml

Confirm Jaeger components are running

You should see collector, query, and agent pods (if enabled) in a Running state.

kubectl get pods -n istio-system -l app.kubernetes.io/name=jaeger

Warning: In-memory storage loses all trace data on pod restart. Do not use this in production. Use Elasticsearch or Cassandra as documented in configuring Jaeger with Elasticsearch backend security and encryption.

Configuring Istio telemetry and trace sampling rates

Create a Telemetry resource

Istio's Telemetry API controls sampling rate and trace propagation headers mesh-wide or per namespace. A 100 percent sample rate is useful for testing but too expensive for production traffic volumes.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-tracing
namespace: istio-system
spec:
tracing:
- providers:
- name: jaeger
randomSamplingPercentage: 10.00

kubectl apply -f telemetry.yaml

Override sampling for a specific namespace

You can apply a higher sampling rate temporarily in a staging or debugging namespace without affecting production traffic elsewhere in the mesh.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: debug-tracing
namespace: demo-app
spec:
tracing:
- providers:
- name: jaeger
randomSamplingPercentage: 100.00

kubectl apply -f telemetry-debug.yaml

For high-volume production environments, tune sampling with head-based and tail-based strategies as covered in setting up Jaeger sampling strategies for high-volume production tracing.

Enabling automatic sidecar trace propagation

Deploy a sample multi-service application

Istio's Envoy sidecars generate spans automatically, but your application code must forward the trace context headers on outbound calls, otherwise spans from different services will not link together.

kubectl apply -n demo-app -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/bookinfo/platform/kube/bookinfo.yaml

Verify header propagation in application code

Each service in the call chain must propagate these headers from incoming requests to any outgoing requests it makes: x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, and b3 or traceparent for W3C trace context.

kubectl logs -n demo-app deploy/productpage-v1 -c istio-proxy | grep -i trace

Expose the sample app through the ingress gateway

This lets you generate real traffic through the mesh entry point to validate end-to-end tracing.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
namespace: demo-app
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
namespace: demo-app
spec:
hosts:
- "*"
gateways:
- bookinfo-gateway
http:
- match:
- uri:
exact: /productpage
route:
- destination:
host: productpage
port:
number: 9080

kubectl apply -f bookinfo-gateway.yaml

Verifying distributed traces across microservices

Generate traffic

Send repeated requests through the ingress gateway to generate a trace chain across productpage, details, reviews, and ratings services.

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
for i in $(seq 1 20); do curl -s -o /dev/null http://$INGRESS_HOST/productpage; done

Port-forward the Jaeger query UI

Access Jaeger locally to confirm traces are arriving with all expected spans.

kubectl port-forward -n istio-system svc/jaeger-query 16686:16686

Open http://localhost:16686, select the productpage.demo-app service, and search for recent traces. Each trace should show connected spans from productpage through details, reviews, and ratings.

Integrating Jaeger UI with Istio ingress gateway

Create a dedicated gateway for Jaeger

Exposing Jaeger through the Istio ingress gateway avoids running a separate load balancer just for observability tools.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: jaeger-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http-jaeger
protocol: HTTP
hosts:
- "jaeger.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: jaeger-vs
namespace: istio-system
spec:
hosts:
- "jaeger.example.com"
gateways:
- jaeger-gateway
http:
- route:
- destination:
host: jaeger-query
port:
number: 16686

kubectl apply -f jaeger-gateway.yaml

Point DNS and test access

Add an A record for jaeger.example.com pointing to the ingress gateway's external IP, then confirm the UI loads.

curl -s -o /dev/null -w "%{http_code}\n" -H "Host: jaeger.example.com" http://203.0.113.10/

Warning: Do not expose the raw Jaeger UI without authentication on a public IP. Add OAuth2 and RBAC as described in configuring Jaeger authentication with OAuth2 and RBAC for enterprise security before exposing it beyond your internal network.

Verify your setup

kubectl get pods -n istio-system
kubectl get telemetry -A
istioctl proxy-config log deploy/productpage-v1.demo-app
kubectl exec -n demo-app deploy/productpage-v1 -c istio-proxy -- curl -s localhost:15000/stats | grep tracing

A healthy setup shows all Istio and Jaeger pods Running, at least one Telemetry resource applied, and non-zero tracing counters on the sidecar's admin stats endpoint.

Common issues
SymptomCauseFixTraces show single spans with no parent-child linksApplication not propagating trace headers on outbound callsForward incoming x-request-id, x-b3-*, and traceparent headers on every outbound HTTP client callNo traces appear in Jaeger UI at allSampling percentage set too low or Telemetry resource not applied to the right namespaceCheck kubectl get telemetry -A and temporarily raise randomSamplingPercentage to 100 for testingJaeger collector pod crashloopingOTLP gRPC port misconfigured or storage backend unreachableCheck kubectl logs -n istio-system deploy/jaeger-collector and confirm storage.type matches your backendSidecar not injected into podsNamespace missing the istio-injection labelRun kubectl label namespace demo-app istio-injection=enabled then restart the deploymentJaeger UI returns 502 through ingress gatewayVirtualService pointing to wrong service name or portConfirm the Jaeger query service name with kubectl get svc -n istio-system and match the port in the VirtualServiceHigh trace volume overwhelming storage backendSampling rate too high for production trafficLower randomSamplingPercentage and use tail-based sampling for error and slow-request capture

Next steps

Implement Istio observability with Jaeger tracing and Kiali dashboard for Kubernetes service mesh

Configure Jaeger distributed tracing on Kubernetes cluster with Helm charts and Elasticsearch backend

Configure Istio distributed tracing with Jaeger and Zipkin for comprehensive microservices observability

Configure advanced Jaeger sampling strategies for high-traffic environments

Set up Istio multi-cluster service mesh with cross-cluster communication

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning for the tracing pipeline, storage retention, sampling tuning as traffic grows, and on-call coverage when the collector falls behind. See how we run infrastructure like this for European teams.

---

### Configure Consul Connect service mesh monitoring with distributed tracing

URL: https://binadit.com/tutorials/configure-consul-connect-service-mesh-monitoring
Category: monitoring
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up comprehensive monitoring for Consul Connect service mesh with Prometheus metrics, Grafana dashboards, Jaeger distributed tracing, and Envoy proxy observability for production-grade service mesh operations.

What this solves

Consul Connect service mesh provides secure service-to-service communication, but operating it reliably requires deep observability into service health, proxy performance, and request flows. This tutorial configures comprehensive monitoring with Prometheus metrics collection, Grafana dashboards for service mesh visualization, and distributed tracing with Jaeger and OpenTelemetry to track requests across your entire service topology.

Prerequisites

You need a running Consul cluster with Connect enabled and at least two services configured to communicate through the service mesh. This tutorial builds on existing Consul Connect infrastructure to add monitoring capabilities.

Update system packages

Start by updating your package manager and installing required dependencies for monitoring components.

sudo apt update && sudo apt upgrade -y
sudo apt install -y wget curl unzip jq

sudo dnf update -y
sudo dnf install -y wget curl unzip jq

Configure Consul metrics collection

Enable Consul telemetry

Configure Consul to export metrics in Prometheus format and enable detailed service mesh telemetry.

telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
metrics_prefix = "consul"
}

connect {
enabled = true
}

ports {
grpc = 8502
http = 8500
}

Configure Connect proxy metrics

Enable detailed metrics collection for Envoy proxies managed by Consul Connect.

connect {
enabled = true

proxy_defaults {
config {
envoy_prometheus_bind_addr = "0.0.0.0:9102"
envoy_stats_bind_addr = "0.0.0.0:9103"
}
}
}

Restart Consul services

Apply the new configuration by restarting Consul on all cluster nodes.

sudo systemctl restart consul
sudo systemctl status consul
curl -s http://localhost:8500/v1/agent/metrics?format=prometheus | head -20

Install and configure Prometheus

Install Prometheus

Download and install the latest version of Prometheus for metrics collection and storage.

wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-2.45.0.linux-amd64.tar.gz
sudo mv prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

Configure Prometheus for Consul metrics

Set up Prometheus to discover and scrape metrics from Consul servers and Connect proxies automatically.

global:
scrape_interval: 15s
evaluation_interval: 15s

scrape_configs:
- job_name: 'consul'
static_configs:
- targets: ['localhost:8500']
metrics_path: /v1/agent/metrics
params:
format: ['prometheus']
scrape_interval: 5s

- job_name: 'consul-connect-proxies'
consul_sd_configs:
- server: 'localhost:8500'
services: []
relabel_configs:
- source_labels: [__meta_consul_service_metadata_proxy_type]
regex: connect-proxy
action: keep
- source_labels: [__meta_consul_service_port]
target_label: __address__
regex: (.*)
replacement: ${1}:9102
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__meta_consul_node]
target_label: node
metrics_path: /metrics
scrape_interval: 5s

- job_name: 'envoy-admin'
consul_sd_configs:
- server: 'localhost:8500'
services: []
relabel_configs:
- source_labels: [__meta_consul_service_metadata_proxy_type]
regex: connect-proxy
action: keep
- source_labels: [__meta_consul_service_port]
target_label: __address__
regex: (.*)
replacement: ${1}:9103
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__meta_consul_node]
target_label: node
metrics_path: /stats/prometheus
scrape_interval: 10s

Create Prometheus systemd service

Configure Prometheus to run as a system service with proper permissions and resource limits.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle \
--storage.tsdb.retention.time=30d

[Install]
WantedBy=multi-user.target

Start Prometheus

Enable and start the Prometheus service, then verify it can scrape Consul metrics.

sudo chown -R prometheus:prometheus /etc/prometheus/
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

Install and configure Grafana dashboards

Install Grafana

Install Grafana for creating comprehensive service mesh monitoring dashboards.

sudo apt install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

sudo tee /etc/yum.repos.d/grafana.repo<

Configure Grafana data source

Add Prometheus as a data source for Grafana to visualize Consul Connect metrics.

apiVersion: 1

datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://localhost:9090
isDefault: true
editable: true

Create Consul Connect dashboard

Deploy a comprehensive dashboard for monitoring Consul Connect service mesh metrics.

{
"dashboard": {
"id": null,
"title": "Consul Connect Service Mesh",
"tags": ["consul", "connect", "service-mesh"],
"timezone": "browser",
"panels": [
{
"title": "Service Health",
"type": "stat",
"targets": [
{
"expr": "consul_health_service_query_tag{status=\"passing\"}",
"legendFormat": "Healthy Services"
}
],
"gridPos": {"h": 8, "w": 6, "x": 0, "y": 0}
},
{
"title": "Proxy Connections",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_cluster_upstream_cx_connect_total[5m])",
"legendFormat": "{{service}} - {{cluster_name}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 6, "y": 0}
},
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_http_inbound_0_0_0_0_20000_http_requests_total[5m])",
"legendFormat": "{{service}} - Requests/sec"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
},
{
"title": "Response Times",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(envoy_http_inbound_0_0_0_0_20000_http_request_duration_milliseconds_bucket[5m]))",
"legendFormat": "{{service}} - 95th percentile"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "10s"
}
}

Start Grafana

Enable and start Grafana, then access the dashboard to verify service mesh metrics visualization.

sudo chown -R grafana:grafana /var/lib/grafana/
sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server
echo "Grafana available at http://localhost:3000 (admin/admin)"

Configure distributed tracing with Jaeger

Install Jaeger

Install Jaeger for distributed tracing across your Consul Connect service mesh.

wget https://github.com/jaegertracing/jaeger/releases/download/v1.47.0/jaeger-1.47.0-linux-amd64.tar.gz
tar -xzf jaeger-1.47.0-linux-amd64.tar.gz
sudo mv jaeger-1.47.0-linux-amd64/jaeger-all-in-one /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false jaeger
sudo mkdir -p /var/lib/jaeger
sudo chown jaeger:jaeger /var/lib/jaeger

Configure Jaeger service

Set up Jaeger to collect traces from Envoy proxies in your service mesh with proper storage configuration.

[Unit]
Description=Jaeger Tracing
After=network.target

[Service]
User=jaeger
Group=jaeger
Type=simple
ExecStart=/usr/local/bin/jaeger-all-in-one \
--collector.grpc-server.host-port=:14250 \
--collector.http-server.host-port=:14268 \
--query.host-port=:16686 \
--memory.max-traces=50000 \
--log-level=info
Restart=always

[Install]
WantedBy=multi-user.target

Configure Envoy tracing

Enable distributed tracing in Consul Connect by configuring Envoy proxies to send trace data to Jaeger.

connect {
enabled = true

proxy_defaults {
config {
envoy_tracing_json = jsonencode({
http = {
name = "envoy.tracers.zipkin"
typed_config = {
"@type" = "type.googleapis.com/envoy.extensions.tracers.zipkin.v3.ZipkinConfig"
collector_cluster = "jaeger_collector"
collector_endpoint_version = "HTTP_JSON"
collector_endpoint = "/api/v2/spans"
shared_span_context = false
}
}
})

envoy_extra_static_clusters_json = jsonencode({
jaeger_collector = {
name = "jaeger_collector"
connect_timeout = "1s"
type = "STRICT_DNS"
lb_policy = "ROUND_ROBIN"
load_assignment = {
cluster_name = "jaeger_collector"
endpoints = [{
lb_endpoints = [{
endpoint = {
address = {
socket_address = {
address = "127.0.0.1"
port_value = 14268
}
}
}
}]
}]
}
}
})
}
}
}

Start Jaeger and restart Connect proxies

Start the Jaeger service and restart your Connect proxies to enable tracing.

sudo systemctl daemon-reload
sudo systemctl enable --now jaeger
sudo systemctl status jaeger

# Restart Consul to pick up tracing configuration
sudo systemctl restart consul

# Restart any existing Connect proxies
consul connect proxy -sidecar-for web-service &
echo "Jaeger UI available at http://localhost:16686"

Configure OpenTelemetry integration

Install OpenTelemetry Collector

Deploy the OpenTelemetry Collector to provide advanced telemetry processing and export capabilities.

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.80.0/otelcol_0.80.0_linux_amd64.tar.gz
tar -xzf otelcol_0.80.0_linux_amd64.tar.gz
sudo mv otelcol /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false otelcol
sudo mkdir -p /etc/otelcol
sudo chown otelcol:otelcol /etc/otelcol

Configure OpenTelemetry for service mesh

Set up the collector to receive traces from Envoy and export them to Jaeger and metrics to Prometheus.

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

zipkin:
endpoint: 0.0.0.0:9411

prometheus:
config:
scrape_configs:
- job_name: 'envoy-metrics'
static_configs:
- targets: ['localhost:9102']

processors:
batch:
timeout: 1s
send_batch_size: 1024

attributes:
actions:
- key: service.name
action: upsert
from_attribute: service_name
- key: service.version
action: upsert
from_attribute: service_version

exporters:
jaeger:
endpoint: http://localhost:14250
tls:
insecure: true

prometheus:
endpoint: "0.0.0.0:8889"

logging:
loglevel: debug

service:
pipelines:
traces:
receivers: [otlp, zipkin]
processors: [batch, attributes]
exporters: [jaeger, logging]

metrics:
receivers: [prometheus]
processors: [batch]
exporters: [prometheus, logging]

Start OpenTelemetry Collector

Create a systemd service for the OpenTelemetry Collector and start it.

[Unit]
Description=OpenTelemetry Collector
After=network.target

[Service]
User=otelcol
Group=otelcol
Type=simple
ExecStart=/usr/local/bin/otelcol --config=/etc/otelcol/config.yaml
Restart=always

[Install]
WantedBy=multi-user.target

sudo chown -R otelcol:otelcol /etc/otelcol/
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol
sudo systemctl status otelcol

Monitor Envoy proxy metrics

Configure enhanced Envoy metrics

Enable comprehensive Envoy metrics collection including circuit breaker status and connection pool metrics.

connect {
enabled = true

proxy_defaults {
config {
envoy_prometheus_bind_addr = "0.0.0.0:9102"
envoy_stats_bind_addr = "0.0.0.0:9103"

# Enable additional Envoy stats
envoy_stats_config_json = jsonencode({
stats_config = {
histogram_bucket_settings = [
{
match = {
prefix = "http.inbound"
}
buckets = [0.5, 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]
}
]
}
})

# Enable circuit breaker and outlier detection stats
envoy_extra_static_clusters_json = jsonencode({
circuit_breakers = {
thresholds = [
{
priority = "DEFAULT"
max_connections = 1024
max_pending_requests = 256
max_requests = 1024
max_retries = 3
}
]
}
outlier_detection = {
consecutive_5xx = 3
interval = "30s"
base_ejection_time = "30s"
max_ejection_percent = 50
}
})
}
}
}

Create Envoy proxy dashboard

Deploy a specialized dashboard for monitoring Envoy proxy performance and health metrics.

{
"dashboard": {
"id": null,
"title": "Envoy Proxy Metrics",
"tags": ["envoy", "proxy", "consul-connect"],
"panels": [
{
"title": "Connection Pool Status",
"type": "graph",
"targets": [
{
"expr": "envoy_cluster_upstream_cx_active",
"legendFormat": "{{cluster_name}} - Active Connections"
},
{
"expr": "envoy_cluster_upstream_cx_overflow",
"legendFormat": "{{cluster_name}} - Overflow"
}
]
},
{
"title": "Circuit Breaker Status",
"type": "stat",
"targets": [
{
"expr": "envoy_cluster_circuit_breakers_default_cx_open",
"legendFormat": "{{cluster_name}} - Circuit Open"
}
]
},
{
"title": "Request Success Rate",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_http_inbound_0_0_0_0_20000_http_requests_total{response_code!~\"5..\"}[5m]) / rate(envoy_http_inbound_0_0_0_0_20000_http_requests_total[5m]) * 100",
"legendFormat": "{{service}} - Success Rate %"
}
]
},
{
"title": "Outlier Detection Events",
"type": "graph",
"targets": [
{
"expr": "rate(envoy_cluster_outlier_detection_ejections_active[5m])",
"legendFormat": "{{cluster_name}} - Ejections"
}
]
}
]
}
}

Restart services for enhanced metrics

Apply the enhanced Envoy configuration by restarting Consul and any running proxies.

sudo systemctl restart consul
sudo systemctl restart grafana-server

# Verify metrics endpoints are responding
curl -s http://localhost:9102/metrics | grep envoy_cluster | head -5
curl -s http://localhost:9103/stats | grep circuit_breakers | head -5

Verify your setup

Test the complete monitoring stack

---

### Configure Kubernetes RBAC with service accounts and cluster roles for secure access control

URL: https://binadit.com/tutorials/configure-kubernetes-rbac-with-service-accounts-and-cluster-roles
Category: devops
Difficulty: intermediate
Time: ~25 minutes
Author: Binadit Tech Team

> Learn to implement Kubernetes Role-Based Access Control (RBAC) with service accounts, cluster roles, and role bindings for granular permissions and secure cluster access management.

What this solves

Kubernetes RBAC provides fine-grained access control for your cluster resources by defining who can perform specific actions on which resources. This tutorial shows you how to create service accounts with appropriate permissions using cluster roles and role bindings, ensuring secure access control while maintaining operational flexibility.

Understanding RBAC components

Kubernetes RBAC consists of four main components that work together to control access. Service accounts represent identities for pods and external systems. Roles and ClusterRoles define permissions for specific actions. RoleBindings and ClusterRoleBindings associate subjects (users, groups, or service accounts) with roles.

Note: ClusterRoles apply cluster-wide while Roles are namespace-specific. ClusterRoleBindings grant cluster-wide permissions while RoleBindings grant namespace-specific permissions.

Step-by-step configuration

Verify RBAC is enabled

Check that RBAC is enabled in your Kubernetes cluster by examining the API server configuration.

kubectl auth can-i list pods --as=system:anonymous
kubectl cluster-info dump | grep -i authorization-mode

Create a dedicated namespace

Create a namespace for testing RBAC configurations to isolate your setup.

kubectl create namespace rbac-demo

Create service accounts

Create service accounts with specific metadata and labels for better organization.

apiVersion: v1
kind: ServiceAccount
metadata:
name: pod-reader
namespace: rbac-demo
labels:
app: rbac-demo
role: reader
annotations:
description: "Service account for reading pod information"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployment-manager
namespace: rbac-demo
labels:
app: rbac-demo
role: manager
annotations:
description: "Service account for managing deployments"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cluster-admin-sa
namespace: rbac-demo
labels:
app: rbac-demo
role: admin
annotations:
description: "Service account with cluster-wide admin privileges"

kubectl apply -f serviceaccounts.yaml

Create cluster roles with specific permissions

Define cluster roles with granular permissions for different access levels.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pod-reader-role
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: deployment-manager-role
labels:
app: rbac-demo
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: namespace-admin-role
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["apps"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["extensions"]
resources: ["*"]
verbs: ["*"]

kubectl apply -f clusterroles.yaml

Create namespace-specific roles

Create roles that are scoped to specific namespaces for more granular control.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: secret-manager
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: service-manager
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch"]

kubectl apply -f roles.yaml

Create cluster role bindings

Bind service accounts to cluster roles to grant cluster-wide permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pod-reader-binding
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: pod-reader
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: pod-reader-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: deployment-manager-binding
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: deployment-manager-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-binding
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: cluster-admin-sa
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io

kubectl apply -f clusterrolebindings.yaml

Create namespace-specific role bindings

Bind service accounts to namespace roles for scoped permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: secret-manager-binding
namespace: rbac-demo
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: rbac-demo
roleRef:
kind: Role
name: secret-manager
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: service-manager-binding
namespace: rbac-demo
labels:
app: rbac-demo
subjects:
- kind: ServiceAccount
name: deployment-manager
namespace: rbac-demo
roleRef:
kind: Role
name: service-manager
apiGroup: rbac.authorization.k8s.io

kubectl apply -f rolebindings.yaml

Configure service account tokens

Create long-lived tokens for service accounts that need persistent access.

apiVersion: v1
kind: Secret
metadata:
name: pod-reader-token
namespace: rbac-demo
annotations:
kubernetes.io/service-account.name: pod-reader
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
name: deployment-manager-token
namespace: rbac-demo
annotations:
kubernetes.io/service-account.name: deployment-manager
type: kubernetes.io/service-account-token

kubectl apply -f serviceaccount-tokens.yaml

Create test pods with service accounts

Deploy pods that use the configured service accounts to test RBAC permissions.

apiVersion: v1
kind: Pod
metadata:
name: pod-reader-test
namespace: rbac-demo
labels:
app: rbac-demo
test: pod-reader
spec:
serviceAccountName: pod-reader
containers:
- name: kubectl
image: bitnami/kubectl:latest
command: ['sleep', '3600']
restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
name: deployment-manager-test
namespace: rbac-demo
labels:
app: rbac-demo
test: deployment-manager
spec:
serviceAccountName: deployment-manager
containers:
- name: kubectl
image: bitnami/kubectl:latest
command: ['sleep', '3600']
restartPolicy: Never

kubectl apply -f test-pods.yaml

Implementing advanced RBAC policies

Create resource-specific permissions

Configure roles with permissions for specific resources and resource names.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: specific-resource-manager
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-secret", "db-secret"]
verbs: ["get", "update"]
- apiGroups: ["apps"]
resources: ["deployments"]
resourceNames: ["app-deployment"]
verbs: ["get", "update", "patch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"]
resourceNames: ["app-config"]

kubectl apply -f resource-specific-role.yaml

Implement attribute-based access control

Create roles that use label selectors and field selectors for fine-grained access.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: label-based-access
labels:
app: rbac-demo
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch"]
resourceNames: []
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]

kubectl apply -f attribute-based-role.yaml

Configure admission control integration

Create roles that work with admission controllers for policy enforcement. This example works well with OPA Gatekeeper configurations.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: admission-controller-reviewer
labels:
app: rbac-demo
rules:
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["validatingadmissionwebhooks", "mutatingadmissionwebhooks"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["authorization.k8s.io"]
resources: ["subjectaccessreviews"]
verbs: ["create"]
- apiGroups: ["authentication.k8s.io"]
resources: ["tokenreviews"]
verbs: ["create"]

kubectl apply -f admission-control-role.yaml

Verify your RBAC setup

Test service account permissions

Verify that each service account has the expected permissions and restrictions.

# Test pod reader permissions
kubectl auth can-i list pods --as=system:serviceaccount:rbac-demo:pod-reader
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:pod-reader

# Test deployment manager permissions
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:deployment-manager
kubectl auth can-i delete secrets --as=system:serviceaccount:rbac-demo:deployment-manager -n rbac-demo

# Test cluster admin permissions
kubectl auth can-i "*" "*" --as=system:serviceaccount:rbac-demo:cluster-admin-sa

Verify permissions from within pods

Test actual API access from the pods using the service accounts.

# Test from pod-reader pod
kubectl exec -n rbac-demo pod-reader-test -- kubectl get pods --all-namespaces
kubectl exec -n rbac-demo pod-reader-test -- kubectl get deployments

# Test from deployment-manager pod
kubectl exec -n rbac-demo deployment-manager-test -- kubectl get deployments
kubectl exec -n rbac-demo deployment-manager-test -- kubectl create deployment test-deploy --image=nginx

Audit RBAC configuration

Review the complete RBAC setup and identify potential security gaps.

# List all service accounts
kubectl get serviceaccounts -n rbac-demo -o wide

# List all role bindings
kubectl get rolebindings,clusterrolebindings -n rbac-demo -o wide

# Check role definitions
kubectl describe clusterrole pod-reader-role
kubectl describe role secret-manager -n rbac-demo

# Verify token secrets
kubectl get secrets -n rbac-demo | grep token

Security best practices

Security Warning: Never use the cluster-admin role for regular applications. Always follow the principle of least privilege when assigning permissions.

Implement these security practices for production RBAC deployments. Use specific resource names when possible instead of wildcard permissions. Regularly audit and rotate service account tokens. Enable audit logging to track RBAC decisions and access patterns.

Create separate service accounts for each application or service component. Group permissions logically and use descriptive names for roles and bindings. Document the purpose and scope of each service account for your team.

For comprehensive cluster security, combine RBAC with network policies and pod security standards to create defense-in-depth protection.

Common issues

SymptomCauseFix

Service account can't access resourcesMissing role bindingCreate appropriate RoleBinding or ClusterRoleBinding
"Forbidden" errors in pod logsInsufficient permissions in roleAdd required verbs and resources to the role definition
Service account token not foundToken secret not createdCreate Secret with kubernetes.io/service-account-token type
Cross-namespace access deniedUsing Role instead of ClusterRoleUse ClusterRole and ClusterRoleBinding for cross-namespace access
Application can't read own metadataMissing self-inspection permissionsAdd permissions for pods/self and configmaps in the same namespace
Service discovery not workingMissing endpoints and services permissionsAdd get/list verbs for services and endpoints resources

Next steps

Implement network policies for pod-to-pod security

Integrate HashiCorp Vault for advanced secrets management

Configure ingress controllers with automated SSL certificate management

Set up Pod Security Standards with admission controllers

Configure Calico CNI for advanced network microsegmentation

Running this in production?

Want this handled for you? Setting up RBAC once is straightforward. Keeping it patched, monitored, backed up and tuned across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.

---

### Implement Deno microservices architecture with service discovery and load balancing

URL: https://binadit.com/tutorials/implement-deno-microservices-architecture
Category: devops
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Build a production-ready Deno microservices architecture with Consul service discovery, HAProxy load balancing, and comprehensive monitoring using Prometheus. This tutorial covers container orchestration, health checks, and automated failover for scalable applications.

What this solves

Modern applications need to scale horizontally by breaking into smaller services, but managing multiple Deno microservices becomes complex without proper service discovery and load balancing. This tutorial builds a production-ready architecture where Deno services automatically register themselves with Consul, HAProxy distributes traffic based on health checks, and Prometheus monitors the entire stack.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you get the latest versions of all components.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget unzip software-properties-common

sudo dnf update -y
sudo dnf install -y curl wget unzip

Install Deno runtime

Install Deno using the official installation script, then verify the installation.

curl -fsSL https://deno.land/install.sh | sh
echo 'export DENO_INSTALL="$HOME/.deno"' >> ~/.bashrc
echo 'export PATH="$DENO_INSTALL/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
deno --version

Install Consul for service discovery

Download and install HashiCorp Consul for service registration and health checking.

CONSUL_VERSION="1.17.0"
wget https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
unzip consul_${CONSUL_VERSION}_linux_amd64.zip
sudo mv consul /usr/local/bin/
sudo chmod +x /usr/local/bin/consul
consul version

Configure Consul server

Create Consul configuration directory and setup the main configuration file.

sudo mkdir -p /etc/consul.d /opt/consul
sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo chown -R consul:consul /etc/consul.d /opt/consul

datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
server = true
bootstrap_expect = 1
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
ui_config {
enabled = true
}
connect {
enabled = true
}
ports {
grpc = 8502
}
acl = {
enabled = false
default_policy = "allow"
}

Create Consul systemd service

Setup Consul to run as a systemd service with automatic restarts.

[Unit]
Description=Consul
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul

Install HAProxy load balancer

Install HAProxy for distributing traffic across Deno microservices with health checks.

sudo apt install -y haproxy

sudo dnf install -y haproxy

Configure HAProxy with Consul integration

Setup HAProxy configuration with service discovery integration and health checks.

global
daemon
maxconn 4096
log stdout local0
stats socket /var/run/haproxy.sock mode 660 level admin
stats timeout 30s

defaults
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
option httplog
option dontlognull
option redispatch
retries 3

frontend api_gateway
bind *:80
bind *:443 ssl crt /etc/ssl/certs/haproxy.pem
redirect scheme https if !{ ssl_fc }

# Route based on URL path
acl is_users_service path_beg /api/users
acl is_orders_service path_beg /api/orders
acl is_health path /health

use_backend users_service if is_users_service
use_backend orders_service if is_orders_service
use_backend health_check if is_health
default_backend api_default

backend users_service
balance roundrobin
option httpchk GET /health
http-check expect status 200
# Dynamic backend discovery via Consul
server-template users 3 _users._tcp.service.consul:8080 check resolvers consul

backend orders_service
balance roundrobin
option httpchk GET /health
http-check expect status 200
server-template orders 3 _orders._tcp.service.consul:8081 check resolvers consul

backend api_default
balance roundrobin
server default 127.0.0.1:8000 check

backend health_check
http-request return status 200 content-type text/plain string "HAProxy healthy"

resolvers consul
nameserver consul 127.0.0.1:8600
accepted_payload_size 8192
hold valid 5s

listen stats
bind *:8404
stats enable
stats uri /
stats refresh 5s
stats admin if TRUE

Install Prometheus for monitoring

Download and install Prometheus to monitor your microservices architecture.

PROMETHEUS_VERSION="2.48.0"
wget https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROMETHEUS_VERSION}.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-${PROMETHEUS_VERSION}.linux-amd64/promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo useradd --system --home /var/lib/prometheus --shell /bin/false prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

Configure Prometheus with service discovery

Setup Prometheus to automatically discover services registered in Consul.

global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "/etc/prometheus/rules/*.yml"

alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

- job_name: 'consul'
static_configs:
- targets: ['localhost:8500']
metrics_path: /v1/agent/metrics
params:
format: ['prometheus']

- job_name: 'haproxy'
static_configs:
- targets: ['localhost:8404']
metrics_path: /stats/prometheus

- job_name: 'consul-services'
consul_sd_configs:
- server: 'localhost:8500'
services: ['users', 'orders']
relabel_configs:
- source_labels: [__meta_consul_service]
target_label: job
- source_labels: [__meta_consul_node]
target_label: instance
- source_labels: [__meta_consul_service_address]
target_label: __address__
- source_labels: [__meta_consul_service_port]
target_label: __address__
regex: '(.*)'
replacement: '${1}:${__meta_consul_service_port}'

- job_name: 'deno-services'
consul_sd_configs:
- server: 'localhost:8500'
tags: ['deno', 'microservice']
relabel_configs:
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__address__]
target_label: __address__
regex: '([^:]+):(\d+)'
replacement: '${1}:${2}'
metrics_path: '/metrics'

Create Prometheus systemd service

Setup Prometheus to run as a systemd service with proper permissions.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Create Deno microservice template

Create a reusable template for Deno microservices with service registration and metrics.

mkdir -p ~/deno-microservices
cd ~/deno-microservices

import { serve } from "https://deno.land/std@0.208.0/http/server.ts";

export interface ServiceConfig {
name: string;
port: number;
version: string;
consulUrl?: string;
}

export class MicroService {
private config: ServiceConfig;
private routes: Map

Create users microservice

Build the first microservice for user management with Consul registration.

import { MicroService } from "./service-base.ts";

const service = new MicroService({
name: "users",
port: 8080,
version: "1.0.0",
});

// Mock user data
const users = new Map([
["1", { id: "1", name: "Alice Johnson", email: "alice@example.com" }],
["2", { id: "2", name: "Bob Smith", email: "bob@example.com" }],
["3", { id: "3", name: "Carol Wilson", email: "carol@example.com" }],
]);

// GET /api/users
service.addRoute("/api/users", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}

return new Response(JSON.stringify(Array.from(users.values())), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});

// GET /api/users/:id
service.addRoute("/api/users/", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}

const url = new URL(req.url);
const id = url.pathname.split("/").pop();

if (!id || !users.has(id)) {
return new Response("User not found", { status: 404 });
}

return new Response(JSON.stringify(users.get(id)), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});

if (import.meta.main) {
service.start();
}

Create orders microservice

Build the second microservice for order management with service discovery.

import { MicroService } from "./service-base.ts";

const service = new MicroService({
name: "orders",
port: 8081,
version: "1.0.0",
});

// Mock order data
const orders = new Map([
["1", { id: "1", userId: "1", items: ["laptop", "mouse"], total: 1299.99, status: "shipped" }],
["2", { id: "2", userId: "2", items: ["phone"], total: 899.99, status: "processing" }],
["3", { id: "3", userId: "1", items: ["keyboard"], total: 129.99, status: "delivered" }],
]);

// GET /api/orders
service.addRoute("/api/orders", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}

const url = new URL(req.url);
const userId = url.searchParams.get("userId");

let result = Array.from(orders.values());

if (userId) {
result = result.filter(order => order.userId === userId);
}

return new Response(JSON.stringify(result), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});

// GET /api/orders/:id
service.addRoute("/api/orders/", async (req: Request) => {
if (req.method !== "GET") {
return new Response("Method Not Allowed", { status: 405 });
}

const url = new URL(req.url);
const id = url.pathname.split("/").pop();

if (!id || !orders.has(id)) {
return new Response("Order not found", { status: 404 });
}

return new Response(JSON.stringify(orders.get(id)), {
status: 200,
headers: { "Content-Type": "application/json" },
});
});

if (import.meta.main) {
service.start();
}

Create systemd services for Deno microservices

Setup systemd services to manage the Deno microservices lifecycle with automatic restarts.

[Unit]
Description=Deno Users Microservice
After=network.target consul.service
Requires=consul.service

[Service]
Type=simple
User=deno
Group=deno
WorkingDirectory=/home/deno/microservices
ExecStart=/home/deno/.deno/bin/deno run --allow-net --allow-env users-service.ts
Restart=always
RestartSec=5
Environment=SERVICE_ADDRESS=localhost
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

[Unit]
Description=Deno Orders Microservice
After=network.target consul.service
Requires=consul.service

[Service]
Type=simple
User=deno
Group=deno
WorkingDirectory=/home/deno/microservices
ExecStart=/home/deno/.deno/bin/deno run --allow-net --allow-env orders-service.ts
Restart=always
RestartSec=5
Environment=SERVICE_ADDRESS=localhost
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Create dedicated user and setup permissions

Create a dedicated user for running Deno services securely with minimal permissions.

sudo useradd --system --home /home/deno --create-home --shell /bin/bash deno
sudo mkdir -p /home/deno/microservices
sudo cp ~/deno-microservices/* /home/deno/microservices/
sudo chown -R deno:deno /home/deno
sudo chmod 755 /home/deno/microservices
sudo chmod 644 /home/deno/microservices/*.ts

Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions like 755 for directories and 644 for files.

Start all services

Enable and start all the services in the correct order with dependency checking.

# Start Consul first
sudo systemctl status consul

# Start HAProxy
sudo

---

### Implement Kubernetes security scanning with Falco and OPA Gatekeeper for runtime protection

URL: https://binadit.com/tutorials/implement-kubernetes-security-scanning-with-falco-and-opa
Category: security
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up comprehensive Kubernetes security with Falco for runtime threat detection and OPA Gatekeeper for admission control policy enforcement. This tutorial covers installation, configuration, and custom security policies.

What this solves

Kubernetes clusters need layered security to detect runtime threats and enforce admission policies. Falco monitors system calls and container behavior for suspicious activity, while OPA Gatekeeper validates resources against security policies before deployment. Together they provide comprehensive protection against malicious workloads, privilege escalation, and policy violations.

Step-by-step installation

Install Helm for package management

Both Falco and Gatekeeper use Helm charts for installation. Install Helm first to manage the deployments.

curl https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz -o helm.tar.gz
tar -zxvf helm.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm version

Add Helm repositories

Add the official repositories for Falco and Gatekeeper charts.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update

Install Falco for runtime security monitoring

Deploy Falco as a DaemonSet to monitor all nodes for suspicious activity. The default configuration includes rules for common threats.

kubectl create namespace falco-system
helm install falco falcosecurity/falco \
--namespace falco-system \
--set driver.kind=ebpf \
--set falco.grpc.enabled=true \
--set falco.grpcOutput.enabled=true

Install OPA Gatekeeper for admission control

Deploy Gatekeeper to validate all resource requests against defined policies before they reach the API server.

kubectl create namespace gatekeeper-system
helm install gatekeeper gatekeeper/gatekeeper \
--namespace gatekeeper-system \
--set replicas=3 \
--set auditInterval=60

Verify installations

Check that both systems are running correctly across all nodes.

kubectl get pods -n falco-system
kubectl get pods -n gatekeeper-system
kubectl get validatingadmissionwebhooks

Configure security policies and rules

Create custom Falco rules

Add detection rules for your specific environment. This example detects unauthorized network connections.

apiVersion: v1
kind: ConfigMap
metadata:
name: falco-custom-rules
namespace: falco-system
data:
custom_rules.yaml: |
- rule: Unexpected outbound connection
desc: Detect unexpected outbound network connections
condition: >
outbound and not fd.typechar=4 and not fd.typechar=6 and 
not proc.name in (curl, wget, apt, yum, dnf) and
not container.image.repository in (docker.io/library/alpine, gcr.io/distroless)
output: >
Unexpected outbound connection (command=%proc.cmdline connection=%fd.name 
user=%user.name container=%container.name image=%container.image.repository)
priority: WARNING
tags: [network, outbound]

- rule: Privileged container spawned
desc: Detect containers running with privileged access
condition: >
spawned_process and container and proc.vpid=1 and 
container.privileged=true
output: >
Privileged container spawned (command=%proc.cmdline user=%user.name 
container=%container.name image=%container.image.repository)
priority: CRITICAL
tags: [container, privilege]

kubectl apply -f falco-custom-rules.yaml

Create Gatekeeper constraint templates

Define reusable policy templates that can be applied to different resource types.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: requiresecuritycontext
spec:
crd:
spec:
names:
kind: RequireSecurityContext
validation:
properties:
runAsNonRoot:
type: boolean
readOnlyRootFilesystem:
type: boolean
allowPrivilegeEscalation:
type: boolean
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requiresecuritycontext

violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.securityContext.runAsNonRoot
msg := "Container must run as non-root user"
}

violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.securityContext.readOnlyRootFilesystem
msg := "Container must use read-only root filesystem"
}

violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
container.securityContext.allowPrivilegeEscalation != false
msg := "Container must not allow privilege escalation"
}

kubectl apply -f require-security-context-template.yaml

Apply security context constraints

Use the template to enforce security contexts on all pods in production namespaces.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireSecurityContext
metadata:
name: must-have-security-context
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production", "staging"]
parameters:
runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false

kubectl apply -f security-context-constraint.yaml

Create resource limit template

Prevent resource exhaustion attacks by requiring CPU and memory limits on all containers.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: requireresources
spec:
crd:
spec:
names:
kind: RequireResources
validation:
properties:
limits:
type: array
items:
type: string
requests:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requireresources

violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
required := input.parameters.limits
provided := container.resources.limits
missing := required[_]
not provided[missing]
msg := sprintf("Container missing required resource limit: %v", [missing])
}

violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
required := input.parameters.requests
provided := container.resources.requests
missing := required[_]
not provided[missing]
msg := sprintf("Container missing required resource request: %v", [missing])
}

kubectl apply -f require-resources-template.yaml

Apply resource limit constraints

Enforce CPU and memory limits on all production workloads.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireResources
metadata:
name: must-have-resource-limits
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production", "staging"]
parameters:
limits: ["cpu", "memory"]
requests: ["cpu", "memory"]

kubectl apply -f resource-limits-constraint.yaml

Set up monitoring and alerting

Configure Falco alerts

Set up webhook notifications to send security alerts to your monitoring system.

apiVersion: v1
kind: ConfigMap
metadata:
name: falco-config-override
namespace: falco-system
data:
falco.yaml: |
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d

json_output: true
json_include_output_property: true

http_output:
enabled: true
url: "http://webhook-service.monitoring.svc.cluster.local:8080/falco"
user_agent: "falco/0.35.0"

priority: WARNING

syscall_event_drops:
actions:
- log
- alert
rate: 0.1
max_burst: 1000

kubectl apply -f falco-config-override.yaml
kubectl rollout restart daemonset/falco -n falco-system

Create alerting webhook service

Deploy a simple webhook receiver to process Falco alerts and forward them to your notification system.

apiVersion: apps/v1
kind: Deployment
metadata:
name: security-webhook
namespace: monitoring
spec:
replicas: 2
selector:
matchLabels:
app: security-webhook
template:
metadata:
labels:
app: security-webhook
spec:
containers:
- name: webhook
image: falcosecurity/falcosidekick:2.25.0
ports:
- containerPort: 2801
env:
- name: WEBHOOK_URL
value: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
- name: DEBUG
value: "true"
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
name: security-webhook
namespace: monitoring
spec:
selector:
app: security-webhook
ports:
- port: 8080
targetPort: 2801
type: ClusterIP

kubectl create namespace monitoring
kubectl apply -f webhook-deployment.yaml

Monitor Gatekeeper violations

Create a monitoring dashboard to track policy violations and system health.

apiVersion: v1
kind: ServiceMonitor
metadata:
name: gatekeeper-metrics
namespace: gatekeeper-system
spec:
selector:
matchLabels:
app: gatekeeper
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: gatekeeper-alerts
namespace: gatekeeper-system
spec:
groups:
- name: gatekeeper.rules
rules:
- alert: GatekeeperViolations
expr: increase(gatekeeper_violations_total[5m]) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "High number of Gatekeeper policy violations"
description: "{{ $value }} policy violations detected in the last 5 minutes"

- alert: GatekeeperDown
expr: up{job="gatekeeper"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Gatekeeper is down"
description: "Gatekeeper admission controller is not responding"

kubectl apply -f gatekeeper-monitoring.yaml

Test security policies

Test Gatekeeper admission control

Verify that policies block non-compliant workloads by trying to deploy a pod without required security context.

apiVersion: v1
kind: Pod
metadata:
name: insecure-test-pod
namespace: production
spec:
containers:
- name: test
image: nginx:latest
ports:
- containerPort: 80

kubectl apply -f test-insecure-pod.yaml

This should fail with a message about missing security context and resource limits.

Test compliant pod deployment

Deploy a pod that meets all security requirements.

apiVersion: v1
kind: Pod
metadata:
name: secure-test-pod
namespace: production
spec:
containers:
- name: test
image: nginx:latest
ports:
- containerPort: 80
securityContext:
runAsNonRoot: true
runAsUser: 1000
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
volumeMounts:
- name: tmp
mountPath: /tmp
- name: var-cache
mountPath: /var/cache/nginx
volumes:
- name: tmp
emptyDir: {}
- name: var-cache
emptyDir: {}

kubectl apply -f test-secure-pod.yaml

Generate test security events

Trigger Falco rules to verify runtime monitoring works correctly.

# Create a test pod that will trigger Falco rules
kubectl run falco-test --image=alpine --rm -it --restart=Never -- sh

# Inside the container, run commands that should trigger alerts:
ps aux
netstat -an
find /etc -name "*passwd*"
wget google.com
exit

Verify your setup

# Check Falco is detecting events
kubectl logs -n falco-system -l app.kubernetes.io/name=falco --tail=50

# Verify Gatekeeper policies are active
kubectl get constraints
kubectl get constrainttemplates

# Check for policy violations
kubectl describe RequireSecurityContext must-have-security-context
kubectl describe RequireResources must-have-resource-limits

# Test webhook connectivity
kubectl logs -n monitoring -l app=security-webhook

Important: Falco generates many events in a typical cluster. Configure appropriate filters and alert thresholds to avoid notification fatigue while ensuring critical security events are not missed.

Common issues

SymptomCauseFix

Falco pods failing to starteBPF driver not supportedUse --set driver.kind=module instead of ebpf
Gatekeeper blocking all podsConstraint too restrictiveAdd namespace exclusions or adjust match criteria
High CPU usage from FalcoToo many syscall eventsTune rules and add filters for noisy processes
Webhook not receiving alertsNetwork policy blocking trafficAllow egress from Falco namespace to webhook service
Policies not applyingGatekeeper not readyWait for all admission webhooks to be ready

Advanced configuration

Performance tuning: In high-traffic clusters, consider using Falco's gRPC output with a separate collector service to reduce resource usage and improve alert processing efficiency.

For production deployments, also consider integrating with Kubernetes RBAC for comprehensive access control and OpenTelemetry monitoring for complete observability across your security stack.

Next steps

Implement Kubernetes RBAC with service accounts for access control

Set up Kubernetes container image security scanning with Trivy

Implement Kubernetes network policies for pod-to-pod security

Configure Kubernetes secrets management with Vault integration

Set up Kubernetes monitoring with Prometheus Operator

Running this in production?

Need this managed? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. See how we run infrastructure like this for European teams.

---

### Configure OpenTelemetry custom metrics for application monitoring with Prometheus and Grafana

URL: https://binadit.com/tutorials/configure-opentelemetry-custom-metrics-for-application-monitoring
Category: monitoring
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team

> Set up OpenTelemetry SDK to collect custom application metrics, export them to Prometheus for storage, and visualize performance data in Grafana dashboards with automated alerting.

What this solves

OpenTelemetry custom metrics give you detailed insights into your application's performance beyond basic system metrics. You can track business-specific metrics like user sign-ups, order completion rates, or API response times. This tutorial shows you how to instrument applications with OpenTelemetry, send metrics to Prometheus, and build Grafana dashboards for monitoring and alerting.

Step-by-step installation

Install OpenTelemetry Collector

The OpenTelemetry Collector receives metrics from your applications and forwards them to Prometheus. Download and install the latest collector binary.

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.91.0/otelcol_0.91.0_linux_amd64.tar.gz
tar -xzf otelcol_0.91.0_linux_amd64.tar.gz
sudo mv otelcol /usr/local/bin/
sudo chmod +x /usr/local/bin/otelcol

Create collector configuration

Configure the collector to receive OTLP metrics and export them to Prometheus format. This config enables metric collection on port 4318 and serves Prometheus metrics on port 8889.

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
limit_mib: 512

exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "app"
const_labels:
environment: "production"

service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
telemetry:
logs:
level: info

Create systemd service for collector

Set up the collector as a systemd service for automatic startup and management.

[Unit]
Description=OpenTelemetry Collector
After=network.target

[Service]
Type=simple
User=nobody
Group=nogroup
ExecStart=/usr/local/bin/otelcol --config=/etc/otelcol-config.yaml
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Start OpenTelemetry Collector

Enable and start the collector service to begin accepting metrics from your applications.

sudo systemctl daemon-reload
sudo systemctl enable --now otelcol
sudo systemctl status otelcol

Install Prometheus

Install Prometheus to scrape metrics from the OpenTelemetry Collector and store them for querying.

sudo apt update
sudo apt install -y prometheus

sudo dnf install -y epel-release
sudo dnf install -y golang-github-prometheus

Configure Prometheus to scrape OpenTelemetry metrics

Add the OpenTelemetry Collector as a scrape target in Prometheus configuration. This tells Prometheus to collect metrics from the collector's Prometheus endpoint.

global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "/etc/prometheus/rules/*.yml"

alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

- job_name: 'otel-collector'
static_configs:
- targets: ['localhost:8889']
scrape_interval: 10s
metrics_path: /metrics

- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']

Create Prometheus alerting rules

Set up alerting rules for custom metrics to notify you when application performance degrades.

sudo mkdir -p /etc/prometheus/rules

groups:
- name: application_metrics
rules:
- alert: HighErrorRate
expr: rate(app_http_requests_total{status=~"5.."}[5m]) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value }} errors per second"

- alert: SlowResponseTime
expr: histogram_quantile(0.95, rate(app_http_request_duration_seconds_bucket[5m])) > 1.0
for: 5m
labels:
severity: critical
annotations:
summary: "Slow response times detected"
description: "95th percentile response time is {{ $value }} seconds"

- alert: LowThroughput
expr: rate(app_http_requests_total[5m]) < 1.0
for: 10m
labels:
severity: warning
annotations:
summary: "Low request throughput"
description: "Request rate is {{ $value }} requests per second"

Start Prometheus

Enable and start Prometheus to begin collecting metrics from the OpenTelemetry Collector.

sudo systemctl enable --now prometheus
sudo systemctl status prometheus

Install Grafana

Install Grafana to create dashboards and visualizations for your OpenTelemetry metrics.

sudo apt install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt update
sudo apt install -y grafana

sudo tee /etc/yum.repos.d/grafana.repo << 'EOF'
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF
sudo dnf install -y grafana

Configure Grafana data source

Add Prometheus as a data source in Grafana to query your OpenTelemetry metrics.

sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server

Access Grafana at http://your-server:3000 with username admin and password admin. Navigate to Configuration > Data Sources and add Prometheus with URL http://localhost:9090.

Install OpenTelemetry SDK in your application

Add OpenTelemetry instrumentation to your application. This example shows Python implementation with custom metrics.

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
import time
import random

# Configure OpenTelemetry
metric_exporter = OTLPMetricExporter(
endpoint="http://localhost:4318/v1/metrics",
headers={}
)

metric_reader = PeriodicExportingMetricReader(
exporter=metric_exporter,
export_interval_millis=5000
)

metrics.set_meter_provider(MeterProvider(metric_readers=[metric_reader]))
meter = metrics.get_meter("app_metrics", "1.0.0")

# Create custom metrics
request_counter = meter.create_counter(
name="http_requests_total",
description="Total number of HTTP requests",
unit="1"
)

response_time_histogram = meter.create_histogram(
name="http_request_duration_seconds",
description="HTTP request duration in seconds",
unit="s"
)

active_connections_gauge = meter.create_up_down_counter(
name="active_connections",
description="Number of active connections",
unit="1"
)

# Example usage
def handle_request(endpoint, status_code):
start_time = time.time()

# Simulate request processing
processing_time = random.uniform(0.1, 2.0)
time.sleep(processing_time)

# Record metrics
request_counter.add(1, {"endpoint": endpoint, "status": str(status_code)})
response_time_histogram.record(processing_time, {"endpoint": endpoint})

return f"Processed {endpoint} in {processing_time:.2f}s"

# Simulate application traffic
if __name__ == "__main__":
endpoints = ["/api/users", "/api/orders", "/api/products"]

for i in range(100):
endpoint = random.choice(endpoints)
status = random.choices([200, 404, 500], weights=[85, 10, 5])[0]

active_connections_gauge.add(1)
result = handle_request(endpoint, status)
active_connections_gauge.add(-1)

print(f"Request {i+1}: {result}")
time.sleep(0.1)

Create Grafana dashboard

Import a custom dashboard configuration to visualize your OpenTelemetry metrics with panels for request rates, response times, and error rates.

{
"dashboard": {
"id": null,
"title": "OpenTelemetry Application Metrics",
"tags": ["opentelemetry", "monitoring"],
"timezone": "browser",
"panels": [
{
"id": 1,
"title": "Request Rate",
"type": "stat",
"targets": [
{
"expr": "rate(app_http_requests_total[5m])",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0
}
},
"gridPos": {"h": 8, "w": 6, "x": 0, "y": 0}
},
{
"id": 2,
"title": "Error Rate",
"type": "stat",
"targets": [
{
"expr": "rate(app_http_requests_total{status=~\"5..\"}[5m]) / rate(app_http_requests_total[5m])",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
}
},
"gridPos": {"h": 8, "w": 6, "x": 6, "y": 0}
},
{
"id": 3,
"title": "Response Time (95th percentile)",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(app_http_request_duration_seconds_bucket[5m]))",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
}
},
"gridPos": {"h": 8, "w": 6, "x": 12, "y": 0}
},
{
"id": 4,
"title": "Active Connections",
"type": "stat",
"targets": [
{
"expr": "app_active_connections",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
}
},
"gridPos": {"h": 8, "w": 6, "x": 18, "y": 0}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "10s"
}
}

Configure firewall rules

Open the necessary ports for OpenTelemetry Collector, Prometheus, and Grafana to communicate properly.

sudo ufw allow 3000/tcp
sudo ufw allow 9090/tcp
sudo ufw allow 4317/tcp
sudo ufw allow 4318/tcp
sudo ufw allow 8889/tcp

sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=4317/tcp
sudo firewall-cmd --permanent --add-port=4318/tcp
sudo firewall-cmd --permanent --add-port=8889/tcp
sudo firewall-cmd --reload

Configure custom metrics collection

Add business metrics to your application

Extend your application with business-specific metrics like user signups, revenue, or feature usage. These metrics provide insights into application performance from a business perspective.

# Additional business metrics
user_signups = meter.create_counter(
name="user_signups_total",
description="Total number of user signups",
unit="1"
)

order_value_histogram = meter.create_histogram(
name="order_value_dollars",
description="Order value in dollars",
unit="USD"
)

feature_usage = meter.create_counter(
name="feature_usage_total",
description="Feature usage by type",
unit="1"
)

# Example business event tracking
def track_user_signup(user_type, source):
user_signups.add(1, {
"user_type": user_type,
"source": source
})

def track_order(order_value, product_category):
order_value_histogram.record(order_value, {
"category": product_category
})

def track_feature_use(feature_name, user_tier):
feature_usage.add(1, {
"feature": feature_name,
"tier": user_tier
})

Set up metric sampling and filtering

Configure the collector to sample high-volume metrics and filter irrelevant data to reduce storage costs and improve query performance.

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

processors:
batch:
timeout: 1s
send_batch_size: 1024

memory_limiter:
limit_mib: 512

filter/drop_debug:
metrics:
exclude:
match_type: regexp
metric_names:
- ".*_debug.*"
- ".*_test.*"

probabilistic_sampler:
sampling_percentage: 10

exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "app"
const_labels:
environment: "production"
version: "1.0.0"

metric_expiration: 180s
enable_open_metrics: true

service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, filter/drop_debug, batch]
exporters: [prometheus]
telemetry:
logs:
level: info
metrics:
address: 0.0.0.0:8888

Set up Grafana dashboards and alerting

Create alerting rules in Grafana

Set up Grafana alerts that trigger notifications when metrics exceed thresholds. This provides proactive monitoring for your application performance.

curl -X POST http://admin:admin@localhost:3000/api/alert-rules \
-H "Content-Type: application/json" \
-d '{
"title": "High Error Rate Alert",
"condition": "B",
"data": [
{
"refId": "A",
"queryType": "",
"relativeTimeRange": {
"from": 300,
"to": 0
},
"model": {
"expr": "rate(app_http_requests_total{status=~\"5..\"}[5m])",
"refId": "A"
}
},
{
"refId": "B",
"queryType": "",
"model": {
"conditions": [
{
"evaluator": {
"params": [0.1],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"params": ["A"]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"refId": "B"
}
}
],
"intervalSeconds": 60,
"noDataState": "NoData",
"execErrState": "Alerting",
"for": "2m"
}'

Configure notification channels

Set up Slack or email notifications for alerts. This ensures your team gets notified when application issues occur.

curl -X POST http://admin:admin@localhost:3000/api/alert-notifications \
-H "Content-Type: application/json" \
-d '{
"name": "slack-alerts",
"type": "slack",
"settings": {
"url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
"username": "Grafana",
"channel": "#alerts",
"title": "Application Alert",
"text": "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ .Annotations.description }}{{ end }}"
}
}'

Import pre-built dashboard

Load a comprehensive dashboard template that includes panels for all common OpenTelemetry metrics and alerts.

curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-d '@otel_dashboard.json'

Verify your setup

# Check OpenTelemetry Collector status
sudo systemctl status otelcol

# Verify collector is receiving metrics
curl http://localhost:8889/metrics | grep app_

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job=="otel-collector")'

# Test metric ingestion
python3 app_metrics.py

# Query metrics in Prometheus
curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=app_http_requests_total'

# Check Grafana data source
curl http://admin:admin@localhost:3000/api/datasources

Common issues

<
SymptomCauseFix

---

### Configure Jaeger with Elasticsearch backend security and encryption

URL: https://binadit.com/tutorials/configure-jaeger-elasticsearch-backend-security
Category: monitoring
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up secure communication between Jaeger and Elasticsearch using TLS encryption, authentication, and production-grade security hardening for distributed tracing infrastructure.

What this solves

Jaeger with Elasticsearch backend requires proper security configuration for production environments. This tutorial configures TLS encryption between Jaeger components and Elasticsearch, sets up authentication mechanisms, and implements security hardening measures. You'll establish secure communication channels, configure X-Pack security features, and implement role-based access control for your distributed tracing infrastructure.

Prerequisites

You need a running Elasticsearch cluster and basic familiarity with Jaeger components. This builds on basic Jaeger installation and requires understanding of TLS certificate management. The configuration assumes you have administrative access to both Jaeger and Elasticsearch instances.

Step-by-step configuration

Install required packages

Install Elasticsearch, Jaeger components, and certificate management tools needed for secure configuration.

sudo apt update
sudo apt install -y elasticsearch jaeger-collector jaeger-query jaeger-agent openssl curl

sudo dnf update -y
sudo dnf install -y elasticsearch jaeger-collector jaeger-query jaeger-agent openssl curl

Configure Elasticsearch X-Pack security

Enable X-Pack security features in Elasticsearch to provide authentication and authorization capabilities for Jaeger access.

cluster.name: jaeger-cluster
node.name: jaeger-node-1
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

# Enable X-Pack security
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

# TLS configuration
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12

# Audit logging for security events
xpack.security.audit.enabled: true

Generate TLS certificates

Create certificate authority and TLS certificates for secure communication between Jaeger and Elasticsearch components.

sudo mkdir -p /etc/elasticsearch/certs
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs
sudo chmod 750 /etc/elasticsearch/certs

# Generate CA and certificates
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/certs/elastic-ca.p12 --pass ""
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-ca.p12 --ca-pass "" --out /etc/elasticsearch/certs/http.p12 --pass ""
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-ca.p12 --ca-pass "" --out /etc/elasticsearch/certs/transport.p12 --pass ""

# Set appropriate permissions
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs/*
sudo chmod 660 /etc/elasticsearch/certs/*

Create Jaeger service account

Set up dedicated Elasticsearch user account for Jaeger with minimal required permissions for tracing data access.

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

# Wait for Elasticsearch to start
sleep 30

# Reset built-in user passwords
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto > /tmp/es-passwords.txt

# Extract elastic password for admin operations
ELASTIC_PASSWORD=$(grep "PASSWORD elastic" /tmp/es-passwords.txt | awk '{print $4}')
echo "Elastic password: $ELASTIC_PASSWORD"

Create Jaeger user and roles

Configure role-based access control with custom roles for Jaeger tracing operations and read-only access patterns.

# Create jaeger_writer role for collector
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_writer" -H "Content-Type: application/json" -d '
{
"indices": [
{
"names": ["jaeger-*"],
"privileges": ["create", "index", "write", "delete", "manage"]
}
]
}'

# Create jaeger_reader role for query service
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_reader" -H "Content-Type: application/json" -d '
{
"indices": [
{
"names": ["jaeger-*"],
"privileges": ["read"]
}
]
}'

# Create jaeger_admin role for index management
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_admin" -H "Content-Type: application/json" -d '
{
"indices": [
{
"names": ["jaeger-*"],
"privileges": ["all"]
}
]
}'

Create Jaeger service users

Set up individual user accounts for Jaeger collector, query service, and administrative operations with appropriate role assignments.

# Create user for Jaeger collector
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_collector" -H "Content-Type: application/json" -d '
{
"password": "JaegerCollector2024!",
"roles": ["jaeger_writer"],
"full_name": "Jaeger Collector Service"
}'

# Create user for Jaeger query
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_query" -H "Content-Type: application/json" -d '
{
"password": "JaegerQuery2024!",
"roles": ["jaeger_reader"],
"full_name": "Jaeger Query Service"
}'

# Create user for Jaeger admin operations
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_admin" -H "Content-Type: application/json" -d '
{
"password": "JaegerAdmin2024!",
"roles": ["jaeger_admin"],
"full_name": "Jaeger Administrator"
}'

Extract certificates for Jaeger

Convert Elasticsearch certificates to PEM format for use by Jaeger components and configure certificate access permissions.

sudo mkdir -p /etc/jaeger/certs

# Extract CA certificate
sudo openssl pkcs12 -in /etc/elasticsearch/certs/elastic-ca.p12 -cacerts -nokeys -out /etc/jaeger/certs/ca.crt -passin pass:""

# Extract client certificate and key
sudo openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -clcerts -nokeys -out /etc/jaeger/certs/client.crt -passin pass:""
sudo openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -nocerts -out /etc/jaeger/certs/client.key -passin pass:"" -passout pass:""

# Set appropriate permissions
sudo chown -R jaeger:jaeger /etc/jaeger/certs
sudo chmod 750 /etc/jaeger/certs
sudo chmod 644 /etc/jaeger/certs/ca.crt
sudo chmod 644 /etc/jaeger/certs/client.crt
sudo chmod 600 /etc/jaeger/certs/client.key

Configure Jaeger collector with security

Set up Jaeger collector with TLS encryption, authentication credentials, and secure connection parameters for Elasticsearch backend.

es:
server-urls: https://localhost:9200
username: jaeger_collector
password: JaegerCollector2024!
tls:
enabled: true
ca: /etc/jaeger/certs/ca.crt
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
server-name: localhost
insecure-skip-verify: false
num-shards: 3
num-replicas: 1
index-prefix: jaeger
create-index-templates: true
timeout: 30s
max-span-age: 72h

collector:
grpc:
host-port: 0.0.0.0:14250
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
http:
host-port: 0.0.0.0:14268
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
zipkin:
host-port: 0.0.0.0:9411

Configure Jaeger query service

Set up Jaeger query service with read-only Elasticsearch access, TLS encryption, and UI security configuration for web interface access.

es:
server-urls: https://localhost:9200
username: jaeger_query
password: JaegerQuery2024!
tls:
enabled: true
ca: /etc/jaeger/certs/ca.crt
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
server-name: localhost
insecure-skip-verify: false
index-prefix: jaeger
timeout: 30s
max-lookback: 168h

query:
http:
host-port: 0.0.0.0:16686
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key
grpc:
host-port: 0.0.0.0:16685
tls:
enabled: true
cert: /etc/jaeger/certs/client.crt
key: /etc/jaeger/certs/client.key

ui:
config:
archive:
enabled: false
dependencies:
menuEnabled: true
tracking:
gaID: ""
menu:
- label: "About Jaeger"
url: "https://jaegertracing.io"

Configure systemd services

Create systemd service files for Jaeger components with proper security context and resource limits for production deployment.

[Unit]
Description=Jaeger Collector
Documentation=https://jaegertracing.io
After=network.target elasticsearch.service
Requires=elasticsearch.service

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/bin/jaeger-collector --config-file=/etc/jaeger/collector.yaml
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30

# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/jaeger
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true

# Resource limits
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

Create Jaeger query service

Set up systemd service for Jaeger query with security hardening and proper dependency management on Elasticsearch availability.

[Unit]
Description=Jaeger Query Service
Documentation=https://jaegertracing.io
After=network.target elasticsearch.service
Requires=elasticsearch.service

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/bin/jaeger-query --config-file=/etc/jaeger/query.yaml
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30

# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/jaeger
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true

# Resource limits
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

Configure log directories and permissions

Set up proper logging directory structure with appropriate permissions for Jaeger service user and log rotation configuration.

sudo mkdir -p /var/log/jaeger
sudo chown jaeger:jaeger /var/log/jaeger
sudo chmod 750 /var/log/jaeger

# Create jaeger user if not exists
sudo useradd -r -s /bin/false jaeger 2>/dev/null || true

# Ensure all certificate permissions are correct
sudo chown -R jaeger:jaeger /etc/jaeger
sudo chmod -R o-rwx /etc/jaeger/certs

Start and enable services

Start Jaeger services with systemd and verify they connect securely to Elasticsearch with proper authentication and encryption.

sudo systemctl daemon-reload
sudo systemctl enable jaeger-collector
sudo systemctl enable jaeger-query
sudo systemctl start jaeger-collector
sudo systemctl start jaeger-query

# Check service status
sudo systemctl status jaeger-collector
sudo systemctl status jaeger-query

Configure firewall rules

Set up specific firewall rules for Jaeger ports with TLS encryption and restrict access to authorized networks only.

sudo ufw allow from 203.0.113.0/24 to any port 14250 comment 'Jaeger gRPC collector'
sudo ufw allow from 203.0.113.0/24 to any port 14268 comment 'Jaeger HTTP collector'
sudo ufw allow from 203.0.113.0/24 to any port 16686 comment 'Jaeger UI'
sudo ufw allow from 203.0.113.0/24 to any port 16685 comment 'Jaeger query gRPC'
sudo ufw reload

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="14250" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="14268" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="16686" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="16685" accept'
sudo firewall-cmd --reload

Verify your setup

Test the secure connection between Jaeger and Elasticsearch and verify all authentication mechanisms are working properly.

# Check Elasticsearch cluster health with authentication
curl -k -u elastic:$ELASTIC_PASSWORD "https://localhost:9200/_cluster/health?pretty"

# Verify Jaeger can connect to Elasticsearch
curl -k "https://localhost:16686/api/services"

# Check TLS certificate validity
echo | openssl s_client -connect localhost:16686 -servername localhost 2>/dev/null | openssl x509 -noout -dates

# Verify collector is receiving data
curl -k "https://localhost:14268/api/traces" -X POST -H "Content-Type: application/json" -d '{
"data": [
{
"traceID": "test-trace-id",
"spans": [
{
"traceID": "test-trace-id",
"spanID": "test-span-id",
"operationName": "test-operation",
"startTime": 1640995200000000,
"duration": 1000000
}
]
}
]
}'

# Check Jaeger indices in Elasticsearch
curl -k -u jaeger_query:JaegerQuery2024! "https://localhost:9200/_cat/indices/jaeger-*?v"

Production security hardening

Configure index lifecycle management

Set up automated index lifecycle policies for secure data retention and automated cleanup of old tracing data.

# Create ILM policy for Jaeger indices
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_ilm/policy/jaeger_policy" -H "Content-Type: application/json" -d '
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50gb",
"max_age": "7d"
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"allocate": {
"number_of_replicas": 0
}
}
},
"delete": {
"min_age": "30d"
}
}
}
}'

# Apply policy to Jaeger index templates
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_index_template/jaeger_template" -H "Content-Type: application/json" -d '
{
"index_patterns": ["jaeger-*"],
"template": {
"settings": {
"index.lifecycle.name": "jaeger_policy",
"index.lifecycle.rollover_alias": "jaeger-write"
}
}
}'

Enable audit logging

Configure comprehensive audit logging for security monitoring and compliance tracking of Jaeger access patterns.

# Add to existing elasticsearch.yml
xpack.security.audit.enabled: true
xpack.security.audit.outputs: [index, logfile]
xpack.security.audit.logfile.events.include: [
"access_denied",
"access_granted",
"anonymous_access_denied",
"authentication_failed",
"connection_denied",
"tampered_request",
"run_as_denied",
"run_as_granted"
]
xpack.security.audit.logfile.events.exclude: ["access_granted"]
xpack.security.audit.index.rollover: "daily"
xpack.security.audit.index.events.include: ["authentication_failed", "access_denied", "tampered_request"]

Set up monitoring alerts

Configure security monitoring and alerting for authentication failures, unauthorized access attempts, and certificate expiration tracking.

# Create watcher for failed authentications
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_watcher/watch/jaeger_auth_failures" -H "Content-Type: application/json" -d '
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [".security-audit-*"],
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"event_type": "authentication_failed"
}
},
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total.value": {
"gt": 5
}
}
},
"actions": {
"log_alert": {
"logging&a

---

### Setup ScyllaDB backup validation and automated restore testing

URL: https://binadit.com/tutorials/setup-scylladb-backup-validation-and-testing
Category: databases
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Configure automated ScyllaDB backups with nodetool snapshots, implement validation scripts to verify backup integrity, and set up systemd timers for automated restore testing with Prometheus monitoring.

What this solves

ScyllaDB backup validation ensures your snapshots are recoverable when disasters strike. This tutorial sets up automated backup creation with nodetool, validates backup integrity through checksums and metadata verification, and implements automated restore testing to catch corruption before you need the backups. You'll also configure Prometheus monitoring to track backup health and receive alerts when validation fails.

Step-by-step configuration

Install backup validation dependencies

Install required tools for backup validation, compression, and monitoring.

sudo apt update
sudo apt install -y python3 python3-pip jq pigz parallel curl

sudo dnf install -y python3 python3-pip jq pigz parallel curl
sudo dnf install -y epel-release

Create backup directory structure

Set up organized directories for backups, validation logs, and restore testing.

sudo mkdir -p /opt/scylladb-backup/{snapshots,validation,restore-test,scripts,logs}
sudo chown -R scylla:scylla /opt/scylladb-backup
sudo chmod -R 755 /opt/scylladb-backup

Configure backup automation script

Create the main backup script that handles snapshot creation, validation, and cleanup.

#!/bin/bash

# ScyllaDB Backup and Validation Script
set -euo pipefail

# Configuration
BACKUP_DIR="/opt/scylladb-backup"
SNAPSHOT_DIR="$BACKUP_DIR/snapshots"
VALIDATION_DIR="$BACKUP_DIR/validation"
LOG_DIR="$BACKUP_DIR/logs"
KEYSPACES="${SCYLLA_KEYSPACES:-system_schema}"
RETENTION_DAYS=${BACKUP_RETENTION_DAYS:-7}
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
SNAPSHOT_TAG="backup_$TIMESTAMP"
LOG_FILE="$LOG_DIR/backup_$TIMESTAMP.log"

# Prometheus metrics file
METRICS_FILE="/var/lib/node_exporter/textfile_collector/scylladb_backup.prom"

# Logging function
log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') $1" | tee -a "$LOG_FILE"
}

# Create snapshot
create_snapshot() {
log "Creating snapshot: $SNAPSHOT_TAG"

# Clear any existing snapshots
nodetool clearsnapshot 2>/dev/null || true

# Create new snapshot
if [ "$KEYSPACES" = "all" ]; then
nodetool snapshot -t "$SNAPSHOT_TAG"
else
for ks in $KEYSPACES; do
nodetool snapshot -t "$SNAPSHOT_TAG" "$ks"
done
fi

log "Snapshot created successfully"
}

# Copy and compress snapshot data
copy_snapshot() {
log "Copying snapshot data to backup directory"

local snapshot_backup_dir="$SNAPSHOT_DIR/$TIMESTAMP"
mkdir -p "$snapshot_backup_dir"

# Find and copy snapshot files
find /var/lib/scylla/data -name "$SNAPSHOT_TAG" -type d | while read -r snap_dir; do
# Extract keyspace and table from path
local rel_path=$(echo "$snap_dir" | sed "s|/var/lib/scylla/data/||")
local dest_dir="$snapshot_backup_dir/$rel_path"

mkdir -p "$(dirname "$dest_dir")"

# Copy with compression
log "Copying $snap_dir to $dest_dir"
tar -I pigz -cf "${dest_dir}.tar.gz" -C "$(dirname "$snap_dir")" "$(basename "$snap_dir")"
done

# Create metadata file
create_metadata "$snapshot_backup_dir"

log "Snapshot copy completed"
}

# Create backup metadata
create_metadata() {
local backup_dir="$1"
local metadata_file="$backup_dir/metadata.json"

log "Creating backup metadata"

# Get cluster information
local cluster_name=$(nodetool describecluster | grep "Name:" | awk '{print $2}')
local node_id=$(nodetool info | grep "ID" | awk '{print $2}')
local datacenter=$(nodetool status | grep "$(hostname -I | awk '{print $1}')" | awk '{print $2}')

# Calculate checksums
local checksums_file="$backup_dir/checksums.sha256"
find "$backup_dir" -name "*.tar.gz" -exec sha256sum {} + > "$checksums_file"

# Create JSON metadata
cat > "$metadata_file" << EOF
{
"timestamp": "$TIMESTAMP",
"snapshot_tag": "$SNAPSHOT_TAG",
"cluster_name": "$cluster_name",
"node_id": "$node_id",
"datacenter": "$datacenter",
"keyspaces": "$KEYSPACES",
"backup_size_bytes": $(du -sb "$backup_dir" | awk '{print $1}'),
"file_count": $(find "$backup_dir" -name "*.tar.gz" | wc -l),
"checksums_file": "checksums.sha256",
"node_hostname": "$(hostname)",
"scylla_version": "$(scylla --version | head -1)"
}
EOF

log "Metadata created: $metadata_file"
}

# Validate backup integrity
validate_backup() {
local backup_dir="$SNAPSHOT_DIR/$TIMESTAMP"
local validation_log="$VALIDATION_DIR/validation_$TIMESTAMP.log"
local validation_status=0

log "Validating backup integrity"

# Check metadata exists
if [ ! -f "$backup_dir/metadata.json" ]; then
echo "FAIL: metadata.json missing" >> "$validation_log"
validation_status=1
else
echo "PASS: metadata.json exists" >> "$validation_log"
fi

# Verify checksums
if [ -f "$backup_dir/checksums.sha256" ]; then
cd "$backup_dir"
if sha256sum -c checksums.sha256 >> "$validation_log" 2>&1; then
echo "PASS: All checksums verified" >> "$validation_log"
else
echo "FAIL: Checksum verification failed" >> "$validation_log"
validation_status=1
fi
else
echo "FAIL: checksums.sha256 missing" >> "$validation_log"
validation_status=1
fi

# Test archive extraction
local test_extract_dir="$VALIDATION_DIR/extract_test_$TIMESTAMP"
mkdir -p "$test_extract_dir"

find "$backup_dir" -name "*.tar.gz" | head -3 | while read -r archive; do
if tar -tzf "$archive" > /dev/null 2>&1; then
echo "PASS: Archive readable - $(basename "$archive")" >> "$validation_log"
else
echo "FAIL: Archive corrupted - $(basename "$archive")" >> "$validation_log"
validation_status=1
fi
done

rm -rf "$test_extract_dir"

# Update metrics
update_prometheus_metrics "$validation_status"

if [ $validation_status -eq 0 ]; then
log "Backup validation PASSED"
else
log "Backup validation FAILED - check $validation_log"
exit 1
fi
}

# Update Prometheus metrics
update_prometheus_metrics() {
local validation_status="$1"
local backup_dir="$SNAPSHOT_DIR/$TIMESTAMP"

if [ -f "$backup_dir/metadata.json" ]; then
local backup_size=$(jq -r '.backup_size_bytes' "$backup_dir/metadata.json")
local file_count=$(jq -r '.file_count' "$backup_dir/metadata.json")
else
local backup_size=0
local file_count=0
fi

cat > "$METRICS_FILE" << EOF
# HELP scylladb_backup_last_success_timestamp Last successful backup timestamp
# TYPE scylladb_backup_last_success_timestamp gauge
scylladb_backup_last_success_timestamp $(date +%s)

# HELP scylladb_backup_size_bytes Size of last backup in bytes
# TYPE scylladb_backup_size_bytes gauge
scylladb_backup_size_bytes $backup_size

# HELP scylladb_backup_file_count Number of files in last backup
# TYPE scylladb_backup_file_count gauge
scylladb_backup_file_count $file_count

# HELP scylladb_backup_validation_status Last backup validation status (0=success, 1=failure)
# TYPE scylladb_backup_validation_status gauge
scylladb_backup_validation_status $validation_status
EOF
}

# Clean old backups
cleanup_old_backups() {
log "Cleaning up backups older than $RETENTION_DAYS days"

find "$SNAPSHOT_DIR" -type d -name "[0-9]*" -mtime +$RETENTION_DAYS -exec rm -rf {} + 2>/dev/null || true
find "$VALIDATION_DIR" -name "*.log" -mtime +$RETENTION_DAYS -delete 2>/dev/null || true
find "$LOG_DIR" -name "*.log" -mtime +$RETENTION_DAYS -delete 2>/dev/null || true

log "Cleanup completed"
}

# Main execution
main() {
log "Starting ScyllaDB backup process"

create_snapshot
copy_snapshot
validate_backup
cleanup_old_backups

# Clear snapshot from ScyllaDB
nodetool clearsnapshot "$SNAPSHOT_TAG" 2>/dev/null || true

log "Backup process completed successfully"
}

main "$@"

sudo chmod +x /opt/scylladb-backup/scripts/backup.sh
sudo chown scylla:scylla /opt/scylladb-backup/scripts/backup.sh

Create restore testing script

Build an automated restore test script that validates backup recoverability.

#!/bin/bash

# ScyllaDB Restore Testing Script
set -euo pipefail

# Configuration
BACKUP_DIR="/opt/scylladb-backup"
SNAPSHOT_DIR="$BACKUP_DIR/snapshots"
RESTORE_TEST_DIR="$BACKUP_DIR/restore-test"
LOG_DIR="$BACKUP_DIR/logs"
TEST_KEYSPACE="backup_test_ks"
TEST_TABLE="backup_test_table"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
LOG_FILE="$LOG_DIR/restore_test_$TIMESTAMP.log"

# Prometheus metrics
METRICS_FILE="/var/lib/node_exporter/textfile_collector/scylladb_restore_test.prom"

# Logging function
log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') $1" | tee -a "$LOG_FILE"
}

# Find latest backup
find_latest_backup() {
local latest_backup=$(find "$SNAPSHOT_DIR" -type d -name "[0-9]*" | sort -r | head -1)

if [ -z "$latest_backup" ]; then
log "ERROR: No backups found in $SNAPSHOT_DIR"
exit 1
fi

echo "$latest_backup"
}

# Create test keyspace and data
create_test_data() {
log "Creating test keyspace and data"

# Drop existing test keyspace if exists
cqlsh -e "DROP KEYSPACE IF EXISTS $TEST_KEYSPACE;" 2>/dev/null || true

# Create test keyspace
cqlsh -e "CREATE KEYSPACE $TEST_KEYSPACE WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};"

# Create test table with sample data
cqlsh -k "$TEST_KEYSPACE" -e "
CREATE TABLE $TEST_TABLE (
id UUID PRIMARY KEY,
name TEXT,
created_at TIMESTAMP,
data BLOB
);

INSERT INTO $TEST_TABLE (id, name, created_at, data)
VALUES (uuid(), 'test_record_1', toTimestamp(now()), 0x123456789abcdef);

INSERT INTO $TEST_TABLE (id, name, created_at, data)
VALUES (uuid(), 'test_record_2', toTimestamp(now()), 0xfedcba987654321);

INSERT INTO $TEST_TABLE (id, name, created_at, data)
VALUES (uuid(), 'test_record_3', toTimestamp(now()), 0x1a2b3c4d5e6f);
"

log "Test data created successfully"
}

# Get test data checksum
get_test_data_checksum() {
local checksum=$(cqlsh -k "$TEST_KEYSPACE" -e "SELECT * FROM $TEST_TABLE;" | md5sum | awk '{print $1}')
echo "$checksum"
}

# Perform snapshot of test data
snapshot_test_data() {
local snapshot_tag="restore_test_$TIMESTAMP"

log "Creating snapshot of test data: $snapshot_tag"
nodetool snapshot -t "$snapshot_tag" "$TEST_KEYSPACE"

echo "$snapshot_tag"
}

# Extract backup for testing
extract_backup() {
local backup_dir="$1"
local extract_dir="$RESTORE_TEST_DIR/extract_$TIMESTAMP"

log "Extracting backup to $extract_dir"
mkdir -p "$extract_dir"

# Extract all archives from the backup
find "$backup_dir" -name "*.tar.gz" | while read -r archive; do
local rel_path=$(echo "$archive" | sed "s|$backup_dir/||" | sed 's|\.tar\.gz$||')
local dest_dir="$extract_dir/$rel_path"

mkdir -p "$(dirname "$dest_dir")"
tar -I pigz -xf "$archive" -C "$(dirname "$dest_dir")"
done

echo "$extract_dir"
}

# Restore keyspace from backup
restore_from_backup() {
local extract_dir="$1"
local target_keyspace="${TEST_KEYSPACE}_restored"

log "Restoring keyspace as $target_keyspace"

# Create target keyspace
cqlsh -e "DROP KEYSPACE IF EXISTS $target_keyspace;" 2>/dev/null || true
cqlsh -e "CREATE KEYSPACE $target_keyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};"

# Find snapshot data for our test keyspace
local snapshot_path=$(find "$extract_dir" -path "*/$TEST_KEYSPACE/*" -name "backup_*" | head -1)

if [ -z "$snapshot_path" ]; then
log "ERROR: No snapshot data found for $TEST_KEYSPACE"
return 1
fi

# Copy snapshot files to target keyspace data directory
local target_data_dir="/var/lib/scylla/data/$target_keyspace"
sudo systemctl stop scylla-server

# Create target directory structure
find "$snapshot_path" -name "*.db" | while read -r db_file; do
local table_dir=$(dirname "$db_file" | sed "s|.*/$TEST_KEYSPACE/|$target_data_dir/|" | sed 's|/backup_[0-9_]*$||')
sudo mkdir -p "$table_dir"
sudo cp "$db_file" "$table_dir/"
sudo chown scylla:scylla "$table_dir"/*.db
done

sudo systemctl start scylla-server

# Wait for ScyllaDB to start
local retries=30
while ! nodetool status >/dev/null 2>&1 && [ $retries -gt 0 ]; do
log "Waiting for ScyllaDB to start... ($retries retries left)"
sleep 10
retries=$((retries - 1))
done

if [ $retries -eq 0 ]; then
log "ERROR: ScyllaDB failed to start after restore"
return 1
fi

log "Restore completed for keyspace $target_keyspace"
}

# Verify restored data
verify_restored_data() {
local target_keyspace="${TEST_KEYSPACE}_restored"
local original_checksum="$1"

log "Verifying restored data integrity"

# Get restored data checksum
local restored_checksum=$(cqlsh -k "$target_keyspace" -e "SELECT * FROM $TEST_TABLE;" | md5sum | awk '{print $1}')

if [ "$original_checksum" = "$restored_checksum" ]; then
log "SUCCESS: Restored data matches original (checksum: $restored_checksum)"
return 0
else
log "ERROR: Restored data does not match original"
log "Original checksum: $original_checksum"
log "Restored checksum: $restored_checksum"
return 1
fi
}

# Update Prometheus metrics
update_restore_metrics() {
local test_status="$1"
local test_duration="$2"

cat > "$METRICS_FILE" << EOF
# HELP scylladb_restore_test_last_run_timestamp Last restore test timestamp
# TYPE scylladb_restore_test_last_run_timestamp gauge
scylladb_restore_test_last_run_timestamp $(date +%s)

# HELP scylladb_restore_test_status Last restore test status (0=success, 1=failure)
# TYPE scylladb_restore_test_status gauge
scylladb_restore_test_status $test_status

# HELP scylladb_restore_test_duration_seconds Duration of last restore test
# TYPE scylladb_restore_test_duration_seconds gauge
scylladb_restore_test_duration_seconds $test_duration
EOF
}

# Cleanup test data
cleanup_test_data() {
log "Cleaning up test data"

# Drop test keyspaces
cqlsh -e "DROP KEYSPACE IF EXISTS $TEST_KEYSPACE;" 2>/dev/null || true
cqlsh -e "DROP KEYSPACE IF EXISTS ${TEST_KEYSPACE}_restored;" 2>/dev/null || true

# Clean snapshot
nodetool clearsnapshot 2>/dev/null || true

# Remove extract directory
rm -rf "$RESTORE_TEST_DIR/extract_$TIMESTAMP" 2>/dev/null || true

log "Cleanup completed"
}

# Main execution
main() {
local start_time=$(date +%s)
local test_status=0

log "Starting ScyllaDB restore test"

# Find latest backup
local backup_dir=$(find_latest_backup)
log "Using backup: $backup_dir"

# Create test data and get checksum
create_test_data
local original_checksum=$(get_test_data_checksum)
log "Original data checksum: $original_checksum"

# Snapshot test data (for comparison)
local test_snapshot=$(snapshot_test_data)

# Extract backup
local extract_dir=$(extract_backup "$backup_dir")

# Restore and verify
if restore_from_backup "$extract_dir" && verify_restored_data "$original_checksum"; then
log "Restore test PASSED"
test_status=0
else
log "Restore test FAILED"
test_status=1
fi

# Calculate duration and update metrics
local end_time=$(date +%s)
local duration=$((end_time - start_time))
update_restore_metrics "$test_status" "$duration"

# Cleanup
cleanup_test_data

log "Restore test completed in ${duration}s with status $test_status"

exit $test_status
}

main "$@"

sudo chmod +x /opt/scylladb-backup/scripts/restore-test.sh
sudo chown scylla:scylla /opt/scylladb-backup/scripts/restore-test.sh

Configure systemd timers for automation

Set up systemd services and timers for automated backup and restore testing.

[Unit]
Description=ScyllaDB Backup Service
After=scylla-server.service
Requires=scylla-server.service

[Service]
Type=oneshot
User=scylla
Group=scylla
ExecStart=/opt/scylladb-backup/scripts/backup.sh
Environment=SCYLLA_KEYSPACES=all
Environment=BACKUP_RETENTION_DAYS=7
St

---

### Implement enterprise network QoS with Cisco integration using FRRouting and traffic shaping

URL: https://binadit.com/tutorials/implement-enterprise-network-qos-with-cisco-integration
Category: networking
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Configure enterprise-grade Quality of Service policies with DSCP marking, traffic shaping using tc and HTB, and seamless integration with Cisco equipment through FRRouting BGP and OSPF routing protocols for comprehensive network bandwidth management.

What this solves

Enterprise networks require sophisticated Quality of Service (QoS) policies to prioritize critical traffic and maintain performance across diverse applications. This tutorial configures advanced QoS with DSCP marking, hierarchical traffic shaping, and seamless integration with Cisco infrastructure through FRRouting's BGP and OSPF implementations.

Prerequisites

You need root access to your Linux servers, basic understanding of networking concepts, and existing network infrastructure with Cisco equipment. Your servers should have multiple network interfaces for testing traffic separation.

Step-by-step configuration

Install FRRouting and traffic control tools

Install FRRouting for Cisco protocol compatibility and Linux traffic control utilities for QoS implementation.

curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add -
echo 'deb https://deb.frrouting.org/frr jammy frr-stable' | sudo tee /etc/apt/sources.list.d/frr.list
sudo apt update
sudo apt install -y frr frr-pythontools iproute2 tc iptables-persistent

sudo dnf install -y epel-release
sudo dnf install -y https://rpm.frrouting.org/repo/rpm-repo-0-*.noarch.rpm
sudo dnf install -y frr iproute tc iptables-services
sudo systemctl enable iptables

Enable FRRouting daemons

Configure FRRouting to enable BGP and OSPF daemons for Cisco integration.

bgpd=yes
ospfd=yes
zebra=yes
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
ospfd_options=" -A 127.0.0.1"

Configure FRRouting BGP for Cisco integration

Set up BGP peering with Cisco equipment and implement route policies for QoS integration.

frr version 8.4
frr defaults traditional
hostname frrouting-qos
log syslog informational
ipv6 forwarding
!
interface eth0
description WAN-Interface
ip address 203.0.113.10/24
!
interface eth1
description LAN-Interface
ip address 192.168.1.1/24
!
router bgp 65001
bgp router-id 203.0.113.10
neighbor 203.0.113.1 remote-as 65000
neighbor 203.0.113.1 description Cisco-BGP-Peer
!
address-family ipv4 unicast
neighbor 203.0.113.1 activate
neighbor 203.0.113.1 route-map QOS-OUT out
neighbor 203.0.113.1 route-map QOS-IN in
exit-address-family
!
router ospf
ospf router-id 192.168.1.1
network 192.168.1.0/24 area 0
network 203.0.113.0/24 area 0
!
route-map QOS-OUT permit 10
set community 65001:100
set ip next-hop unchanged
!
route-map QOS-IN permit 10
match community PRIORITY-TRAFFIC
set local-preference 200
!
ip community-list standard PRIORITY-TRAFFIC permit 65000:100
!
line vty
!

Start and enable FRRouting services

Enable FRRouting services and verify BGP neighbor establishment.

sudo systemctl enable --now frr
sudo systemctl status frr
sudo vtysh -c "show ip bgp summary"
sudo vtysh -c "show ip ospf neighbor"

Configure DSCP marking with iptables

Implement DSCP marking for traffic classification and QoS policy enforcement.

# DSCP marking for enterprise QoS
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# Voice traffic - EF (Expedited Forwarding) DSCP 46
-A POSTROUTING -p udp --dport 5060:5090 -j DSCP --set-dscp 46
-A POSTROUTING -p tcp --dport 5060:5090 -j DSCP --set-dscp 46
-A POSTROUTING -p udp --dport 16384:32767 -j DSCP --set-dscp 46

# Video conferencing - AF41 DSCP 34
-A POSTROUTING -p tcp --dport 443 -m string --string "zoom" --algo bm -j DSCP --set-dscp 34
-A POSTROUTING -p udp --dport 8801:8810 -j DSCP --set-dscp 34
-A POSTROUTING -p tcp --dport 1935 -j DSCP --set-dscp 34

# Database traffic - AF31 DSCP 26
-A POSTROUTING -p tcp --dport 3306 -j DSCP --set-dscp 26
-A POSTROUTING -p tcp --dport 5432 -j DSCP --set-dscp 26
-A POSTROUTING -p tcp --dport 1433 -j DSCP --set-dscp 26

# Management traffic - CS6 DSCP 48
-A POSTROUTING -p tcp --dport 22 -j DSCP --set-dscp 48
-A POSTROUTING -p tcp --dport 161:162 -j DSCP --set-dscp 48
-A POSTROUTING -p tcp --dport 179 -j DSCP --set-dscp 48

# Web traffic - AF21 DSCP 18
-A POSTROUTING -p tcp --dport 80 -j DSCP --set-dscp 18
-A POSTROUTING -p tcp --dport 443 -j DSCP --set-dscp 18

# Best effort - default DSCP 0
-A POSTROUTING -j DSCP --set-dscp 0

COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

Apply iptables rules

Load the DSCP marking rules and make them persistent across reboots.

sudo iptables-restore < /etc/iptables/rules.v4
sudo systemctl enable netfilter-persistent
sudo netfilter-persistent save

sudo iptables-restore < /etc/iptables/rules.v4
sudo service iptables save
sudo systemctl enable iptables

Configure HTB traffic shaping

Implement Hierarchical Token Bucket (HTB) for bandwidth management and traffic prioritization.

#!/bin/bash

# Interface configuration
INTERFACE="eth0"
BANDWIDTH="100mbit"

# Clear existing qdisc
tc qdisc del dev $INTERFACE root 2>/dev/null

# Create root HTB qdisc
tc qdisc add dev $INTERFACE root handle 1: htb default 60

# Create root class with total bandwidth
tc class add dev $INTERFACE parent 1: classid 1:1 htb rate $BANDWIDTH

# Voice traffic - 20% guaranteed, 50% max
tc class add dev $INTERFACE parent 1:1 classid 1:10 htb rate 20mbit ceil 50mbit prio 1
tc filter add dev $INTERFACE parent 1: protocol ip prio 1 u32 match ip tos 0xb8 0xfc flowid 1:10

# Video traffic - 30% guaranteed, 60% max
tc class add dev $INTERFACE parent 1:1 classid 1:20 htb rate 30mbit ceil 60mbit prio 2
tc filter add dev $INTERFACE parent 1: protocol ip prio 2 u32 match ip tos 0x88 0xfc flowid 1:20

# Database traffic - 15% guaranteed, 40% max
tc class add dev $INTERFACE parent 1:1 classid 1:30 htb rate 15mbit ceil 40mbit prio 3
tc filter add dev $INTERFACE parent 1: protocol ip prio 3 u32 match ip tos 0x68 0xfc flowid 1:30

# Management traffic - 10% guaranteed, 30% max
tc class add dev $INTERFACE parent 1:1 classid 1:40 htb rate 10mbit ceil 30mbit prio 4
tc filter add dev $INTERFACE parent 1: protocol ip prio 4 u32 match ip tos 0xc0 0xfc flowid 1:40

# Web traffic - 15% guaranteed, 50% max
tc class add dev $INTERFACE parent 1:1 classid 1:50 htb rate 15mbit ceil 50mbit prio 5
tc filter add dev $INTERFACE parent 1: protocol ip prio 5 u32 match ip tos 0x48 0xfc flowid 1:50

# Best effort - 10% guaranteed, remaining available
tc class add dev $INTERFACE parent 1:1 classid 1:60 htb rate 10mbit ceil $BANDWIDTH prio 6

# Add fair queuing to each class for better distribution
tc qdisc add dev $INTERFACE parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:30 handle 30: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:40 handle 40: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:50 handle 50: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:60 handle 60: sfq perturb 10

echo "QoS configuration applied successfully"
tc class show dev $INTERFACE

Make QoS script executable and apply configuration

Set proper permissions and apply the traffic shaping configuration.

sudo chmod +x /usr/local/bin/setup-qos.sh
sudo /usr/local/bin/setup-qos.sh

Create systemd service for persistent QoS

Ensure QoS configuration persists across system reboots with a systemd service.

[Unit]
Description=Network QoS Configuration
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/setup-qos.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable QoS service and configure bandwidth monitoring

Enable the QoS service and set up monitoring for traffic analysis.

sudo systemctl daemon-reload
sudo systemctl enable --now network-qos
sudo systemctl status network-qos

Configure advanced OSPF for Cisco integration

Set up OSPF areas and route redistribution for enterprise network integration. For complex QoS routing, you might also want to explore OSPF multi-area design with FRRouting for larger deployments.

sudo vtysh

configure terminal
router ospf
area 0 authentication message-digest
area 1 stub
network 10.0.0.0/8 area 1
redistribute connected metric 20 metric-type 2
redistribute static metric 10 metric-type 1
passive-interface default
no passive-interface eth0
no passive-interface eth1
timers throttle spf 200 1000 5000
max-metric router-lsa on-startup 60
!
interface eth0
ip ospf message-digest-key 1 md5 cisco123
ip ospf priority 100
ip ospf cost 10
!
interface eth1
ip ospf message-digest-key 1 md5 cisco123
ip ospf priority 200
ip ospf cost 5
!
write memory
exit

Configure traffic monitoring and QoS statistics

Set up monitoring scripts to track QoS effectiveness and bandwidth utilization.

#!/bin/bash

INTERFACE="eth0"
LOGFILE="/var/log/qos-stats.log"

echo "$(date): QoS Statistics for $INTERFACE" >> $LOGFILE
echo "===========================================" >> $LOGFILE

# Display class statistics
tc -s class show dev $INTERFACE >> $LOGFILE

# Display current DSCP markings
echo "\nDSCP Statistics:" >> $LOGFILE
iptables -t mangle -L POSTROUTING -v -n | grep DSCP >> $LOGFILE

# BGP neighbor status
echo "\nBGP Neighbor Status:" >> $LOGFILE
vtysh -c "show ip bgp summary" >> $LOGFILE

# OSPF neighbor status
echo "\nOSPF Neighbor Status:" >> $LOGFILE
vtysh -c "show ip ospf neighbor" >> $LOGFILE

echo "" >> $LOGFILE

Set up automated QoS monitoring

Create a cron job for regular QoS monitoring and performance tracking.

sudo chmod +x /usr/local/bin/qos-monitor.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/qos-monitor.sh") | sudo crontab -

Configure Cisco integration verification

Verify BGP and OSPF peering with Cisco equipment and QoS policy synchronization.

# Verify BGP neighbor establishment
sudo vtysh -c "show ip bgp neighbor 203.0.113.1"

# Check OSPF database
sudo vtysh -c "show ip ospf database"

# Verify route redistribution
sudo vtysh -c "show ip route ospf"
sudo vtysh -c "show ip route bgp"

Verify your setup

Test the QoS configuration and verify Cisco integration with comprehensive checks.

# Check QoS class configuration
tc class show dev eth0

# Verify DSCP marking rules
iptables -t mangle -L POSTROUTING -v -n

# Test BGP connectivity
sudo vtysh -c "show ip bgp summary"
sudo vtysh -c "show ip bgp neighbors"

# Check OSPF neighbors
sudo vtysh -c "show ip ospf neighbor"
sudo vtysh -c "show ip ospf interface"

# Monitor traffic classification
watch -n 2 'tc -s class show dev eth0'

# Test DSCP marking with ping
ping -Q 184 203.0.113.1 # Test AF21 marking
ping -Q 136 203.0.113.1 # Test AF31 marking

# Check routing table
ip route show
sudo vtysh -c "show ip route"

Performance tuning

Optimize QoS performance for high-throughput enterprise environments.

ParameterDefaultOptimizedPurpose

HTB quantum15008000Better packet scheduling
SFQ perturb105More frequent hash regeneration
OSPF SPF throttle50001000Faster convergence
BGP keepalive6030Faster failure detection

Apply performance optimizations:

# Optimize network buffers
echo 'net.core.rmem_max = 16777216' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 16777216' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 65536 16777216' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 16777216' | sudo tee -a /etc/sysctl.conf

# Apply changes
sudo sysctl -p

# Optimize BGP timers
sudo vtysh -c "configure terminal"
sudo vtysh -c "router bgp 65001"
sudo vtysh -c "neighbor 203.0.113.1 timers 30 90"
sudo vtysh -c "write memory"

Integration with advanced iptables QoS

For more sophisticated packet marking and filtering rules, consider implementing advanced iptables QoS with fwmark to complement your FRRouting setup.

Common issues

SymptomCauseFix

BGP neighbor won't establishFirewall blocking port 179sudo iptables -A INPUT -p tcp --dport 179 -j ACCEPT
OSPF adjacency failsArea ID mismatchVerify area configuration with show ip ospf interface
Traffic not shaped correctlyWrong interface specifiedCheck with ip link show and update script
DSCP marking not workingiptables rules not appliedsudo iptables-save | grep DSCP to verify
QoS classes emptyFilter rules not matchingUse tcpdump -i eth0 -v to check DSCP values
Route redistribution failsMetric conflictsAdjust redistribution metrics in OSPF config

Security considerations

Security note: QoS configurations can be exploited for DoS attacks. Always implement rate limiting and monitor for unusual traffic patterns.

Implement security hardening for QoS infrastructure:

# Protect against QoS abuse
iptables -A INPUT -p tcp --dport 179 -m limit --limit 10/minute -j ACCEPT
iptables -A INPUT -p tcp --dport 179 -j DROP

# Monitor for DSCP abuse
echo '*/10 * * * * /usr/bin/iptables -t mangle -Z' | sudo crontab -

# BGP authentication
sudo vtysh -c "configure terminal"
sudo vtysh -c "router bgp 65001"
sudo vtysh -c "neighbor 203.0.113.1 password CiscoQoSAuth2024"
sudo vtysh -c "write memory"

Next steps

Implement OSPF multi-area design with FRRouting and advanced routing policies

Configure advanced iptables QoS with fwmark and multiple interfaces

Configure enterprise BGP route filtering with FRRouting communities

Implement network automation with FRRouting SNMP monitoring

Setup enterprise network redundancy with VRRP failover

Running this in production?

Need this managed? Running enterprise QoS at scale adds complexity: capacity planning, policy optimization, multi-vendor integration, and 24/7 monitoring. Our managed platform covers monitoring, optimization and incident response for European enterprise networks.

---

### Implement Kubernetes RBAC with service accounts and role-based access control

URL: https://binadit.com/tutorials/implement-kubernetes-rbac-with-service-accounts
Category: devops
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team

> Configure Kubernetes role-based access control (RBAC) with service accounts, roles, and role bindings to enforce secure access policies and namespace isolation in your cluster.

What this solves

Kubernetes RBAC (Role-Based Access Control) controls who can access what resources in your cluster. By default, Kubernetes operates on a principle of least privilege, but without proper RBAC configuration, you may grant excessive permissions or block legitimate access. This tutorial shows you how to create service accounts, define roles with specific permissions, and bind them together for secure, granular access control.

Understanding Kubernetes RBAC fundamentals

RBAC in Kubernetes consists of four main components that work together. Service accounts provide identity for pods and applications. Roles define what actions can be performed on which resources. ClusterRoles work like Roles but apply cluster-wide. RoleBindings and ClusterRoleBindings connect subjects (users, groups, service accounts) to roles.

RBAC operates on the principle of explicit allow - everything is denied by default unless explicitly permitted. Permissions are additive, meaning multiple roles can grant different permissions to the same subject. Namespace-level roles only apply within their specific namespace, while cluster roles can access resources across all namespaces.

Step-by-step configuration

Verify RBAC is enabled

Check that RBAC authorization is enabled in your Kubernetes cluster before proceeding with configuration.

kubectl api-versions | grep rbac
kubectl auth can-i create roles --as=system:serviceaccount:default:default

Create a dedicated namespace

Create a test namespace to demonstrate RBAC policies without affecting existing workloads.

kubectl create namespace rbac-demo
kubectl get namespaces rbac-demo

Create a service account

Service accounts provide an identity for processes that run in pods. Create a service account that will be used by applications needing specific permissions.

apiVersion: v1
kind: ServiceAccount
metadata:
name: app-reader
namespace: rbac-demo
automountServiceAccountToken: true

kubectl apply -f rbac-serviceaccount.yaml
kubectl get serviceaccount -n rbac-demo

Create a namespace-level role

Define a role that grants read-only permissions to pods and services within the rbac-demo namespace. This follows the principle of least privilege by only granting necessary permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "services"]
verbs: ["get", "watch", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]

kubectl apply -f rbac-role.yaml
kubectl get role -n rbac-demo

Create a role binding

Bind the service account to the role, granting the app-reader service account the permissions defined in the pod-reader role.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: rbac-demo
subjects:
- kind: ServiceAccount
name: app-reader
namespace: rbac-demo
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io

kubectl apply -f rbac-rolebinding.yaml
kubectl get rolebinding -n rbac-demo

Create a cluster-level role

Define a ClusterRole for permissions that span multiple namespaces or cluster-wide resources like nodes. This example grants read access to nodes and persistent volumes.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: node-reader
rules:
- apiGroups: [""]
resources: ["nodes", "persistentvolumes"]
verbs: ["get", "watch", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["nodes", "pods"]
verbs: ["get", "list"]

kubectl apply -f rbac-clusterrole.yaml
kubectl get clusterrole node-reader

Create a cluster role binding

Bind the service account to the cluster role, giving it cluster-wide read permissions for nodes and metrics.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: read-nodes
subjects:
- kind: ServiceAccount
name: app-reader
namespace: rbac-demo
roleRef:
kind: ClusterRole
name: node-reader
apiGroup: rbac.authorization.k8s.io

kubectl apply -f rbac-clusterrolebinding.yaml
kubectl get clusterrolebinding read-nodes

Create a role for write operations

Create a separate role with write permissions for deployment management. This demonstrates separation of concerns between read and write access.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: deployment-manager
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]

kubectl apply -f rbac-deployment-role.yaml

Create a service account for deployment operations

Create a separate service account for deployment operations to demonstrate role separation.

apiVersion: v1
kind: ServiceAccount
metadata:
name: app-deployer
namespace: rbac-demo

kubectl apply -f rbac-deployer-sa.yaml

Bind the deployer role

Create a role binding that grants the app-deployer service account deployment management permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: manage-deployments
namespace: rbac-demo
subjects:
- kind: ServiceAccount
name: app-deployer
namespace: rbac-demo
roleRef:
kind: Role
name: deployment-manager
apiGroup: rbac.authorization.k8s.io

kubectl apply -f rbac-deployer-binding.yaml

Testing RBAC policies

Test service account permissions

Use kubectl auth can-i to test what each service account can do. This helps verify your RBAC configuration works as expected.

kubectl auth can-i get pods --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
kubectl auth can-i create deployments --as=system:serviceaccount:rbac-demo:app-deployer -n rbac-demo

Create a test pod with service account

Deploy a pod that uses one of your service accounts to verify the RBAC configuration works in practice.

apiVersion: v1
kind: Pod
metadata:
name: rbac-test
namespace: rbac-demo
spec:
serviceAccountName: app-reader
containers:
- name: kubectl-test
image: bitnami/kubectl:latest
command: ['sleep', '3600']
restartPolicy: Never

kubectl apply -f rbac-test-pod.yaml
kubectl get pod rbac-test -n rbac-demo

Test permissions from inside the pod

Execute commands inside the test pod to verify that the service account can only perform allowed operations.

kubectl exec -it rbac-test -n rbac-demo -- kubectl get pods -n rbac-demo
kubectl exec -it rbac-test -n rbac-demo -- kubectl get nodes
kubectl exec -it rbac-test -n rbac-demo -- kubectl create deployment test --image=nginx -n rbac-demo

Implementing advanced RBAC patterns

Create resource-specific permissions

Configure RBAC to allow access to specific resources by name, not just resource types. This provides fine-grained control over individual resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: rbac-demo
name: specific-config-reader
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["app-config", "database-config"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-secret"]
verbs: ["get"]

kubectl apply -f rbac-specific-resource.yaml

Create role aggregation

Use ClusterRole aggregation to combine multiple roles automatically. This is useful for creating composite roles from smaller, reusable components.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: monitoring-reader
labels:
rbac.example.com/aggregate-to-monitoring: "true"
rules:
- apiGroups: [""]
resources: ["pods", "services", "endpoints"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: aggregate-monitoring
aggregationRule:
clusterRoleSelectors:
- matchLabels:
rbac.example.com/aggregate-to-monitoring: "true"
rules: []

kubectl apply -f rbac-aggregated-role.yaml

Verify your setup

kubectl get serviceaccounts -n rbac-demo
kubectl get roles -n rbac-demo
kubectl get rolebindings -n rbac-demo
kubectl get clusterroles | grep -E "node-reader|aggregate-monitoring"
kubectl get clusterrolebindings | grep read-nodes
kubectl auth can-i --list --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo

Troubleshooting access issues

Debug RBAC permissions

Use these commands to troubleshoot when applications can't access required resources.

kubectl auth can-i create pods --as=system:serviceaccount:rbac-demo:app-reader -n rbac-demo
kubectl describe rolebinding -n rbac-demo
kubectl describe clusterrolebinding read-nodes

Check service account tokens

Verify that service accounts have valid tokens and can authenticate to the API server.

kubectl get serviceaccount app-reader -n rbac-demo -o yaml
kubectl get secrets -n rbac-demo | grep app-reader
kubectl describe secret $(kubectl get serviceaccount app-reader -n rbac-demo -o jsonpath='{.secrets[0].name}') -n rbac-demo

Common issues

SymptomCauseFix

Pod gets "forbidden" errorsService account lacks permissionsCreate role and rolebinding with required verbs and resources
RoleBinding doesn't workWrong namespace or subject referenceVerify namespace matches and subject kind/name are correct
Can't access cluster resourcesUsing Role instead of ClusterRoleCreate ClusterRole and ClusterRoleBinding for cluster-scoped resources
Service account not foundautomountServiceAccountToken disabledSet automountServiceAccountToken: true in ServiceAccount
Permissions too broadUsing wildcards in rulesSpecify exact resources and verbs instead of "*"
Multiple bindings conflictOverlapping role definitionsRBAC permissions are additive - review all bindings for the subject

Next steps

Implement Kubernetes network policies for pod-to-pod security and traffic isolation

Configure Kubernetes secrets management with Vault integration for secure container orchestration

Set up Kubernetes monitoring with Prometheus Operator and custom metrics

Configure advanced Kubernetes Pod Security Standards with admission controllers

Implement Kubernetes policy enforcement with Open Policy Agent Gatekeeper

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.

---

### Configure ScyllaDB cluster monitoring with Prometheus and Grafana dashboards

URL: https://binadit.com/tutorials/configure-scylladb-cluster-monitoring-with-prometheus
Category: monitoring
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team

> Set up comprehensive monitoring for ScyllaDB clusters using Prometheus metrics collection and Grafana visualization dashboards. Configure alerting rules for performance monitoring and health checks.

What this solves

ScyllaDB provides extensive metrics through its built-in monitoring endpoints, but collecting and visualizing these metrics requires proper setup. This tutorial shows you how to configure Prometheus to scrape ScyllaDB metrics, set up Grafana dashboards for cluster visualization, and implement alerting rules for proactive monitoring of your NoSQL database cluster.

Step-by-step configuration

Install Prometheus

First, install Prometheus to collect metrics from your ScyllaDB cluster.

sudo apt update
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xzf prometheus-2.48.0.linux-amd64.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus
sudo useradd --system --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /opt/prometheus

sudo dnf update -y
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xzf prometheus-2.48.0.linux-amd64.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus
sudo useradd --system --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /opt/prometheus

Configure Prometheus for ScyllaDB

Create the Prometheus configuration file with ScyllaDB scrape targets. ScyllaDB exposes metrics on port 9180 by default.

global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "scylla_rules.yml"

alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

- job_name: 'scylla'
static_configs:
- targets:
- '203.0.113.10:9180'
- '203.0.113.11:9180'
- '203.0.113.12:9180'
scrape_interval: 10s
metrics_path: /metrics
params:
format: [prometheus]

- job_name: 'scylla-manager'
static_configs:
- targets:
- '203.0.113.10:56090'
scrape_interval: 30s

Create ScyllaDB alerting rules

Define alerting rules specific to ScyllaDB performance and health monitoring.

groups:
- name: scylla.rules
rules:
- alert: ScyllaNodeDown
expr: up{job="scylla"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "ScyllaDB node is down"
description: "ScyllaDB node {{ $labels.instance }} has been down for more than 1 minute."

- alert: ScyllaHighCPU
expr: scylla_reactor_utilization > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU utilization on ScyllaDB node"
description: "CPU utilization is {{ $value }} on {{ $labels.instance }}"

- alert: ScyllaHighLatency
expr: scylla_storage_proxy_coordinator_read_latency{quantile="0.99"} > 100000
for: 2m
labels:
severity: warning
annotations:
summary: "High read latency detected"
description: "99th percentile read latency is {{ $value }}us on {{ $labels.instance }}"

- alert: ScyllaLowDiskSpace
expr: scylla_node_filesystem_avail_bytes / scylla_node_filesystem_size_bytes < 0.1
for: 1m
labels:
severity: critical
annotations:
summary: "Low disk space on ScyllaDB node"
description: "Available disk space is below 10% on {{ $labels.instance }}"

- alert: ScyllaCompactionBacklog
expr: scylla_compaction_manager_pending_tasks > 100
for: 10m
labels:
severity: warning
annotations:
summary: "High compaction backlog"
description: "Compaction backlog has {{ $value }} pending tasks on {{ $labels.instance }}"

- alert: ScyllaHighMemoryUsage
expr: scylla_memory_allocated_bytes / scylla_memory_total_bytes > 0.9
for: 5m
labels:
severity: critical
annotations:
summary: "High memory usage on ScyllaDB node"
description: "Memory usage is {{ $value | humanizePercentage }} on {{ $labels.instance }}"

- alert: ScyllaTimeouts
expr: rate(scylla_storage_proxy_coordinator_read_timeouts_total[5m]) > 1
for: 2m
labels:
severity: warning
annotations:
summary: "High timeout rate detected"
description: "Read timeout rate is {{ $value }}/sec on {{ $labels.instance }}"

- alert: ScyllaErrorRate
expr: rate(scylla_storage_proxy_coordinator_read_errors_total[5m]) > 0.1
for: 1m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Read error rate is {{ $value }}/sec on {{ $labels.instance }}"

- alert: ScyllaStreamingErrors
expr: rate(scylla_streaming_total_incoming_bytes[5m]) == 0 and scylla_node_operation_mode{mode="NORMAL"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "No streaming activity detected"
description: "No incoming streaming detected on {{ $labels.instance }} during repair/bootstrap"

- alert: ScyllaLargePartitions
expr: scylla_large_partition_exceeding_threshold_total > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Large partitions detected"
description: "{{ $value }} large partitions found on {{ $labels.instance }}"

- alert: ScyllaGCPressure
expr: rate(scylla_memory_free_bytes[5m]) < 0
for: 5m
labels:
severity: warning
annotations:
summary: "Memory pressure detected"
description: "Decreasing free memory trend on {{ $labels.instance }}"

- alert: ScyllaConnectionErrors
expr: rate(scylla_cql_connections_rejected_total[5m]) > 1
for: 2m
labels:
severity: critical
annotations:
summary: "High connection rejection rate"
description: "CQL connection rejection rate is {{ $value }}/sec on {{ $labels.instance }}"

- alert: ScyllaRepairProgress
expr: scylla_repair_segment_total == 0 and on(instance) scylla_node_operation_mode{mode="NORMAL"} == 1
for: 24h
labels:
severity: warning
annotations:
summary: "No repair activity in 24 hours"
description: "Node {{ $labels.instance }} has not run repair in over 24 hours"

Create Prometheus systemd service

Set up Prometheus as a systemd service for automatic startup and management.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus \
--config.file=/opt/prometheus/prometheus.yml \
--storage.tsdb.path=/opt/prometheus/data \
--web.console.templates=/opt/prometheus/consoles \
--web.console.libraries=/opt/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle

[Install]
WantedBy=multi-user.target

Install and configure Grafana

Install Grafana for creating dashboards and visualizations of ScyllaDB metrics.

sudo apt install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

sudo tee /etc/yum.repos.d/grafana.repo <

Start monitoring services

Enable and start both Prometheus and Grafana services.

sudo mkdir -p /opt/prometheus/data
sudo chown prometheus:prometheus /opt/prometheus/data
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server

Configure Grafana data source

Add Prometheus as a data source in Grafana and import ScyllaDB dashboards.

curl -X POST http://admin:admin@localhost:3000/api/datasources \
-H "Content-Type: application/json" \
-d '{
"name": "Prometheus",
"type": "prometheus",
"url": "http://localhost:9090",
"access": "proxy",
"basicAuth": false
}'

Import ScyllaDB dashboard

Create a comprehensive dashboard for ScyllaDB cluster monitoring with key performance indicators.

curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-d @- <<'EOF'
{
"dashboard": {
"id": null,
"title": "ScyllaDB Cluster Overview",
"tags": ["scylla", "database"],
"timezone": "browser",
"panels": [
{
"id": 1,
"title": "Node Status",
"type": "stat",
"targets": [
{
"expr": "up{job=\"scylla\"}",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0}
},
{
"id": 2,
"title": "CPU Utilization",
"type": "graph",
"targets": [
{
"expr": "scylla_reactor_utilization",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4}
},
{
"id": 3,
"title": "Read Latency (99th percentile)",
"type": "graph",
"targets": [
{
"expr": "scylla_storage_proxy_coordinator_read_latency{quantile=\"0.99\"}",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 4}
},
{
"id": 4,
"title": "Write Latency (99th percentile)",
"type": "graph",
"targets": [
{
"expr": "scylla_storage_proxy_coordinator_write_latency{quantile=\"0.99\"}",
"legendFormat": "{{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 12}
},
{
"id": 5,
"title": "Operations per Second",
"type": "graph",
"targets": [
{
"expr": "rate(scylla_cql_reads_total[5m])",
"legendFormat": "Reads - {{instance}}"
},
{
"expr": "rate(scylla_cql_inserts_total[5m])",
"legendFormat": "Writes - {{instance}}"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 12}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "5s"
}
}
EOF

Install Alertmanager

Set up Alertmanager to handle alerts generated by Prometheus rules.

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xzf alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager
sudo useradd --system --shell /bin/false alertmanager
sudo chown -R alertmanager:alertmanager /opt/alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xzf alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64 /opt/alertmanager
sudo useradd --system --shell /bin/false alertmanager
sudo chown -R alertmanager:alertmanager /opt/alertmanager

Configure Alertmanager

Set up email notifications for ScyllaDB alerts.

global:
smtp_smarthost: 'localhost:587'
smtp_from: 'alertmanager@example.com'
smtp_auth_username: 'alertmanager@example.com'
smtp_auth_password: 'your-email-password'

route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'web.hook'
routes:
- match:
severity: critical
receiver: 'critical-email'
- match:
severity: warning
receiver: 'warning-email'

receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://127.0.0.1:5001/'

- name: 'critical-email'
email_configs:
- to: 'admin@example.com'
subject: 'CRITICAL: ScyllaDB Alert - {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
Severity: {{ .Labels.severity }}
{{ end }}

- name: 'warning-email'
email_configs:
- to: 'monitoring@example.com'
subject: 'WARNING: ScyllaDB Alert - {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
{{ end }}

Create Alertmanager service

Set up Alertmanager as a systemd service.

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/opt/alertmanager/alertmanager \
--config.file=/opt/alertmanager/alertmanager.yml \
--storage.path=/opt/alertmanager/data

[Install]
WantedBy=multi-user.target

Start Alertmanager

Enable and start the Alertmanager service.

sudo mkdir -p /opt/alertmanager/data
sudo chown alertmanager:alertmanager /opt/alertmanager/data
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager

Configure ScyllaDB monitoring agent

Install and configure the ScyllaDB monitoring agent for enhanced metrics collection.

wget https://github.com/scylladb/scylla-monitoring/archive/refs/tags/scylla-monitoring-4.5.0.tar.gz
tar xzf scylla-monitoring-4.5.0.tar.gz
sudo mv scylla-monitoring-4.5.0 /opt/scylla-monitoring
sudo chown -R prometheus:prometheus /opt/scylla-monitoring

Import advanced ScyllaDB dashboards

Import official ScyllaDB Grafana dashboards for comprehensive monitoring.

cd /opt/scylla-monitoring
sudo -u prometheus ./start-grafana.sh -s prometheus_servers.yml -n node_exporter_servers.yml -G

Verify your setup

Check that all monitoring components are running and collecting metrics properly.

sudo systemctl status prometheus
sudo systemctl status grafana-server
sudo systemctl status alertmanager
curl http://localhost:9090/api/v1/targets
curl http://localhost:3000/api/health
curl http://localhost:9093/api/v1/status

Access Grafana at http://your-server:3000 (admin/admin) and verify that ScyllaDB metrics are being collected. Check the dashboard shows current cluster status and performance metrics.

Common issues

SymptomCauseFix

No metrics from ScyllaDBWrong port or endpointVerify ScyllaDB metrics endpoint: curl http://node:9180/metrics
Prometheus can't scrape targetsFirewall blocking accessOpen port 9180: sudo ufw allow 9180
Grafana shows no dataData source not configuredCheck Prometheus data source URL in Grafana settings
Alerts not firingAlertmanager not connectedVerify Alertmanager target in Prometheus: http://localhost:9090/alerts
Dashboard import failsJSON format errorUse Grafana UI to import dashboard ID 9614 for ScyllaDB
High memory usageToo many metrics retainedAdjust Prometheus retention: --storage.tsdb.retention.time=30d

Next steps

Configure ScyllaDB SSL encryption and authentication for production security

Configure Prometheus Alertmanager with Slack integration for team notifications

Setup ScyllaDB backup automation with S3 integration

Implement ScyllaDB performance tuning and optimization

Configure advanced Grafana dashboards and alerting with custom metrics

Running this in production?

Need help with the operational load? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.

---

### Implement OpenResty rate limiting and DDoS protection with advanced Lua rules

URL: https://binadit.com/tutorials/implement-openresty-rate-limiting-and-ddos-protection
Category: security
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up comprehensive rate limiting and DDoS protection for OpenResty using nginx directives, Redis-backed Lua middleware, and advanced security rules with monitoring and alerting.

What this solves

OpenResty combines NGINX with embedded Lua scripting to create powerful rate limiting and DDoS protection systems. This tutorial shows you how to implement multi-layered defense using nginx.conf directives for basic protection, Redis-backed Lua scripts for sophisticated rate limiting, and custom middleware for advanced threat detection and mitigation.

Prerequisites

You'll need a server with root access and basic familiarity with NGINX configuration. Redis will be used for distributed rate limiting state management. We'll also integrate with existing monitoring systems for security event tracking.

Step-by-step installation

Install OpenResty and dependencies

OpenResty provides NGINX with Lua scripting capabilities built-in. We'll also install Redis for state management and development tools.

sudo apt update
wget -qO - https://openresty.org/package/pubkey.gpg | sudo apt-key add -
echo "deb http://openresty.org/package/ubuntu $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/openresty.list
sudo apt update
sudo apt install -y openresty redis-server lua-cjson lua-resty-redis

sudo dnf update -y
sudo dnf install -y wget
wget https://openresty.org/package/centos/openresty.repo
sudo mv openresty.repo /etc/yum.repos.d/
sudo dnf install -y openresty redis lua-cjson

Configure Redis for rate limiting state

Redis will store rate limiting counters and DDoS detection state across OpenResty worker processes. Configure it for persistence and security.

# Basic security and persistence
bind 127.0.0.1
port 6379
requirepass OpenResty_RateLimit_2024!
save 900 1
save 300 10
save 60 10000

# Memory optimization for rate limiting
maxmemory 256mb
maxmemory-policy allkeys-lru

# Faster key expiration
hz 50

sudo systemctl enable --now redis-server
sudo systemctl status redis-server

Create directory structure for Lua scripts

Organize Lua scripts in a dedicated directory with proper permissions for the OpenResty worker processes.

sudo mkdir -p /etc/openresty/lua/ratelimit
sudo mkdir -p /var/log/openresty/security
sudo chown -R nobody:nogroup /etc/openresty/lua
sudo chown -R www-data:www-data /var/log/openresty
sudo chmod 755 /etc/openresty/lua/ratelimit

Create Redis connection module

This Lua module handles Redis connections with connection pooling and error handling for rate limiting operations.

local redis = require "resty.redis"
local cjson = require "cjson"

local _M = {}

function _M.new()
local red = redis:new()
red:set_timeout(1000) -- 1 second

local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.log(ngx.ERR, "failed to connect to redis: ", err)
return nil, err
end

-- Authenticate
local res, err = red:auth("OpenResty_RateLimit_2024!")
if not res then
ngx.log(ngx.ERR, "failed to authenticate with redis: ", err)
return nil, err
end

return red
end

function _M.close(red)
if red then
local ok, err = red:set_keepalive(10000, 100)
if not ok then
ngx.log(ngx.ERR, "failed to set keepalive: ", err)
end
end
end

function _M.increment_counter(key, window, limit)
local red, err = _M.new()
if not red then
return nil, err
end

-- Use Redis pipeline for atomic operations
red:init_pipeline()
red:incr(key)
red:expire(key, window)

local results, err = red:commit_pipeline()
if not results then
_M.close(red)
return nil, err
end

local current_count = results[1]
_M.close(red)

return tonumber(current_count)
end

return _M

Create basic rate limiting module

This module implements sliding window rate limiting with different limits for different request types and client classes.

local redis_client = require "ratelimit.redis_client"
local cjson = require "cjson"

local _M = {}

-- Rate limit configurations
local rate_limits = {
default = { requests = 100, window = 60 },
api = { requests = 1000, window = 60 },
login = { requests = 5, window = 300 },
upload = { requests = 10, window = 60 }
}

local function get_client_ip()
local headers = ngx.var.http_x_forwarded_for
if headers then
local ip = headers:match("([^,]+)")
return ip:gsub("%s+", "")
end
return ngx.var.remote_addr
end

local function get_rate_limit_config(uri)
if string.match(uri, "/api/") then
return rate_limits.api
elseif string.match(uri, "/login") or string.match(uri, "/auth") then
return rate_limits.login
elseif string.match(uri, "/upload") then
return rate_limits.upload
else
return rate_limits.default
end
end

function _M.check_rate_limit()
local client_ip = get_client_ip()
local uri = ngx.var.uri
local config = get_rate_limit_config(uri)

local key = "rate_limit:" .. client_ip .. ":" .. uri
local current_count, err = redis_client.increment_counter(key, config.window, config.requests)

if not current_count then
ngx.log(ngx.ERR, "Rate limiting error: ", err)
-- Fail open - allow request if Redis is down
return true
end

-- Log rate limiting decisions
local log_data = {
timestamp = ngx.time(),
client_ip = client_ip,
uri = uri,
current_count = current_count,
limit = config.requests,
window = config.window,
allowed = current_count <= config.requests
}

local log_file = io.open("/var/log/openresty/security/rate_limit.log", "a")
if log_file then
log_file:write(cjson.encode(log_data) .. "\n")
log_file:close()
end

if current_count > config.requests then
-- Set rate limit headers
ngx.header["X-RateLimit-Limit"] = config.requests
ngx.header["X-RateLimit-Remaining"] = 0
ngx.header["X-RateLimit-Reset"] = ngx.time() + config.window

ngx.status = 429
ngx.header.content_type = "application/json"
ngx.say(cjson.encode({
error = "Rate limit exceeded",
limit = config.requests,
window = config.window,
retry_after = config.window
}))
ngx.exit(429)
end

-- Set informational headers
ngx.header["X-RateLimit-Limit"] = config.requests
ngx.header["X-RateLimit-Remaining"] = math.max(0, config.requests - current_count)

return true
end

return _M

Create DDoS detection module

This advanced module detects DDoS patterns by analyzing request patterns, response times, and implementing progressive penalties.

local redis_client = require "ratelimit.redis_client"
local cjson = require "cjson"

local _M = {}

-- DDoS detection thresholds
local ddos_config = {
burst_threshold = 50, -- requests per 10 seconds
burst_window = 10,
penalty_multiplier = 2, -- increase penalty each time
max_penalty_time = 3600, -- 1 hour max penalty
suspicious_ua_patterns = {
"curl", "wget", "python", "bot", "crawler", "scanner"
}
}

local function get_client_ip()
local headers = ngx.var.http_x_forwarded_for
if headers then
local ip = headers:match("([^,]+)")
return ip:gsub("%s+", "")
end
return ngx.var.remote_addr
end

local function is_suspicious_user_agent(ua)
if not ua then return true end

ua = ua:lower()
for _, pattern in ipairs(ddos_config.suspicious_ua_patterns) do
if string.find(ua, pattern) then
return true
end
end
return false
end

local function calculate_request_score()
local score = 1

-- Check User-Agent
local user_agent = ngx.var.http_user_agent
if is_suspicious_user_agent(user_agent) then
score = score + 3
end

-- Check for missing common headers
if not ngx.var.http_accept then score = score + 2 end
if not ngx.var.http_accept_language then score = score + 1 end

-- Check request method
if ngx.var.request_method == "POST" then
score = score + 1
end

-- Check for rapid requests (implemented via Redis timing)
local client_ip = get_client_ip()
local timing_key = "timing:" .. client_ip
local red, err = redis_client.new()

if red then
local last_request = red:get(timing_key)
local current_time = ngx.time()

if last_request and last_request ~= ngx.null then
local time_diff = current_time - tonumber(last_request)
if time_diff < 1 then -- Less than 1 second between requests
score = score + 5
end
end

red:setex(timing_key, 60, current_time)
redis_client.close(red)
end

return score
end

function _M.check_ddos_pattern()
local client_ip = get_client_ip()
local current_time = ngx.time()

-- Check if client is currently penalized
local penalty_key = "penalty:" .. client_ip
local red, err = redis_client.new()

if not red then
ngx.log(ngx.ERR, "DDoS detector Redis connection failed: ", err)
return true -- Fail open
end

local penalty_end = red:get(penalty_key)
if penalty_end and penalty_end ~= ngx.null then
if current_time < tonumber(penalty_end) then
redis_client.close(red)

-- Log penalty enforcement
local log_data = {
timestamp = current_time,
client_ip = client_ip,
action = "penalty_enforced",
penalty_end = penalty_end,
uri = ngx.var.uri
}

local log_file = io.open("/var/log/openresty/security/ddos.log", "a")
if log_file then
log_file:write(cjson.encode(log_data) .. "\n")
log_file:close()
end

ngx.status = 403
ngx.header.content_type = "application/json"
ngx.say(cjson.encode({
error = "Access temporarily blocked due to suspicious activity",
retry_after = tonumber(penalty_end) - current_time
}))
ngx.exit(403)
else
-- Penalty expired, remove it
red:del(penalty_key)
end
end

-- Calculate request score
local request_score = calculate_request_score()

-- Track burst requests
local burst_key = "burst:" .. client_ip
local burst_count = redis_client.increment_counter(burst_key, ddos_config.burst_window, ddos_config.burst_threshold)

if not burst_count then
redis_client.close(red)
return true -- Fail open
end

-- Apply scoring to burst detection
local weighted_burst = burst_count * (request_score / 2)

if weighted_burst > ddos_config.burst_threshold then
-- Get current penalty count
local penalty_count_key = "penalty_count:" .. client_ip
local current_penalties = red:get(penalty_count_key)
current_penalties = (current_penalties and current_penalties ~= ngx.null) and tonumber(current_penalties) or 0

-- Calculate penalty duration
local penalty_duration = math.min(
300 * math.pow(ddos_config.penalty_multiplier, current_penalties),
ddos_config.max_penalty_time
)

-- Set penalty
red:setex(penalty_key, penalty_duration, current_time + penalty_duration)
red:incr(penalty_count_key)
red:expire(penalty_count_key, 86400) -- Reset penalty count after 24 hours

-- Log DDoS detection
local log_data = {
timestamp = current_time,
client_ip = client_ip,
action = "ddos_detected",
burst_count = burst_count,
request_score = request_score,
weighted_burst = weighted_burst,
penalty_duration = penalty_duration,
penalty_count = current_penalties + 1,
user_agent = ngx.var.http_user_agent,
uri = ngx.var.uri
}

local log_file = io.open("/var/log/openresty/security/ddos.log", "a")
if log_file then
log_file:write(cjson.encode(log_data) .. "\n")
log_file:close()
end

redis_client.close(red)

ngx.status = 403
ngx.header.content_type = "application/json"
ngx.say(cjson.encode({
error = "Suspicious activity detected - access blocked",
penalty_duration = penalty_duration
}))
ngx.exit(403)
end

redis_client.close(red)
return true
end

return _M

Configure OpenResty with rate limiting

Set up the main OpenResty configuration with nginx.conf directives for basic protection and Lua integration.

worker_processes auto;
error_log /var/log/openresty/error.log warn;
worker_rlimit_nofile 65535;

events {
worker_connections 4096;
use epoll;
multi_accept on;
}

http {
include mime.types;
default_type application/octet-stream;

# Basic security headers
add_header X-Frame-Options SAMEORIGIN always;
add_header X-Content-Type-Options nosniff always;
add_header X-XSS-Protection "1; mode=block" always;

# Rate limiting zones (nginx level)
limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=login:10m rate=1r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;

# Connection limiting
limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;

# Lua package path
lua_package_path "/etc/openresty/lua/?.lua;;";

# Shared dictionaries for caching
lua_shared_dict rate_limit_cache 10m;
lua_shared_dict ddos_cache 10m;

# Logging format
log_format security_log '$remote_addr - $remote_user [$time_local] '
'"$request" $status $bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time '
'$http_x_forwarded_for';

# Main server block
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;

access_log /var/log/openresty/access.log security_log;

# Basic nginx rate limiting (first layer)
limit_req zone=general burst=20 nodelay;
limit_conn conn_limit_per_ip 20;

# Block common attack patterns
location ~ \.(env|git|svn)$ {
deny all;
return 404;
}

# API endpoints with advanced protection
location /api/ {
limit_req zone=api burst=50 nodelay;

# Lua-based rate limiting and DDoS detection
access_by_lua_block {
local ddos_detector = require "ratelimit.ddos_detector"
local basic_limiter = require "ratelimit.basic_limiter"

-- Run DDoS detection first
ddos_detector.check_ddos_pattern()

-- Then apply rate limiting
basic_limiter.check_rate_limit()
}

# Your API backend
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

# Login endpoints with stricter limits
location ~ ^/(login|auth|signin) {
limit_req zone=login burst=3 nodelay;

access_by_lua_block {
local ddos_detector = require "ratelimit.ddos_detector"
local basic_limiter = require "ratelimit.basic_limiter"

ddos_detector.check_ddos_pattern()
basic_limiter.check_rate_limit()
}

proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

# Default location with basic protection
location / {
access_by_lua_block {
local ddos_detector = require "ratelimit.ddos_detector"
local basic_limiter = require "ratelimit.basic_limiter"

ddos_detector.check_ddos_pattern()
basic_limiter.check_rate_limit()
}

proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

# Health check endpoint (no rate limiting)
location /health {
access_log off;
return 200 "OK";
add_header Content-Type text/plain;
}
}
}

Create monitoring script

This script monitors security events and generates alerts for suspicious activity patterns.

#!/bin/bash

# Security monitoring script for OpenResty rate limiting
LOG_DIR="/var/log/openresty/security"
RATE_LIMIT_LOG="$LOG_DIR/rate_limit.log"
DDOS_LOG="$LOG_DIR/ddos.log"
ALERT_THRESHOLD=10
EMAIL_ALERT="admin@example.com"

# Create summary report
generate_security_report() {
local report_file="$LOG_DIR/security_summary_$(date +%Y%m%d_%H%M).txt"

echo "OpenResty Security Report - $(date)" > "$report_file"
echo "======================

---

### Integrate ArgoCD with External Secrets Operator for secure Kubernetes secret management

URL: https://binadit.com/tutorials/integrate-argocd-with-external-secret-operator-for-kubernetes
Category: devops
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up External Secrets Operator to sync secrets from HashiCorp Vault and AWS Secrets Manager into your ArgoCD GitOps workflow, enabling secure automated secret management across multiple environments without storing sensitive data in Git repositories.

What this solves

Managing secrets in GitOps workflows presents a security challenge: you need secrets in Kubernetes but can't store them in Git repositories. The External Secrets Operator (ESO) bridges this gap by automatically synchronizing secrets from external systems like HashiCorp Vault and AWS Secrets Manager into your cluster. This tutorial shows you how to integrate ESO with ArgoCD for secure, automated secret management across multiple environments.

Prerequisites

You'll need a working Kubernetes cluster with ArgoCD installed and either HashiCorp Vault or AWS Secrets Manager access. If you haven't set up ArgoCD yet, check our ArgoCD installation guide.

Step-by-step installation

Install External Secrets Operator

Deploy the External Secrets Operator using Helm, which provides the most flexible installation method.

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

Verify operator installation

Check that all ESO components are running correctly before proceeding.

kubectl get pods -n external-secrets-system
kubectl get crd | grep external-secrets

Create HashiCorp Vault SecretStore

Configure a SecretStore that connects to your HashiCorp Vault instance. This example uses Kubernetes authentication.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: argocd
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "external-secrets"
serviceAccountRef:
name: "external-secrets-sa"

Create AWS Secrets Manager SecretStore

Alternative SecretStore configuration for AWS Secrets Manager using IAM roles.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-backend
namespace: argocd
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa

Set up service account and RBAC

Create the service account and necessary permissions for External Secrets Operator.

apiVersion: v1
kind: ServiceAccount
metadata:
name: external-secrets-sa
namespace: argocd
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ExternalSecretsRole
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: external-secrets-role
namespace: argocd
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "update", "get", "list", "watch"]

Apply the configurations

Deploy all the External Secrets Operator configurations to your cluster.

kubectl apply -f rbac.yaml
kubectl apply -f vault-secret-store.yaml
# OR if using AWS
</code><h2><code>kubectl apply -f aws-secret-store.yaml</code></h2></pre>
</div>

# Step-by-step secret synchronization

<div class="step">
### Create an ExternalSecret resource
<p>Define which secrets to sync from your external provider into Kubernetes.</p>
<pre class="terminal"><code>apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: argocd-repo-creds
namespace: argocd
spec:
refreshInterval: 30s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: argocd-repo-server-tls-certs-secret
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: argocd/repo-credentials
property: username
- secretKey: password
remoteRef:
key: argocd/repo-credentials
property: password

Configure ArgoCD repository credentials

Create an ExternalSecret that manages ArgoCD's private repository credentials.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: private-repo-creds
namespace: argocd
spec:
refreshInterval: 60s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: private-repo
creationPolicy: Owner
template:
type: Opaque
metadata:
labels:
argocd.argoproj.io/secret-type: repository
data:
type: git
url: https://github.com/example/private-repo
username: "{{ .username }}"
password: "{{ .password }}"
data:
- secretKey: username
remoteRef:
key: github/credentials
property: username
- secretKey: password
remoteRef:
key: github/credentials
property: token

Set up application secrets

Create ExternalSecrets for application-specific secrets that ArgoCD will deploy.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: production
spec:
refreshInterval: 300s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: postgres-credentials
creationPolicy: Owner
data:
- secretKey: POSTGRES_USER
remoteRef:
key: database/production
property: username
- secretKey: POSTGRES_PASSWORD
remoteRef:
key: database/production
property: password
- secretKey: POSTGRES_DB
remoteRef:
key: database/production
property: database

Apply secret configurations

Deploy the ExternalSecret resources to start automatic synchronization.

kubectl apply -f external-secret.yaml
kubectl apply -f argocd-repo-secret.yaml
kubectl apply -f app-secrets.yaml

ArgoCD GitOps integration

Create ArgoCD Application with External Secrets

Configure ArgoCD to manage ExternalSecret resources as part of your GitOps workflow.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: external-secrets-config
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/example/k8s-config
targetRevision: HEAD
path: external-secrets/
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

Configure sync waves for proper ordering

Use ArgoCD sync waves to ensure ExternalSecrets are created before applications that depend on them.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-config
namespace: production
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
refreshInterval: 300s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: app-config-secret
creationPolicy: Owner
data:
- secretKey: api-key
remoteRef:
key: application/config
property: api-key

Deploy application with dependency

Create an application deployment that uses the synchronized secrets.

apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web-app
image: nginx:1.21
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: app-config-secret
key: api-key
- name: DB_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: POSTGRES_USER

Multi-environment configuration

Create ClusterSecretStore for shared access

Use ClusterSecretStore when multiple namespaces need access to the same secret backend.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-cluster-backend
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "external-secrets-cluster"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "external-secrets-system"

Configure environment-specific secrets

Create ExternalSecrets that pull different values based on environment using path templating.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: env-database-creds
namespace: staging
spec:
refreshInterval: 300s
secretStoreRef:
name: vault-cluster-backend
kind: ClusterSecretStore
target:
name: database-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: database/staging/postgres
property: username
- secretKey: password
remoteRef:
key: database/staging/postgres
property: password

Apply multi-environment configuration

Deploy the cluster-wide and environment-specific configurations.

kubectl apply -f cluster-secret-store.yaml
kubectl apply -f env-specific-secret.yaml

Verify your setup

Test that External Secrets Operator is properly synchronizing secrets and ArgoCD can access them.

# Check ESO operator status
kubectl get pods -n external-secrets-system

# Verify SecretStore connection
kubectl describe secretstore vault-backend -n argocd

# Check ExternalSecret synchronization
kubectl get externalsecrets -n argocd
kubectl describe externalsecret argocd-repo-creds -n argocd

# Verify secrets were created
kubectl get secrets -n argocd | grep argocd-repo
kubectl get secrets -n production | grep postgres-credentials

# Check ArgoCD can access private repositories
kubectl get applications -n argocd
kubectl describe application external-secrets-config -n argocd

Security note: Never use kubectl get secret -o yaml to view secret contents in production logs or CI/CD output, as this exposes sensitive data in plaintext.

Advanced configuration

Configure secret rotation and refresh

Set up automatic secret rotation with shorter refresh intervals for critical secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: rotating-api-key
namespace: production
spec:
refreshInterval: 30s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: api-credentials
creationPolicy: Owner
deletionPolicy: Retain
data:
- secretKey: api-key
remoteRef:
key: rotating/api-credentials
property: current-key

Set up monitoring and alerts

Configure monitoring for External Secrets Operator to track synchronization health.

apiVersion: v1
kind: ServiceMonitor
metadata:
name: external-secrets-metrics
namespace: external-secrets-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: external-secrets
endpoints:
- port: metrics
interval: 30s
path: /metrics

Common issues

SymptomCauseFix

ExternalSecret stuck in "SecretSyncError"Invalid credentials or vault pathCheck SecretStore configuration and vault permissions
ArgoCD can't access private repositoryRepository secret not properly labeledAdd argocd.argoproj.io/secret-type: repository label
Secrets not refreshingService account lacks permissionsVerify RBAC configuration and vault policies
Application pods failing with secret mount errorsSecret not synchronized before pod creationUse ArgoCD sync waves to order resource creation
ClusterSecretStore connection failingService account in wrong namespaceEnsure service account exists in external-secrets-system namespace

Security best practices

Follow these security guidelines when integrating External Secrets Operator with ArgoCD for production use.

Important: Always use least-privilege access when configuring vault policies and Kubernetes RBAC. Grant only the minimum permissions required for each component to function.

Configure vault policies to restrict access to specific secret paths per environment

Use different service accounts for different environments and applications

Set appropriate refresh intervals based on secret sensitivity

Monitor External Secrets Operator logs for failed authentication attempts

Regularly rotate service account tokens and vault authentication credentials

Next steps

Advanced Vault integration with Kubernetes

ArgoCD ApplicationSets for multi-environment GitOps workflows

Integrate ArgoCD with SonarQube for deployment validation

Implement secrets backup and disaster recovery strategies

Configure External Secrets Operator with Azure Key Vault

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. Our managed platform covers monitoring, backups and 24/7 response by default.

---

### Configure advanced iptables QoS with fwmark and multiple interfaces

URL: https://binadit.com/tutorials/configure-advanced-iptables-qos-with-fwmark-and-multiple-interfaces
Category: networking
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up traffic shaping with iptables packet marking, HTB queueing discipline, and multi-interface QoS policies for bandwidth management and network performance optimization.

What this solves

This tutorial shows you how to configure Quality of Service (QoS) with iptables packet marking and traffic control (tc) to prioritize network traffic across multiple interfaces. You'll use fwmark to tag packets in iptables, then apply Hierarchical Token Bucket (HTB) queueing to shape bandwidth based on those marks. This is essential when you need to guarantee bandwidth for critical services while limiting less important traffic.

Step-by-step configuration

Install traffic control tools

Install the necessary packages for advanced traffic shaping and QoS management.

sudo apt update
sudo apt install -y iproute2 iptables-persistent netfilter-persistent

sudo dnf install -y iproute-tc iptables-services
sudo systemctl enable iptables

Configure network interfaces

Set up your network interfaces with proper IP addresses. We'll use eth0 for internal traffic and eth1 for external connections.

sudo ip addr add 192.168.1.10/24 dev eth0
sudo ip addr add 203.0.113.10/24 dev eth1
sudo ip link set eth0 up
sudo ip link set eth1 up

Create iptables mangle rules with fwmark

Set up packet marking rules to classify traffic by service type, source, and destination. These marks will be used by tc for traffic shaping.

# Clear existing mangle rules
sudo iptables -t mangle -F
sudo iptables -t mangle -X

# Mark SSH traffic as high priority (mark 1)
sudo iptables -t mangle -A OUTPUT -p tcp --dport 22 -j MARK --set-mark 1
sudo iptables -t mangle -A INPUT -p tcp --sport 22 -j MARK --set-mark 1

# Mark HTTP/HTTPS traffic as medium priority (mark 2)
sudo iptables -t mangle -A OUTPUT -p tcp --dport 80 -j MARK --set-mark 2
sudo iptables -t mangle -A OUTPUT -p tcp --dport 443 -j MARK --set-mark 2
sudo iptables -t mangle -A INPUT -p tcp --sport 80 -j MARK --set-mark 2
sudo iptables -t mangle -A INPUT -p tcp --sport 443 -j MARK --set-mark 2

# Mark database traffic as high priority (mark 3)
sudo iptables -t mangle -A OUTPUT -p tcp --dport 3306 -j MARK --set-mark 3
sudo iptables -t mangle -A OUTPUT -p tcp --dport 5432 -j MARK --set-mark 3
sudo iptables -t mangle -A INPUT -p tcp --sport 3306 -j MARK --set-mark 3
sudo iptables -t mangle -A INPUT -p tcp --sport 5432 -j MARK --set-mark 3

# Mark bulk transfer traffic as low priority (mark 4)
sudo iptables -t mangle -A OUTPUT -p tcp --dport 21 -j MARK --set-mark 4
sudo iptables -t mangle -A OUTPUT -p tcp --dport 20 -j MARK --set-mark 4
sudo iptables -t mangle -A INPUT -p tcp --sport 21 -j MARK --set-mark 4
sudo iptables -t mangle -A INPUT -p tcp --sport 20 -j MARK --set-mark 4

Set up HTB root qdisc on primary interface

Configure the Hierarchical Token Bucket queueing discipline on your primary interface (eth0) with a total bandwidth limit.

# Remove existing qdisc
sudo tc qdisc del dev eth0 root 2>/dev/null || true

# Create HTB root qdisc with 100Mbit total bandwidth
sudo tc qdisc add dev eth0 root handle 1: htb default 40

# Create root class with total bandwidth
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit

Configure HTB classes for different priorities

Create HTB classes that correspond to your iptables marks, with guaranteed and maximum bandwidth allocations.

# High priority class for SSH and database (mark 1 and 3) - 40% guaranteed, can use up to 80%
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 40mbit ceil 80mbit prio 1

# Medium priority class for HTTP/HTTPS (mark 2) - 30% guaranteed, can use up to 60%
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 30mbit ceil 60mbit prio 2

# Low priority class for bulk transfers (mark 4) - 10% guaranteed, can use up to 40%
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 10mbit ceil 40mbit prio 3

# Default class for unmarked traffic - 20% guaranteed, can use up to 50%
sudo tc class add dev eth0 parent 1:1 classid 1:40 htb rate 20mbit ceil 50mbit prio 4

Add queueing disciplines to leaf classes

Attach Stochastic Fair Queueing (SFQ) to each HTB class to ensure fair distribution among flows within each priority level.

# Add SFQ to high priority class
sudo tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10

# Add SFQ to medium priority class
sudo tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10

# Add SFQ to low priority class
sudo tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10

# Add SFQ to default class
sudo tc qdisc add dev eth0 parent 1:40 handle 40: sfq perturb 10

Create tc filters based on fwmark

Set up traffic control filters that direct packets to appropriate HTB classes based on their iptables fwmark values.

# Filter for SSH traffic (mark 1) -> high priority class
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:10

# Filter for HTTP/HTTPS traffic (mark 2) -> medium priority class
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 2 handle 2 fw flowid 1:20

# Filter for database traffic (mark 3) -> high priority class
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 1:10

# Filter for bulk transfer traffic (mark 4) -> low priority class
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 3 handle 4 fw flowid 1:30

Configure QoS on secondary interface

Set up similar traffic shaping on your secondary interface (eth1) with different bandwidth allocations.

# Remove existing qdisc from eth1
sudo tc qdisc del dev eth1 root 2>/dev/null || true

# Create HTB root qdisc with 50Mbit total bandwidth for external interface
sudo tc qdisc add dev eth1 root handle 2: htb default 240

# Create root class
sudo tc class add dev eth1 parent 2: classid 2:1 htb rate 50mbit

# High priority class - 20Mbit guaranteed, 40Mbit max
sudo tc class add dev eth1 parent 2:1 classid 2:10 htb rate 20mbit ceil 40mbit prio 1

# Medium priority class - 15Mbit guaranteed, 30Mbit max
sudo tc class add dev eth1 parent 2:1 classid 2:20 htb rate 15mbit ceil 30mbit prio 2

# Low priority class - 5Mbit guaranteed, 20Mbit max
sudo tc class add dev eth1 parent 2:1 classid 2:30 htb rate 5mbit ceil 20mbit prio 3

# Default class - 10Mbit guaranteed, 25Mbit max
sudo tc class add dev eth1 parent 2:1 classid 2:40 htb rate 10mbit ceil 25mbit prio 4

Add SFQ and filters to secondary interface

Complete the QoS setup on eth1 with fair queueing and fwmark-based filters.

# Add SFQ to all classes on eth1
sudo tc qdisc add dev eth1 parent 2:10 handle 210: sfq perturb 10
sudo tc qdisc add dev eth1 parent 2:20 handle 220: sfq perturb 10
sudo tc qdisc add dev eth1 parent 2:30 handle 230: sfq perturb 10
sudo tc qdisc add dev eth1 parent 2:40 handle 240: sfq perturb 10

# Add filters based on fwmark
sudo tc filter add dev eth1 protocol ip parent 2:0 prio 1 handle 1 fw flowid 2:10
sudo tc filter add dev eth1 protocol ip parent 2:0 prio 2 handle 2 fw flowid 2:20
sudo tc filter add dev eth1 protocol ip parent 2:0 prio 1 handle 3 fw flowid 2:10
sudo tc filter add dev eth1 protocol ip parent 2:0 prio 3 handle 4 fw flowid 2:30

Create advanced marking rules for subnet-based QoS

Add more sophisticated iptables rules that classify traffic based on source and destination subnets for better traffic management.

# Mark internal management traffic as high priority
sudo iptables -t mangle -A OUTPUT -s 192.168.1.0/24 -d 192.168.1.0/24 -j MARK --set-mark 1
sudo iptables -t mangle -A INPUT -s 192.168.1.0/24 -d 192.168.1.0/24 -j MARK --set-mark 1

# Mark traffic to/from DMZ as medium priority
sudo iptables -t mangle -A OUTPUT -s 192.168.2.0/24 -j MARK --set-mark 2
sudo iptables -t mangle -A OUTPUT -d 192.168.2.0/24 -j MARK --set-mark 2
sudo iptables -t mangle -A INPUT -s 192.168.2.0/24 -j MARK --set-mark 2

# Mark guest network traffic as low priority
sudo iptables -t mangle -A OUTPUT -s 192.168.3.0/24 -j MARK --set-mark 4
sudo iptables -t mangle -A OUTPUT -d 192.168.3.0/24 -j MARK --set-mark 4
sudo iptables -t mangle -A INPUT -s 192.168.3.0/24 -j MARK --set-mark 4

# Mark traffic based on packet size for bulk transfers
sudo iptables -t mangle -A OUTPUT -m length --length 1000:65535 -j MARK --set-mark 4
sudo iptables -t mangle -A INPUT -m length --length 1000:65535 -j MARK --set-mark 4

Save configuration for persistence

Save your iptables rules and create systemd service for tc rules to ensure they persist after reboot.

sudo netfilter-persistent save
sudo systemctl enable netfilter-persistent

sudo iptables-save > /etc/sysconfig/iptables
sudo systemctl enable iptables

[Unit]
Description=QoS Traffic Control Setup
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/setup-qos.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Create QoS setup script

Create a script to automatically configure tc rules on system startup.

#!/bin/bash

# Setup QoS on eth0
tc qdisc del dev eth0 root 2>/dev/null || true
tc qdisc add dev eth0 root handle 1: htb default 40
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 40mbit ceil 80mbit prio 1
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 30mbit ceil 60mbit prio 2
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 10mbit ceil 40mbit prio 3
tc class add dev eth0 parent 1:1 classid 1:40 htb rate 20mbit ceil 50mbit prio 4

# Add SFQ
tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10
tc qdisc add dev eth0 parent 1:40 handle 40: sfq perturb 10

# Add filters
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:10
tc filter add dev eth0 protocol ip parent 1:0 prio 2 handle 2 fw flowid 1:20
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 1:10
tc filter add dev eth0 protocol ip parent 1:0 prio 3 handle 4 fw flowid 1:30

# Setup QoS on eth1
tc qdisc del dev eth1 root 2>/dev/null || true
tc qdisc add dev eth1 root handle 2: htb default 240
tc class add dev eth1 parent 2: classid 2:1 htb rate 50mbit
tc class add dev eth1 parent 2:1 classid 2:10 htb rate 20mbit ceil 40mbit prio 1
tc class add dev eth1 parent 2:1 classid 2:20 htb rate 15mbit ceil 30mbit prio 2
tc class add dev eth1 parent 2:1 classid 2:30 htb rate 5mbit ceil 20mbit prio 3
tc class add dev eth1 parent 2:1 classid 2:40 htb rate 10mbit ceil 25mbit prio 4

# Add SFQ
tc qdisc add dev eth1 parent 2:10 handle 210: sfq perturb 10
tc qdisc add dev eth1 parent 2:20 handle 220: sfq perturb 10
tc qdisc add dev eth1 parent 2:30 handle 230: sfq perturb 10
tc qdisc add dev eth1 parent 2:40 handle 240: sfq perturb 10

# Add filters
tc filter add dev eth1 protocol ip parent 2:0 prio 1 handle 1 fw flowid 2:10
tc filter add dev eth1 protocol ip parent 2:0 prio 2 handle 2 fw flowid 2:20
tc filter add dev eth1 protocol ip parent 2:0 prio 1 handle 3 fw flowid 2:10
tc filter add dev eth1 protocol ip parent 2:0 prio 3 handle 4 fw flowid 2:30

sudo chmod +x /usr/local/bin/setup-qos.sh
sudo systemctl enable qos-setup.service
sudo systemctl start qos-setup.service

Monitor QoS performance

Set up monitoring to track QoS effectiveness and bandwidth utilization across your interfaces.

#!/bin/bash

echo "=== QoS Statistics for eth0 ==="
tc -s class show dev eth0

echo -e "\n=== QoS Statistics for eth1 ==="
tc -s class show dev eth1

echo -e "\n=== iptables Packet Counters ==="
iptables -t mangle -L -v -n

echo -e "\n=== Interface Statistics ==="
ip -s link show eth0
ip -s link show eth1

sudo chmod +x /usr/local/bin/qos-monitor.sh

Verify your setup

Check that your QoS configuration is working correctly and traffic is being classified properly.

# View current tc configuration
sudo tc qdisc show
sudo tc class show dev eth0
sudo tc filter show dev eth0

# Check iptables mangle rules
sudo iptables -t mangle -L -v -n

# Monitor real-time traffic classification
sudo /usr/local/bin/qos-monitor.sh

# Test bandwidth limits with iperf3 (if available)
iperf3 -c example.com -p 80 -t 10

You can also monitor QoS effectiveness by examining the packet and byte counters:

# Watch real-time statistics
watch -n 2 'tc -s class show dev eth0'

# Check specific class statistics
tc -s class show dev eth0 classid 1:10

Common issues

Symptom
Cause
Fix

Traffic not being shaped
fwmark not set or filters missing
Check iptables rules with iptables -t mangle -L -v

HTB classes not working
Incorrect parent-child relationships
Verify class hierarchy with tc class show dev ethX

No bandwidth limiting
Default class not configured
Ensure HTB has default parameter set

Filters not matching
Wrong handle values in filters
Match filter handles to iptables marks exactly

Configuration lost after reboot
Rules not persisted
Enable systemd service: systemctl enable qos-setup

High latency on priority traffic
SFQ not attached to classes
Add SFQ qdisc to each HTB leaf class

Next steps

Configure advanced iptables firewall rules with logging and DDoS protection

Set up Linux network traffic shaping with tc and QoS for bandwidth management

Monitor network performance with Prometheus and Grafana dashboards

Install and configure ntopng for comprehensive network monitoring

Running this in production?

Need this managed? Running QoS at scale adds complexity: capacity planning, performance tuning, and monitoring across multiple interfaces. See how we run infrastructure like this for European teams who need guaranteed network performance.

---

### Configure Redis 7 cluster SSL encryption and authentication for production security

URL: https://binadit.com/tutorials/configure-redis-cluster-ssl-encryption-and-authentication
Category: security
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Secure your Redis 7 cluster with TLS encryption, client authentication, and inter-node SSL communication for production environments. Includes certificate generation, authentication setup, and security validation.

What this solves

This tutorial configures Redis 7 cluster with SSL/TLS encryption and authentication for production security. You'll set up encrypted communication between cluster nodes, secure client connections with certificates, and implement authentication to protect against unauthorized access. Essential for compliance requirements and securing sensitive data in distributed Redis deployments.

Step-by-step configuration

Update system packages and install dependencies

Start by updating your package manager and installing required tools for SSL certificate generation and Redis cluster management.

sudo apt update && sudo apt upgrade -y
sudo apt install -y redis-server redis-tools openssl wget curl

sudo dnf update -y
sudo dnf install -y redis redis-tools openssl wget curl

Create SSL certificate directory structure

Set up dedicated directories for SSL certificates with proper permissions for Redis cluster security.

sudo mkdir -p /etc/redis/ssl/ca
sudo mkdir -p /etc/redis/ssl/certs
sudo mkdir -p /etc/redis/ssl/private
sudo chmod 755 /etc/redis/ssl
sudo chmod 700 /etc/redis/ssl/private

Generate Certificate Authority (CA) for cluster

Create a private CA to sign certificates for Redis cluster nodes and clients. This establishes trust between all cluster components.

cd /etc/redis/ssl/ca
sudo openssl genrsa -out redis-ca-key.pem 4096
sudo openssl req -new -x509 -days 365 -key redis-ca-key.pem -out redis-ca-cert.pem -subj "/C=US/ST=State/L=City/O=Organization/OU=IT/CN=Redis-CA"

Generate server certificates for cluster nodes

Create individual SSL certificates for each Redis cluster node. Replace the IP addresses with your actual Redis cluster node IPs.

cd /etc/redis/ssl
# Generate server private key
sudo openssl genrsa -out private/redis-server-key.pem 2048

# Create certificate signing request
sudo openssl req -new -key private/redis-server-key.pem -out redis-server.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=IT/CN=redis-cluster"

# Create certificate with SAN for cluster IPs
sudo tee server-cert-config.conf > /dev/null << 'EOF'
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]

[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = redis-cluster
IP.1 = 127.0.0.1
IP.2 = 203.0.113.10
IP.3 = 203.0.113.11
IP.4 = 203.0.113.12
EOF

# Sign the certificate
sudo openssl x509 -req -in redis-server.csr -CA ca/redis-ca-cert.pem -CAkey ca/redis-ca-key.pem -CAcreateserial -out certs/redis-server-cert.pem -days 365 -extensions v3_req -extfile server-cert-config.conf

Generate client certificates for authentication

Create client certificates for secure authentication to the Redis cluster. These will be used by applications and Redis CLI tools.

# Generate client private key
sudo openssl genrsa -out private/redis-client-key.pem 2048

# Create client certificate signing request
sudo openssl req -new -key private/redis-client-key.pem -out redis-client.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=IT/CN=redis-client"

# Sign client certificate
sudo openssl x509 -req -in redis-client.csr -CA ca/redis-ca-cert.pem -CAkey ca/redis-ca-key.pem -CAcreateserial -out certs/redis-client-cert.pem -days 365

# Clean up CSR files
sudo rm redis-server.csr redis-client.csr server-cert-config.conf

Set proper SSL certificate ownership and permissions

Configure secure file permissions for SSL certificates. Redis user needs read access to certificates but private keys must be protected.

sudo chown -R redis:redis /etc/redis/ssl
sudo chmod 644 /etc/redis/ssl/ca/redis-ca-cert.pem
sudo chmod 644 /etc/redis/ssl/certs/*.pem
sudo chmod 600 /etc/redis/ssl/private/*.pem
sudo chmod 600 /etc/redis/ssl/ca/redis-ca-key.pem

Never use chmod 777. It gives every user on the system full access to your private keys. Instead, use restrictive permissions (600) for private keys and appropriate ownership with chown.

Configure Redis cluster authentication

Create a strong password for Redis cluster authentication. This will be used for both cluster communication and client connections.

# Generate strong password for Redis authentication
REDIS_PASSWORD=$(openssl rand -base64 32)
echo "Generated Redis password: $REDIS_PASSWORD"

# Store password securely for later use
echo "$REDIS_PASSWORD" | sudo tee /etc/redis/redis-auth-password > /dev/null
sudo chmod 600 /etc/redis/redis-auth-password
sudo chown redis:redis /etc/redis/redis-auth-password

Configure first Redis cluster node

Set up the primary Redis configuration with SSL encryption, authentication, and cluster settings. This will serve as the template for other nodes.

# Network and cluster configuration
port 0
tls-port 6380
bind 0.0.0.0
protected-mode yes
cluster-enabled yes
cluster-config-file nodes-6380.conf
cluster-node-timeout 5000
cluster-announce-port 6380
cluster-announce-bus-port 16380

# SSL/TLS configuration
tls-cert-file /etc/redis/ssl/certs/redis-server-cert.pem
tls-key-file /etc/redis/ssl/private/redis-server-key.pem
tls-ca-cert-file /etc/redis/ssl/ca/redis-ca-cert.pem
tls-dh-params-file /etc/redis/ssl/redis-dh.pem

# SSL security settings
tls-protocols "TLSv1.2 TLSv1.3"
tls-ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"
tls-ciphersuites "TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256"
tls-prefer-server-ciphers yes
tls-session-caching no
tls-session-cache-size 5000
tls-session-cache-timeout 60

# Client authentication
tls-auth-clients yes
tls-cluster yes

# Authentication
requirepass REPLACE_WITH_PASSWORD
masterauth REPLACE_WITH_PASSWORD

# Security settings
tcp-keepalive 300
timeout 0
maxclients 10000

# Persistence
save 900 1
save 300 10
save 60 10000
dir /var/lib/redis
dbfilename dump-6380.rdb
appendonly yes
appendfilename "appendonly-6380.aof"
appendfsync everysec

# Logging
loglevel notice
logfile /var/log/redis/redis-server-6380.log
syslog-enabled yes
syslog-ident redis-6380

Generate Diffie-Hellman parameters for SSL

Create strong DH parameters for SSL key exchange. This improves the security of TLS connections to the Redis cluster.

sudo openssl dhparam -out /etc/redis/ssl/redis-dh.pem 2048
sudo chown redis:redis /etc/redis/ssl/redis-dh.pem
sudo chmod 644 /etc/redis/ssl/redis-dh.pem

Apply authentication password to configuration

Replace the password placeholder in the Redis configuration with the generated strong password.

REDIS_PASSWORD=$(cat /etc/redis/redis-auth-password)
sudo sed -i "s/REPLACE_WITH_PASSWORD/$REDIS_PASSWORD/g" /etc/redis/redis-node-1.conf

# Set proper ownership and permissions
sudo chown redis:redis /etc/redis/redis-node-1.conf
sudo chmod 640 /etc/redis/redis-node-1.conf

Create systemd service for first cluster node

Set up a dedicated systemd service for the first Redis cluster node with proper security settings and resource limits.

[Unit]
Description=Redis Cluster Node 6380
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/redis-server /etc/redis/redis-node-1.conf
TimeoutStopSec=0
Restart=always
User=redis
Group=redis

# Security settings
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/redis /var/log/redis

# Resource limits
LimitNOFILE=65535
MemoryAccounting=true
MemoryMax=2G

[Install]
WantedBy=multi-user.target

Create additional cluster node configurations

Set up configurations for additional cluster nodes on different ports. This example creates two more nodes for a minimal 3-node cluster.

# Create second node configuration
sudo cp /etc/redis/redis-node-1.conf /etc/redis/redis-node-2.conf
sudo sed -i 's/6380/6381/g' /etc/redis/redis-node-2.conf
sudo sed -i 's/16380/16381/g' /etc/redis/redis-node-2.conf
sudo sed -i 's/nodes-6380.conf/nodes-6381.conf/g' /etc/redis/redis-node-2.conf
sudo sed -i 's/dump-6380.rdb/dump-6381.rdb/g' /etc/redis/redis-node-2.conf
sudo sed -i 's/appendonly-6380.aof/appendonly-6381.aof/g' /etc/redis/redis-node-2.conf
sudo sed -i 's/redis-server-6380.log/redis-server-6381.log/g' /etc/redis/redis-node-2.conf
sudo sed -i 's/redis-6380/redis-6381/g' /etc/redis/redis-node-2.conf

# Create third node configuration
sudo cp /etc/redis/redis-node-1.conf /etc/redis/redis-node-3.conf
sudo sed -i 's/6380/6382/g' /etc/redis/redis-node-3.conf
sudo sed -i 's/16380/16382/g' /etc/redis/redis-node-3.conf
sudo sed -i 's/nodes-6380.conf/nodes-6382.conf/g' /etc/redis/redis-node-3.conf
sudo sed -i 's/dump-6380.rdb/dump-6382.rdb/g' /etc/redis/redis-node-3.conf
sudo sed -i 's/appendonly-6380.aof/appendonly-6382.aof/g' /etc/redis/redis-node-3.conf
sudo sed -i 's/redis-server-6380.log/redis-server-6381.log/g' /etc/redis/redis-node-3.conf
sudo sed -i 's/redis-6380/redis-6382/g' /etc/redis/redis-node-3.conf

Create systemd services for additional nodes

Set up systemd services for the remaining cluster nodes with the same security settings.

# Create service for second node
sudo cp /etc/systemd/system/redis-cluster-6380.service /etc/systemd/system/redis-cluster-6381.service
sudo sed -i 's/6380/6381/g' /etc/systemd/system/redis-cluster-6381.service
sudo sed -i 's/redis-node-1.conf/redis-node-2.conf/g' /etc/systemd/system/redis-cluster-6381.service

# Create service for third node
sudo cp /etc/systemd/system/redis-cluster-6380.service /etc/systemd/system/redis-cluster-6382.service
sudo sed -i 's/6380/6382/g' /etc/systemd/system/redis-cluster-6382.service
sudo sed -i 's/redis-node-1.conf/redis-node-3.conf/g' /etc/systemd/system/redis-cluster-6382.service

Create Redis data and log directories

Set up proper directories for Redis data files and logs with correct ownership and permissions.

sudo mkdir -p /var/lib/redis /var/log/redis
sudo chown redis:redis /var/lib/redis /var/log/redis
sudo chmod 755 /var/lib/redis /var/log/redis

# Enable and start all cluster services
sudo systemctl daemon-reload
sudo systemctl enable redis-cluster-6380 redis-cluster-6381 redis-cluster-6382
sudo systemctl start redis-cluster-6380 redis-cluster-6381 redis-cluster-6382

Initialize Redis cluster with SSL

Create the Redis cluster using redis-cli with SSL authentication. This establishes the cluster topology and enables distributed operation.

REDIS_PASSWORD=$(cat /etc/redis/redis-auth-password)

# Initialize cluster with SSL
redis-cli --tls \
--cert /etc/redis/ssl/certs/redis-client-cert.pem \
--key /etc/redis/ssl/private/redis-client-key.pem \
--cacert /etc/redis/ssl/ca/redis-ca-cert.pem \
--cluster create 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 \
--cluster-replicas 0 \
-a $REDIS_PASSWORD \
--cluster-yes

Configure Redis CLI for SSL connections

Set up a Redis CLI configuration file to simplify SSL connections to the cluster with authentication.

# Redis CLI SSL Configuration
tls-cert-file /etc/redis/ssl/certs/redis-client-cert.pem
tls-key-file /etc/redis/ssl/private/redis-client-key.pem
tls-ca-cert-file /etc/redis/ssl/ca/redis-ca-cert.pem
tls
port 6380

sudo chown redis:redis /etc/redis/redis-cli.conf
sudo chmod 644 /etc/redis/redis-cli.conf

Set up client authentication script

Create a helper script for connecting to the Redis cluster with SSL and authentication for easier administration.

#!/bin/bash
# Redis Cluster SSL Connection Script

REDIS_PASSWORD=$(cat /etc/redis/redis-auth-password 2>/dev/null)
SSL_DIR="/etc/redis/ssl"

if [ -z "$REDIS_PASSWORD" ]; then
echo "Error: Redis password not found"
exit 1
fi

# Connect to cluster with SSL and auth
exec redis-cli --tls \
--cert "$SSL_DIR/certs/redis-client-cert.pem" \
--key "$SSL_DIR/private/redis-client-key.pem" \
--cacert "$SSL_DIR/ca/redis-ca-cert.pem" \
-c \
-h 127.0.0.1 \
-p 6380 \
-a "$REDIS_PASSWORD" \
"$@"

sudo chmod +x /usr/local/bin/redis-cluster-cli

Verify your setup

Test the Redis cluster SSL configuration and authentication to ensure everything is working correctly.

# Check cluster services status
sudo systemctl status redis-cluster-6380 redis-cluster-6381 redis-cluster-6382

# Test cluster connectivity and SSL
redis-cluster-cli ping
redis-cluster-cli cluster nodes
redis-cluster-cli cluster info

# Test SSL certificate information
echo | openssl s_client -connect 127.0.0.1:6380 -cert /etc/redis/ssl/certs/redis-client-cert.pem -key /etc/redis/ssl/private/redis-client-key.pem -CAfile /etc/redis/ssl/ca/redis-ca-cert.pem 2>/dev/null | openssl x509 -noout -subject -issuer

# Test data operations across cluster
redis-cluster-cli set test:ssl:key "SSL encryption working"
redis-cluster-cli get test:ssl:key

# Verify authentication is required
redis-cli --tls --cert /etc/redis/ssl/certs/redis-client-cert.pem --key /etc/redis/ssl/private/redis-client-key.pem --cacert /etc/redis/ssl/ca/redis-ca-cert.pem -h 127.0.0.1 -p 6380 ping

Expected output: The authenticated connection should return "PONG" while the unauthenticated connection should fail with "NOAUTH Authentication required"

Application connection examples

Configure applications to connect to your secured Redis cluster using SSL and authentication.

Python Redis connection

import redis
import ssl

# SSL context configuration
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.load_verify_locations('/etc/redis/ssl/ca/redis-ca-cert.pem')
ssl_context.load_cert_chain('/etc/redis/ssl/certs/redis-client-cert.pem', 
'/etc/redis/ssl/private/redis-client-key.pem')

# Redis cluster connection
from rediscluster import RedisCluster

startup_nodes = [
{"host": "127.0.0.1", "port": "6380"},
{"host": "127.0.0.1", "port": "6381"},
{"host": "127.0.0.1", "port": "6382"}
]

rc = RedisCluster(
startup_nodes=startup_nodes,
password='YOUR_REDIS_PASSWORD',
ssl=True,
ssl_context=ssl_context,
decode_responses=True
)

# Test connection
rc.set('python:test', 'SSL connection successful')
print(rc.get('python:test'))

Node.js Redis connection

const Redis = require('ioredis');
const fs = require('fs');

const cluster = new Redis.Cluster(
[
{ host: '127.0.0.1', port: 6380 },
{ host: '127.0.0.1', port: 6381 },
{ host: '127.0.0.1', port: 6382 }
],
{
redisOptions: {
password: 'YOUR_REDIS_PASSWORD',
tls: {
ca: fs.readFileSync('/etc/redis/ssl/ca/redis-ca-cert.pem'),
cert: fs.readFileSync('/etc/redis/ssl/certs/redis-client-cert.pem'),
key: fs.readFileSync('/etc/redis/ssl/private/redis-client-key.pem'),
checkServerIdentity: () => undefined
}
}
}
);

// Test connection
cluster.set('nodejs:test', 'SSL connection successful')
.then(() => cluster.get('nodejs:test'))
.then(result => console.log(result))
.catch(err => console.error('Connection failed:', err));

Security monitoring and maintenance

Set up monitoring and maintenance procedures for your secured Redis cluster.

Create SSL certificate monitoring script

Monitor SSL certificate expiration to ensure continuous security and prevent connection failures.

#!/bin/bash
# Redis SSL Certificate Monitoring Script

SSL_DIR="/etc/redis/ssl"
WARN_DAYS=30
CRIT_DAYS=7

# Check certificate expiration
check_cert() {
local cert_file="$1"
local cert_name="$2"

if [ ! -f "$cert_file" ]; then
echo "ERROR: Certificate $cert_name not found at $cert_file"
return 1
fi

local exp_date=$(openssl x509 -enddate -noout -in "$cert_file" | cut -d= -f2)
local exp_epoch=$(date -d "$exp_date" +%s)
local now_epoch=$(date +%s)
local days_left=$(( (exp_epoch - now_epoch) / 86400 ))

if [ $days_left -lt $CRIT_DAYS ]; then
echo "CRITICAL: $cert_name expires in $days_left days"
return 2
elif [ $days_left -lt $WARN_DAYS ]; then
echo "WARNING: $cert_name expires in $days_left days"
return 1
else
echo "OK: $cert_name expires in $days_left days"
return 0
fi
}

# Check all certificates
echo "Redis SSL Certificate Status:"
check_cert "$SSL_DIR/ca/redis-ca-cert.pem" "CA Certificate"
check_cert "$SSL_DIR/certs/redis-server-cert.pem" "Server Certificate"
check_cert "$SSL_DIR/certs/redis-client-cert.pem" "Client Certificate"

# Test cluster connectivity
echo "\nCluster Connectivity Test:"
REDIS_PASSWORD=$(cat /etc/redis/redis-auth-password 2>/dev/null)
if redis-cli --tls --cert "$SSL_DIR/certs/redis-client-cert.pem" --key "$SSL_DIR/private/redis-client-key.pem" --cacert "$SSL_DIR/ca/redis-ca-cert.pem" -h 127.0.0.1 -p 6380 -a "$REDIS_PASSWORD" ping > /dev/null 2>&1; then
echo "OK: Redis cluster SSL connectivity working"
else
echo "ERROR: Redis cluster SSL connectivity failed"
fi

---

### Setup OpenResty load balancing with health checks and automatic failover

URL: https://binadit.com/tutorials/setup-openresty-load-balancing-health-checks
Category: hosting
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team

> Configure OpenResty with upstream backend servers, implement health monitoring, and set up automatic failover for high availability load balancing.

What this solves

OpenResty provides advanced load balancing capabilities with built-in health checks and automatic failover. This setup distributes traffic across multiple backend servers while continuously monitoring their availability and removing unhealthy servers from rotation.

Step-by-step installation

Install OpenResty

OpenResty extends Nginx with Lua scripting capabilities for advanced load balancing features.

wget -qO - https://openresty.org/package/pubkey.gpg | sudo apt-key add -
echo "deb http://openresty.org/package/ubuntu $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/openresty.list
sudo apt update
sudo apt install -y openresty

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://openresty.org/package/centos/openresty.repo
sudo yum install -y openresty

Install lua-resty-upstream-healthcheck

This module provides active health checking capabilities for upstream servers.

sudo apt install -y lua-resty-upstream-healthcheck
sudo apt install -y lua-resty-lock

sudo yum install -y lua-resty-upstream-healthcheck
sudo yum install -y lua-resty-lock

Create OpenResty configuration directory

Set up the directory structure for OpenResty configuration files.

sudo mkdir -p /usr/local/openresty/nginx/conf/conf.d
sudo mkdir -p /usr/local/openresty/nginx/logs
sudo mkdir -p /var/log/openresty

Configure main OpenResty configuration

Create the main configuration file with load balancing and health check modules.

user www-data;
worker_processes auto;
error_log /var/log/openresty/error.log warn;
pid /var/run/openresty.pid;

events {
worker_connections 1024;
use epoll;
multi_accept on;
}

http {
include /usr/local/openresty/nginx/conf/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'upstream: $upstream_addr response_time: $upstream_response_time';

access_log /var/log/openresty/access.log main;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;

# Shared memory for health checks
lua_shared_dict healthcheck 1m;
lua_shared_dict locks 1m;

# Load health check module
lua_package_path "/usr/local/openresty/lualib/?.lua;;";
init_worker_by_lua_block {
local hc = require "resty.upstream.healthcheck"

local ok, err = hc.spawn_checker {
shm = "healthcheck",
upstream = "backend_servers",
type = "http",
http_req = "GET /health HTTP/1.0\r\nHost: backend\r\n\r\n",
interval = 2000, -- 2 seconds
timeout = 1000, -- 1 second
fall = 3, -- # successive failures before marking unhealthy
rise = 2, -- # successive successes before marking healthy
valid_statuses = {200, 302},
concurrency = 10,
}
if not ok then
ngx.log(ngx.ERR, "failed to spawn health checker: ", err)
return
end
}

# Upstream backend servers
upstream backend_servers {
server 203.0.113.10:8080 max_fails=0 fail_timeout=0;
server 203.0.113.11:8080 max_fails=0 fail_timeout=0;
server 203.0.113.12:8080 max_fails=0 fail_timeout=0;

# Load balancing method
least_conn;

# Connection keepalive
keepalive 32;
}

# Health check status endpoint
server {
listen 8090;
location /status {
access_log off;
allow 127.0.0.1;
allow 203.0.113.0/24;
deny all;

content_by_lua_block {
local hc = require "resty.upstream.healthcheck"
ngx.say("Nginx health check status page")
hc.status_page()
}
}
}

include /usr/local/openresty/nginx/conf/conf.d/*.conf;
}

Configure load balancer virtual host

Create the main load balancer configuration with failover logic.

server {
listen 80;
server_name example.com;

# Real IP configuration for proxy
set_real_ip_from 203.0.113.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

# Proxy settings
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# Timeouts
proxy_connect_timeout 5s;
proxy_send_timeout 10s;
proxy_read_timeout 30s;

# Buffer settings
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;

# Main location with health-aware load balancing
location / {
proxy_pass http://backend_servers;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
proxy_next_upstream_tries 3;
proxy_next_upstream_timeout 10s;

# Add response headers for debugging
add_header X-Upstream-Server $upstream_addr always;
add_header X-Response-Time $upstream_response_time always;
}

# Health check endpoint for backends
location /health {
access_log off;
return 200 "healthy";
add_header Content-Type text/plain;
}

# Load balancer status page
location /lb-status {
access_log off;
allow 127.0.0.1;
allow 203.0.113.0/24;
deny all;

content_by_lua_block {
local upstream = require "ngx.upstream"
local hc = require "resty.upstream.healthcheck"

ngx.say("Backend Server Status:")
ngx.say("========================")

local ups = upstream.get_servers("backend_servers")
if not ups then
ngx.say("No upstream servers found")
return
end

for _, server in ipairs(ups) do
local status = "unknown"
if server.backup then
status = "backup"
elseif server.down then
status = "down"
elseif server.fail_timeout and server.max_fails then
status = "active"
end

ngx.say(string.format("%s:%s - %s (weight: %d)", 
server.name or server.addr, 
server.port, 
status, 
server.weight or 1))
end
}
}
}

Create SSL configuration for production

Add HTTPS support with automatic HTTP to HTTPS redirection.

server {
listen 80;
server_name example.com;
return 301 https://$server_name$request_uri;
}

server {
listen 443 ssl http2;
server_name example.com;

ssl_certificate /etc/ssl/certs/example.com.pem;
ssl_certificate_key /etc/ssl/private/example.com.key;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;

# HSTS
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload" always;

# Security headers
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
add_header Referrer-Policy strict-origin-when-cross-origin always;

# Real IP configuration
set_real_ip_from 203.0.113.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

# Proxy settings
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# Main location
location / {
proxy_pass http://backend_servers;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
proxy_next_upstream_tries 3;
proxy_next_upstream_timeout 10s;

add_header X-Upstream-Server $upstream_addr always;
add_header X-Response-Time $upstream_response_time always;
}
}

Create systemd service file

Set up OpenResty to run as a system service with proper permissions.

[Unit]
Description=OpenResty (nginx)
Documentation=https://openresty.org/
After=network.target
Wants=network.target

[Service]
Type=forking
PIDFile=/var/run/openresty.pid
ExecStartPre=/usr/local/openresty/bin/openresty -t
ExecStart=/usr/local/openresty/bin/openresty
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Set proper permissions

Configure file ownership and permissions for OpenResty directories.

sudo chown -R www-data:www-data /var/log/openresty
sudo chmod 755 /var/log/openresty
sudo chown www-data:www-data /var/run/openresty.pid 2>/dev/null || true
sudo chmod 644 /usr/local/openresty/nginx/conf/nginx.conf
sudo chmod 644 /usr/local/openresty/nginx/conf/conf.d/*.conf

Configure log rotation

Set up automatic log rotation to prevent disk space issues.

/var/log/openresty/*.log {
daily
missingok
rotate 30
compress
delaycompress
notifempty
copytruncate
postrotate
if [ -f /var/run/openresty.pid ]; then
kill -USR1 `cat /var/run/openresty.pid`
fi
endscript
}

Start and enable OpenResty

Enable the service to start automatically and verify it's running.

sudo systemctl daemon-reload
sudo systemctl enable openresty
sudo systemctl start openresty
sudo systemctl status openresty

Configure upstream backend monitoring

Create advanced health check configuration

Implement custom health check logic with detailed monitoring.

# Advanced health check monitoring
server {
listen 8091;
server_name localhost;

location /health-check {
access_log off;
allow 127.0.0.1;
allow 203.0.113.0/24;
deny all;

content_by_lua_block {
local json = require "cjson"
local hc = require "resty.upstream.healthcheck"
local upstream = require "ngx.upstream"

local function check_backend(host, port)
local sock = ngx.socket.tcp()
sock:settimeout(1000)

local ok, err = sock:connect(host, port)
if not ok then
return false, err
end

local req = "GET /health HTTP/1.0\r\nHost: " .. host .. "\r\n\r\n"
local bytes, err = sock:send(req)
if not bytes then
sock:close()
return false, err
end

local line, err = sock:receive("*l")
sock:close()

if line and string.find(line, "200") then
return true, "healthy"
else
return false, "unhealthy response"
end
end

local backends = {
{host = "203.0.113.10", port = 8080},
{host = "203.0.113.11", port = 8080},
{host = "203.0.113.12", port = 8080}
}

local status = {}
local healthy_count = 0
local total_count = #backends

for _, backend in ipairs(backends) do
local is_healthy, msg = check_backend(backend.host, backend.port)
status[backend.host .. ":" .. backend.port] = {
healthy = is_healthy,
message = msg,
checked_at = ngx.now()
}
if is_healthy then
healthy_count = healthy_count + 1
end
end

local response = {
timestamp = ngx.now(),
total_backends = total_count,
healthy_backends = healthy_count,
status = status,
load_balancer_healthy = healthy_count > 0
}

ngx.header.content_type = "application/json"
ngx.say(json.encode(response))
}
}
}

Add performance monitoring

Configure metrics collection for load balancing performance.

server {
listen 8092;
server_name localhost;

location /metrics {
access_log off;
allow 127.0.0.1;
allow 203.0.113.0/24;
deny all;

content_by_lua_block {
local json = require "cjson"
local upstream = require "ngx.upstream"

-- Get basic upstream statistics
local ups = upstream.get_servers("backend_servers")
local metrics = {
timestamp = ngx.now(),
upstream_servers = {},
requests_total = 0,
active_connections = 0
}

if ups then
for _, server in ipairs(ups) do
local server_info = {
address = server.name or (server.addr .. ":" .. server.port),
weight = server.weight or 1,
max_fails = server.max_fails or 1,
fail_timeout = server.fail_timeout or 10,
backup = server.backup or false,
down = server.down or false
}
table.insert(metrics.upstream_servers, server_info)
end
end

ngx.header.content_type = "application/json"
ngx.say(json.encode(metrics))
}
}

location /prometheus-metrics {
access_log off;
allow 127.0.0.1;
allow 203.0.113.0/24;
deny all;

content_by_lua_block {
local upstream = require "ngx.upstream"

ngx.say("# HELP openresty_upstream_servers_total Total number of upstream servers")
ngx.say("# TYPE openresty_upstream_servers_total gauge")

local ups = upstream.get_servers("backend_servers")
local total_servers = 0
local active_servers = 0

if ups then
for _, server in ipairs(ups) do
total_servers = total_servers + 1
if not server.down then
active_servers = active_servers + 1
end
end
end

ngx.say(string.format("openresty_upstream_servers_total{upstream=\"backend_servers\"} %d", total_servers))
ngx.say(string.format("openresty_upstream_servers_active{upstream=\"backend_servers\"} %d", active_servers))
}
}
}

Implement automatic failover mechanisms

Configure backup server

Add a backup server that activates when all primary servers fail.

# Enhanced upstream with backup server
upstream backend_servers_with_backup {
# Primary servers
server 203.0.113.10:8080 max_fails=3 fail_timeout=30s;
server 203.0.113.11:8080 max_fails=3 fail_timeout=30s;
server 203.0.113.12:8080 max_fails=3 fail_timeout=30s;

# Backup server - only used when all primary servers are down
server 203.0.113.20:8080 backup;

# Load balancing settings
least_conn;
keepalive 32;
keepalive_requests 100;
keepalive_timeout 60s;
}

server {
listen 81;
server_name example.com;

# Enhanced error handling
error_page 502 503 504 /maintenance.html;

location = /maintenance.html {
root /usr/local/openresty/nginx/html;
internal;
}

location / {
# Try backup upstream if primary fails
proxy_pass http://backend_servers_with_backup;

# Aggressive failover settings
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_next_upstream_tries 5;
proxy_next_upstream_timeout 15s;

# Shorter timeouts for faster failover
proxy_connect_timeout 3s;
proxy_send_timeout 5s;
proxy_read_timeout 15s;

# Headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# Failover debugging
add_header X-Upstream-Server $upstream_addr always;
add_header X-Response-Time $upstream_response_time always;
add_header X-Upstream-Status $upstream_status always;
}
}

Create maintenance page

Design a user-friendly maintenance page for complete outages.

---

### Configure Jaeger authentication with OAuth2 and RBAC for enterprise security

URL: https://binadit.com/tutorials/configure-jaeger-authentication-with-oauth2-and-rbac
Category: security
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up enterprise-grade authentication for Jaeger distributed tracing using OAuth2 with Keycloak integration and role-based access control policies for secure production deployments.

What this solves

Jaeger provides powerful distributed tracing capabilities, but out of the box it lacks authentication and authorization. This tutorial configures enterprise-grade security for Jaeger using OAuth2 authentication with Keycloak, implementing role-based access control (RBAC) policies, and adding SSL/TLS encryption for production environments.

Step-by-step configuration

Install Jaeger components

Start by installing Jaeger collector, query, and agent components with their dependencies.

sudo apt update
sudo apt install -y curl wget gnupg2 software-properties-common
wget https://github.com/jaegertracing/jaeger/releases/download/v1.52.0/jaeger-1.52.0-linux-amd64.tar.gz
tar -xzf jaeger-1.52.0-linux-amd64.tar.gz
sudo cp jaeger-1.52.0-linux-amd64/jaeger-* /usr/local/bin/
sudo chmod +x /usr/local/bin/jaeger-*

sudo dnf update -y
sudo dnf install -y curl wget gnupg2
wget https://github.com/jaegertracing/jaeger/releases/download/v1.52.0/jaeger-1.52.0-linux-amd64.tar.gz
tar -xzf jaeger-1.52.0-linux-amd64.tar.gz
sudo cp jaeger-1.52.0-linux-amd64/jaeger-* /usr/local/bin/
sudo chmod +x /usr/local/bin/jaeger-*

Install and configure Elasticsearch backend

Set up Elasticsearch as the storage backend for Jaeger traces with security enabled.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install -y elasticsearch

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
echo '[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md' | sudo tee /etc/yum.repos.d/elasticsearch.repo
sudo dnf install -y --enablerepo=elasticsearch elasticsearch

cluster.name: jaeger-cluster
node.name: jaeger-node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch

Install and configure Keycloak for OAuth2

Set up Keycloak as the OAuth2 identity provider for Jaeger authentication.

wget https://github.com/keycloak/keycloak/releases/download/22.0.5/keycloak-22.0.5.tar.gz
tar -xzf keycloak-22.0.5.tar.gz
sudo mv keycloak-22.0.5 /opt/keycloak
sudo useradd -r -s /bin/false keycloak
sudo chown -R keycloak:keycloak /opt/keycloak

[Unit]
Description=Keycloak Authentication Server
After=network.target

[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/bin/kc.sh start --http-enabled=true --http-host=0.0.0.0 --http-port=8080
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now keycloak
sudo systemctl status keycloak

Configure Keycloak realm and client

Create a Keycloak realm and OAuth2 client for Jaeger authentication.

curl -X POST http://localhost:8080/admin/realms \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(curl -X POST http://localhost:8080/realms/master/protocol/openid-connect/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=admin&password=admin&grant_type=password&client_id=admin-cli" \
| jq -r .access_token)" \
-d '{
"realm": "jaeger",
"enabled": true,
"displayName": "Jaeger Tracing"
}'

curl -X POST http://localhost:8080/admin/realms/jaeger/clients \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"clientId": "jaeger-query",
"enabled": true,
"protocol": "openid-connect",
"publicClient": false,
"redirectUris": ["https://example.com:16686/oauth/callback"],
"webOrigins": ["https://example.com:16686"]
}'

Create Jaeger service configurations

Configure the Jaeger collector and query services with OAuth2 authentication.

[Unit]
Description=Jaeger Collector
After=network.target elasticsearch.service

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/local/bin/jaeger-collector \
--es.server-urls=http://localhost:9200 \
--es.username=elastic \
--es.password=changeme \
--collector.grpc-tls.enabled=true \
--collector.grpc-tls.cert=/etc/jaeger/tls/collector.crt \
--collector.grpc-tls.key=/etc/jaeger/tls/collector.key
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

[Unit]
Description=Jaeger Query Service
After=network.target elasticsearch.service keycloak.service

[Service]
Type=simple
User=jaeger
Group=jaeger
Environment=OAUTH2_PROXY_PROVIDER=keycloak-oidc
Environment=OAUTH2_PROXY_KEYCLOAK_GROUP=jaeger-users
Environment=OAUTH2_PROXY_CLIENT_ID=jaeger-query
Environment=OAUTH2_PROXY_CLIENT_SECRET=your-client-secret
Environment=OAUTH2_PROXY_OIDC_ISSUER_URL=http://localhost:8080/realms/jaeger
Environment=OAUTH2_PROXY_REDIRECT_URL=https://example.com:16686/oauth/callback
ExecStart=/usr/local/bin/jaeger-query \
--es.server-urls=http://localhost:9200 \
--es.username=elastic \
--es.password=changeme \
--query.bearer-token-propagation=true \
--query.ui-config=/etc/jaeger/ui-config.json
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Install OAuth2 proxy for authentication

Set up OAuth2 proxy to handle authentication between Jaeger and Keycloak.

wget https://github.com/oauth2-proxy/oauth2-proxy/releases/download/v7.5.1/oauth2-proxy-v7.5.1.linux-amd64.tar.gz
tar -xzf oauth2-proxy-v7.5.1.linux-amd64.tar.gz
sudo cp oauth2-proxy-v7.5.1.linux-amd64/oauth2-proxy /usr/local/bin/
sudo chmod +x /usr/local/bin/oauth2-proxy

http_address = "0.0.0.0:4180"
upstreams = [
"http://127.0.0.1:16686"
]

provider = "keycloak-oidc"
oidc_issuer_url = "http://localhost:8080/realms/jaeger"
client_id = "jaeger-query"
client_secret = "your-client-secret-here"

email_domains = [
"*"
]

cookie_secret = "$(openssl rand -base64 32 | head -c 32)"
cookie_secure = true
cookie_httponly = true
cookie_samesite = "lax"

set_xauthrequest = true
pass_authorization_header = true
pass_access_token = true
pass_user_headers = true

authorized_groups = [
"/jaeger-admins",
"/jaeger-users"
]

Configure RBAC policies

Set up role-based access control with different permission levels for Jaeger users.

apiVersion: rbac.jaeger.io/v1
kind: RoleBinding
metadata:
name: jaeger-admin-binding
subjects:
- kind: Group
name: jaeger-admins
apiGroup: rbac.jaeger.io
roleRef:
kind: Role
name: admin
apiGroup: rbac.jaeger.io
---
apiVersion: rbac.jaeger.io/v1
kind: RoleBinding
metadata:
name: jaeger-user-binding
subjects:
- kind: Group
name: jaeger-users
apiGroup: rbac.jaeger.io
roleRef:
kind: Role
name: viewer
apiGroup: rbac.jaeger.io
---
apiVersion: rbac.jaeger.io/v1
kind: Role
metadata:
name: admin
rules:
- apiGroups: [""]
resources: ["traces", "services", "operations"]
verbs: ["get", "list", "create", "update", "delete"]
---
apiVersion: rbac.jaeger.io/v1
kind: Role
metadata:
name: viewer
rules:
- apiGroups: [""]
resources: ["traces", "services", "operations"]
verbs: ["get", "list"]

Generate SSL certificates

Create SSL certificates for secure communication between Jaeger components.

sudo mkdir -p /etc/jaeger/tls
sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/jaeger/tls/jaeger.key -out /etc/jaeger/tls/jaeger.crt -days 365 -nodes -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=example.com"
sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/jaeger/tls/collector.key -out /etc/jaeger/tls/collector.crt -days 365 -nodes -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=collector.example.com"
sudo chown -R jaeger:jaeger /etc/jaeger/tls
sudo chmod 600 /etc/jaeger/tls/*.key
sudo chmod 644 /etc/jaeger/tls/*.crt

Configure NGINX reverse proxy

Set up NGINX as a reverse proxy with SSL termination for Jaeger services.

sudo apt install -y nginx

sudo dnf install -y nginx

upstream oauth2_proxy {
server 127.0.0.1:4180;
}

upstream jaeger_query {
server 127.0.0.1:16686;
}

server {
listen 443 ssl http2;
server_name example.com;

ssl_certificate /etc/jaeger/tls/jaeger.crt;
ssl_certificate_key /etc/jaeger/tls/jaeger.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

add_header Strict-Transport-Security "max-age=63072000" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;

location /oauth/ {
proxy_pass http://oauth2_proxy;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

location / {
auth_request /oauth/auth;
error_page 401 = @error401;

proxy_pass http://jaeger_query;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Authorization $http_authorization;
}

location @error401 {
return 302 /oauth/start?rd=$request_uri;
}
}

server {
listen 80;
server_name example.com;
return 301 https://$server_name$request_uri;
}

sudo ln -s /etc/nginx/sites-available/jaeger /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Create system users and directories

Set up the jaeger system user and required directories with proper permissions.

sudo useradd -r -s /bin/false jaeger
sudo mkdir -p /var/log/jaeger /etc/jaeger
sudo chown -R jaeger:jaeger /var/log/jaeger /etc/jaeger
sudo chmod 755 /var/log/jaeger /etc/jaeger

Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions like 755 for directories and 644 for files.

Start and enable all services

Enable and start all Jaeger services in the correct order.

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl enable --now keycloak
sudo systemctl enable --now jaeger-collector
sudo systemctl enable --now jaeger-query

[Unit]
Description=OAuth2 Proxy
After=network.target keycloak.service

[Service]
Type=simple
User=oauth2-proxy
Group=oauth2-proxy
ExecStart=/usr/local/bin/oauth2-proxy --config=/etc/oauth2-proxy/oauth2-proxy.cfg
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo useradd -r -s /bin/false oauth2-proxy
sudo chown -R oauth2-proxy:oauth2-proxy /etc/oauth2-proxy
sudo systemctl enable --now oauth2-proxy

Configure Keycloak groups and users

Create user groups

Set up Keycloak groups for different access levels in Jaeger.

TOKEN=$(curl -X POST http://localhost:8080/realms/master/protocol/openid-connect/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=admin&password=admin&grant_type=password&client_id=admin-cli" \
| jq -r .access_token)

curl -X POST http://localhost:8080/admin/realms/jaeger/groups \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "jaeger-admins"}'

curl -X POST http://localhost:8080/admin/realms/jaeger/groups \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "jaeger-users"}'

Create test users

Add test users with different permission levels for validation.

curl -X POST http://localhost:8080/admin/realms/jaeger/users \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"username": "admin@example.com",
"email": "admin@example.com",
"enabled": true,
"credentials": [{
"type": "password",
"value": "SecureP@ssw0rd123",
"temporary": false
}]
}'

curl -X POST http://localhost:8080/admin/realms/jaeger/users \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"username": "viewer@example.com",
"email": "viewer@example.com",
"enabled": true,
"credentials": [{
"type": "password",
"value": "ViewerP@ssw0rd123",
"temporary": false
}]
}'

Verify your setup

Test that all components are running and authentication is working properly.

sudo systemctl status elasticsearch keycloak jaeger-collector jaeger-query oauth2-proxy nginx
curl -k https://example.com/api/services
curl http://localhost:8080/realms/jaeger/.well-known/openid_configuration

Access the Jaeger UI at https://example.com and verify that:

You are redirected to Keycloak for authentication

Login works with the test credentials

Different users have appropriate access levels

SSL certificates are properly configured

Common issues
SymptomCauseFixOAuth2 redirect failsIncorrect redirect URI in KeycloakUpdate client redirect URIs in Keycloak adminSSL certificate errorsSelf-signed certificatesUse Let's Encrypt or proper CA-signed certificatesElasticsearch connection failsAuthentication credentialsVerify ES username/password in service configsNGINX proxy errorsUpstream service not runningCheck sudo systemctl status jaeger-query oauth2-proxyKeycloak groups not workingGroup mapping misconfigurationVerify group assignments in Keycloak admin consoleRBAC permissions deniedRole binding configurationCheck /etc/jaeger/rbac-config.yaml syntax

Next steps

Configure Jaeger alerting with Prometheus and Grafana for monitoring authentication events

Setup Jaeger sampling strategies for high-volume production tracing to optimize performance

Configure Keycloak high availability clustering for production for enterprise deployments

Implement mutual TLS authentication between Jaeger components

Configure Jaeger backup and disaster recovery procedures

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. See how we run infrastructure like this for European teams.

---

### Setup Jaeger sampling strategies for high-volume production tracing

URL: https://binadit.com/tutorials/setup-jaeger-sampling-strategies-for-high-volume-tracing
Category: monitoring
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Configure advanced Jaeger sampling strategies to efficiently capture traces in high-traffic production environments while controlling storage costs and maintaining observability.

What this solves

In high-volume production environments, tracing every request creates overwhelming data volumes and storage costs. Jaeger sampling strategies help you capture meaningful traces while controlling resource usage. This tutorial shows you how to implement adaptive sampling, per-service policies, and remote sampling configuration for production-scale distributed tracing.

Prerequisites

You need a running Jaeger deployment with Elasticsearch or another storage backend. If you don't have this yet, follow our Jaeger Kubernetes deployment guide.

Understanding sampling strategies

Jaeger supports several sampling strategies that determine which traces to collect:

Strategy TypeUse CaseConfiguration

ConstFixed percentage samplingAlways sample X% of traces
ProbabilisticRandom samplingSample based on trace ID
RateLimitingMaximum traces per secondCap at N traces/second
AdaptiveDynamic adjustmentAdjust based on traffic patterns
PerServiceService-specific rulesDifferent rates per service

Step-by-step configuration

Create sampling strategies configuration

Create a JSON configuration file that defines your sampling strategies. This file tells Jaeger how to sample traces for different services and operations.

{
"default_strategy": {
"type": "probabilistic",
"param": 0.1
},
"per_service_strategies": [
{
"service": "frontend-service",
"type": "probabilistic",
"param": 0.5,
"max_traces_per_second": 100
},
{
"service": "payment-service",
"type": "probabilistic",
"param": 1.0,
"max_traces_per_second": 50
},
{
"service": "logging-service",
"type": "probabilistic",
"param": 0.01,
"max_traces_per_second": 10
},
{
"service": "health-check",
"type": "probabilistic",
"param": 0.001
}
],
"per_operation_strategies": [
{
"service": "frontend-service",
"operation": "GET /health",
"type": "probabilistic",
"param": 0.001
},
{
"service": "api-gateway",
"operation": "POST /api/orders",
"type": "probabilistic",
"param": 0.8,
"max_traces_per_second": 200
}
]
}

Configure Jaeger Collector with sampling strategies

Update your Jaeger Collector configuration to use the sampling strategies file. This enables remote sampling where the collector serves sampling decisions to clients.

sampling:
strategies-file: /etc/jaeger/sampling_strategies.json
strategies-reload-interval: 30s

http-server:
host-port: :14268

grpc-server:
host-port: :14250

processors:
batch:
timeout: 1s
send-batch-size: 1024
send-batch-max-size: 2048

Setup adaptive sampling with volume control

Create an advanced configuration that adapts sampling rates based on traffic volume and service importance.

{
"default_strategy": {
"type": "adaptive",
"max_traces_per_second": 500,
"param": 0.1
},
"per_service_strategies": [
{
"service": "user-service",
"type": "adaptive",
"param": 0.3,
"max_traces_per_second": 100,
"operation_strategies": [
{
"operation": "login",
"type": "probabilistic",
"param": 0.8
},
{
"operation": "register",
"type": "probabilistic",
"param": 1.0
}
]
},
{
"service": "database-service",
"type": "rate_limiting",
"param": 50
},
{
"service": "cache-service",
"type": "probabilistic",
"param": 0.05,
"max_traces_per_second": 20
}
]
}

Configure environment-specific sampling

Create different sampling configurations for development, staging, and production environments.

{
"default_strategy": {
"type": "probabilistic",
"param": 0.01
},
"per_service_strategies": [
{
"service": "critical-payment-service",
"type": "probabilistic",
"param": 0.5,
"max_traces_per_second": 1000
},
{
"service": "user-analytics",
"type": "probabilistic",
"param": 0.001,
"max_traces_per_second": 10
}
]
}

{
"default_strategy": {
"type": "probabilistic",
"param": 1.0
},
"per_service_strategies": [
{
"service": "test-service",
"type": "probabilistic",
"param": 1.0
}
]
}

Enable remote sampling in Jaeger Collector

Configure the Jaeger Collector to serve sampling strategies to client applications over HTTP.

sudo systemctl stop jaeger-collector

[Unit]
Description=Jaeger Collector
After=network.target

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/local/bin/jaeger-collector \
--config-file=/etc/jaeger/collector.yaml \
--sampling.strategies-file=/etc/jaeger/production_sampling.json \
--sampling.strategies-reload-interval=60s \
--collector.http-server.host-port=:14268 \
--collector.grpc-server.host-port=:14250
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start jaeger-collector
sudo systemctl status jaeger-collector

Configure client applications for remote sampling

Update your application configuration to fetch sampling strategies from the Jaeger Collector instead of using local configuration.

package main

import (
"github.com/uber/jaeger-client-go/config"
"github.com/uber/jaeger-client-go"
)

func initJaeger() {
cfg := config.Configuration{
ServiceName: "my-service",
Sampler: &config.SamplerConfig{
Type: jaeger.SamplerTypeRemote,
Param: 0.1, // fallback sampling rate
SamplingServerURL: "http://jaeger-collector:14268/api/sampling",
SamplingRefreshInterval: 60,
},
Reporter: &config.ReporterConfig{
LocalAgentHostPort: "jaeger-agent:6831",
},
}

tracer, closer, err := cfg.NewTracer()
if err != nil {
panic(err)
}
defer closer.Close()
}

Setup sampling strategy monitoring

Create a monitoring script to track sampling effectiveness and adjust strategies based on metrics.

#!/bin/bash

# Get sampling stats from Jaeger
SAMPLING_URL="http://localhost:14268/api/sampling"
METRICS_URL="http://localhost:14269/metrics"

# Check current sampling strategies
echo "Current sampling strategies:"
curl -s $SAMPLING_URL | jq .

# Get trace volume metrics
echo "\nTrace volume metrics:"
curl -s $METRICS_URL | grep jaeger_collector_traces_received_total

# Check storage usage
echo "\nStorage usage:"
curl -s $METRICS_URL | grep jaeger_collector_spans_saved_total

# Calculate sampling efficiency
RECEIVED=$(curl -s $METRICS_URL | grep jaeger_collector_traces_received_total | tail -1 | awk '{print $2}')
SAVED=$(curl -s $METRICS_URL | grep jaeger_collector_spans_saved_total | tail -1 | awk '{print $2}')

if [ "$RECEIVED" -gt 0 ]; then
EFFICIENCY=$(echo "scale=2; $SAVED / $RECEIVED * 100" | bc)
echo "\nSampling efficiency: $EFFICIENCY%"
fi

sudo chmod +x /usr/local/bin/monitor-sampling.sh

Create automated sampling adjustment script

Implement a script that automatically adjusts sampling rates based on system load and storage capacity.

#!/usr/bin/env python3
import json
import requests
import time
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class SamplingAdjuster:
def __init__(self, collector_url, strategies_file):
self.collector_url = collector_url
self.strategies_file = strategies_file

def get_current_load(self):
"""Get current trace volume from metrics"""
try:
response = requests.get(f"{self.collector_url}/metrics")
metrics = response.text

# Extract trace rate (traces per second)
for line in metrics.split('\n'):
if 'jaeger_collector_traces_received_total' in line:
return float(line.split()[-1])
except Exception as e:
logger.error(f"Failed to get metrics: {e}")
return 0

def adjust_sampling_rate(self, current_load):
"""Adjust sampling based on load"""
with open(self.strategies_file, 'r') as f:
strategies = json.load(f)

# Adjust default strategy based on load
if current_load > 10000: # High load
strategies['default_strategy']['param'] = 0.01
elif current_load > 1000: # Medium load 
strategies['default_strategy']['param'] = 0.05
else: # Low load
strategies['default_strategy']['param'] = 0.1

# Write updated strategies
with open(self.strategies_file, 'w') as f:
json.dump(strategies, f, indent=2)

logger.info(f"Adjusted sampling for load: {current_load}")

def main():
adjuster = SamplingAdjuster(
collector_url="http://localhost:14268",
strategies_file="/etc/jaeger/production_sampling.json"
)

while True:
load = adjuster.get_current_load()
adjuster.adjust_sampling_rate(load)
time.sleep(300) # Check every 5 minutes

if __name__ == "__main__":
main()

sudo chmod +x /usr/local/bin/adjust-sampling.py

Setup sampling strategy validation

Create a validation script to ensure sampling configurations are working correctly.

#!/bin/bash

JAEGER_COLLECTOR="http://localhost:14268"
JAEGER_QUERY="http://localhost:16686"

echo "Validating Jaeger sampling configuration..."

# Test sampling endpoint
echo "1. Testing sampling endpoint:"
SAMPLING_RESPONSE=$(curl -s -w "%{http_code}" $JAEGER_COLLECTOR/api/sampling)
HTTP_CODE=${SAMPLING_RESPONSE: -3}

if [ "$HTTP_CODE" = "200" ]; then
echo "✓ Sampling endpoint accessible"
else
echo "✗ Sampling endpoint failed (HTTP $HTTP_CODE)"
exit 1
fi

# Validate JSON structure
echo "\n2. Validating sampling strategy JSON:"
SAMPLING_JSON=$(curl -s $JAEGER_COLLECTOR/api/sampling)
echo $SAMPLING_JSON | jq . > /dev/null 2>&1
if [ $? -eq 0 ]; then
echo "✓ Valid JSON structure"
else
echo "✗ Invalid JSON structure"
exit 1
fi

# Check for required fields
echo "\n3. Checking required fields:"
HAS_DEFAULT=$(echo $SAMPLING_JSON | jq -r '.default_strategy.type')
if [ "$HAS_DEFAULT" != "null" ] && [ "$HAS_DEFAULT" != "" ]; then
echo "✓ Default strategy configured"
else
echo "✗ Missing default strategy"
fi

# Test trace collection
echo "\n4. Testing trace collection:"
TRACE_COUNT=$(curl -s "$JAEGER_QUERY/api/traces?limit=1" | jq -r '.data | length')
if [ "$TRACE_COUNT" -gt 0 ]; then
echo "✓ Traces are being collected"
else
echo "! No recent traces found (this may be normal)"
fi

echo "\nSampling validation complete."

sudo chmod +x /usr/local/bin/validate-sampling.sh

Configure per-service sampling policies

Create service-tier based sampling

Implement different sampling rates based on service criticality and business importance.

{
"default_strategy": {
"type": "probabilistic",
"param": 0.1
},
"per_service_strategies": [
{
"service": "tier1-payment-gateway",
"type": "probabilistic",
"param": 0.8,
"max_traces_per_second": 500,
"operation_strategies": [
{
"operation": "process_payment",
"type": "probabilistic",
"param": 1.0
},
{
"operation": "refund_payment",
"type": "probabilistic",
"param": 1.0
}
]
},
{
"service": "tier2-user-service",
"type": "probabilistic",
"param": 0.3,
"max_traces_per_second": 200
},
{
"service": "tier3-analytics",
"type": "probabilistic",
"param": 0.05,
"max_traces_per_second": 50
},
{
"service": "tier4-background-jobs",
"type": "probabilistic",
"param": 0.01,
"max_traces_per_second": 10
}
]
}

Setup error-based sampling boost

Configure higher sampling rates for services experiencing errors to improve debugging.

{
"default_strategy": {
"type": "probabilistic",
"param": 0.1
},
"per_service_strategies": [
{
"service": "error-prone-service",
"type": "probabilistic",
"param": 0.5,
"max_traces_per_second": 100,
"operation_strategies": [
{
"operation": "failing_endpoint",
"type": "probabilistic",
"param": 1.0
}
]
}
],
"per_operation_strategies": [
{
"service": "*",
"operation": "*error*",
"type": "probabilistic",
"param": 0.8
},
{
"service": "*", 
"operation": "*exception*",
"type": "probabilistic",
"param": 0.8
}
]
}

Setup remote sampling with Jaeger Collector

Configure collector for high availability

Setup multiple Jaeger Collectors with load balancing for sampling strategy distribution.

sampling:
strategies-file: /etc/jaeger/production_sampling.json
strategies-reload-interval: 30s

http-server:
host-port: 0.0.0.0:14268

grpc-server: 
host-port: 0.0.0.0:14250

span-storage:
type: elasticsearch

elasticsearch:
server-urls: http://elasticsearch-1:9200,http://elasticsearch-2:9200
index-prefix: jaeger

processors:
batch:
timeout: 1s
send-batch-size: 2048
send-batch-max-size: 4096

metrics-storage:
type: prometheus

Create sampling strategy hot reload

Implement a system to update sampling strategies without restarting the collector.

#!/bin/bash

STRATEGIES_FILE="/etc/jaeger/production_sampling.json"
COLLECTOR_PID_FILE="/var/run/jaeger-collector.pid"
BACKUP_DIR="/var/backups/jaeger"

# Create backup
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
sudo mkdir -p $BACKUP_DIR
sudo cp $STRATEGIES_FILE "$BACKUP_DIR/sampling_strategies_$TIMESTAMP.json"

# Validate new configuration
echo "Validating new sampling configuration..."
if ! jq . "$STRATEGIES_FILE" > /dev/null 2>&1; then
echo "Error: Invalid JSON in strategies file"
exit 1
fi

# Send SIGHUP to collector for hot reload
if [ -f "$COLLECTOR_PID_FILE" ]; then
PID=$(cat $COLLECTOR_PID_FILE)
if kill -0 $PID 2>/dev/null; then
echo "Reloading sampling strategies..."
kill -HUP $PID
echo "Sampling strategies reloaded successfully"
else
echo "Collector process not found, restarting service..."
sudo systemctl restart jaeger-collector
fi
else
echo "PID file not found, restarting service..."
sudo systemctl restart jaeger-collector
fi

# Verify reload
sleep 2
echo "Verifying configuration reload..."
curl -s http://localhost:14268/api/sampling | jq . > /dev/null
if [ $? -eq 0 ]; then
echo "✓ Sampling strategies successfully reloaded"
else
echo "✗ Failed to reload sampling strategies"
exit 1
fi

sudo chmod +x /usr/local/bin/reload-sampling.sh

Monitor and optimize sampling performance

Setup Prometheus metrics collection

Configure Prometheus to scrape Jaeger metrics for sampling analysis. This helps you monitor sampling effectiveness and storage impact.

global:
scrape_interval: 15s

scrape_configs:
- job_name: 'jaeger-collector'
static_configs:
- targets: ['localhost:14269']
scrape_interval: 10s
metrics_path: /metrics

- job_name: 'jaeger-agent'
static_configs:
- targets: ['localhost:14271']
scrape_interval: 30s

- job_name: 'jaeger-query'
static_configs:
- targets: ['localhost:16687']
scrape_interval: 30s

Create sampling performance dashboard

Setup Grafana dashboard to visualize sampling metrics and trace volumes.

---

### Setup S3-compatible disaster recovery with cross-region replication using MinIO

URL: https://binadit.com/tutorials/setup-s3-compatible-disaster-recovery-with-cross-region-replication
Category: devops
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Configure MinIO clusters across multiple regions with automated cross-region bucket replication, SSL encryption, and comprehensive monitoring for enterprise-grade disaster recovery.

What this solves

Enterprise applications need reliable disaster recovery with geographically distributed storage that can survive regional outages. MinIO provides S3-compatible object storage that you can deploy across multiple data centers with automated replication. This tutorial shows you how to build a production-ready disaster recovery system using MinIO clusters in different regions with SSL encryption, automated failover testing, and comprehensive monitoring through Prometheus and Grafana.

Step-by-step configuration

Install MinIO server on both regions

Download and install MinIO on your primary and disaster recovery servers. We'll set up a minimum of 4 nodes per region for high availability.

wget https://dl.min.io/server/minio/release/linux-amd64/minio
sudo chmod +x minio
sudo mv minio /usr/local/bin/
sudo useradd -r minio-user
sudo mkdir -p /etc/minio /opt/minio/data{1..4}
sudo chown -R minio-user:minio-user /opt/minio /etc/minio

wget https://dl.min.io/server/minio/release/linux-amd64/minio
sudo chmod +x minio
sudo mv minio /usr/local/bin/
sudo useradd -r minio-user
sudo mkdir -p /etc/minio /opt/minio/data{1..4}
sudo chown -R minio-user:minio-user /opt/minio /etc/minio

Create SSL certificates for secure communication

Generate SSL certificates for encrypted communication between MinIO clusters. Replace example.com with your actual domain.

sudo mkdir -p /etc/minio/certs
sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 \
-keyout /etc/minio/certs/private.key \
-out /etc/minio/certs/public.crt \
-subj "/C=US/ST=State/L=City/O=Organization/CN=minio1.example.com" \
-addext "subjectAltName=DNS:minio1.example.com,DNS:minio2.example.com,DNS:minio3.example.com,DNS:minio4.example.com"
sudo chown -R minio-user:minio-user /etc/minio/certs
sudo chmod 600 /etc/minio/certs/private.key
sudo chmod 644 /etc/minio/certs/public.crt

Configure MinIO environment for primary region

Set up the primary MinIO cluster configuration with strong credentials and distributed storage.

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=SuperSecureMinIOPassword123!
MINIO_VOLUMES="https://minio1.example.com:9000/opt/minio/data1 https://minio2.example.com:9000/opt/minio/data2 https://minio3.example.com:9000/opt/minio/data3 https://minio4.example.com:9000/opt/minio/data4"
MINIO_OPTS="--console-address :9001 --certs-dir /etc/minio/certs"
MINIO_PROMETHEUS_AUTH_TYPE=public

sudo chown minio-user:minio-user /etc/minio/minio.conf
sudo chmod 640 /etc/minio/minio.conf

Create systemd service for MinIO primary cluster

Configure MinIO to run as a systemd service with automatic restart on failure.

[Unit]
Description=MinIO Object Storage Server
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/
User=minio-user
Group=minio-user
ProtectProc=invisible
EnvironmentFile=/etc/minio/minio.conf
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/minio/minio.conf\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
Restart=always
LimitNOFILE=65536
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

Configure MinIO disaster recovery region

Set up the secondary MinIO cluster in your disaster recovery region with different endpoints.

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=SuperSecureMinIOPassword123!
MINIO_VOLUMES="https://minio-dr1.example.com:9000/opt/minio/data1 https://minio-dr2.example.com:9000/opt/minio/data2 https://minio-dr3.example.com:9000/opt/minio/data3 https://minio-dr4.example.com:9000/opt/minio/data4"
MINIO_OPTS="--console-address :9001 --certs-dir /etc/minio/certs"
MINIO_PROMETHEUS_AUTH_TYPE=public

Start MinIO services on both regions

Enable and start MinIO on all nodes in both the primary and disaster recovery regions.

sudo systemctl daemon-reload
sudo systemctl enable minio
sudo systemctl start minio
sudo systemctl status minio

Install and configure MinIO client

Install the MinIO client to manage both clusters and configure replication.

wget https://dl.min.io/client/mc/release/linux-amd64/mc
sudo chmod +x mc
sudo mv mc /usr/local/bin/
mc alias set primary https://minio1.example.com:9000 minioadmin SuperSecureMinIOPassword123!
mc alias set disaster-recovery https://minio-dr1.example.com:9000 minioadmin SuperSecureMinIOPassword123!

Create buckets and configure cross-region replication

Set up buckets on both clusters and configure automated replication from primary to disaster recovery.

mc mb primary/production-data
mc mb primary/backups
mc mb disaster-recovery/production-data
mc mb disaster-recovery/backups

mc replicate add primary/production-data --remote-bucket disaster-recovery/production-data
mc replicate add primary/backups --remote-bucket disaster-recovery/backups

Configure bucket versioning and lifecycle policies

Enable versioning for data protection and set up lifecycle policies for automatic cleanup.

mc version enable primary/production-data
mc version enable primary/backups
mc version enable disaster-recovery/production-data
mc version enable disaster-recovery/backups

{
"Rules": [
{
"ID": "DeleteOldVersions",
"Status": "Enabled",
"NoncurrentVersionExpiration": {
"NoncurrentDays": 30
}
},
{
"ID": "AbortIncompleteMultipartUploads",
"Status": "Enabled",
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
}
]
}

mc ilm import primary/production-data < /tmp/lifecycle-policy.json
mc ilm import disaster-recovery/production-data < /tmp/lifecycle-policy.json

Set up Prometheus monitoring for MinIO clusters

Configure Prometheus to monitor both MinIO clusters with custom metrics collection.

wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
sudo mv prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
sudo useradd -r prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
sudo mv prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
sudo useradd -r prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Configure Prometheus for MinIO monitoring

Set up Prometheus configuration to scrape metrics from both MinIO clusters.

global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "minio_alerts.yml"

scrape_configs:
- job_name: 'minio-primary'
metrics_path: /minio/v2/metrics/cluster
scheme: https
tls_config:
insecure_skip_verify: true
static_configs:
- targets: ['minio1.example.com:9000']
relabel_configs:
- source_labels: [__address__]
target_label: cluster
replacement: 'primary'

- job_name: 'minio-disaster-recovery'
metrics_path: /minio/v2/metrics/cluster
scheme: https
tls_config:
insecure_skip_verify: true
static_configs:
- targets: ['minio-dr1.example.com:9000']
relabel_configs:
- source_labels: [__address__]
target_label: cluster
replacement: 'disaster-recovery'

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create MinIO alerting rules

Configure Prometheus alerting rules to monitor replication health and cluster status.

groups:
- name: minio_alerts
rules:
- alert: MinIOClusterDown
expr: up{job=~"minio.*"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "MinIO cluster {{ $labels.cluster }} is down"
description: "MinIO cluster {{ $labels.cluster }} has been down for more than 2 minutes."

- alert: MinIOReplicationFailure
expr: increase(minio_replication_failed_bytes_total[5m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "MinIO replication failure detected"
description: "MinIO replication has failed for cluster {{ $labels.cluster }}."

- alert: MinIOHighDiskUsage
expr: (minio_cluster_disk_total_bytes - minio_cluster_disk_free_bytes) / minio_cluster_disk_total_bytes * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "MinIO cluster disk usage is high"
description: "MinIO cluster {{ $labels.cluster }} disk usage is above 80%."

- alert: MinIOReplicationLag
expr: time() - minio_replication_last_activity_time > 3600
for: 5m
labels:
severity: warning
annotations:
summary: "MinIO replication lag detected"
description: "MinIO replication for cluster {{ $labels.cluster }} has been inactive for over 1 hour."

sudo chown prometheus:prometheus /etc/prometheus/minio_alerts.yml

Create Prometheus systemd service

Configure Prometheus to run as a system service with automatic restart capabilities.

[Unit]
Description=Prometheus Monitoring System
Documentation=https://prometheus.io/docs/
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Install and configure Grafana for visualization

Set up Grafana to create dashboards for monitoring MinIO cluster health and replication status.

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

sudo dnf install -y https://dl.grafana.com/oss/release/grafana-10.2.0-1.x86_64.rpm

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Create disaster recovery testing automation

Set up automated scripts to test disaster recovery procedures and validate replication integrity.

#!/bin/bash

# MinIO Disaster Recovery Test Script
set -e

TEST_BUCKET="production-data"
TEST_FILE="dr-test-$(date +%Y%m%d-%H%M%S).txt"
TEST_CONTENT="Disaster recovery test at $(date)"
MAX_WAIT=300

echo "Starting disaster recovery test..."

# Create test file in primary cluster
echo "$TEST_CONTENT" | mc pipe primary/$TEST_BUCKET/$TEST_FILE
echo "Test file uploaded to primary cluster: $TEST_FILE"

# Wait for replication to DR cluster
echo "Waiting for replication to disaster recovery cluster..."
start_time=$(date +%s)
while true; do
if mc stat disaster-recovery/$TEST_BUCKET/$TEST_FILE >/dev/null 2>&1; then
echo "File replicated successfully to DR cluster"
break
fi

current_time=$(date +%s)
elapsed=$((current_time - start_time))

if [ $elapsed -gt $MAX_WAIT ]; then
echo "ERROR: Replication timeout after $MAX_WAIT seconds"
exit 1
fi

sleep 5
done

# Verify file integrity
PRIMARY_HASH=$(mc cat primary/$TEST_BUCKET/$TEST_FILE | sha256sum | cut -d' ' -f1)
DR_HASH=$(mc cat disaster-recovery/$TEST_BUCKET/$TEST_FILE | sha256sum | cut -d' ' -f1)

if [ "$PRIMARY_HASH" = "$DR_HASH" ]; then
echo "SUCCESS: File integrity verified across clusters"
else
echo "ERROR: File integrity check failed"
exit 1
fi

# Cleanup test files
mc rm primary/$TEST_BUCKET/$TEST_FILE
mc rm disaster-recovery/$TEST_BUCKET/$TEST_FILE

echo "Disaster recovery test completed successfully"
echo "Replication time: $elapsed seconds"

sudo chmod +x /opt/minio/dr-test.sh
sudo chown minio-user:minio-user /opt/minio/dr-test.sh

Schedule automated disaster recovery testing

Create a systemd timer to run disaster recovery tests automatically every 6 hours.

[Unit]
Description=MinIO Disaster Recovery Test
After=minio.service

[Service]
Type=oneshot
User=minio-user
Group=minio-user
WorkingDirectory=/opt/minio
ExecStart=/opt/minio/dr-test.sh
StandardOutput=journal
StandardError=journal

[Unit]
Description=Run MinIO Disaster Recovery Test
Requires=minio-dr-test.service

[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable minio-dr-test.timer
sudo systemctl start minio-dr-test.timer

Configure MinIO access policies for applications

Create specific access policies for applications to use the MinIO clusters securely.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::production-data",
"arn:aws:s3:::production-data/*",
"arn:aws:s3:::backups",
"arn:aws:s3:::backups/*"
]
}
]
}

mc admin policy create primary app-access-policy /tmp/app-policy.json
mc admin policy create disaster-recovery app-access-policy /tmp/app-policy.json

mc admin user add primary app-user SecureAppPassword123!
mc admin user add disaster-recovery app-user SecureAppPassword123!

mc admin policy attach primary app-access-policy --user app-user
mc admin policy attach disaster-recovery app-access-policy --user app-user

Set up automated failover procedures

Create scripts for automated failover to the disaster recovery cluster during outages.

#!/bin/bash

# MinIO Failover Script
set -e

PRIMARY_ENDPOINT="https://minio1.example.com:9000"
DR_ENDPOINT="https://minio-dr1.example.com:9000"
HEALTH_CHECK_TIMEOUT=10

check_cluster_health() {
local endpoint=$1
local cluster_name=$2

if timeout $HEALTH_CHECK_TIMEOUT mc admin info "$cluster_name" >/dev/null 2>&1; then
return 0
else
return 1
fi
}

echo "Checking primary cluster health..."
if check_cluster_health "$PRIMARY_ENDPOINT" "primary"; then
echo "Primary cluster is healthy - no failover needed"
exit 0
fi

echo "Primary cluster unhealthy - initiating failover to DR cluster"

# Check DR cluster health
if ! check_cluster_health "$DR_ENDPOINT" "disaster-recovery"; then
echo "ERROR: Disaster recovery cluster is also unhealthy!"
exit 1
fi

# Update application configuration to use DR cluster
echo "Updating application endpoints to DR cluster..."
# This would typically update load balancer configuration
# or application configuration files

# Log failover event
echo "$(date): Failover completed - applications now using DR cluster" >> /var/log/minio-failover.log

# Send notification (configure with your notification system)
echo "Failover to disaster recovery cluster completed at $(date)" | \
mail -s "MinIO Failover Alert" admin@example.com 2>/dev/null || true

echo "Failover procedure completed successfully"

sudo chmod +x /opt/minio/failover.sh
sudo chown minio-user:minio-user /opt/minio/failover.sh
sudo touch /var/log/minio-failover.log
sudo chown minio-user:minio-user /var/log/minio-failover.log

Verify your setup

Test your disaster recovery configuration to ensure everything works correctly.

# Check MinIO cluster status
mc admin info primary
mc admin info disaster-recovery

# Verify replication configuration
mc replicate ls primary/production-data

# Test replication with a sample file
echo "Test data" | mc pipe primary/production-data/test.txt
sleep 30
mc cat disaster-recovery/production-data/test.txt

# Check Promethe

---

### Configure Apache reverse proxy with caching for microservices

URL: https://binadit.com/tutorials/configure-apache-reverse-proxy-with-caching
Category: hosting
Difficulty: intermediate
Time: ~25 minutes
Author: Binadit Tech Team

> Set up Apache HTTP Server as a reverse proxy with intelligent caching for microservices architectures. This tutorial covers mod_proxy, mod_cache configuration, cache policies, and monitoring for high-performance service delivery.

What this solves

A reverse proxy with caching reduces load on backend microservices by serving cached responses for repeated requests. Apache's mod_proxy and mod_cache modules provide flexible routing, load balancing, and intelligent caching that can dramatically improve response times and reduce resource usage across your service architecture.

Step-by-step configuration

Update system packages

Start by updating your package manager to ensure you have the latest security patches and package versions.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install Apache HTTP Server

Install Apache web server which includes the proxy and caching modules we need.

sudo apt install -y apache2 apache2-utils

sudo dnf install -y httpd httpd-tools

Enable required Apache modules

Enable the proxy, cache, and related modules needed for reverse proxy functionality with caching.

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod proxy_balancer
sudo a2enmod lbmethod_byrequests
sudo a2enmod cache
sudo a2enmod cache_disk
sudo a2enmod headers
sudo a2enmod rewrite

# Modules are loaded via LoadModule directives in configuration
</code><h2><code>We'll configure these in the next steps</code></h2></pre>
</div>
</div>
</div>

<div class="step">
### Create cache directory
<p>Create a dedicated directory for Apache to store cached content with proper permissions.</p>
<pre class="terminal"><code>sudo mkdir -p /var/cache/apache2/mod_cache_disk
sudo chown www-data:www-data /var/cache/apache2/mod_cache_disk
sudo chmod 755 /var/cache/apache2/mod_cache_disk

Never use chmod 777. It gives every user on the system full access to your files. The web server only needs write access to the cache directory, which we provide with proper ownership.

Configure main Apache settings

Configure the main Apache configuration with performance optimizations for proxy and caching workloads.

# Add these settings at the end of the file

# Performance tuning for proxy workloads
MaxRequestWorkers 400
ThreadsPerChild 25
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250

# Proxy timeout settings
ProxyTimeout 300
ProxyPreserveHost On

# Security headers
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
Header always set Referrer-Policy strict-origin-when-cross-origin

# Add these settings at the end of the file

# Load required modules
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule cache_module modules/mod_cache.so
LoadModule cache_disk_module modules/mod_cache_disk.so
LoadModule headers_module modules/mod_headers.so
LoadModule rewrite_module modules/mod_rewrite.so

# Performance tuning for proxy workloads
MaxRequestWorkers 400
ThreadsPerChild 25
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250

# Proxy timeout settings
ProxyTimeout 300
ProxyPreserveHost On

# Security headers
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
Header always set Referrer-Policy strict-origin-when-cross-origin

Create virtual host configuration

Create a virtual host that configures reverse proxy with caching for your microservices.

Create cache directory for Red Hat systems

On Red Hat-based systems, create the cache directory with SELinux context.

# Cache directory already created in previous step

sudo mkdir -p /var/cache/httpd/mod_cache_disk
sudo chown apache:apache /var/cache/httpd/mod_cache_disk
sudo chmod 755 /var/cache/httpd/mod_cache_disk
sudo setsebool -P httpd_can_network_connect 1
sudo semanage fcontext -a -t httpd_cache_t "/var/cache/httpd/mod_cache_disk(/.*)?" || true
sudo restorecon -R /var/cache/httpd/mod_cache_disk

Enable the virtual host

Enable the new virtual host configuration and disable the default site.

sudo a2ensite microservices-proxy.conf
sudo a2dissite 000-default.conf

# Configuration is automatically loaded from conf.d directory

Configure cache cleanup

Set up automatic cache cleanup to prevent disk space issues. Create a systemd timer to clean old cache files.

[Unit]
Description=Apache Cache Cleanup
After=network.target

[Service]
Type=oneshot
User=www-data
Group=www-data
ExecStart=/usr/bin/find /var/cache/apache2/mod_cache_disk -type f -mtime +7 -delete
ExecStart=/usr/bin/find /var/cache/apache2/mod_cache_disk -type d -empty -delete

[Unit]
Description=Run Apache Cache Cleanup Daily
Requires=apache-cache-cleanup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now apache-cache-cleanup.timer

Test configuration and start Apache

Test the Apache configuration for syntax errors and start the service.

sudo apache2ctl configtest
sudo systemctl enable --now apache2
sudo systemctl status apache2

sudo httpd -t
sudo systemctl enable --now httpd
sudo systemctl status httpd

Configure firewall rules

Open HTTP and HTTPS ports in the firewall for web traffic.

sudo ufw allow 'Apache Full'
sudo ufw reload

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Configure cache policies and optimization

Advanced cache configuration

Fine-tune cache behavior with advanced policies for different types of content and API responses.

# Cache size limits and memory settings
CacheMaxFileSize 10000000
CacheMinFileSize 1
CacheReadSize 102400
CacheReadTime 3600

# Ignore certain headers that prevent caching
CacheIgnoreHeaders Set-Cookie
CacheIgnoreQueryString Off
CacheKeyBaseURL http://api.example.com

# Cache based on Accept-Encoding for compressed content
CacheStorePrivate On
CacheStoreNoStore Off

# Advanced cache policies

# Don't cache authentication endpoints

# Cache size limits and memory settings
CacheMaxFileSize 10000000
CacheMinFileSize 1
CacheReadSize 102400
CacheReadTime 3600

# Ignore certain headers that prevent caching
CacheIgnoreHeaders Set-Cookie
CacheIgnoreQueryString Off
CacheKeyBaseURL http://api.example.com

# Cache based on Accept-Encoding for compressed content
CacheStorePrivate On
CacheStoreNoStore Off

# Advanced cache policies

# Don't cache authentication endpoints

Enable cache optimization

Enable the cache optimization configuration and reload Apache.

sudo a2enconf cache-optimization
sudo apache2ctl configtest
sudo systemctl reload apache2

sudo httpd -t
sudo systemctl reload httpd

Monitor and troubleshoot reverse proxy

Set up cache monitoring script

Create a monitoring script to track cache performance and hit rates.

#!/bin/bash

# Apache cache monitoring script
CACHE_DIR="/var/cache/apache2/mod_cache_disk"
LOG_FILE="/var/log/apache2/microservices_cache.log"

# Cache disk usage
echo "Cache Directory Usage:"
du -sh $CACHE_DIR
echo ""

# Cache files count
echo "Cache Files Count:"
find $CACHE_DIR -type f | wc -l
echo ""

# Recent cache statistics from logs
echo "Cache Hit/Miss Statistics (last 1000 requests):"
if [ -f "$LOG_FILE" ]; then
tail -1000 $LOG_FILE | awk '{
if ($NF ~ /HIT/) hits++;
else if ($NF ~ /MISS/) misses++;
total++
} END {
if (total > 0) {
hit_rate = (hits/total)*100;
printf "Hits: %d (%.1f%%)\n", hits, hit_rate;
printf "Misses: %d (%.1f%%)\n", misses, 100-hit_rate;
printf "Total: %d\n", total;
} else {
print "No cache data found in logs";
}
}'
else
echo "Cache log file not found"
fi

# Top requested URLs being cached
echo ""
echo "Top 10 Cached URLs:"
if [ -f "$LOG_FILE" ]; then
tail -1000 $LOG_FILE | awk '{print $7}' | sort | uniq -c | sort -nr | head -10
fi

sudo chmod +x /usr/local/bin/apache-cache-stats.sh

Configure l

---

### Setup ArgoCD ApplicationSets for multi-environment GitOps workflows with automated deployment pipelines

URL: https://binadit.com/tutorials/setup-argocd-application-sets-for-multi-environment-gitops
Category: devops
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Configure ArgoCD ApplicationSets to automate deployments across multiple environments using GitOps patterns. Learn to create templates, generators, and automated promotion workflows for production-grade Kubernetes deployments.

What this solves

ArgoCD ApplicationSets automate the deployment of applications across multiple environments, clusters, or namespaces using template-based configurations. Instead of manually creating individual ArgoCD Applications for dev, staging, and production environments, ApplicationSets use generators to dynamically create and manage these applications from a single configuration.

Prerequisites

Requirements: You need a running Kubernetes cluster with ArgoCD already installed. This tutorial assumes you have cluster-admin privileges and kubectl configured.

Install ArgoCD if not already present

Skip this step if ArgoCD is already running in your cluster. Otherwise, install ArgoCD in the argocd namespace.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Wait for ArgoCD components to be ready

Verify all ArgoCD components are running before proceeding with ApplicationSet configuration.

kubectl wait --for=condition=ready pod -l app.kubernetes.io/part-of=argocd -n argocd --timeout=300s

Install ArgoCD ApplicationSet Controller

The ApplicationSet controller is included in recent ArgoCD versions but may need explicit installation on older versions.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/applicationset/v0.4.1/manifests/install.yaml

Step-by-step configuration

Create Git repository structure

Set up a proper GitOps repository structure to support multi-environment deployments with ApplicationSets.

mkdir -p gitops-demo/{apps,environments/{dev,staging,prod},clusters}
cd gitops-demo

Create base application manifests

Create a base application configuration that will be templated across environments.

apiVersion: v1
kind: Namespace
metadata:
name: nginx-{{.environment}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: nginx-{{.environment}}
spec:
replicas: {{.replicas}}
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
environment: {{.environment}}
spec:
containers:
- name: nginx
image: nginx:{{.version}}
ports:
- containerPort: 80
resources:
requests:
cpu: {{.cpu_request}}
memory: {{.memory_request}}
limits:
cpu: {{.cpu_limit}}
memory: {{.memory_limit}}
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: nginx-{{.environment}}
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
type: ClusterIP

Create environment-specific configurations

Define different resource allocations and configurations for each environment.

{
"environment": "dev",
"replicas": 1,
"version": "1.21",
"cpu_request": "100m",
"memory_request": "128Mi",
"cpu_limit": "200m",
"memory_limit": "256Mi",
"cluster": "dev-cluster",
"namespace": "nginx-dev"
}

Create staging environment configuration

Configure staging with higher resource allocation and stable image versions.

{
"environment": "staging",
"replicas": 2,
"version": "1.21",
"cpu_request": "200m",
"memory_request": "256Mi",
"cpu_limit": "500m",
"memory_limit": "512Mi",
"cluster": "staging-cluster",
"namespace": "nginx-staging"
}

Create production environment configuration

Configure production with high availability and resource allocation suitable for production workloads.

{
"environment": "prod",
"replicas": 3,
"version": "1.20",
"cpu_request": "500m",
"memory_request": "512Mi",
"cpu_limit": "1000m",
"memory_limit": "1Gi",
"cluster": "prod-cluster",
"namespace": "nginx-prod"
}

Create cluster configuration files

Define cluster-specific settings that ApplicationSets can reference for multi-cluster deployments.

apiVersion: v1
kind: Secret
metadata:
name: dev-cluster-secret
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
name: dev-cluster
server: https://dev.example.com
config: |
{
"bearerToken": "

Create Git generator ApplicationSet

Create an ApplicationSet that uses Git file generator to automatically discover environments from your repository structure.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: nginx-environments
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/your-org/gitops-demo
revision: HEAD
files:
- path: "environments/*/config.json"
template:
metadata:
name: 'nginx-{{environment}}'
spec:
project: default
source:
repoURL: https://github.com/your-org/gitops-demo
targetRevision: HEAD
path: apps
helm:
valueFiles:
- "../environments/{{environment}}/config.json"
destination:
server: https://kubernetes.default.svc
namespace: 'nginx-{{environment}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

Create list generator ApplicationSet for specific clusters

Use list generator when you need explicit control over which clusters and environments to deploy to.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: nginx-multi-cluster
namespace: argocd
spec:
generators:
- list:
elements:
- cluster: dev-cluster
url: https://dev.example.com
environment: dev
replicas: "1"
version: "1.21"
- cluster: staging-cluster
url: https://staging.example.com
environment: staging
replicas: "2"
version: "1.21"
- cluster: prod-cluster
url: https://prod.example.com
environment: prod
replicas: "3"
version: "1.20"
template:
metadata:
name: 'nginx-{{environment}}-{{cluster}}'
spec:
project: default
source:
repoURL: https://github.com/your-org/gitops-demo
targetRevision: HEAD
path: apps
helm:
parameters:
- name: environment
value: '{{environment}}'
- name: replicas
value: '{{replicas}}'
- name: version
value: '{{version}}'
destination:
server: '{{url}}'
namespace: 'nginx-{{environment}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

Create cluster generator ApplicationSet

Automatically discover registered clusters and deploy applications to all of them.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: nginx-all-clusters
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
environment: production
template:
metadata:
name: 'nginx-{{name}}'
spec:
project: default
source:
repoURL: https://github.com/your-org/gitops-demo
targetRevision: HEAD
path: apps
helm:
parameters:
- name: environment
value: '{{metadata.labels.environment}}'
- name: cluster
value: '{{name}}'
destination:
server: '{{server}}'
namespace: nginx
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

Apply ApplicationSet configurations

Deploy the ApplicationSets to your cluster and verify they create the expected Application resources.

kubectl apply -f applicationset-git-generator.yaml
kubectl apply -f applicationset-list-generator.yaml

Configure RBAC for ApplicationSets

Set up proper permissions for ApplicationSet controller to manage applications across namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: applicationset-controller
rules:
- apiGroups: ["argoproj.io"]
resources: ["applications", "applicationsets"]
verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: applicationset-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: applicationset-controller
subjects:
- kind: ServiceAccount
name: applicationset-controller
namespace: argocd

Apply RBAC configuration

Apply the RBAC rules to ensure ApplicationSet controller has necessary permissions.

kubectl apply -f applicationset-rbac.yaml

Create progressive deployment ApplicationSet

Implement automated promotion workflow where successful deployment to dev triggers staging deployment.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: nginx-progressive
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/your-org/gitops-demo
revision: HEAD
files:
- path: "environments/*/config.json"
template:
metadata:
name: 'nginx-{{environment}}'
annotations:
argocd-image-updater.argoproj.io/image-list: nginx=nginx
argocd-image-updater.argoproj.io/write-back-method: git
spec:
project: default
source:
repoURL: https://github.com/your-org/gitops-demo
targetRevision: HEAD
path: apps
destination:
server: https://kubernetes.default.svc
namespace: 'nginx-{{environment}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
syncWindows:
- kind: allow
schedule: '0 2 * * *'
duration: 1h
applications:
- 'nginx-prod'

Advanced ApplicationSet patterns

Create matrix generator for complex scenarios

Use matrix generator to combine multiple generators for complex deployment scenarios.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: nginx-matrix
namespace: argocd
spec:
generators:
- matrix:
generators:
- git:
repoURL: https://github.com/your-org/gitops-demo
revision: HEAD
directories:
- path: environments/*
- clusters:
selector:
matchLabels:
environment: '{{path.basename}}'
template:
metadata:
name: 'nginx-{{path.basename}}-{{name}}'
spec:
project: default
source:
repoURL: https://github.com/your-org/gitops-demo
targetRevision: HEAD
path: apps
helm:
parameters:
- name: environment
value: '{{path.basename}}'
- name: cluster
value: '{{name}}'
destination:
server: '{{server}}'
namespace: 'nginx-{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true

Configure ApplicationSet with Helm chart source

Use ApplicationSets with Helm charts for more complex application packaging and configuration.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: nginx-helm
namespace: argocd
spec:
generators:
- list:
elements:
- environment: dev
values: |
replicaCount: 1
image:
tag: "1.21"
resources:
requests:
cpu: 100m
memory: 128Mi
- environment: prod
values: |
replicaCount: 3
image:
tag: "1.20"
resources:
requests:
cpu: 500m
memory: 512Mi
template:
metadata:
name: 'nginx-helm-{{environment}}'
spec:
project: default
source:
repoURL: https://charts.bitnami.com/bitnami
chart: nginx
targetRevision: 13.2.23
helm:
values: '{{values}}'
destination:
server: https://kubernetes.default.svc
namespace: 'nginx-{{environment}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

Monitoring and troubleshooting

Set up ApplicationSet monitoring

Create ServiceMonitor for Prometheus to scrape ApplicationSet controller metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: applicationset-controller-metrics
namespace: argocd
spec:
selector:
matchLabels:
app.kubernetes.io/component: applicationset-controller
app.kubernetes.io/name: argocd-applicationset-controller
endpoints:
- port: metrics

Configure logging for ApplicationSet controller

Increase log level for debugging ApplicationSet issues and generator behavior.

kubectl patch deployment argocd-applicationset-controller -n argocd -p '{
"spec": {
"template": {
"spec": {
"containers": [{
"name": "applicationset-controller",
"args": [
"--logLevel", "debug",
"--metrics-addr", "0.0.0.0:8080"
]
}]
}
}
}
}'

Verify your setup

Check that ApplicationSets are creating the expected Applications and that they sync successfully.

# Check ApplicationSet status
kubectl get applicationsets -n argocd

# Verify generated Applications
kubectl get applications -n argocd

# Check ApplicationSet controller logs
kubectl logs -n argocd deployment/argocd-applicationset-controller

# Verify applications are synced
kubectl get applications -n argocd -o jsonpath='{range .items[*]}{.metadata.name}: {.status.sync.status}{"\n"}{end}'

You can also check the ArgoCD UI to see the ApplicationSet and its generated Applications visually.

Common issues

Symptom
Cause
Fix

Applications not created
Generator not finding files/clusters
Check generator paths and cluster labels with kubectl describe applicationset

Template parameters not resolved
Missing or incorrect generator field references
Verify generator output fields match template variables

ApplicationSet controller crashloop
RBAC permissions missing
Apply comprehensive RBAC rules with cluster-admin if needed for testing

Git generator not detecting changes
Repository access issues or path mismatch
Check repository credentials and file paths in git generator

Applications stuck in sync
Resource conflicts or namespace issues
Check Application events and sync policy configuration

Cluster generator empty results
No clusters match selector labels
Verify cluster secrets have correct labels in argocd namespace

Next steps

Configure ArgoCD Image Updater for automated container deployments

Set up ArgoCD notifications for Slack and Microsoft Teams

Configure ArgoCD with Vault for secure secrets management

Implement ArgoCD multi-cluster GitOps with cross-cluster application promotion

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. See how we run infrastructure like this for European teams.

---

### Configure Redis cluster monitoring with custom Grafana dashboards

URL: https://binadit.com/tutorials/configure-redis-cluster-monitoring-with-custom-dashboards
Category: monitoring
Difficulty: intermediate
Time: ~25 minutes
Author: Binadit Tech Team

> Set up comprehensive monitoring for your Redis cluster using redis_exporter, Prometheus, and Grafana. Configure custom dashboards and alerting rules to track performance metrics, cluster health, and resource utilization.

What this solves

Redis cluster monitoring requires tracking multiple nodes, replication status, memory usage, and performance metrics across your distributed cache infrastructure. This tutorial sets up comprehensive monitoring using redis_exporter to collect metrics, Prometheus to store them, and Grafana to visualize cluster health with custom dashboards and alerting rules.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you have the latest packages available.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install required dependencies

Install wget and systemd utilities needed for downloading and managing the monitoring services.

sudo apt install -y wget curl systemd tar

sudo dnf install -y wget curl systemd tar

Create monitoring user

Create a dedicated system user for running the redis_exporter service securely.

sudo useradd --no-create-home --shell /bin/false redis_exporter

Download and install redis_exporter

Download the latest redis_exporter binary from the official GitHub releases and install it system-wide.

cd /tmp
wget https://github.com/oliver006/redis_exporter/releases/download/v1.55.0/redis_exporter-v1.55.0.linux-amd64.tar.gz
tar xzf redis_exporter-v1.55.0.linux-amd64.tar.gz
sudo cp redis_exporter-v1.55.0.linux-amd64/redis_exporter /usr/local/bin/
sudo chown redis_exporter:redis_exporter /usr/local/bin/redis_exporter
sudo chmod 755 /usr/local/bin/redis_exporter

Configure redis_exporter service

Create a systemd service file to manage the redis_exporter process with automatic startup and monitoring.

[Unit]
Description=Redis Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=redis_exporter
Group=redis_exporter
Type=simple
ExecStart=/usr/local/bin/redis_exporter \
-redis.addr=redis://localhost:6379 \
-redis.addr=redis://localhost:6380 \
-redis.addr=redis://localhost:6381 \
-web.listen-address=:9121
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Start and enable redis_exporter

Enable the redis_exporter service to start on boot and start it immediately.

sudo systemctl daemon-reload
sudo systemctl enable redis_exporter
sudo systemctl start redis_exporter
sudo systemctl status redis_exporter

Configure Prometheus scraping

Add the redis_exporter endpoints to your Prometheus configuration to collect Redis cluster metrics.

global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "redis_cluster_rules.yml"

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

- job_name: 'redis-cluster'
static_configs:
- targets: ['localhost:9121']
scrape_interval: 10s
metrics_path: /metrics
params:
format: [prometheus]

alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093

Create Redis cluster alerting rules

Define Prometheus alerting rules to monitor critical Redis cluster conditions and performance thresholds.

groups:
- name: redis_cluster_alerts
rules:
- alert: RedisDown
expr: redis_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Redis instance is down"
description: "Redis instance {{ $labels.instance }} has been down for more than 1 minute."

- alert: RedisHighMemoryUsage
expr: (redis_memory_used_bytes / redis_memory_max_bytes) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "Redis memory usage is high"
description: "Redis instance {{ $labels.instance }} memory usage is {{ $value }}%"

- alert: RedisSlowlog
expr: increase(redis_slowlog_length[5m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Redis slow queries detected"
description: "Redis instance {{ $labels.instance }} has {{ $value }} slow queries in the last 5 minutes"

- alert: RedisClusterNodeFailure
expr: redis_cluster_nodes{role="master",state!="ok"} > 0
for: 2m
labels:
severity: critical
annotations:
summary: "Redis cluster master node failure"
description: "Redis cluster master node {{ $labels.instance }} is in {{ $labels.state }} state"

- alert: RedisHighConnections
expr: redis_connected_clients > redis_config_maxclients * 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Redis connection count is high"
description: "Redis instance {{ $labels.instance }} has {{ $value }} connections (>80% of max)"

Restart Prometheus to load new configuration

Reload Prometheus configuration to start collecting Redis metrics and enable the new alerting rules.

sudo systemctl restart prometheus
sudo systemctl status prometheus

Import Redis cluster dashboard in Grafana

Create a comprehensive Grafana dashboard configuration for monitoring Redis cluster performance and health metrics.

{
"dashboard": {
"id": null,
"title": "Redis Cluster Monitoring",
"description": "Comprehensive Redis cluster monitoring dashboard",
"tags": ["redis", "cluster", "monitoring"],
"timezone": "browser",
"panels": [
{
"id": 1,
"title": "Redis Instances Status",
"type": "stat",
"targets": [
{
"expr": "redis_up",
"legendFormat": "{{ instance }}"
}
],
"fieldConfig": {
"defaults": {
"mappings": [
{
"options": {
"0": {"text": "DOWN", "color": "red"},
"1": {"text": "UP", "color": "green"}
},
"type": "value"
}
]
}
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
},
{
"id": 2,
"title": "Memory Usage",
"type": "timeseries",
"targets": [
{
"expr": "redis_memory_used_bytes",
"legendFormat": "{{ instance }} - Used Memory"
},
{
"expr": "redis_memory_max_bytes",
"legendFormat": "{{ instance }} - Max Memory"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
},
{
"id": 3,
"title": "Commands Per Second",
"type": "timeseries",
"targets": [
{
"expr": "rate(redis_commands_processed_total[5m])",
"legendFormat": "{{ instance }} - Commands/sec"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
},
{
"id": 4,
"title": "Connected Clients",
"type": "timeseries",
"targets": [
{
"expr": "redis_connected_clients",
"legendFormat": "{{ instance }} - Clients"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
},
{
"id": 5,
"title": "Cluster Slots Assignment",
"type": "table",
"targets": [
{
"expr": "redis_cluster_slots_assigned",
"legendFormat": "Assigned Slots",
"format": "table"
}
],
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 16}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "30s"
}
}

Import dashboard via Grafana API

Use curl to import the Redis cluster dashboard into Grafana via the API.

curl -X POST \
http://admin:admin@localhost:3000/api/dashboards/db \
-H 'Content-Type: application/json' \
-d @/tmp/redis-cluster-dashboard.json

Configure Grafana data source

Add Prometheus as a data source in Grafana to connect your Redis metrics for visualization.

curl -X POST \
http://admin:admin@localhost:3000/api/datasources \
-H 'Content-Type: application/json' \
-d '{
"name": "prometheus-redis",
"type": "prometheus",
"url": "http://localhost:9090",
"access": "proxy",
"isDefault": true
}'

Configure dashboard alerts

Set up Grafana alerting rules that complement the Prometheus alerts for Redis cluster monitoring.

{
"alert": {
"name": "Redis High Memory Usage Alert",
"message": "Redis memory usage is above 80%",
"frequency": "10s",
"conditions": [
{
"query": {
"queryType": "prometheus",
"refId": "A",
"expr": "(redis_memory_used_bytes / redis_memory_max_bytes) * 100"
},
"reducer": {
"type": "last",
"params": []
},
"evaluator": {
"params": [80],
"type": "gt"
}
}
],
"executionErrorState": "alerting",
"noDataState": "no_data",
"for": "5m"
},
"notificationChannels": [
{
"name": "email-alerts",
"type": "email",
"settings": {
"addresses": "admin@example.com",
"subject": "Redis Cluster Alert"
}
}
]
}

Enable firewall rules for monitoring ports

Configure firewall rules to allow access to the redis_exporter and monitoring services.

sudo ufw allow 9121/tcp comment "Redis Exporter"
sudo ufw allow 9090/tcp comment "Prometheus"
sudo ufw allow 3000/tcp comment "Grafana"
sudo ufw reload

sudo firewall-cmd --permanent --add-port=9121/tcp
sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --reload

Verify your setup

Test that all monitoring components are working correctly and collecting Redis cluster metrics.

# Check redis_exporter status and metrics
sudo systemctl status redis_exporter
curl http://localhost:9121/metrics | grep redis_up

# Verify Prometheus is scraping Redis metrics
curl http://localhost:9090/api/v1/query?query=redis_up

# Check Grafana dashboard access
curl -I http://localhost:3000/dashboards

# Test Redis cluster connectivity
redis-cli -c -p 6379 cluster nodes
redis-cli -c -p 6379 cluster info

Common issues

Symptom
Cause
Fix

redis_exporter fails to start
Redis connection refused
Check Redis is running: sudo systemctl status redis

No metrics in Grafana
Prometheus not scraping
Verify targets in Prometheus: http://localhost:9090/targets

Dashboard shows "No data"
Data source misconfigured
Check Grafana data source URL points to Prometheus

Alerts not firing
Alertmanager not configured
Configure notification channels in Grafana settings

Permission denied on metrics
User permissions issue
Ensure redis_exporter user has access: sudo chmod 755 /usr/local/bin/redis_exporter

High memory false alerts
Incorrect memory calculation
Verify Redis maxmemory setting: redis-cli CONFIG GET maxmemory

Next steps

Setup MinIO monitoring with Prometheus and Grafana dashboards for object storage observability

Configure Prometheus Alertmanager with Slack integration for team notifications to enhance your alerting workflow

Implement Redis backup automation with compression and encryption for data protection

Configure Redis cluster SSL encryption and authentication for enhanced security

Setup centralized log aggregation with ELK Stack for comprehensive monitoring

Running this in production?

Want this handled for you? Setting up monitoring once is straightforward. Keeping it patched, monitored, backed up and tuned across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.

---

### Configure ArgoCD Image Updater for automated container deployments

URL: https://binadit.com/tutorials/configure-argocd-image-updater-for-automated-container-deployments
Category: devops
Difficulty: intermediate
Time: ~25 minutes
Author: Binadit Tech Team

> Set up ArgoCD Image Updater to automatically detect and deploy new container image versions in your GitOps workflow. Includes Git repository integration, webhook configuration, and monitoring setup.

What this solves

ArgoCD Image Updater automatically monitors container registries for new image versions and updates your Kubernetes manifests in Git repositories. This eliminates manual image version updates while maintaining GitOps principles and audit trails.

Step-by-step installation

Update system packages

Ensure your package manager has the latest package information before installing dependencies.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install kubectl and required tools

Install kubectl for Kubernetes cluster interaction and curl for API calls.

sudo apt install -y curl git
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

sudo dnf install -y curl git
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Verify ArgoCD installation

Confirm that ArgoCD is already running in your cluster. ArgoCD Image Updater requires an existing ArgoCD installation.

kubectl get pods -n argocd
kubectl get svc -n argocd

Note: This tutorial assumes you have ArgoCD already installed. If not, check out our ArgoCD installation guide first.

Create ArgoCD Image Updater namespace and RBAC

Set up the namespace and service account with appropriate permissions for the Image Updater to function.

apiVersion: v1
kind: ServiceAccount
metadata:
name: argocd-image-updater
namespace: argocd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: argocd-image-updater
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- argoproj.io
resources:
- applications
- appprojects
verbs:
- get
- list
- update
- patch
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: argocd-image-updater
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: argocd-image-updater
subjects:
- kind: ServiceAccount
name: argocd-image-updater
namespace: argocd

kubectl apply -f /tmp/argocd-image-updater-rbac.yaml

Deploy ArgoCD Image Updater

Install the Image Updater deployment with the latest stable version from the official repository.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

Configure Image Updater settings

Create a ConfigMap with global configuration settings for the Image Updater behavior and logging.

apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-image-updater-config
namespace: argocd
data:
registries.conf: |
registries:
- name: Docker Hub
api_url: https://registry-1.docker.io
prefix: docker.io
ping: yes
credentials: secret:argocd/docker-registry#creds
credsexpire: 10h
- name: Quay
api_url: https://quay.io
prefix: quay.io
ping: yes
- name: GitHub Container Registry
api_url: https://ghcr.io
prefix: ghcr.io
ping: yes
git.conf: |
git:
user: argocd-image-updater
email: argocd-image-updater@example.com
log.conf: |
log:
level: info

kubectl apply -f /tmp/argocd-image-updater-config.yaml

Create Git repository access secret

Configure credentials for the Image Updater to commit changes back to your Git repository.

kubectl create secret generic git-credentials \
--from-literal=username=your-git-username \
--from-literal=password=your-git-token \
-n argocd

kubectl label secret git-credentials \
"argocd.argoproj.io/secret-type=repo-creds" \
-n argocd

Security: Replace your-git-username and your-git-token with your actual Git credentials. Use a personal access token, not your password, for better security.

Configure container registry credentials

Set up authentication for private container registries that require credentials.

kubectl create secret generic docker-registry \
--from-literal=creds=username:password \
-n argocd

kubectl label secret docker-registry \
"argocd.argoproj.io/secret-type=repository" \
-n argocd

Configure image update automation

Annotate ArgoCD Application for automatic updates

Add annotations to your existing ArgoCD Application to enable automatic image updates.

kubectl patch application my-app -n argocd --type merge --patch '{
"metadata": {
"annotations": {
"argocd-image-updater.argoproj.io/image-list": "myapp=docker.io/myuser/myapp:latest",
"argocd-image-updater.argoproj.io/write-back-method": "git",
"argocd-image-updater.argoproj.io/git-branch": "main"
}
}
}'

Configure update strategy

Define how the Image Updater should determine which new versions to deploy.

kubectl patch application my-app -n argocd --type merge --patch '{
"metadata": {
"annotations": {
"argocd-image-updater.argoproj.io/myapp.update-strategy": "semver",
"argocd-image-updater.argoproj.io/myapp.allow-tags": "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$",
"argocd-image-updater.argoproj.io/myapp.ignore-tags": "latest,dev,staging"
}
}
}'

Enable Helm chart image updates

For Helm-based applications, configure the Image Updater to modify Helm values.

kubectl patch application my-helm-app -n argocd --type merge --patch '{
"metadata": {
"annotations": {
"argocd-image-updater.argoproj.io/image-list": "myapp=docker.io/myuser/myapp",
"argocd-image-updater.argoproj.io/write-back-method": "git",
"argocd-image-updater.argoproj.io/myapp.helm.image-name": "image.repository",
"argocd-image-updater.argoproj.io/myapp.helm.image-tag": "image.tag"
}
}
}'

Set up Git repository integration

Configure Git write-back settings

Set up how the Image Updater commits changes back to your Git repository with proper commit messages.

kubectl patch configmap argocd-image-updater-config -n argocd --patch '{
"data": {
"git.conf": "git:\n user: argocd-image-updater\n email: argocd-image-updater@example.com\n commit_user: ArgoCD Image Updater\n commit_email: argocd-image-updater@example.com\n commit_message_template: |\n build: automatic update of {{ .AppName }}\n \n {{ range .AppChanges -}}\n updates image {{ .Image }} tag '{{ .OldTag }}' to '{{ .NewTag }}'\n {{ end -}}"
}
}'

Configure webhook notifications

Set up webhook notifications to receive updates when images are automatically updated.

apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-image-updater-config
namespace: argocd
data:
webhooks.conf: |
webhooks:
- name: slack
url: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
method: POST
headers:
Content-Type: application/json
template: |
{
"text": "Image updated: {{ .AppName }}",
"attachments": [
{
"color": "good",
"fields": [
{{ range .AppChanges -}}
{
"title": "{{ .Image }}",
"value": "{{ .OldTag }} → {{ .NewTag }}",
"short": true
}{{ if not (last .) }},{{ end }}
{{ end -}}
]
}
]
}

kubectl patch configmap argocd-image-updater-config -n argocd --patch-file /tmp/webhook-config.yaml

Configure update schedules

Set up cron-based schedules for when the Image Updater should check for new images.

kubectl patch application my-app -n argocd --type merge --patch '{
"metadata": {
"annotations": {
"argocd-image-updater.argoproj.io/myapp.update-schedule": "0 2 * * *",
"argocd-image-updater.argoproj.io/myapp.platforms": "linux/amd64,linux/arm64"
}
}
}'

Monitor and troubleshoot deployments

Enable detailed logging

Configure verbose logging to help with troubleshooting and monitoring update activities.

kubectl patch deployment argocd-image-updater -n argocd --patch '{
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "argocd-image-updater",
"args": [
"--interval", "2m",
"--loglevel", "debug",
"--metrics-port", "8080",
"--health-probe-port", "8081",
"--argocd-server-addr", "argocd-server.argocd.svc.cluster.local:443",
"--insecure"
]
}
]
}
}
}
}'

Set up Prometheus metrics

Enable metrics collection for monitoring the Image Updater performance and activity.

apiVersion: v1
kind: Service
metadata:
name: argocd-image-updater-metrics
namespace: argocd
labels:
app.kubernetes.io/component: image-updater
app.kubernetes.io/name: argocd-image-updater
app.kubernetes.io/part-of: argocd
spec:
ports:
- name: metrics
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: argocd-image-updater

kubectl apply -f /tmp/metrics-service.yaml

Create ServiceMonitor for Prometheus

Set up automatic metrics scraping if you have Prometheus Operator installed.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: argocd-image-updater
namespace: argocd
labels:
app.kubernetes.io/component: image-updater
app.kubernetes.io/name: argocd-image-updater
spec:
endpoints:
- interval: 30s
path: /metrics
port: metrics
namespaceSelector:
matchNames:
- argocd
selector:
matchLabels:
app.kubernetes.io/name: argocd-image-updater

kubectl apply -f /tmp/servicemonitor.yaml

Verify your setup

kubectl get pods -n argocd | grep image-updater
kubectl logs -n argocd deployment/argocd-image-updater
kubectl get applications -n argocd -o json | jq '.items[].metadata.annotations | select(. != null) | to_entries[] | select(.key | contains("argocd-image-updater"))'
curl -s http://localhost:8080/metrics | grep argocd_image_updater

Check that your application has the correct annotations:

kubectl describe application my-app -n argocd | grep -A 10 Annotations

Common issues

Symptom
Cause
Fix

Image Updater pod not starting
Missing RBAC permissions
Verify ClusterRole and ClusterRoleBinding are applied correctly

No image updates happening
Incorrect application annotations
Check annotation syntax and ensure image-list matches your container registry

Git commits failing
Invalid Git credentials
Verify git-credentials secret has correct username and token

Registry authentication errors
Missing or invalid registry credentials
Create proper docker-registry secret with valid credentials

Webhook notifications not working
Incorrect webhook URL or format
Test webhook URL manually and verify JSON template syntax

Metrics not available
Metrics port not exposed
Ensure deployment args include --metrics-port 8080 and service is created

Next steps

Set up comprehensive Kubernetes monitoring to track your automated deployments

Configure ArgoCD notifications for deployment status updates

Implement Vault integration for secure credential management

Set up security scanning in your GitOps pipeline

Configure advanced RBAC policies for better security

Running this in production?

Need this managed? Setting up ArgoCD Image Updater once is straightforward. Keeping it patched, monitored, backed up and tuned across environments is the harder part. See how we run infrastructure like this for European teams.

---

### Setup MinIO monitoring with Prometheus and Grafana dashboards

URL: https://binadit.com/tutorials/setup-minio-monitoring-with-prometheus-and-grafana
Category: monitoring
Difficulty: intermediate
Time: ~45 minutes
Author: Binadit Tech Team

> Configure comprehensive monitoring for MinIO object storage with Prometheus metrics collection and Grafana dashboards for performance, capacity, and health tracking.

What this solves

MinIO provides detailed metrics for monitoring object storage performance, API requests, and resource usage. This tutorial sets up Prometheus to scrape MinIO metrics and configures Grafana dashboards for real-time visibility into your storage cluster's health, capacity utilization, and request patterns.

Step-by-step configuration

Install and configure Prometheus

First, install Prometheus to collect metrics from MinIO. We'll create a dedicated user and configure it as a systemd service.

sudo useradd --no-create-home --shell /bin/false prometheus
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xzf prometheus-2.45.0.linux-amd64.tar.gz
sudo cp prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

sudo useradd --no-create-home --shell /bin/false prometheus
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xzf prometheus-2.45.0.linux-amd64.tar.gz
sudo cp prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Create Prometheus directories and configuration

Set up the required directories and create the main configuration file that will include MinIO as a scrape target.

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Configure MinIO metrics endpoint

MinIO exposes Prometheus metrics on port 9000 by default. Add these environment variables to your MinIO configuration to ensure metrics are properly exposed.

MINIO_PROMETHEUS_AUTH_TYPE=public
MINIO_PROMETHEUS_URL=http://localhost:9000/minio/v2/metrics/cluster
MINIO_PROMETHEUS_JOB_ID=minio-job

Note: If you're using MinIO in distributed mode, each node will expose its own metrics endpoint. Make sure this configuration is applied to all nodes.

Create Prometheus configuration file

Configure Prometheus to scrape MinIO metrics. This configuration includes both cluster and node-level metrics collection.

global:
scrape_interval: 15s
evaluation_interval: 15s

rule_files:
- "minio_alerts.yml"

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

- job_name: 'minio-cluster'
metrics_path: /minio/v2/metrics/cluster
static_configs:
- targets: ['localhost:9000']
scrape_interval: 30s
scrape_timeout: 10s

- job_name: 'minio-node'
metrics_path: /minio/v2/metrics/node
static_configs:
- targets: ['localhost:9000']
scrape_interval: 30s
scrape_timeout: 10s

- job_name: 'minio-bucket'
metrics_path: /minio/v2/metrics/bucket
static_configs:
- targets: ['localhost:9000']
scrape_interval: 60s
scrape_timeout: 15s

Create MinIO alerting rules

Define alert conditions for critical MinIO metrics like disk usage, API errors, and node availability.

groups:
- name: minio
rules:
- alert: MinIODiskUsageHigh
expr: (minio_cluster_capacity_usable_free_bytes / minio_cluster_capacity_usable_total_bytes) * 100 < 10
for: 5m
labels:
severity: critical
annotations:
summary: "MinIO cluster disk usage is critically high"
description: "MinIO cluster has less than 10% free disk space remaining."

- alert: MinIONodeOffline
expr: up{job="minio-node"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "MinIO node is offline"
description: "MinIO node {{ $labels.instance }} has been offline for more than 2 minutes."

- alert: MinIOHighAPIErrors
expr: rate(minio_s3_requests_errors_total[5m]) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "High MinIO API error rate"
description: "MinIO is experiencing high API error rate: {{ $value }} errors/second."

- alert: MinIOHighRequestLatency
expr: histogram_quantile(0.99, rate(minio_s3_requests_ttfb_seconds_bucket[5m])) > 10
for: 10m
labels:
severity: warning
annotations:
summary: "MinIO high request latency"
description: "MinIO 99th percentile request latency is {{ $value }} seconds."

Set file ownership and create systemd service

Set proper permissions and create a systemd service file to manage Prometheus.

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sudo chown prometheus:prometheus /etc/prometheus/minio_alerts.yml

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle \
--storage.tsdb.retention.time=30d

[Install]
WantedBy=multi-user.target

Start Prometheus service

Enable and start the Prometheus service, then verify it's collecting MinIO metrics.

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Install Grafana

Install Grafana to visualize MinIO metrics collected by Prometheus.

sudo apt-get install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana

sudo tee /etc/yum.repos.d/grafana.repo<

Configure Grafana data source

Start Grafana and add Prometheus as a data source. We'll configure it via the API for automation.

sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Add Prometheus data source via API:

curl -X POST \
http://admin:admin@localhost:3000/api/datasources \
-H 'Content-Type: application/json' \
-d '{
"name": "Prometheus",
"type": "prometheus",
"url": "http://localhost:9090",
"access": "proxy",
"isDefault": true
}'

Import MinIO Grafana dashboard

Create a comprehensive MinIO dashboard that displays key metrics for monitoring storage performance and health.

{
"dashboard": {
"id": null,
"title": "MinIO Metrics",
"tags": ["minio", "storage"],
"timezone": "browser",
"panels": [
{
"title": "Cluster Storage Usage",
"type": "stat",
"targets": [
{
"expr": "minio_cluster_capacity_usable_total_bytes",
"legendFormat": "Total Capacity"
},
{
"expr": "minio_cluster_capacity_usable_free_bytes",
"legendFormat": "Free Space"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
},
{
"title": "API Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(minio_s3_requests_total[5m])",
"legendFormat": "{{ method }} requests/sec"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
},
{
"title": "Request Errors",
"type": "graph",
"targets": [
{
"expr": "rate(minio_s3_requests_errors_total[5m])",
"legendFormat": "{{ method }} errors/sec"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
},
{
"title": "Request Duration",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.99, rate(minio_s3_requests_ttfb_seconds_bucket[5m]))",
"legendFormat": "99th percentile"
},
{
"expr": "histogram_quantile(0.95, rate(minio_s3_requests_ttfb_seconds_bucket[5m]))",
"legendFormat": "95th percentile"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
}
],
"time": {
"from": "now-1h",
"to": "now"
},
"refresh": "30s"
}
}

Import the dashboard:

curl -X POST \
http://admin:admin@localhost:3000/api/dashboards/db \
-H 'Content-Type: application/json' \
-d @/tmp/minio-dashboard.json

Configure alerting in Grafana

Set up Grafana alerting to receive notifications for critical MinIO issues via email or Slack.

[smtp]
enabled = true
host = localhost:587
user = 
password = 
from_address = alerts@example.com
from_name = Grafana

[alerting]
enabled = true
execute_alerts = true

Restart Grafana to apply email configuration:

sudo systemctl restart grafana-server

Configure firewall rules

Open the required ports for Prometheus and Grafana access.

sudo ufw allow 9090/tcp comment 'Prometheus'
sudo ufw allow 3000/tcp comment 'Grafana'
sudo ufw reload

sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --reload

Verify your setup

Check that all services are running and collecting metrics properly:

# Check Prometheus status and targets
sudo systemctl status prometheus
curl http://localhost:9090/api/v1/targets

# Check Grafana status
sudo systemctl status grafana-server

# Verify MinIO metrics are being scraped
curl http://localhost:9090/api/v1/query?query=minio_cluster_capacity_usable_total_bytes

# Test MinIO metrics endpoint directly
curl http://localhost:9000/minio/v2/metrics/cluster

Access Grafana at http://your-server-ip:3000 using admin/admin credentials. You should see the MinIO dashboard with live metrics data.

Common issues

Symptom
Cause
Fix

Prometheus can't scrape MinIO metrics
MinIO metrics endpoint not enabled
Add MINIO_PROMETHEUS_AUTH_TYPE=public to MinIO config and restart

"No data" in Grafana dashboard
Prometheus data source not configured
Verify Prometheus URL is http://localhost:9090 in Grafana data sources

Grafana shows connection refused
Services not started or firewall blocking
Check systemctl status grafana-server and firewall rules

Alerts not firing
Alert rules syntax error or thresholds not met
Validate rules with promtool check rules /etc/prometheus/minio_alerts.yml

Missing bucket-level metrics
Bucket metrics endpoint not scraped
Verify /minio/v2/metrics/bucket target is active in Prometheus

Next steps

Implement MinIO security hardening with IAM policies to secure your monitored storage

Configure MinIO backup and disaster recovery for data protection

Configure Prometheus Alertmanager with Slack integration for team notifications

Setup MinIO multi-tenant monitoring with Prometheus for complex deployments

Integrate MinIO monitoring with ELK stack for log correlation

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.

---

### Implement Consul multi-datacenter replication with WAN federation

URL: https://binadit.com/tutorials/implement-consul-multi-datacenter-replication
Category: networking
Difficulty: advanced
Time: ~45 minutes
Author: Binadit Tech Team

> Set up Consul WAN federation to replicate services and configuration across multiple datacenters with ACL token replication, health monitoring, and automatic failover capabilities.

What this solves

Consul WAN federation connects multiple Consul datacenters for service discovery, configuration replication, and cross-datacenter communication. This setup provides geographic redundancy, disaster recovery capabilities, and centralized service mesh management across distributed infrastructure.

Step-by-step configuration

Install Consul on all datacenter nodes

Install Consul on each node that will participate in the WAN federation. We'll use the official HashiCorp repository for consistent versions.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y consul

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul

Create Consul user and directories

Set up the required user account and directory structure for Consul to run securely.

sudo useradd --system --home /etc/consul --shell /bin/false consul
sudo mkdir -p /opt/consul /etc/consul.d /var/lib/consul
sudo chown -R consul:consul /opt/consul /etc/consul.d /var/lib/consul
sudo chmod 755 /opt/consul /etc/consul.d /var/lib/consul

Generate encryption keys and certificates

Create the gossip encryption key and generate TLS certificates for secure communication between datacenters.

consul keygen

Save this encryption key for use in all datacenter configurations. Next, generate TLS certificates:

consul tls ca create
consul tls cert create -server -dc dc1
consul tls cert create -server -dc dc2

Note: Replace dc1 and dc2 with your actual datacenter names. Copy the CA certificate to all nodes and the appropriate server certificates to each datacenter.

Configure primary datacenter (DC1)

Create the Consul configuration for the primary datacenter that will be the source of truth for ACL replication.

datacenter = "dc1"
data_dir = "/var/lib/consul"
log_level = "INFO"
node_name = "consul-dc1-01"
bind_addr = "203.0.113.10"
client_addr = "0.0.0.0"

server = true
bootstrap_expect = 3

ui_config {
enabled = true
}

connect {
enabled = true
}

encrypt = "your-gossip-encryption-key-here"

tls {
defaults {
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}

acl = {
enabled = true
default_policy = "deny"
enable_token_persistence = true
tokens = {
initial_management = "your-bootstrap-token-here"
}
}

retry_join = ["203.0.113.11", "203.0.113.12"]
retry_join_wan = ["203.0.113.20", "203.0.113.21", "203.0.113.22"]

ports {
grpc = 8502
grpc_tls = 8503
}

performance {
raft_multiplier = 1
}

Configure secondary datacenter (DC2)

Configure the secondary datacenter to replicate from the primary and participate in WAN federation.

datacenter = "dc2"
data_dir = "/var/lib/consul"
log_level = "INFO"
node_name = "consul-dc2-01"
bind_addr = "203.0.113.20"
client_addr = "0.0.0.0"

server = true
bootstrap_expect = 3

ui_config {
enabled = true
}

connect {
enabled = true
}

encrypt = "your-gossip-encryption-key-here"

tls {
defaults {
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc2-server-consul-0.pem"
key_file = "/etc/consul.d/dc2-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}

acl = {
enabled = true
default_policy = "deny"
enable_token_persistence = true
enable_token_replication = true
tokens = {
replication = "your-replication-token-here"
}
}

primary_datacenter = "dc1"

retry_join = ["203.0.113.21", "203.0.113.22"]
retry_join_wan = ["203.0.113.10", "203.0.113.11", "203.0.113.12"]

ports {
grpc = 8502
grpc_tls = 8503
}

performance {
raft_multiplier = 1
}

Set up systemd service files

Create systemd unit files to manage Consul as a system service with proper resource limits.

[Unit]
Description=Consul
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=65536
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Configure firewall rules

Open the required ports for Consul communication between datacenters.

sudo ufw allow 8300/tcp comment "Consul server RPC"
sudo ufw allow 8301/tcp comment "Consul serf LAN"
sudo ufw allow 8301/udp comment "Consul serf LAN"
sudo ufw allow 8302/tcp comment "Consul serf WAN"
sudo ufw allow 8302/udp comment "Consul serf WAN"
sudo ufw allow 8500/tcp comment "Consul HTTP API"
sudo ufw allow 8501/tcp comment "Consul HTTPS API"
sudo ufw allow 8502/tcp comment "Consul gRPC"
sudo ufw allow 8503/tcp comment "Consul gRPC TLS"
sudo ufw reload

sudo firewall-cmd --permanent --add-port=8300/tcp
sudo firewall-cmd --permanent --add-port=8301/tcp
sudo firewall-cmd --permanent --add-port=8301/udp
sudo firewall-cmd --permanent --add-port=8302/tcp
sudo firewall-cmd --permanent --add-port=8302/udp
sudo firewall-cmd --permanent --add-port=8500/tcp
sudo firewall-cmd --permanent --add-port=8501/tcp
sudo firewall-cmd --permanent --add-port=8502/tcp
sudo firewall-cmd --permanent --add-port=8503/tcp
sudo firewall-cmd --reload

Start Consul services

Enable and start Consul on all nodes, starting with the primary datacenter first.

sudo systemctl daemon-reload
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul

Bootstrap ACL system

Initialize the ACL system on the primary datacenter and create replication tokens.

consul acl bootstrap

Save the bootstrap token and create a replication token for the secondary datacenter:

export CONSUL_HTTP_TOKEN="your-bootstrap-token-here"
consul acl policy create -name "replication" -rules 'acl = "write" operator = "write" service_prefix "" { policy = "read" intentions = "read" } node_prefix "" { policy = "write" } namespace_prefix "" { policy = "read" }'
consul acl token create -description "ACL Token Replication" -policy-name "replication"

Configure ACL token replication

Set up automatic ACL token replication from primary to secondary datacenter.

consul acl replication-token create -description "DC2 Replication Token"

Update the secondary datacenter configuration to include the replication token and restart Consul:

sudo systemctl restart consul

Join datacenters via WAN

Connect the datacenters using WAN federation to enable cross-datacenter service discovery.

consul join -wan 203.0.113.20

Verify the WAN federation status:

consul members -wan

Configure health monitoring

Set up cross-datacenter health checks and monitoring for service failover capabilities.

{
"checks": [
{
"id": "wan-connectivity",
"name": "WAN Connectivity Check",
"script": "consul members -wan | grep -q alive",
"interval": "30s",
"timeout": "10s"
},
{
"id": "acl-replication",
"name": "ACL Replication Status",
"http": "https://localhost:8501/v1/acl/replication?token=your-token-here",
"tls_skip_verify": false,
"interval": "60s",
"timeout": "10s"
}
]
}

Configure automatic failover

Set up prepared queries for automatic service failover between datacenters.

consul prepared-query create -name="web-failover" -service="web" -failover-datacenters="dc2" -token="your-token-here"

Create a sample service registration for testing:

{
"service": {
"name": "web",
"tags": ["v1"],
"port": 80,
"check": {
"http": "http://localhost:80/health",
"interval": "10s"
}
}
}

Verify your setup

Check that WAN federation is working correctly and services are replicating across datacenters:

consul members -wan
consul catalog services
consul acl replication-status
consul operator raft list-peers

Test cross-datacenter service discovery:

dig @127.0.0.1 -p 8600 web.service.dc2.consul
consul catalog services -datacenter=dc2

Monitor health checks and replication status:

consul monitor -log-level=INFO
curl -k https://127.0.0.1:8501/v1/health/state/any

Common issues

SymptomCauseFix

WAN join failsFirewall blocking portsEnsure ports 8302 TCP/UDP are open between datacenters
ACL replication not workingMissing or invalid replication tokenCheck token permissions with consul acl token read -id TOKEN
Service discovery fails across DCDNS configuration incorrectVerify DNS forwarding to port 8600 and check service registration
TLS certificate errorsHostname verification failingEnsure certificates match server hostnames and CA is properly distributed
Raft leader election issuesNetwork partitions or clock driftCheck NTP synchronization and network connectivity between nodes
High memory usageLarge number of services/nodesTune performance.raft_multiplier and enable metrics monitoring

Next steps

Monitor Consul with Prometheus and Grafana for comprehensive observability

Configure Consul Connect service mesh with Envoy for secure microservices communication

Implement Consul backup and disaster recovery for production resilience

Configure Consul mesh gateways for cross-datacenter communication

Setup Consul intentions and service segmentation

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. Our managed platform covers monitoring, backups and 24/7 response by default.

---

## Contact

Binadit B.V., Seinhuiswachter 2, 3034 KH Rotterdam, Netherlands
Email: contact@binadit.com · Phone: +31 10 477 5362
KvK 80923216 · VAT NL861852990B01

Content may be cited by AI assistants with canonical URL attribution.