How Docker networking broke checkout under load: a container ecommerce infrastructure case study

The situation: a growing marketplace runs into container networking limits

The platform was a multi-vendor marketplace built on a PHP monolith, containerized about a year before we got involved. Roughly 40,000 daily active users, peak traffic around 900 requests per second during evening hours, and a catalog large enough that search and checkout both hit the database hard.

The engineering team had done the container migration themselves: application, Redis, and a background worker queue all running as Docker services on a single host, orchestrated with Docker Compose, sitting behind a managed load balancer. It worked. Deploys got faster. Local development matched production more closely. Nobody was unhappy with the decision to containerize.

The problem showed up six months later, and only under load. Checkout latency was fine most of the day and then, during traffic spikes, p95 response times on the checkout endpoint would climb from around 280ms to over 2.1 seconds. Support tickets mentioned "the site freezing" during flash sales. The team had already added more application containers, assuming it was a CPU or memory ceiling. It wasn't. Adding containers made things marginally worse.

This is a common pattern: teams scale the thing they can see (container count, CPU graphs) while the actual bottleneck sits in the network layer connecting those containers, where default tooling gives almost no visibility.

What we found during the audit

We started with the assumption that this was a database problem, because checkout latency almost always looks like a database problem. It wasn't, or at least not primarily. Query times on Postgres were stable, averaging 8-14ms even during the spike windows.

The actual issue was in how Docker's default bridge network was handling inter-container traffic at scale. Three specific findings:

• Every packet between containers was traversing userland proxying. The Compose setup used the default bridge driver without any tuning. Under low traffic this is invisible. Under 900 req/s with each request making 3-4 internal calls (app to Redis, app to Postgres, app to a search service), the overhead compounded. We measured an average of 1.8ms added latency per hop through the default bridge, which sounds small until you multiply it by every internal call on every request.
• Connection tracking (conntrack) was hitting its table limit. The host's nf_conntrack_max was left at the kernel default of 65,536 entries. With containers opening and closing short-lived connections to Redis and Postgres rapidly, the conntrack table filled up during spikes. Once full, new connections got dropped or delayed, and the kernel started logging "table full, dropping packet" without anyone noticing because syslog wasn't being shipped anywhere useful.
• DNS resolution inside the Docker network was adding unpredictable latency. Docker's embedded DNS server (127.0.0.11) was resolving service names for every new connection rather than the app caching resolved addresses. Under normal load this is a rounding error. Under spike load, with connection churn, DNS lookups were queuing behind each other. We've written about this resolution path in more depth in how DNS resolution works under the hood, and the same fundamentals apply inside a container network, not just at the edge.

None of these three things alone would have caused a 2-second checkout. Together, under concurrent load, they created exactly the kind of compounding latency that's invisible in a staging environment with ten test users and very visible with nine hundred real ones.

The approach we took and why

We ruled out two tempting options early.

The first was "just move to Kubernetes." It would not have fixed anything here. The problems were in networking fundamentals (bridge driver behavior, conntrack limits, DNS caching), not in orchestration. Kubernetes has its own version of these same issues (CNI plugin choice, kube-proxy mode, CoreDNS caching) and swapping orchestrators without understanding the root cause would have just relocated the problem.

The second was rewriting the application to reduce internal network calls. Also unnecessary. The call pattern (app to Redis, app to Postgres, app to search) is normal and not excessive. The network layer needed to handle that pattern efficiently, not the other way around.

Our approach was to fix the network path itself, in three layers, matched to the audit findings: kernel-level tuning for conntrack, a driver change for inter-container traffic, and application-level DNS caching. This kept the change surface small and testable, and it meant zero application code changes, which mattered because the team's next planned release was already in QA and we didn't want to block it.

We also treated this as a chance to move the checkout path toward genuine high availability without a rewrite, since the same single-host container setup was also a single point of failure. That became phase two of the work.

Implementation details

1. Conntrack and kernel network tuning

We raised the connection tracking table and tuned the timeout for established connections, since most of the churn was short-lived Redis and Postgres connections that didn't need to linger in the table:

net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 4096

These went into /etc/sysctl.d/99-docker-network.conf and were applied with sysctl --system. We also added conntrack table utilization as a monitored metric going forward, since "table full" is exactly the kind of failure that produces no obvious symptom in application logs.

2. Replacing the default bridge with a tuned macvlan setup for internal services

For inter-container east-west traffic (app to Redis, app to Postgres proxy), we moved from Docker's default bridge to a dedicated user-defined bridge network with jumbo frame support enabled and ICC (inter-container communication) explicitly configured, rather than relying on default NAT-based routing for every hop:

docker network create \
  --driver bridge \
  --opt com.docker.network.bridge.enable_icc=true \
  --opt com.docker.network.driver.mtu=9000 \
  --subnet 172.28.0.0/16 \
  internal-services

Services that only talked to each other internally (app containers, Redis, the search sidecar) were attached to this network instead of the default bridge, keeping their traffic off the NAT path Docker uses for the default network. The load balancer facing traffic still terminated on a separate public-facing network, so we weren't changing the external attack surface, just the internal routing.

3. Static internal DNS with local caching

Rather than resolving service names through Docker's embedded DNS on every new connection, we added a lightweight local DNS cache (dnsmasq running as a sidecar) in front of the Docker embedded resolver, with short but non-zero TTLs:

# dnsmasq.conf
no-resolv
server=127.0.0.11
cache-size=1000
local-ttl=10
neg-ttl=5

A 10-second local TTL was enough to absorb the connection churn during spikes without causing stale resolution problems when containers were replaced during deploys. We tested failover behavior specifically to confirm that a container restart was still picked up within one TTL window.

4. Application-side connection pooling

The last piece was at the application layer, but it was a configuration change, not a rewrite: enabling persistent connection pooling to Postgres via PgBouncer, which reduced the connection churn that was stressing conntrack in the first place.

[databases]
marketplace = host=postgres-primary port=5432 dbname=marketplace

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 50

This mattered more than it might look. Every new short-lived connection is a new conntrack entry and a new DNS resolution. Pooling connections at the application boundary reduced both problems simultaneously, which is why we sequenced it after the network fixes rather than before: it made the underlying fixes more effective, but it wasn't a substitute for them.

Results with real numbers

We rolled this out over two maintenance windows, three weeks apart, to isolate the impact of each layer.

The most telling number is the gap between peak and normal load before the fix: 280ms normal, 2,140ms peak, almost an 8x difference under the same application code and same database. After the fix, that gap closed to roughly 1.6x, which is what you'd expect from legitimate resource contention rather than a network layer choking on connection churn.

The conntrack utilization number explains why the team's earlier fix (adding more app containers) made things worse. More containers meant more concurrent connections to Redis and Postgres, which meant the conntrack table filled up faster, which meant more dropped packets. They were scaling the exact resource that was already the bottleneck.

Infrastructure cost didn't change meaningfully. This wasn't a "add servers" fix; it was a "use the network correctly" fix. The only new cost was the monitoring for conntrack utilization and DNS cache hit rate, which added about €40/month in observability overhead across the fleet.

What we'd do differently next time

A few things we'd change if we ran this again from day one.

We would check conntrack limits before looking at anything else. It's a five-minute check (sysctl net.netfilter.nf_conntrack_count versus nf_conntrack_max) and it should be step one for any "intermittent slowness under load" ticket involving containers, ahead of database profiling. We spent longer than we'd like ruling out the database first, mostly because that's where the client's own monitoring pointed.

We would push harder, earlier, on separating the single-host container topology from the network fix. Fixing conntrack and DNS caching solved the immediate latency problem, but the underlying architecture still had a single Docker host as a single point of failure for the entire checkout path. We did eventually move this to a proper multi-host setup with the app tier load balanced across two hosts, but that happened as a phase two project rather than being bundled into the initial fix. In hindsight, given how closely the two problems were related, we'd scope both from the start rather than treating high availability as a separate conversation.

We would also set up conntrack and DNS resolution metrics as day-one monitoring for any container-based ecommerce infrastructure, rather than something added after an incident. These are cheap to monitor and expensive to debug blind. The default Docker and Compose setup gives you almost no visibility into either, which is exactly why this kind of problem tends to surface in production during a sale event rather than in a load test three weeks earlier.

Closing thoughts

Docker networking works well by default for development and for low-to-moderate production traffic. It stops working well by default once you have real concurrency: enough simultaneous connections to fill a conntrack table, enough churn to make DNS resolution matter, enough internal hops to make bridge overhead visible. None of that shows up in a staging environment with a handful of test requests, which is exactly why it tends to surface for the first time during a traffic spike that actually matters to the business.

The fix is rarely "add more containers" or "move to Kubernetes." It's usually tuning the kernel-level and network-level defaults that nobody set intentionally in the first place, and pairing that with connection pooling so the application isn't generating more network churn than it needs to.

Facing a similar challenge? Tell us about your setup and we will outline an approach.