From 4.2s to 380ms: debugging latency in a high availability infrastructure setup

The situation: a slow, gradual degradation with no obvious cause

The platform was a B2B SaaS product used for scheduling and resource planning, with around 40,000 active users across several hundred customer accounts. Traffic was steady during business hours in European time zones, with predictable spikes on Monday mornings.

The engineering team had not shipped anything unusual. No major feature releases, no big schema changes, no new integrations. But over roughly six months, p95 API response times had drifted from around 400ms to over 4.2 seconds. Support tickets mentioned the dashboard 'feeling laggy' long before anyone had hard numbers to point to.

This is a common pattern in distributed systems: nothing breaks outright, so nothing triggers an incident. Latency just creeps until it becomes a business problem. In this case, the business impact was concrete. Trial-to-paid conversion had dropped 11% over the same period, and the sales team was fielding questions about performance during renewal conversations.

The team had already tried the obvious fixes: bumping instance sizes, adding a read replica, restarting services on a schedule. None of it moved the needle for long. That is usually a sign the bottleneck is architectural, not capacity related, which is why they brought us in to audit the stack before committing to a bigger infrastructure overhaul.

What we found during the audit

We spent the first week purely on measurement, not fixing anything. Changing things before you understand the system just adds noise to your data. We instrumented request tracing across the API gateway, application servers, database, and cache layer, and let it run for five business days under normal load.

Five distinct issues emerged, and none of them alone explained the full 4.2 seconds. That is often how latency problems in high availability infrastructure actually work: it is rarely one dramatic bottleneck, it is several smaller ones stacking on top of each other.

• N+1 queries in a hot path. A dashboard endpoint that loaded a customer's active projects was making one query for the project list, then a separate query per project to fetch its status. At 40 projects per account, that was 41 round trips to the database for a single page load.
• Connection pool exhaustion under load. The application was running with a PostgreSQL connection pool of 20 per app server across 8 servers, but the database's max_connections was set to 100. During peak hours, requests were queueing for a connection slot before they ever reached a query.
• Cache invalidation was too aggressive. A Redis cache in front of the project status data had a 30 second TTL, which sounds reasonable, but a background job was also flushing entire cache namespaces on any write, including unrelated writes. Cache hit ratio was measured at 34%, far lower than the team assumed.
• Cross-AZ chatter between services. The application tier and the cache cluster were not pinned to the same availability zone. Roughly 40% of Redis calls were crossing zones, adding 2 to 4ms per call that, at high call volumes, added up to hundreds of milliseconds per request.
• A synchronous third-party call in the request path. A usage-tracking webhook to an external analytics provider was being called synchronously during the request cycle. When that provider was slow, and it averaged 800ms to 1.5s on a bad day, every request downstream waited for it.

None of these were exotic. They were the kind of accumulated technical debt you find in a system that has grown organically for a few years without a dedicated performance review. That is worth naming clearly, because a lot of teams assume a latency problem this size must mean rearchitecting everything. It usually does not.

The approach we took and why

We prioritized fixes by expected impact versus risk, not by ease of implementation. It is tempting to fix the easiest thing first, but that is not always the thing costing you the most milliseconds.

We ranked the five issues like this:

1. Synchronous third-party call: highest impact, low risk to fix (move to async queue)
2. N+1 queries: high impact, low risk (query batching, well-understood pattern)
3. Connection pool exhaustion: high impact, medium risk (requires careful tuning to avoid overloading the database)
4. Cache invalidation strategy: medium impact, medium risk (requires understanding all write paths)
5. Cross-AZ placement: medium impact, low risk once the migration window was scheduled

We deliberately did not touch instance sizing again. The team had already scaled vertically twice without improvement, which was itself a useful data point: if adding CPU and memory does not help, the bottleneck is not compute, it is architecture or I/O. This is a distinction we cover in more depth in our piece on when high availability infrastructure becomes a bottleneck: past a certain point, more servers just mean more machines waiting on the same slow dependency.

We also made a call early on to fix things incrementally and measure after each change, rather than bundling everything into one release. Distributed systems have enough moving parts that a single combined deploy makes it nearly impossible to know which change actually helped, or whether one change quietly made something else worse.

Implementation details

Moving the analytics call off the request path

The synchronous webhook call was replaced with a message queue (Redis-backed, using the existing infrastructure rather than adding a new dependency). The request handler now pushes a lightweight event onto the queue and returns immediately. A separate worker process consumes the queue and handles delivery to the analytics provider, with retry logic and a dead-letter queue for failures.

// before: synchronous call blocking the response
await analyticsClient.track(event); // 800ms-1.5s on slow days
return response;

// after: fire-and-forget via queue
await queue.push('analytics.track', event); // ~2ms
return response;

This single change removed 800ms to 1.5 seconds from the median request in the affected endpoints. It also meant that if the analytics provider had an outage, which happened twice during our engagement, it no longer affected the user-facing application at all.

Fixing the N+1 pattern

The project status lookup was rewritten to use a single query with a join, rather than a loop of per-project queries.

-- before: 1 query for projects, then N queries for status
SELECT * FROM projects WHERE account_id = $1;
-- then, per project:
SELECT * FROM project_status WHERE project_id = $1;

-- after: one query
SELECT p.*, s.status, s.updated_at
FROM projects p
JOIN project_status s ON s.project_id = p.id
WHERE p.account_id = $1;

For an account with 40 active projects, this took the endpoint from 41 database round trips to 1. Query time for that endpoint dropped from an average of 620ms to 45ms.

Retuning the connection pool

We reduced per-server pool size from 20 to 12 (8 servers x 12 = 96, leaving headroom under the 100 connection limit) and added PgBouncer in transaction pooling mode in front of PostgreSQL. This let us actually raise the effective connection ceiling the application could use, since PgBouncer multiplexes many client connections onto a smaller number of real database connections.

[databases]
app_db = host=127.0.0.1 port=5432 dbname=app_production

[pgbouncer]
pool_mode = transaction
max_client_conn = 500
default_pool_size = 25
reserve_pool_size = 5
reserve_pool_timeout = 3

Connection wait time, which had been averaging 180ms during peak hours, dropped to under 5ms.

Rebuilding the cache invalidation logic

Rather than flushing entire namespaces on write, we moved to key-level invalidation tied to the specific project being updated. TTL stayed at 30 seconds as a safety net, but the primary invalidation mechanism became explicit and targeted.

Cache hit ratio moved from 34% to 91% within the first week after deployment. This is the kind of change that looks small in a code diff but has an outsized effect on database load, because every cache miss becomes a query.

Fixing AZ placement

We moved the Redis cluster and pinned application servers to the same availability zone, using zone-aware routing at the load balancer level so requests preferentially stayed local. This was the most operationally sensitive change, since it touched the load balancing and failover configuration, so we scheduled it during a low-traffic window and kept the previous configuration ready to roll back for 48 hours.

Results: before and after

The cost reduction is worth noting specifically. Once the actual bottlenecks were fixed, the team no longer needed the oversized instances they had added earlier in an attempt to outrun the latency problem. Right-sizing after the fix, rather than before it, meant the infrastructure spend went down while performance went up.

Trial-to-paid conversion recovered over the following quarter, though we are careful not to claim a direct causal number there. Latency was one factor among several, and conversion metrics have too many inputs to attribute cleanly to an infrastructure change. What we can say with confidence is the technical numbers above, measured with the same tracing setup before and after.

What we'd do differently next time

A few things we would change in hindsight.

We should have set up distributed tracing before the audit even started, as a standing part of the client's monitoring, rather than as a temporary measurement tool for the engagement. The team's existing monitoring showed server-level metrics (CPU, memory, request count) but nothing that connected a slow user-facing request to the specific downstream call causing it. That gap is what let the problem compound silently for six months. We have written before about how monitoring can give you a false sense of security when it only tracks infrastructure health and not request-level behavior, and this engagement was a clean example of exactly that gap.

We also would have pushed harder, earlier, to separate the AZ migration from the other changes. It carried more operational risk than the others, and while it went smoothly, bundling the planning for it alongside four other workstreams stretched the team thinner than it needed to be during that week.

Finally, we would have proposed load testing the fixes against production-like traffic patterns before the final rollout, rather than relying on staged rollout and close monitoring. It worked out, but synthetic load testing ahead of time would have caught the connection pool sizing question with more confidence and less babysitting during the rollout window.

Facing something similar

Latency problems like this rarely have one cause, and they rarely show up in a single dashboard metric. They show up as a slow accumulation of small inefficiencies that compound under real traffic, in ways that are hard to see until you trace individual requests end to end. If you want the methodology in more detail, we have written a broader walkthrough on how to trace performance bottlenecks end-to-end.

Facing a similar challenge? Tell us about your setup and we will outline an approach.