Benchmarking API reliability under load: when zero downtime migration becomes critical

The question: at what load do APIs actually break?

Most engineering teams discover their API's breaking point during peak traffic, not during testing. The business impact is immediate: failed requests mean lost transactions, frustrated users, and revenue walking out the door.

We wanted real numbers. At what concurrent request levels do different API configurations start failing? How do error rates climb? When does response time become unacceptable?

To find out, we benchmarked the same REST API across different infrastructure setups, measuring exactly when reliability degrades and by how much.

Methodology: controlled load testing across infrastructure patterns

We built a simple e-commerce API handling product lookups, user authentication, and order processing. Three endpoints: GET /products, POST /auth/login, and POST /orders.

Test environment specifications:

• Application: Node.js 18.17.0 with Express 4.18.2
• Database: PostgreSQL 15.3 with 2GB RAM allocation
• Server: 4 CPU cores, 8GB RAM, NVMe storage
• Network: 1Gbps connection, 5ms baseline latency
• Cache: Redis 7.0.11 for session storage

Infrastructure configurations tested:

1. Single server: all components on one machine
2. Database separation: app server + dedicated database server
3. Load balanced: 2 app servers + shared database + Redis cluster
4. Auto-scaling: 2-6 app servers with horizontal scaling triggers

Load profile:

We used Artillery.io to generate realistic traffic patterns. Starting at 10 concurrent users, we increased load every 2 minutes: 10, 25, 50, 100, 250, 500, 750, 1000, 1500, 2000 concurrent users.

Each user session included: browsing products (60%), logging in (20%), placing orders (15%), and admin actions (5%). This mirrors real e-commerce traffic distribution.

We measured response time (p50, p95, p99), error rate, CPU utilization, memory usage, and database connection pool status every 30 seconds.

Results: reliability degrades predictably, but breaking points vary dramatically

The numbers reveal clear patterns in how APIs fail under load.

Single server configuration:

Load balanced configuration:

Auto-scaling configuration:

The database became the bottleneck in every configuration. Connection pool exhaustion started affecting response times before CPU or memory limits were reached.

Analysis: what these numbers mean for production systems

The single server configuration failed catastrophically at 500 concurrent users. Response times jumped from acceptable (178ms p50) to unusable (1,234ms p50) with nearly 25% error rates.

Load balancing pushed the breaking point to 1,500 concurrent users, but the degradation pattern remained similar. Once database connections saturated, error rates climbed exponentially.

Auto-scaling provided the most graceful degradation. Even at 2,000 concurrent users, error rates stayed under 4% and response times remained manageable.

The critical insight: reliability doesn't decline gradually. It cliff-dives once resource limits are exceeded. The database connection pool became exhausted before CPU reached 60% utilization in every test.

For business context: an e-commerce platform processing 500 concurrent users might handle 2,000-3,000 daily active users depending on usage patterns. The single server configuration would start failing during modest traffic spikes.

Teams planning zero downtime migration strategies need this data before peak seasons, not during them. Migration complexity increases significantly once you're already experiencing reliability issues.

Caveats and what we'd test differently

Our testing methodology had limitations that affect real-world applicability.

Database optimization was minimal. We used default PostgreSQL settings without connection pooling, read replicas, or query optimization. Production systems typically perform better than our baseline numbers.

Load pattern was synthetic. Real users don't generate perfectly distributed traffic. Actual breaking points might occur at lower concurrent user counts during traffic spikes or at higher counts during steady-state load.

Geographic distribution wasn't tested. All load originated from the same region. Global user bases introduce network latency variations that affect perceived performance differently.

Application complexity was limited. Our test API performed basic CRUD operations. Real applications with complex business logic, external API calls, or heavy computational tasks would show different performance characteristics.

Failure modes were incomplete. We focused on response time and error rates. Production systems also fail through memory leaks, disk space exhaustion, and cascading service dependencies.

For more comprehensive testing, we'd include database performance degradation patterns, network partition scenarios, and longer-duration tests to capture memory leak effects.

Takeaways: plan your zero downtime migration before you need it

Three key lessons from these benchmarks:

Resource exhaustion creates cliff-edge failures. Systems perform acceptably until they don't. There's typically a narrow band between "working fine" and "completely broken."

Database connections limit scaling more than CPU or memory. Every configuration hit database bottlenecks first. Connection pooling and read replicas should be architectural decisions, not performance optimizations you add later.

Infrastructure changes under load are risky. The gap between single server and auto-scaling capabilities is significant. Teams need zero downtime migration strategies planned before reliability becomes a daily concern.

The numbers show why timing matters for infrastructure decisions. Moving from single server to distributed architecture is straightforward during low-traffic periods but becomes complex once you're already experiencing reliability issues.

Want these kinds of numbers for your own stack? Request a performance audit.