How to stabilize a system while it is actively failing

What you will achieve and why it matters

This guide walks through the sequence for stabilizing a production system that is degrading right now, before you know the root cause. The goal is not to fix the underlying bug during the incident. The goal is to reduce user impact, buy diagnostic time, and avoid the two most common mistakes teams make under pressure: changing too many things at once, and diagnosing before stabilizing.

This applies whether you are running a single application server or a full high availability infrastructure setup with load balancers and read replicas. The steps below come from real incident response patterns we use across managed infrastructure for SaaS platforms and e-commerce stores.

Prerequisites and assumptions

Before you can stabilize anything, you need visibility and control. This guide assumes:

• You have shell access to the affected hosts (SSH, or a bastion into your cloud provider's console)
• You have basic monitoring in place: at minimum, CPU, memory, disk I/O, and application error rates. Prometheus, Datadog, or even htop and application logs count
• You have a load balancer or reverse proxy in front of your application (Nginx, HAProxy, or a cloud load balancer)
• You know how to roll back your last deployment
• You have at least one other person who can act as a second pair of eyes, even if that is just a Slack thread

If none of this exists yet, the real prerequisite is building it before the next incident. That is a separate project, covered in the next steps section.

Step by step: stabilizing the system

Step 1: freeze changes

The first action is not technical. It is a decision: stop deploying, stop running migrations, stop anyone from touching config. Post in your team channel:

INCIDENT: elevated error rate on checkout-api since 14:32 UTC.
No deploys, no config changes, no manual DB edits until stabilized.
Incident channel: #inc-2024-checkout

This single step prevents the most common failure mode in live incidents: a well-meaning engineer pushes a "quick fix" that interacts badly with whatever is already going wrong, and now you have two problems layered on top of each other with no clean way to tell them apart.

Step 2: identify what changed recently

Most active failures trace back to a change in the last 24 hours: a deploy, a config edit, a traffic spike, or an upstream dependency update. Check in this order:

# Recent deploys
git log --since="24 hours ago" --oneline

# Recent config changes (if using version-controlled infra)
git -C /etc/nginx log --since="24 hours ago"

# System-level changes
last -x | head -20
cat /var/log/dpkg.log | grep "$(date +%Y-%m-%d)"

If a deploy or config change lines up with the onset of the problem, rolling it back is your fastest stabilization path. Do this before deep diagnosis, not after.

# Example: rolling back a bad deploy with a tagged release process
git checkout tags/v2.14.1
./deploy.sh production

# Or with a container-based setup
kubectl rollout undo deployment/checkout-api

Step 3: shed load before you shed correctness

If there is no clear recent change, or the rollback does not resolve it, the next lever is reducing load on the failing component. This is where you trade some functionality for stability, temporarily.

Common load-shedding techniques, roughly in order of how disruptive they are:

• Rate limit at the edge. Add a rate limit in Nginx or your load balancer to cap requests per IP or per token
• Disable non-critical endpoints. Turn off recommendation engines, analytics beacons, or background report generation
• Enable a maintenance page for non-essential routes while keeping checkout or login alive
• Scale horizontally if the bottleneck is compute, not a shared resource like a database

Example: adding an emergency rate limit in Nginx:

limit_req_zone $binary_remote_addr zone=emergency:10m rate=5r/s;

server {
    location /api/ {
        limit_req zone=emergency burst=10 nodelay;
        limit_req_status 503;
        proxy_pass http://backend;
    }
}

Reload without dropping connections:

nginx -t && nginx -s reload

Step 4: protect the database first

In most stacks we manage, the database is the resource that fails last and recovers slowest. If connections are maxing out or queries are queueing, address this before anything else, because a saturated database will keep the rest of the system unstable no matter what you do upstream.

# Check active connections vs max on PostgreSQL
psql -c "SELECT count(*) FROM pg_stat_activity;"
psql -c "SHOW max_connections;"

# Find long-running queries
psql -c "SELECT pid, now() - query_start AS duration, query
         FROM pg_stat_activity
         WHERE state = 'active'
         ORDER BY duration DESC LIMIT 10;"

If a handful of runaway queries are holding locks, killing them is often the fastest stabilization move available:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid = 14832;

If connection exhaustion is the pattern, a connection pooler like PgBouncer absorbs this without touching application code. If you do not have one running, this incident is a strong signal to add it once things settle.

Step 5: fail over, do not fight in place

If a single node is the problem, whether from disk pressure, a memory leak, or a bad kernel state, it is usually faster to remove it from rotation than to debug it live. This is the core value of high availability infrastructure: you have somewhere to fail over to.

# Remove a node from the load balancer pool (HAProxy example)
echo "disable server backend/web03" | socat stdio /var/run/haproxy.sock

# Or drain it gracefully in Kubernetes
kubectl cordon node-3
kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data

Once removed, that node becomes a forensic artifact, not a liability. You can debug it at your own pace without affecting live traffic.

Step 6: communicate a stable status, not a guess

Once error rates flatten and the change freeze is holding, post a status update with what you know and do not know:

STATUS 15:10 UTC: error rate back to baseline (0.2%) after rolling back
deploy v2.14.2 and draining node-3. Root cause under investigation.
Next update in 30 min or on change.

Resist the urge to declare the incident closed the moment metrics look normal. Stability under reduced load or a smaller traffic window does not confirm the fix.

Verification: how to confirm the system is actually stable

Do not rely on a single green dashboard. Confirm stability across at least three signals held over a meaningful window, typically 30 to 60 minutes at representative traffic:

• Error rate: back to baseline, not just trending down. Check 5xx rate over 5-minute buckets, not instant values
• Latency percentiles: p50, p95, and p99 all recovered, not just the average. A recovered average can hide a p99 that is still five times normal
• Resource headroom: CPU, memory, and connection pools have margin, not just "not maxed"
• Queue depth: if you use background jobs, confirm the backlog is draining, not just growing more slowly

Useful spot checks:

# Tail error rate from Nginx access logs
tail -n 5000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

# Watch active connections in real time
watch -n 2 "ss -s"

# Confirm database connection headroom
psql -c "SELECT count(*), max_conn FROM pg_stat_activity, (SELECT setting::int AS max_conn FROM pg_settings WHERE name='max_connections') s GROUP BY max_conn;"

If you have historical dashboards, compare the current traffic-normalized error rate to the same time last week, not just to five minutes ago. A quiet period after load shedding can look deceptively healthy.

Common pitfalls to avoid

• Fixing root cause mid-incident. Diagnosing while the system is unstable adds risk and rarely speeds up recovery. Stabilize first, root-cause after
• Rolling back and rolling forward in the same window. Give the rollback time to prove itself before trying anything else
• Declaring victory too early. A dashboard that looks calm for five minutes is not the same as a stable system under real traffic
• Skipping the change freeze. Uncoordinated fixes during an incident are one of the most common causes of a second, unrelated outage layered on top of the first
• Not documenting what you tried. If the same instability recurs, you want a timestamped record of every action taken, not a reconstructed memory two days later

Next steps and related reading

Once the immediate instability is resolved, the real work is making sure the pattern does not repeat. A few natural follow-ups:

• Run a proper post-incident review while details are fresh. See post-incident reviews that actually improve things for a format that produces real action items instead of vague summaries
• If load shedding was your main lever, it is worth reviewing whether your architecture has enough redundancy built in. Best practices for horizontal scaling in high availability infrastructure covers how to build that headroom in advance
• If a deploy caused the instability, tightening your rollback process pays for itself the next time this happens. Our 6-phase zero-downtime migration playbook outlines how to structure changes so rollback is always a single command
• If database exhaustion was the bottleneck, look at connection pooling and query-level monitoring before the next spike

Teams that handle this well tend to invest ahead of time in the boring parts: alerting thresholds tuned to catch drift early, a documented rollback procedure, and a database that has headroom before it is needed. That groundwork is most of what separates a five-minute blip from a multi-hour incident.

Getting this right consistently

Stabilizing an actively failing system is a skill that improves with repetition, but most teams only get the practice during real incidents, which is an expensive way to learn. This is also where dedicated infrastructure management services earn their keep: having engineers who have run this exact sequence dozens of times, on dozens of different stacks, means the freeze, the rollback, and the load-shedding decisions happen in minutes instead of being figured out live. A managed infrastructure partner brings that muscle memory as a standing capability rather than something your team assembles under pressure.

Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.