How to debug memory leaks in production without a zero downtime migration going sideways

What you will achieve and why it matters

A slow memory leak is rarely an emergency on day one. It becomes one three weeks later, when a process that used to run fine at 400MB is now getting OOM-killed every 36 hours and nobody remembers which deploy started it. This guide walks through finding the source of a leak in a running production service, confirming the fix in isolation, and shipping it through a zero downtime migration so you are not trading a memory problem for an availability problem.

By the end you will have a repeatable process: capture evidence from the live process, reproduce the leak locally, patch it, and roll the fix out behind a process that keeps serving traffic the entire time.

Prerequisites and assumptions

This guide assumes:

• You have shell access to the affected server or container host, with permission to attach profilers to a running process.
• The service is running behind a load balancer or reverse proxy (Nginx, HAProxy, or a cloud load balancer), which is what makes a zero downtime migration possible in the first place.
• You have at least basic metrics already: CPU, memory (RSS), and request latency, scraped at intervals of 30 seconds or less. If you do not, set up node_exporter or your platform equivalent before continuing. You cannot debug a trend you cannot see.
• You can run a second instance of the service alongside the current one, even temporarily, for canary testing.

The examples below use Node.js and a generic Linux/systemd setup, but the method applies the same way to Ruby, Python, Java, or Go services. The tools differ, the approach does not.

Step by step: finding, fixing, and shipping the leak

Step 1: Confirm it is actually a leak

Before chasing a leak, rule out normal behavior. A process that climbs in memory and then plateaus is probably just warming caches. A leak keeps climbing, indefinitely, correlated with uptime rather than traffic.

# Watch RSS over time for a specific PIDnwatch -n 5 'ps -o pid,rss,etime,cmd -p 1234'

Plot this against your existing metrics. If RSS grows roughly linearly over hours or days regardless of traffic volume, and only resets on restart, you have a leak. If it grows with traffic and plateaus, it is probably cache sizing, which is a different fix.

Step 2: Capture evidence from the live process

Do not restart the process before capturing data. That is the single most common mistake here, and it throws away the evidence you need.

For Node.js, trigger a heap snapshot without stopping the process:

# Requires --inspect or a signal handler wired up in advancenkill -USR2 1234   # if you have a SIGUSR2 handler writing heap snapshotsn# or, with --inspect already enabled:nnode --inspect=127.0.0.1:9229 server.jsn# then connect Chrome DevTools remotely via SSH tunnel:nssh -L 9229:127.0.0.1:9229 user@server

For Java, use jmap against the running PID:

jmap -dump:live,format=b,file=heap.bin 1234

For Python, py-spy gives you a profile without touching the process state:

py-spy dump --pid 1234npy-spy record -o profile.svg --pid 1234 --duration 60

Pull two snapshots, 30 to 60 minutes apart, under normal load. The diff between them is where the leak lives, not the absolute numbers.

Step 3: Diagnose the diff

Load both heap snapshots into Chrome DevTools (Memory tab, Load) and use the comparison view. Look for object counts that grow between snapshots without a corresponding drop. Common culprits worth checking first:

• Event listeners never removed. Every new WebSocket connection or request handler that adds a listener without a matching removeListener call.
• Unbounded caches or Maps. An in-memory cache with no TTL and no max size, especially one keyed by user ID or request ID.
• Closures holding references. A callback that captures a large object in its closure scope and outlives the request that created it.
• Connection pools misconfigured to grow without a ceiling, common with database drivers under retry logic.

For Java heaps, run the dump through Eclipse MAT and use "Leak Suspects" reporting. For Python, compare tracemalloc snapshots:

import tracemallocntracemalloc.start()n# ... later ...nsnapshot1 = tracemalloc.take_snapshot()n# ... wait ...nsnapshot2 = tracemalloc.take_snapshot()ntop_stats = snapshot2.compare_to(snapshot1, 'lineno')nfor stat in top_stats[:10]:n    print(stat)

Step 4: Reproduce locally under load

Once you have a suspect, confirm it with a load test, not a guess. Use k6 or autocannon to simulate sustained traffic locally while watching RSS:

autocannon -c 50 -d 300 http://localhost:3000/your-endpointn# in a second terminalnwhile true; do ps -o rss= -p $(pgrep -f server.js); sleep 10; done

If memory climbs locally under the same pattern you saw in production, you have confirmed the leak and have a repeatable test you can use to verify the fix.

Step 5: Patch and verify in isolation

Apply the fix: add the TTL to the cache, remove the listener, bound the pool. Re-run the same load test. Memory should plateau within a few minutes of steady-state traffic, not keep climbing.

Do not stop here and ship straight to production. The fix working locally tells you the code is correct. It does not tell you the rollout is safe.

Step 6: Roll out through a zero downtime migration

This is where most teams get sloppy, because the instinct after finding a leak is to restart everything immediately. A better approach treats the fix like any other production change: a controlled, zero downtime migration from the old process to the new one.

If you are running behind a load balancer with multiple instances:

# Deploy the fix to one instance firstnansible-playbook -i inventory deploy.yml --limit app-node-1nn# Drain connections gracefully before replacing the old processncurl -X POST http://localhost:8080/-/drain   # app-specific drain endpointn# or, for Nginx upstream:n# mark the backend as 'down' in upstream config, reload, then stop the old processnnginx -s reload

With systemd and a single-instance service, use a blue-green pattern on the same host rather than a hard restart:

# Start the patched version on a new portnPORT=3001 systemctl start app-patched.servicenn# Health check before switching trafficncurl -f http://localhost:3001/health || echo "not ready"nn# Flip the load balancer or reverse proxy upstreamnsed -i 's/3000/3001/' /etc/nginx/conf.d/upstream.confnnginx -s reloadnn# Only now stop the old processnsystemctl stop app.service

The sequence matters: start new, verify healthy, switch traffic, then stop old. Reversing any of those steps is how a memory leak fix turns into an outage. For a deeper walkthrough of this pattern across database and application layers together, see our 6-phase zero downtime migration playbook.

If you are running Kubernetes, the same principle applies through a rolling update with a proper readiness probe:

kubectl set image deployment/app app=registry/app:patchednkubectl rollout status deployment/app

Make sure readinessProbe is actually checking application health, not just that the port is open. A pod that accepts connections but has not finished warming up will cause dropped requests during the rollout, which defeats the purpose.

Verification: how to confirm it worked

Do not close the ticket the moment the deploy finishes. Confirm the fix held under real traffic for at least one full business cycle, ideally 24 to 48 hours.

• RSS trend: graph memory per instance over 48 hours. It should plateau, not just rise more slowly.
• Restart count: check your process manager or orchestrator for OOM kills. journalctl -u app.service | grep -i "oom" or, on Kubernetes, kubectl get events --field-selector reason=OOMKilling.
• Request error rate during rollout: your 5xx rate during the migration window should not spike above baseline. If it does, your drain or readiness logic has a gap.
• GC pause times (for GC'd languages): a shrinking live heap usually shows up as shorter, more predictable GC pauses. Watch p95 GC pause time before and after.

A concrete target worth setting: RSS variance over a 48-hour window should stay within 10-15% of its post-restart baseline. Anything climbing steadily beyond that means the leak was only partially fixed, or there is a second one.

Common pitfalls to avoid

• Restarting before capturing a heap dump. This destroys the evidence and guarantees you are debugging blind next time.
• Fixing the symptom with a scheduled restart. A cron job that bounces the process nightly hides the leak instead of fixing it, and it will eventually collide with a traffic spike.
• Skipping the drain step during rollout. Killing the old process before in-flight requests finish causes real user-facing errors, even if your total downtime is zero seconds.
• Assuming one fix solves it. Memory leaks often stack: a cache without a TTL and a connection pool without a ceiling can both be present in the same service. Re-profile after the first fix before declaring victory.

Next steps and related reading

Once the leak is resolved, it is worth checking whether the underlying rollout mechanism needs hardening more broadly, not just for memory fixes. If your deploys regularly involve this kind of careful choreography, you may be better served by a proper GitOps pipeline, see our guide to GitOps workflows for infrastructure management, which removes the manual sequencing risk entirely.

If the leak was database-connection related, it is also worth reviewing how your migrations and schema changes are rolled out, since the same drain-then-switch principle applies at the data layer too. See how zero downtime database migrations work for that pattern specifically.

Teams running on managed infrastructure for SaaS platforms often bake this kind of canary and drain logic into their standard deploy pipeline once, rather than re-deriving it under pressure every time a leak shows up. That is usually the better long-term investment: fix the process, not just the bug.

Bringing it together

Debugging a memory leak is a two-part problem: finding the root cause in the code, and shipping the fix without trading one incident for another. The profiling steps above will get you to the cause. The deploy sequence will get you there without downtime. Treat both as equally important, and you will stop seeing the same leak pattern show up again three months later under a different symptom.

Need this running in production without building it yourself? See our managed infrastructure services or schedule a call.