Who this guide is for
Small engineering teams face a unique challenge: maintaining high availability infrastructure without the luxury of dedicated SRE teams or 24/7 operations staff. Whether you're running a SaaS platform with 50,000 users or managing e-commerce infrastructure that processes thousands of orders daily, your team needs sustainable on-call practices that prevent both downtime and engineer burnout.
This guide covers 12 practical approaches that help small teams maintain reliable systems while keeping on-call duties manageable. These practices work whether you have 3 engineers or 15, and they're designed to grow with your team.
12 practices for sustainable on-call
1. Define clear escalation boundaries
Every on-call engineer needs to know exactly when to wake up their manager or senior staff. Create specific criteria for escalation: customer-facing services down for more than 15 minutes, data corruption detected, or security incidents. This prevents junior engineers from struggling alone with critical issues and stops senior engineers from being woken up for routine alerts.
2. Build runbooks that actually work under pressure
Write runbooks as if you're explaining the fix to yourself at 3 AM after being woken up. Include the exact commands to run, what the expected output looks like, and when to give up and escalate. Test your runbooks by having team members who didn't write them follow the steps during non-emergency situations.
# Service restart procedure
# Expected time: 2-3 minutes
# Escalate if: service doesn't respond after 2 restart attempts
1. Check service status:
systemctl status webapp
2. If failed, restart:
sudo systemctl restart webapp
3. Verify recovery:
curl -f https://api.example.com/health
4. If still failing after 2 attempts, escalate immediately3. Implement alert fatigue protection
Too many alerts train your team to ignore notifications, which leads to missing real problems. Set up alert routing that sends different severity levels to different channels: critical alerts go to phones, warnings go to Slack, and informational alerts go to email or dedicated monitoring channels.
4. Use intelligent alert grouping
When your database goes down, you don't need 47 separate alerts about every service that depends on it. Configure your monitoring to group related alerts and suppress downstream notifications when upstream services fail. This turns a overwhelming flood of notifications into a single, actionable alert.
5. Establish on-call handoff rituals
Create a structured handoff process where the outgoing on-call person briefs their replacement on current system health, ongoing issues, and anything that needs attention. Schedule these handoffs for a specific time, not just "whenever convenient," so both engineers can plan their day around the transition.
6. Build automated recovery for common issues
Identify the problems your team fixes manually more than twice per month and automate them. Disk space cleanup, service restarts, and clearing stuck queues are perfect candidates. Your high availability infrastructure should handle routine failures without human intervention.
# Automated disk cleanup script
#!/bin/bash
DISK_USAGE=$(df /var/log | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 85 ]; then
find /var/log -name "*.log" -mtime +7 -delete
systemctl restart rsyslog
fi7. Create dedicated incident communication channels
Set up separate Slack channels or communication tools specifically for incident response. This keeps urgent technical discussion separate from general team chat and makes it easier to track resolution progress. Include relevant stakeholders like customer success or sales leadership when incidents affect customers.
8. Schedule regular on-call retrospectives
After significant incidents or at least monthly, review what happened during on-call periods. Focus on systemic improvements rather than individual blame: what tools would have helped, which runbooks need updating, and what monitoring gaps exist. These reviews help evolve your practices based on real experience.
9. Implement graceful degradation monitoring
Monitor not just whether services are up or down, but whether they're operating in degraded modes. Track response times, queue depths, and error rates that indicate your system is struggling before it completely fails. This gives on-call engineers early warning and time to take preventive action.
10. Use time-boxed investigation periods
Set specific time limits for troubleshooting before escalating or implementing temporary fixes. For example, spend maximum 30 minutes investigating a performance issue before switching to a known-good configuration. This prevents engineers from spending hours debugging during critical outages when restoration should be the priority.
11. Build multiple communication paths
Don't rely solely on Slack or email for critical alerts. Use multiple notification channels: SMS for critical alerts, phone calls for extended outages, and push notifications through dedicated on-call apps. Test these channels regularly to ensure they work when needed.
12. Maintain on-call compensation and time boundaries
Fairly compensate engineers for on-call duties through additional pay, time off, or schedule flexibility. Set clear expectations about response times: acknowledge alerts within 15 minutes, begin investigation within 30 minutes. This creates mutual respect between the business needs and engineer well-being.
Rolling out sustainable on-call practices
Start with the practices that address your team's biggest pain points. If engineers complain about being woken up for minor issues, begin with escalation boundaries and alert routing. If incident response feels chaotic, focus on communication channels and runbooks first.
Implement changes gradually over 2-3 months rather than overhauling everything at once. Pick 3-4 practices to start with, get them working smoothly, then add more. This prevents overwhelming your team and gives you time to adjust each practice based on real usage.
Measure the impact of changes by tracking metrics like mean time to resolution, number of escalations, and engineer satisfaction with on-call duties. Proper reliability metrics help you understand whether your sustainable practices are actually improving both system uptime and team experience.
Remember that sustainable on-call practices require ongoing refinement. What works for a team of 5 engineers might need adjustment when you grow to 12. Regular retrospectives and honest feedback help evolve your approach as your team and infrastructure scale together.
Building reliability without burnout
Sustainable on-call practices protect both your systems and your engineers. They ensure that your high availability infrastructure stays reliable while keeping your team healthy and motivated. The goal isn't to eliminate all incidents, but to handle them efficiently when they occur.
These practices work because they acknowledge that small teams have limited resources while still maintaining the reliability standards that growing businesses require. They're designed to scale with your team and evolve as your systems become more complex.
If implementing these yourself is not the best use of your engineering time, our managed services cover all of them by default.
