The situation: when good uptime metrics hide bad user experience
A European SaaS platform serving 15,000 active users was celebrating their monitoring dashboard. Uptime: 99.94%. Response times: looking green. Server resources: well within limits. Yet customer support was drowning in complaints about slow loading times and "glitchy" behavior during peak hours.
The platform processed financial data for small businesses across multiple EU markets. Users logged in primarily during business hours, creating predictable traffic spikes between 9-11 AM and 2-4 PM CET. During these periods, support tickets spiked too.
The contradiction was stark: monitoring said everything was fine, but customers were threatening to cancel subscriptions. The CEO estimated they were losing roughly €4,200 per month in recurring revenue to churn that seemed directly tied to performance complaints.
Their existing setup ran on a managed hosting provider with basic monitoring included. The dashboard showed server uptime, CPU usage, memory consumption, and simple HTTP checks. All green lights, all the time.
What we found during the infrastructure audit
Within the first day of our audit, the disconnect became clear. Their monitoring was measuring the wrong things and missing the user experience entirely.
The HTTP health checks hit a lightweight endpoint that returned a 200 status code in under 100ms. But real user workflows involved complex database queries, API calls to third-party financial services, and heavy JavaScript execution. The health check bore no resemblance to actual usage patterns.
We deployed real user monitoring (RUM) and synthetic transaction monitoring to measure what customers actually experienced. The results were sobering:
- Dashboard loading: 847ms average during peak hours (their monitoring showed 120ms for the health check)
- Financial report generation: 12.3 seconds at 95th percentile (completely invisible to their existing monitoring)
- API response times for core features: 2.1 seconds average during traffic spikes
- Time to interactive for the main application: 4.7 seconds on average
The application server logs revealed another issue their monitoring missed entirely. During peak periods, the connection pool to their PostgreSQL database was exhausting. Requests queued for up to 8 seconds waiting for available connections, but since the server stayed online, their uptime monitoring registered everything as healthy.
Memory usage looked fine on their dashboard because they were monitoring total system memory. But the application's memory allocation told a different story. Garbage collection pauses were hitting 300-500ms every few minutes during peak traffic, causing visible freezes in the user interface.
We also found that their CDN was configured incorrectly. Static assets that should have been cached were hitting the origin server on every request. During peak hours, this generated unnecessary load that cascaded through their entire high availability infrastructure.
The approach we took and why
Rather than adding more monitoring tools, we first defined what actually mattered for their business. For a financial SaaS platform, user experience translates directly to trust and retention. We identified five critical user journeys and established performance budgets for each.
The core principle: monitor what users do, not what servers do. Server metrics matter, but only when they correlate with user experience degradation.
We implemented monitoring at three levels:
User experience monitoring: Real user monitoring (RUM) to capture actual performance as experienced by customers in different geographic locations and network conditions. This included Core Web Vitals, time to interactive, and completion rates for critical workflows.
Synthetic transaction monitoring: Automated tests that simulate real user journeys every minute from multiple EU locations. These tests logged into the application, generated reports, processed transactions, and measured end-to-end performance including all third-party dependencies.
Infrastructure correlation monitoring: Traditional server metrics, but configured to alert only when they correlated with user experience degradation. CPU spikes that don't impact users don't need immediate attention.
We also restructured their alerting philosophy. Instead of alerting on server metrics crossing arbitrary thresholds, alerts fired when user experience degraded beyond acceptable limits. This eliminated noise and focused attention on what actually impacted the business.
Implementation details with high availability infrastructure specifics
The monitoring architecture required careful planning to avoid adding overhead to an already stressed system. We deployed monitoring agents strategically across their infrastructure stack.
For real user monitoring, we implemented a lightweight JavaScript agent that sampled 25% of user sessions to capture performance data without impacting the user experience. The agent collected:
// Performance observer for Core Web Vitals
const observer = new PerformanceObserver((list) => {
for (const entry of list.getEntries()) {
if (entry.entryType === 'navigation') {
// Capture Time to Interactive
sendMetric('tti', entry.domInteractive - entry.fetchStart);
}
if (entry.entryType === 'measure' && entry.name === 'report-generation') {
// Custom business metric
sendMetric('report_duration', entry.duration);
}
}
});
observer.observe({entryTypes: ['navigation', 'measure']});The synthetic monitoring required building realistic test scenarios. We created Puppeteer scripts that replicated actual customer workflows:
// Synthetic test for report generation workflow
const testReportGeneration = async (page) => {
const start = Date.now();
await page.goto(process.env.APP_URL + '/login');
await page.fill('[name="email"]', process.env.TEST_USER_EMAIL);
await page.fill('[name="password"]', process.env.TEST_USER_PASSWORD);
await page.click('button[type="submit"]');
await page.waitForSelector('[data-testid="dashboard"]');
await page.click('[data-testid="generate-report"]');
// Wait for report to complete, with 15s timeout
await page.waitForSelector('[data-testid="report-complete"]', { timeout: 15000 });
const duration = Date.now() - start;
return { success: true, duration };
};For database monitoring, we implemented connection pool visibility and query performance tracking:
# PostgreSQL configuration for connection monitoring
log_min_duration_statement = 1000
log_connections = on
log_disconnections = on
# Connection pool monitoring in application
pool.on('acquire', (client) => {
metrics.increment('db.connections.acquired');
});
pool.on('release', (client) => {
metrics.increment('db.connections.released');
});
pool.on('error', (err, client) => {
metrics.increment('db.connections.error');
logger.error('Database connection error', { error: err.message });
});We configured alerting rules based on user experience thresholds rather than infrastructure metrics:
# Alert when user experience degrades
- alert: SlowReportGeneration
expr: avg_over_time(report_generation_p95[5m]) > 8000
for: 2m
labels:
severity: critical
annotations:
summary: "Report generation is too slow"
description: "95th percentile report generation time is {{ $value }}ms"
- alert: HighErrorRate
expr: rate(user_workflow_errors[5m]) > 0.05
for: 1m
labels:
severity: criticalThe correlation monitoring required custom dashboards that showed infrastructure metrics alongside user experience data. When users experienced slow performance, engineers could immediately see which infrastructure component was the bottleneck.
Results with real numbers: from blind spots to clarity
The impact was immediate and measurable. Within two weeks of implementing comprehensive monitoring, we identified and resolved performance issues that had been invisible for months.
User experience improvements:
- Dashboard loading time: 847ms → 312ms average during peak hours
- Report generation P95: 12.3s → 4.1s
- API response times: 2.1s → 680ms average
- Time to interactive: 4.7s → 2.1s
- User workflow completion rate: 78% → 94%
Business impact:
- Support tickets related to performance: reduced by 73%
- Customer churn attributed to performance: dropped from €4,200/month to €800/month
- Customer satisfaction scores (CSAT) for application performance: 6.2/10 → 8.7/10
The monitoring changes also revealed optimization opportunities we quantified:
- Database connection pool exhaustion eliminated by increasing pool size from 10 to 25 connections
- CDN hit rate improved from 67% to 91% by fixing cache headers
- Memory garbage collection pauses reduced from 300-500ms to under 50ms by tuning JVM parameters
- Third-party API timeout issues identified and resolved (they were causing 12% of workflow failures)
Perhaps most importantly, the development team gained confidence in their deployments. Previously, they deployed cautiously because they couldn't measure real impact. With proper monitoring, they could deploy and immediately see if users were affected.
The false positive alert rate dropped dramatically. Before, they received 15-20 alerts per week, mostly about server metrics that didn't impact users. After implementing user-focused monitoring, they received 2-3 alerts per week, all of which required action.
Revenue retention improved measurably. Six months after implementing the new monitoring approach, monthly recurring revenue churn dropped from 3.2% to 1.8%. The CEO attributed roughly €31,000 in retained annual revenue directly to the performance improvements that were only possible once they could actually measure user experience.
What we'd do differently next time
The implementation went smoothly, but we learned several lessons that would accelerate future projects.
We should have started with business impact analysis sooner. We spent the first few days analyzing their existing technical monitoring setup, but the real breakthrough came when we mapped their user workflows to revenue impact. Starting with business metrics and working backward to technical implementation would have been more efficient.
The synthetic monitoring tests required more maintenance than expected. As the application evolved, the tests broke frequently when UI elements changed. Building tests that use stable data attributes rather than CSS selectors would have reduced maintenance overhead.
We also underestimated the cultural change required. The engineering team was accustomed to monitoring infrastructure metrics and needed time to adjust to user-experience-focused alerting. More upfront training on interpreting the new metrics would have helped.
For similar engagements, we'd implement staged rollouts of the monitoring changes. We replaced their alerting system completely, which created a brief period where the team felt uncertain about system visibility. Gradually transitioning from old alerts to new ones while running both systems in parallel would have been less disruptive.
The real user monitoring sample rate of 25% generated more data than necessary for this scale. For a platform with 15,000 active users, 10% sampling would have provided sufficient statistical significance while reducing storage and processing overhead.
Finally, we should have documented the correlation between infrastructure metrics and user experience more systematically from the beginning. While we established these correlations during the project, having a formal runbook that maps specific user experience degradation patterns to likely infrastructure causes would have been valuable for the client's team.
Beyond monitoring: building reliable high availability infrastructure
This project reinforced a fundamental principle: monitoring is only valuable when it measures what actually matters to your business. Traditional infrastructure monitoring creates a dangerous false sense of security by focusing on server health rather than user experience.
The most reliable systems are those where monitoring directly correlates with business outcomes. When alerts fire, they should indicate that customers are experiencing problems, not that a server metric crossed an arbitrary threshold.
For SaaS platforms especially, user experience monitoring isn't optional. Your infrastructure might be perfectly healthy while your application performance drives customers away. The gap between what traditional monitoring shows and what users experience can be the difference between growth and stagnation.
The key insight: reliable infrastructure isn't about keeping servers running. It's about ensuring user workflows complete successfully within acceptable timeframes. Once you monitor what matters, you can optimize what matters.
This extends beyond monitoring into architectural decisions. When you can measure real user impact, you make better tradeoffs between performance, cost, and complexity. Infrastructure decisions become data-driven rather than assumption-driven.
Facing a similar challenge? Tell us about your setup and we will outline an approach.
