The four golden signals
Google's SRE book nails it: latency, traffic, errors, saturation. Everything else is supplementary.
1. Latency: p50, p95, p99
Average latency is a lie. If 99 percent of requests take 50ms and 1 percent take 5 seconds, your average is 100ms — which tells you nothing about the users having a terrible experience.
// Using prom-client
const httpDuration = new Histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration in seconds',
labelNames: ['method', 'route', 'status'],
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});
app.use((req, res, next) => {
const end = httpDuration.startTimer();
res.on('finish', () => {
end({ method: req.method, route: req.route?.path || 'unknown', status: res.statusCode });
});
next();
});
2. Error rate by type
Not just "errors per second" but broken down by status code and endpoint. A spike in 429s means rate limiting is working. A spike in 500s on /checkout means revenue is at risk.
3. Event loop lag
This is Node.js-specific and critical. If the event loop is blocked, every request waits:
const eventLoopLag = new Gauge({
name: 'nodejs_eventloop_lag_seconds',
help: 'Event loop lag in seconds',
});
setInterval(() => {
const start = process.hrtime.bigint();
setImmediate(() => {
const lag = Number(process.hrtime.bigint() - start) / 1e9;
eventLoopLag.set(lag);
});
}, 1000);
Healthy: under 10ms. Warning: over 50ms. Critical: over 100ms.
4. Database query duration
Slow queries are the number one cause of slow APIs. Instrument your ORM:
const queryDuration = new Histogram({
name: 'db_query_duration_seconds',
help: 'Database query duration',
labelNames: ['operation', 'model'],
});
5. External dependency health
Track latency and error rate for every external call: Redis, S3, Stripe, third-party APIs. When your API slows down, you need to know if it is your code or a dependency.
Alerting rules I start with
- p99 latency > 2s for 5 minutes
- Error rate > 5 percent for 2 minutes
- Event loop lag > 100ms for 1 minute
- Any 5xx on payment endpoints (immediate)
The dashboard that matters
One page, four panels: request rate, error rate, p95 latency, event loop lag. If all four are green, the system is healthy. If any is red, you know exactly where to look.