Classic SLO examples are about request-response services: 99.9% of HTTP requests succeed, 99% complete within 300 ms. Many of our systems do not look like that. An order is accepted by an API in 80 ms, then travels through an event bus, two queues, four Lambda functions and a database before the customer gets a confirmation and the warehouse gets a pick list. The API's latency SLO can be perfect while confirmations arrive an hour late.

This post describes how we define SLOs for event-driven pipelines: what to measure per hop, the end-to-end objectives users actually care about, how reliability combines across steps, and how we alert.

Start from what the user experiences

For the order pipeline, we asked what "working" means from the outside:

  • The customer receives a confirmation email shortly after ordering.
  • The warehouse receives the order in time to ship the same day.
  • Nothing is lost, duplicated or wrong.

That gives three kinds of objectives:

Kind Question Example SLI
Freshness How long from event to outcome? Share of orders confirmed within 2 minutes of acceptance
Completeness Did every event reach the end? Share of accepted orders with a confirmation within 1 hour
Correctness Is the outcome right? Share of orders whose warehouse record matches the order

Request success rate at the API is still useful, but it describes only the entry point.

Measuring end-to-end freshness

Freshness needs a timestamp at the start and at the end of the journey for each event. We added an acceptedAt timestamp to the OrderPlaced event and recorded a metric when the confirmation was sent:

from datetime import datetime, timezone

def on_confirmation_sent(order):
    lag = (datetime.now(timezone.utc) - order.accepted_at).total_seconds()
    ORDER_CONFIRMATION_LAG.observe(lag)      # histogram with buckets around the target

The SLI is then the share of confirmations faster than the target:

sum(rate(order_confirmation_lag_seconds_bucket{le="120"}[28d]))
  /
sum(rate(order_confirmation_lag_seconds_count[28d]))

This includes everything: queue backlogs, retries, cold starts, throttling and email provider latency. That is the point.

Completeness needs a reconciliation

Freshness only counts events that reached the end. An order that is lost never produces a lag sample. We run a reconciliation job every ten minutes that compares accepted orders older than an hour with confirmations sent, and records the count of missing ones:

SELECT count(*) AS missing
FROM orders o
LEFT JOIN confirmations c ON c.order_id = o.id
WHERE o.accepted_at < now() - interval '1 hour'
  AND o.accepted_at > now() - interval '25 hours'
  AND c.order_id IS NULL;

The completeness SLI is 1 - missing / accepted over the window. It caught two silent drops in its first quarter, both from failed messages that no one had redriven from a dead-letter queue.

Per-hop SLIs for diagnosis

End-to-end SLOs tell you that something is wrong; per-hop SLIs tell you where. For each hop we track:

Hop Availability SLI Latency or freshness SLI
API (accept order) Successful requests / total p99 latency
Event bus to queue Delivered / matched events Delivery latency
Queue consumer (Lambda) Successful invocations / total, DLQ arrivals Age of oldest message
Database writes Successful writes / attempted Write latency
Email provider Accepted sends / attempts Provider acknowledgement time

The single most useful per-hop signal in queue-based systems is the age of the oldest message. It rises whenever consumers fall behind, for any reason, and it directly predicts freshness violations.

How reliability combines

When steps are in series and each must succeed, their availabilities multiply:

0.999 (API) x 0.9995 (bus) x 0.999 (consumer) x 0.9995 (DB) x 0.999 (email) ~= 0.996

Five steps at "three nines" each give about 99.6% end to end, not 99.9%. This matters when you set targets: an end-to-end objective of 99.9% requires the hops to be considerably better, or retries that recover failures.

Retries change the maths. A hop whose individual attempts fail 0.1% of the time, with three independent retries, loses far fewer events, as long as failures are transient and the operation is idempotent. That is why our completeness SLO is much better than the product of per-hop availabilities, and why idempotency is a reliability requirement, not just a correctness one.

Parallel paths (for example two email providers with failover) combine the other way: the path fails only if both fail.

Choosing targets

We set targets from three inputs:

  1. What users need: the warehouse needs orders within 30 minutes for same-day shipping; customers expect a confirmation within a couple of minutes.
  2. What the system achieved historically: setting a target far above recent performance guarantees a permanently exhausted error budget.
  3. What dependencies can deliver: if the email provider's own target is 99.9%, our confirmation SLO cannot be stricter without a second provider.

The result:

SLO Target Window
Orders confirmed within 2 minutes 99% 28 days
Orders confirmed within 1 hour (completeness) 99.95% 28 days
Orders in warehouse system within 30 minutes 99.5% 28 days

Alerting on burn rate

An error budget is the allowed failure: for a 99% target over 28 days, 1% of orders may be slower than two minutes. We alert on how fast the budget is being consumed, using two windows to catch both sharp and slow problems:

- alert: OrderConfirmationFreshnessBurnFast
  expr: |
    (1 - (sum(rate(order_confirmation_lag_seconds_bucket{le="120"}[1h]))
          / sum(rate(order_confirmation_lag_seconds_count[1h])))) > (14.4 * 0.01)
    and
    (1 - (sum(rate(order_confirmation_lag_seconds_bucket{le="120"}[5m]))
          / sum(rate(order_confirmation_lag_seconds_count[5m])))) > (14.4 * 0.01)
  labels: { severity: page }

A burn rate of 14.4 sustained for one hour consumes about 2% of a 28-day error budget; left alone, it would exhaust the whole budget in under two days, which is worth waking someone for. The short window makes the alert stop quickly once the problem is fixed. A slower burn, around 1 sustained over three days, opens a ticket instead.

Common mistakes

Mistake Why it hurts Do this instead
Only API SLOs for a pipeline Late or lost outcomes go unseen End-to-end freshness and completeness
Freshness without completeness Lost events produce no samples Reconcile inputs against outputs
Per-hop targets equal to the end-to-end target The product is lower than any hop Budget reliability across hops
No idempotency Retries cause duplicates, so teams avoid retries Make every consumer idempotent
Alerting on raw error counts Noisy and blind to slow burns Multi-window burn-rate alerts

Key takeaways

  • For pipelines, define SLOs on freshness, completeness and correctness of outcomes.
  • Measure freshness with timestamps carried in events, and completeness with reconciliation.
  • Use per-hop SLIs, especially age of oldest message, to locate problems.
  • Availability multiplies across serial steps; retries with idempotency recover it.
  • Alert on error budget burn rate with fast and slow windows.

References