Root cause analysis that finds causes, not culprits: 5 whys, fishbone, causal charts

On a Tuesday morning about 3,100 customers received a second copy of their monthly invoice, and roughly 400 of them were charged twice by direct debit. The first root cause analysis was finished by lunchtime. It said: "An engineer manually re-ran the billing job after it appeared to fail. Action item: remind engineers not to re-run billing jobs." Everyone involved knew this was unsatisfying, and the same team had a different double-charge incident two months later.

We reran the analysis properly, and the difference was not effort but method. This post covers the three techniques we now combine, using that incident as the worked example: branching 5 whys, a fishbone diagram for breadth, and a cause-and-effect chart that ties evidence to every cause. How to write and share the postmortem document itself is a separate topic; this is about the analysis that goes into it.

What "root cause" should mean

Complex system failures rarely have a single root cause. They happen when several conditions line up, and each one alone was survivable. The useful goal of RCA is to find the set of causes that, if any one had been different, would have prevented or contained the incident, and to pick changes that remove the most of them cheaply.

Two rules follow:

• "Human error" is a starting point, not a conclusion. When a person's action is in the chain, the next question is what made that action reasonable at the time.
• Every cause needs evidence. A log line, a config diff, a metric, a timestamped message. A cause nobody can show is a hypothesis.

Step 1: a timeline before any whys

We build the timeline first, from logs, deploy history and chat, before anyone proposes causes. Analysis done without a timeline tends to follow whoever speaks first.

The timeline already shows something the first RCA missed: the engineer did exactly what the runbook said, based on what the dashboard said.

Step 2: 5 whys, but branching

The classic 5 whys asks "why" repeatedly down one chain. The weakness is that each "why" usually has more than one honest answer, and a single chain picks one and drops the rest. We force branching: at each level, list every answer supported by evidence, then continue down each branch.

Problem: ~3,100 customers invoiced twice, ~400 charged twice

Why were invoices created twice?
  A. The billing job ran twice for the same period.
  B. The job does not check whether an invoice already exists for the period.

Why did the job run twice?  (branch A)
  A1. On-call started a manual run at 06:20.
      Why? The dashboard showed "failed" and the runbook says re-run on failure.
        Why did the dashboard show "failed"? Status is written to a separate store;
          a lost connection to that store is recorded as job failure.
        Why does the runbook say re-run? It was written when the job was
          all-or-nothing in one transaction; that stopped being true a year ago.
  A2. Nothing prevented a second concurrent or repeated run.
      Why? No lock or run record keyed by billing period.

Why did the job not detect existing invoices?  (branch B)
  B1. Invoices are keyed by a generated id, with no unique constraint on
      (customer_id, billing_period).
      Why? The constraint was dropped during a migration to support
      mid-month plan changes, which create a second invoice per period.

Why were customers charged, not just invoiced?
  C1. Debit requests are sent as invoices are created, not after a review step.
  C2. The idempotency key sent to the payment provider was the invoice id,
      which was new for each duplicate invoice.

Branching 5 whys turned one cause into six. It also killed the original action item. "Remind engineers not to re-run" contradicts a runbook that tells them to.

Step 3: a fishbone for breadth

5 whys goes deep. A fishbone (Ishikawa) diagram goes wide, so you notice categories nobody thought to ask about. The classic manufacturing categories do not fit software well, so we use our own:

Code ---------\      Data ----------\      Process/docs ---\
 no period lock \     no unique key  \     stale runbook    \
 key = invoice id \   mid-month plans \    no debit review    \
                   >------------------->--------------------->  DUPLICATE CHARGES
Tooling ---------/   Change ---------/     People/context ---/
 status error =   /  migration without /    early page, open /
 job failure     /   runbook update   /     alert, clear doc/

The fishbone added one cause the 5 whys missed: the alert had been open for over three hours, which pushed the on-call engineer to act quickly rather than investigate. That came from asking "what was the context" as its own category.

Step 4: a cause-and-effect chart with evidence

The final step joins everything into one chart where every effect has its causes and every cause has evidence. We borrowed the idea from reality charting (the Apollo method): for each effect, look for at least one action (something that happened) and at least one condition (something that already existed and allowed the action to matter). Incidents need both, and RCAs that only list actions end up blaming people.

EFFECT: customers charged twice
 |- action:    second run sent debit requests              [provider log 06:21-07:30]
 |- condition: debits sent without review step             [billing code, send_debits()]
 |- condition: idempotency key = invoice id                [request payloads]
 |
 EFFECT: second run created invoices
  |- action:    manual run at 06:20                         [shell history]
  |- condition: no unique (customer_id, period) constraint  [migration 2025_03_14]
  |- condition: no run record / lock per period             [job code]
  |
  EFFECT: on-call started manual run
   |- action:    dashboard showed "failed"                  [dashboard screenshot, log]
   |- condition: runbook says "re-run on failure"           [runbook v3]
   |- condition: alert open 3.5 h, page at 06:12            [alert history]
   |
   EFFECT: dashboard showed "failed"
    |- action:    status-store connection dropped 02:41     [dashboard error log]
    |- condition: status write error treated as job failure [dashboard code]

Each line with evidence in brackets was verified in the RCA meeting. Two proposed causes were dropped because nobody could produce evidence for them.

Step 5: action items that change conditions

With the chart, choosing actions becomes a matter of picking conditions to remove. We rate each candidate by how many failure paths it closes and how much it costs.

We avoid action items of the form "be more careful", "add more tests" or "improve monitoring" without a specific condition attached. If an action does not remove or weaken a condition on the chart, it does not go in.

Running the session

• Facilitator is not the incident commander and not from the owning team, so they can ask naive questions.
• Timeline circulated a day early, so the meeting is spent on causes, not reconstructing events.
• Language rule: describe what the system did and what information a person had, not what they "should have" done.
• Time box: 90 minutes for a medium incident. If the chart is not finished, schedule a second session rather than rushing to action items.
• Follow-up: action items get ticket links and due dates; the RCA is reopened if the same chart node causes another incident.

Common mistakes

Key takeaways

• Build the timeline from evidence before anyone proposes causes.
• Use branching 5 whys for depth and a software-specific fishbone for breadth.
• Chart each effect with both actions and conditions, and require evidence for every node.
• "Human error" starts the analysis; the runbook, dashboard or alert that shaped the decision is usually the real cause.
• Choose action items by which conditions they remove, preferring changes that make the risky action safe by design.

References

• Google SRE book, Postmortem culture: https://sre.google/sre-book/postmortem-culture/
• Google SRE workbook, Postmortem culture: learning from failure: https://sre.google/workbook/postmortem-culture/
• NIST SP 800-61 Rev. 3, Incident Response Recommendations: https://csrc.nist.gov/pubs/sp/800/61/r3/final
• Richard I. Cook, How Complex Systems Fail: https://how.complexsystems.fail/