When we went back through a year of postmortems for our order and billing services, very few incidents traced back to an exotic distributed-systems failure. Most started as a small habit that looked harmless in a pull request: a catch block that logged nothing, a list endpoint with no limit, a two-step write without a transaction. Each one was fine at a hundred users and painful at a hundred thousand.

This post lists the ten habits that showed up most often, why each one bites, and the concrete rule we now apply in code review. It is aimed at backend engineers and reviewers who want a short checklist grounded in real failure modes rather than style preferences.

1. Catching an exception and doing nothing with it

A bare catch (Exception e) {} turns a loud, debuggable failure into silent data corruption. In our case a payment-provider timeout was swallowed, the order stayed in pending forever and support found out from customers a week later.

// What we reject in review
try {
    gateway.capture(orderId);
} catch (Exception e) {
    // ignore
}

// What we accept
try {
    gateway.capture(orderId);
} catch (GatewayTimeoutException e) {
    log.warn("capture timed out, scheduling retry order={}", orderId, e);
    retryQueue.enqueue(new CaptureRetry(orderId));
}

Review rule: every catch block either handles a specific exception type with a real recovery action, or rethrows. Catching the base Exception needs a written justification.

2. Queries inside loops (N+1)

Loading a list and then querying once per row makes database round trips grow linearly with data. It hides in ORMs because the code looks like plain property access. A dashboard that ran 3 queries in staging ran 1,400 in production for a large tenant.

Review rule: any loop that touches a relation must show eager loading or a batched WHERE id IN (...) query. We also turn on lazy-loading detection in development (for example Laravel's Model::preventLazyLoading() or Django's query count assertions in tests) so the mistake fails locally.

3. Filtering on columns with no index

A query like this is instant on a fresh table and a full scan later:

SELECT id, total
FROM orders
WHERE customer_id = 4821 AND status = 'paid';

Without an index on (customer_id, status) the database reads every row. The symptom is usually a slow creep in p95 latency rather than a sudden failure, which is why it survives so long.

Review rule: every new query on a hot path ships with the EXPLAIN output in the pull request description, and a migration adding the index if the plan shows a sequential scan on a large table.

4. Multi-step writes without a transaction

Debiting a balance and then creating an order as two independent statements means a crash between them leaves money gone and no order. This class of bug is rare per request and guaranteed over millions of requests.

# Both writes commit together or not at all
with db.transaction():
    accounts.debit(user_id, amount)
    orders.create(user_id, cart_id, amount)

Review rule: if a function writes to more than one table, or writes and then calls an external service, the reviewer asks what state is left behind if the process dies on each line. External calls go after commit, driven by an outbox table, not inside the transaction.

5. Returning whole models from the API

Serialising an ORM entity straight to JSON leaks whatever columns exist today and whatever columns are added next year. We found a password reset token in a user search response because someone added a column and the serializer picked it up automatically.

Review rule: API responses are built from explicit response objects or resources with an allow-list of fields. Never "return the model". This also shrinks payloads, which matters on mobile.

6. Trusting client input

Unvalidated input is the root of injection bugs and also of plain crashes: a string where a number was expected, a 40 MB JSON body, a negative quantity. Building SQL with string concatenation is the extreme case, but missing range checks cause more day-to-day incidents.

# Parameterised query, never string concatenation
cur.execute("SELECT id, name FROM users WHERE email = %s", (email,))

Review rule: validation happens at the edge with a schema (Form Requests, Pydantic, Joi, Bean Validation), with explicit types, ranges and max sizes. Queries are always parameterised. Body size limits are set at the proxy as well as in the app.

7. APIs that ignore HTTP semantics

Endpoints like GET /deleteUser?id=1 or a 200 OK with {"error": "not found"} break caches, retries and monitoring. A crawler following links once deleted test data through a GET endpoint in our staging environment. Load balancers and clients retry idempotent methods automatically, so a state-changing GET can run more than once.

Review rule: resources are nouns, methods carry the verb, status codes are honest (404, 409, 422, 429), and any state change uses POST, PUT, PATCH or DELETE. Error bodies follow one shape across the API; we use RFC 9457 problem details.

8. List endpoints with no pagination

An endpoint that returns "all invoices for this account" works until the biggest account has 600,000 invoices. Then it times out, retries pile up and memory spikes on every app server at once.

Review rule: every list endpoint has a default and a maximum page size enforced on the server. For large or frequently appended tables we prefer cursor (keyset) pagination over offsets. Exports go through a background job, not a synchronous request.

9. Hardcoded URLs, keys and limits

A hardcoded sandbox URL that slipped into production, an API key committed to the repository, a retry limit buried in a helper: all of these turn a config change into a deploy, and the committed secret becomes a security incident.

# config read from the environment, validated at boot
payments:
  base_url: ${PAYMENTS_BASE_URL}
  api_key: ${PAYMENTS_API_KEY}
  timeout_ms: ${PAYMENTS_TIMEOUT_MS:-3000}

Review rule: environment-specific values come from configuration, secrets come from a secret manager, and the app fails fast at startup if a required value is missing. A secret scanner runs in CI on every push.

10. Thin logs and no alerts

The most expensive incidents are the ones nobody notices for hours. Logs like error occurred with no request id, user id or upstream status make even a known failure slow to diagnose.

Review rule: logs are structured (JSON), carry a correlation id, and include the identifiers needed to find the affected record. Every new external dependency gets a latency and error-rate metric plus an alert threshold before it goes live. We never log secrets, tokens or full card numbers.

How we enforce this without slowing reviews

A checklist only works if it is cheap to apply. We split ours by how it is enforced:

Habit Enforced by
Swallowed exceptions Static analysis rule plus reviewer
N+1 queries Lazy-loading detection in dev, query count assertions in tests
Missing indexes EXPLAIN in PR template, slow query log alert
Missing transactions Reviewer question: "what if it dies here?"
Over-exposed responses Response resources only, serializer tests
Unvalidated input Schema validation at the edge, SAST in CI
Non-REST endpoints API linter on the OpenAPI spec
No pagination Server-side max page size, contract tests
Hardcoded config Secret scanning, boot-time config validation
Thin logging Log schema review, alert required for new dependency

Anything a tool can catch, a tool catches. Reviewers spend their attention on the two items tools cannot judge well: transaction boundaries and what happens when a dependency fails.

Key takeaways

  • Most backend incidents grow from small, repeatable habits, not rare design flaws.
  • Make the safe path the default: schema validation, response resources, parameterised queries, required config.
  • Push detection left: lazy-loading checks, query count tests and secret scanning fail before production does.
  • Ask "what state is left if the process dies on this line?" for every multi-step write.
  • No new dependency ships without metrics, an alert and logs that carry a correlation id.

References