One nested GraphQL query pinned our API at 100% CPU: depth limits, cost analysis and introspection

On a Sunday evening, all pods of our public GraphQL API hit 100% CPU, latency for every client rose to seconds, and autoscaling added pods that immediately saturated too. The traffic volume was normal. The load balancer logs showed a handful of requests per minute from one client IP that each took the full 30 second timeout.

Finding the query

Our GraphQL server logged operation names and a hash of the query, but the slow requests had no operation name. We temporarily logged full query documents for requests over 5 seconds and found this shape:

query {
  posts(first: 100) {
    author {
      posts(first: 100) {
        comments(first: 100) {
          author {
            posts(first: 100) {
              comments(first: 100) {
                author { name }
              }
            }
          }
        }
      }
    }
  }
}

Each level multiplies the work: up to 100 posts, each with an author, each author with 100 posts, each with 100 comments, and so on. In the worst case that is trillions of resolver calls; in practice, the resolvers and DataLoader batches ran until the timeout, and each request consumed a full CPU core for 30 seconds.

Is it an attack?

The client IP belonged to a cloud provider, the user agent was a common GraphQL tool, and the queries came in a slow, steady stream. It looked like someone exploring the schema, possibly testing for denial of service. Whatever the intent, the problem was that our API allowed it.

What we ruled out

• A database problem. The database was busy but responsive; the API CPU, running resolvers and serialising huge responses, was the bottleneck.
• A recent deploy. The schema had allowed this shape since launch.
• Normal client behaviour. Our own web and mobile clients never nested beyond three levels.

Root cause

The schema contains cycles (Post.author -> User.posts -> Post.comments -> Comment.author -> ...) and list fields that accept large page sizes. The server had no limits on query depth, query cost or response size, so any client could request an exponentially expensive query in a single HTTP request, bypassing our request-per-minute rate limit entirely.

Why GraphQL needs different limits

In a REST API, each endpoint has a roughly known cost, so rate limiting requests limits work. In GraphQL, one request can ask for almost anything the schema allows, so cost varies by orders of magnitude between requests. Limits must be based on what a query asks for:

Introspection

Introspection lets clients query the schema itself. It powers tooling and is useful in development. In production, for a public API whose schema is not meant to be public, it makes discovery of expensive paths easier. Disabling it is not a security boundary, since fields can still be guessed, but it reduces casual exploration. For a public, documented API, keeping it enabled can be the right choice; then limits matter even more.

Who is affected

Any GraphQL API whose schema has cycles between types (users have posts, posts have authors) and list fields with client-controlled sizes, which describes most real schemas. Public APIs and APIs reachable by any logged-in user are the most exposed. Server libraries generally ship without depth or cost limits by default, so they must be added deliberately.

Environment

Immediate: block and cap

We blocked the client IP at the edge, then deployed a depth limit and a lower maximum page size within the hour:

import depthLimit from 'graphql-depth-limit';

const server = new ApolloServer({
  schema,
  validationRules: [depthLimit(6)],
});

Our deepest legitimate query had a depth of four, so six left headroom. In resolvers, first was capped:

const pageSize = (first) => Math.min(Math.max(first ?? 20, 1), 50);

Cost analysis

A depth limit alone does not stop a wide query at depth three with large lists. We added complexity analysis, assigning a cost to fields and multiplying by requested list sizes:

import { createComplexityRule, simpleEstimator, fieldExtensionsEstimator } from 'graphql-query-complexity';

const complexityRule = createComplexityRule({
  maximumComplexity: 2000,
  estimators: [
    fieldExtensionsEstimator(),       // per-field costs declared in the schema
    simpleEstimator({ defaultComplexity: 1 }),
  ],
  onComplete: (complexity) => metrics.histogram('graphql_query_complexity').observe(complexity),
});

List fields declare their cost relative to the requested size:

posts: {
  type: new GraphQLList(Post),
  args: { first: { type: GraphQLInt } },
  extensions: {
    complexity: ({ args, childComplexity }) => pageSize(args.first) * childComplexity,
  },
}

We set the maximum from the cost of our real clients' heaviest queries, measured over a week with the histogram, times two.

Persisted queries for first-party clients

Our web and mobile apps now send only registered operation hashes. Their builds extract every operation into a manifest that is uploaded at deploy time; the server rejects unknown hashes for first-party client credentials. Partners using the public API still send arbitrary queries, subject to the limits above and a cost-based rate limit per API key: each key has a budget of complexity points per minute instead of a request count.

Introspection policy

Introspection is disabled for unauthenticated requests in production and enabled for partner keys, which need it for their tooling. Field suggestions in error messages ("Did you mean ...?") are disabled in production as well.

Alternatives and trade-offs

• Only persisted queries. The strongest control, but impossible for a public API meant for arbitrary queries.
• Query timeouts alone. A 30-second timeout limited each request, but a few concurrent requests were still enough to saturate the service.

How to test the fix

# deep query rejected at validation, before any resolver runs
curl -s https://api.staging.example.com/graphql -H 'content-type: application/json' \
  -d '{"query":"{ posts(first:100){ author{ posts(first:100){ comments(first:100){ author{ posts(first:100){ comments(first:100){ author{ name } } } } } } } } }"}' \
  | jq '.errors[0].message'
# "'<anonymous>' exceeds maximum operation depth of 6"

A test suite of our real client operations runs against the limits in CI, so a limit change can never break a legitimate client unnoticed.

Confirming in production

The blocked query shapes now fail in under 5 ms at validation. Over the following month, the complexity histogram showed our real clients' queries below 600, and 0.02% of partner requests were rejected for complexity, each followed up with the partner. CPU usage returned to normal and never saturated again under similar probing.

Regression guard

• Alerts on the rate of complexity and depth rejections, which also detect probing early.
• New list fields require a complexity extension in code review.
• The persisted query manifest is checked in CI against the client code.

Security implications

This was an availability attack surface: a single client could take down the API for everyone. The controls are defensive and layered, because each one alone has gaps. Logging full query documents helped the investigation but can capture personal data in arguments; we kept it limited to slow requests, redacted variables and retained those logs for seven days.

Prevention checklist

• Add depth limits and cost analysis before launching a public GraphQL API.
• Cap list sizes in arguments and resolvers.
• Rate limit by query cost, not request count.
• Use persisted queries for first-party clients.
• Decide on an introspection policy deliberately, per audience.

References

• OWASP GraphQL Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/GraphQL_Cheat_Sheet.html
• GraphQL, Security: https://graphql.org/learn/security/
• Apollo Server, Persisted queries: https://www.apollographql.com/docs/apollo-server/performance/apq
• graphql-query-complexity: https://github.com/slicknode/graphql-query-complexity