Our load test said latency stayed flat. The load generator was the bottleneck

Before a big seasonal sale, we load tested the product API. The goal was 4,000 requests per second, about twice the previous year's peak. The report looked reassuring:

scenarios: (100.00%) 1 scenario, 200 max VUs, 10m30s max duration
     http_req_duration..............: avg=81.2ms  min=12.4ms  med=74.0ms  max=1.9s  p(90)=118ms  p(95)=141ms
     http_reqs......................: 1468311  2447.18/s
     vus............................: 200     min=200  max=200

Latency was fine. The team signed off. Nobody focused on the line that says 2,447 requests per second, well short of the 4,000 target, because the script was "configured for 4,000".

On sale day, at about 2,100 requests per second of real traffic, p95 latency went past two seconds and errors began. The load test had told us we could handle double that.

What the script did

import http from "k6/http";
import { sleep } from "k6";

export const options = {
  vus: 200,
  duration: "10m",
};

export default function () {
  http.get(`${__ENV.BASE}/products/${randomId()}`);
  sleep(0.03);
}

The author's reasoning: 200 virtual users, each sending a request every 50 ms (roughly 20 ms response plus 30 ms sleep), gives 200 × 20 = 4,000 requests per second. That holds only if responses take 20 ms. Each virtual user sends its next request only after the previous response arrives. When responses slowed to 80 ms, each user sent about 9 requests per second instead of 20, and the total rate fell to 2,400. The test throttled itself to whatever rate the system could serve comfortably, then reported comfortable latency.

The generator was also struggling

We re-ran the test with the generator's own metrics visible. The single test machine was at 96 percent CPU, and k6 printed a warning we had missed in the scroll:

WARN[0312] The test has generated metrics with 1002345 unique time series, which is higher than the suggested limit of 100000

That warning came from using the full product URL as a metric tag, which made k6 itself slower. A load generator near its own limits adds latency to its measurements and sends fewer requests than configured.

Closed and open workload models

There are two ways to generate load:

• Closed model: a fixed number of users, each waiting for a response before sending the next request. Throughput depends on response time. This is k6's default vus + duration setup.
• Open model: requests arrive at a fixed rate, regardless of how long previous ones take. If the system slows down, requests pile up, exactly as real traffic does.

Real users of a public web API behave like an open model. Thousands of independent people do not wait for other people's responses before clicking. When our API slowed down on sale day, new requests kept arriving at the same rate, queues grew, and latency exploded. A closed-model test cannot show that: when the system slows, the test slows with it.

Coordinated omission

The term, coined by Gil Tene, describes what happens to measurements in a closed model. Suppose the server stalls for one second. A closed-model user waiting on that request records one slow sample of 1,000 ms, then continues. But at the intended rate of one request every 50 ms, twenty requests should have been sent during that second, and each of them would have waited too. The tester "coordinated" with the system by not sending them, and "omitted" their latency from the results.

The effect is dramatic on percentiles. One stall that should have produced twenty slow samples produces one, so p95 and p99 look far better than what users would experience.

Environment

1. Use an arrival-rate executor

k6 has open-model executors that start iterations at a fixed rate, allocating more virtual users as needed when responses slow down:

export const options = {
  scenarios: {
    sale_peak: {
      executor: "ramping-arrival-rate",
      startRate: 500,
      timeUnit: "1s",
      preAllocatedVUs: 500,
      maxVUs: 4000,
      stages: [
        { target: 2000, duration: "5m" },
        { target: 4000, duration: "5m" },
        { target: 4000, duration: "10m" },
      ],
    },
  },
  thresholds: {
    http_req_duration: ["p(95)<300", "p(99)<800"],
    http_req_failed: ["rate<0.01"],
    dropped_iterations: ["count<100"],
  },
};

export default function () {
  http.get(`${__ENV.BASE}/products/${randomId()}`, { tags: { name: "GET /products/:id" } });
}

Key points:

• No sleep. The executor controls the rate. Sleep only belongs in scripts that model a user's think time in a multi-step journey.
• maxVUs high enough. If the system slows down, k6 needs more concurrent users to keep the rate. If it runs out, it skips iterations and reports them in dropped_iterations.
• dropped_iterations as a threshold. Dropped iterations mean the test did not deliver the load it claims. A test with many dropped iterations has failed, whatever its latency numbers say.
• A name tag. Grouping all product URLs under one name keeps metric cardinality low.

2. Check that the generator delivered the load

Every report now starts with two numbers before anyone reads latency: the target rate and the achieved rate. If they differ by more than 2 percent, the result is invalid. We also watch the generator's CPU and network. For 4,000 requests per second we now use three generator machines through k6's distributed execution, each kept under 60 percent CPU.

3. Re-run, and believe the result

The open-model test against the same build:

     http_req_duration..............: avg=1.41s  med=310ms  p(90)=3.9s  p(95)=5.2s
     http_reqs......................: 3791284  3159.40/s
     dropped_iterations.............: 251203
     http_req_failed................: 6.31%
     dropped_iterations: count<100 ... FAILED
     http_req_duration: p(95)<300 ... FAILED

Both thresholds failed. That is what sale day looked like. Latency climbed steeply above about 2,000 requests per second, matching production.

The bottleneck, now visible in the test, was the product API's database connection pool: requests queued for a connection when an uncached category query ran slowly under load. We fixed the query and added a cache for category listings. The next open-model run held p95 at 140 ms at 4,000 requests per second with no dropped iterations.

1. Replay of sale day. We built a scenario from sale-day access logs, matching the arrival rate minute by minute, and ran it against the old build. It reproduced the failure at the same rate as production, which gave us confidence the method now matches reality.

2. Tests in CI. A smaller open-model test runs nightly at 50 percent of expected peak against staging, with the same thresholds, including dropped_iterations. A regression in tail latency fails the nightly job rather than surfacing on the next busy day.

3. Result template. Load test reports now have a fixed header: target rate, achieved rate, dropped iterations, generator CPU, test duration, warm-up excluded, and the latency percentiles from p50 to p99.9. A report missing any of those is not accepted.

4. Security implications. Load tests against production-like environments use synthetic accounts and data, never real customer credentials, and generators run from our own network with the target's rate limiter configured to allow them by an explicit token. Turning off the rate limiter for a test, as had been done before, also removes a defence you need on the real sale day.

Takeaways

• A fixed number of users that wait for responses (closed model) throttles itself and hides queueing.
• Public traffic behaves like an open model. Test with a fixed arrival rate.
• Coordinated omission makes percentiles look far better than reality in closed-model tests.
• Check achieved rate, dropped iterations and generator load before reading any latency number.

References: k6 open and closed models, k6 ramping arrival rate executor, k6 dropped iterations, k6 thresholds.