Health checks timed out during report exports: CPU work blocked the Node.js event loop

Our admin API, a Node.js service, lets staff export orders to CSV. For months, nobody complained. Then the finance team started exporting full quarters, about 200,000 rows, and two things happened during each export: the whole admin panel froze for everyone, and Kubernetes restarted the pod handling the export, which killed the export itself.

The pod events:

Warning  Unhealthy  pod/admin-api-6c8b7-qk2fd  Liveness probe failed: Get "http://10.20.33.17:3000/healthz":
         context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal   Killing    pod/admin-api-6c8b7-qk2fd  Container admin-api failed liveness probe, will be restarted

The /healthz endpoint does nothing but return 200 OK. It timed out because the process could not run any JavaScript to answer it.

The export handler

app.get('/exports/orders.csv', async (req, res) => {
  const rows = await db.query(
    'SELECT * FROM orders WHERE created_at >= $1 AND created_at < $2', [req.query.from, req.query.to]);

  const lines = rows.map(r => [
    r.id, r.customer_name, r.email, formatMoney(r.total, r.currency),
    new Date(r.created_at).toLocaleString('en-GB', { timeZone: 'Europe/London' }),
  ].map(csvEscape).join(','));

  res.type('text/csv').send(['id,customer,email,total,created_at', ...lines].join('\n'));
});

The database query itself was asynchronous and fine. Everything after it was synchronous: mapping 200,000 rows, formatting money and dates with Intl (relatively expensive per call), escaping, joining into one large string.

Measuring the block

Node's perf_hooks can measure event loop delay with a histogram:

import { monitorEventLoopDelay } from 'node:perf_hooks';
const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();
setInterval(() => {
  logger.info({ p99_ms: h.percentile(99) / 1e6, max_ms: h.max / 1e6 }, 'event loop delay');
  h.reset();
}, 10_000);

During a quarterly export:

event loop delay p99_ms=6210 max_ms=11840

The event loop was blocked for up to twelve seconds at a time. A CPU profile (node --cpu-prof) showed almost all of that time in toLocaleString, formatMoney and string concatenation.

What we ruled out

• The database. The query returned in about 900 ms and did not block the loop.
• Memory pressure. Heap grew during the export but stayed well below the limit; garbage collection pauses were short.
• The liveness probe settings. A longer timeout would have hidden the problem, not fixed it.

Root cause

Node.js runs JavaScript on a single main thread per process. The export handler did seconds of synchronous CPU work on that thread, so the event loop could not process anything else: other requests, health checks, timers. The liveness probe failed, Kubernetes restarted the container, and the export never completed.

The event loop in one paragraph

Node.js handles concurrency by never blocking: I/O operations are started, and their callbacks run later when results are ready. This works as long as each callback finishes quickly. A callback that computes for five seconds holds the only JavaScript thread for five seconds, and every other request waits. Async functions do not change that: code between await points runs synchronously.

Where to move CPU work

Environment

Step 1: stop creating formatters per row

toLocaleString with options creates or looks up a formatter internally on every call. Creating the formatters once is much faster:

const dateFmt = new Intl.DateTimeFormat('en-GB', {
  timeZone: 'Europe/London', dateStyle: 'short', timeStyle: 'medium' });
const moneyFmts = new Map();
const money = (amount, currency) => {
  if (!moneyFmts.has(currency)) {
    moneyFmts.set(currency, new Intl.NumberFormat('en-GB', { style: 'currency', currency }));
  }
  return moneyFmts.get(currency).format(amount);
};

That alone cut CPU time for the export by about 70%, but 200,000 rows still blocked the loop for over a second, and exports were growing.

Step 2: stream from the database to the response

Instead of loading all rows and building one string, the handler now streams rows through a transform to the response. Each chunk is small, so other work gets a turn between chunks:

import { pipeline } from 'node:stream/promises';
import { Transform } from 'node:stream';
import QueryStream from 'pg-query-stream';

app.get('/exports/orders.csv', async (req, res) => {
  const client = await pool.connect();
  try {
    const query = new QueryStream(
      'SELECT id, customer_name, email, total, currency, created_at FROM orders WHERE created_at >= $1 AND created_at < $2',
      [req.query.from, req.query.to], { batchSize: 1000 });
    const toCsv = new Transform({
      objectMode: true,
      transform(r, _enc, cb) {
        cb(null, [r.id, r.customer_name, r.email, money(r.total, r.currency), dateFmt.format(r.created_at)]
          .map(csvEscape).join(',') + '\n');
      },
    });
    res.type('text/csv');
    res.write('id,customer,email,total,created_at\n');
    await pipeline(client.query(query), toCsv, res);
  } finally {
    client.release();
  }
});

Streams also apply backpressure: if the client downloads slowly, the database cursor pauses, and memory stays flat instead of holding the whole file.

Step 3: worker threads for heavy formatting

One export type, a PDF-friendly report with aggregation and charts, was CPU-heavy even per chunk. It now runs in a worker thread pool using Piscina:

import Piscina from 'piscina';
const reportPool = new Piscina({ filename: new URL('./report-worker.js', import.meta.url).href, maxThreads: 2 });

app.post('/exports/quarterly-report', async (req, res) => {
  const result = await reportPool.run({ from: req.body.from, to: req.body.to });
  res.json(result);
});

The main thread stays responsive; the work runs on other cores. With 1 vCPU per pod, we limited the pool to two threads and raised the pod's CPU request for this service.

Step 4: health checks that reflect the event loop

We kept the liveness probe simple, but added a readiness check that fails when event loop delay is high, so Kubernetes stops sending new traffic to a busy pod instead of killing it:

app.get('/readyz', (_req, res) =>
  h.percentile(99) / 1e6 > 500 ? res.status(503).end() : res.status(200).end());

Alternatives and trade-offs

• A background job with an emailed download link for very large exports. We use it now for exports over 500,000 rows, where even streaming takes minutes.
• setImmediate chunking without streams works, but is easy to get wrong and does not bound memory.

How to test the fix

A load test runs a quarterly export while sending health and normal API requests:

(curl -s -o /dev/null -w 'export %{time_total}s\n' 'https://admin.staging.example.com/exports/orders.csv?from=2025-01-01&to=2025-04-01') &
for i in $(seq 1 60); do curl -s -o /dev/null -w '%{time_total}\n' https://admin.staging.example.com/healthz; sleep 0.5; done | sort -n | tail -1

Expected: health check worst case under 100 ms during the export.

Confirming in production

Quarterly exports completed in about 14 seconds, streamed, with event loop delay p99 under 40 ms during the export. No liveness failures or restarts since. Other admin users no longer noticed exports happening.

Regression guard

• Event loop delay p99 is exported as a metric with an alert above 200 ms for five minutes.
• A performance test with a 200,000-row export runs in CI against a seeded database.
• Code review guideline: loops over unbounded result sets in request handlers must stream or use a worker.

Security implications

Any endpoint that does synchronous work proportional to user-controlled input is a denial of service vector: one large request can make a service unavailable for everyone. Streaming, worker threads, input limits on date ranges and authorization on export endpoints reduce that risk. Exports contain personal data, so streaming also avoids holding entire files in memory and logs.

Prevention checklist

• Keep synchronous work per callback short; measure event loop delay.
• Stream large results from database to response.
• Create Intl formatters once, not per item.
• Move CPU-heavy work to worker threads or background jobs.
• Make readiness reflect event loop health so busy pods are not killed.

References

• Node.js, Don't Block the Event Loop: https://nodejs.org/en/learn/asynchronous-work/dont-block-the-event-loop
• Node.js, perf_hooks.monitorEventLoopDelay: https://nodejs.org/docs/latest-v22.x/api/perf_hooks.html#perf_hooksmonitoreventloopdelayoptions
• Node.js, worker_threads: https://nodejs.org/docs/latest-v22.x/api/worker_threads.html
• Node.js, stream.pipeline: https://nodejs.org/docs/latest-v22.x/api/stream.html#streampipelinesource-transforms-destination-options