Node.js worker crashed on an unhandled promise rejection: one fire-and-forget call took it down

Our invoice worker, a Node.js process consuming jobs from a Redis-backed queue, restarted three or four times a day. Each restart dropped the jobs it was processing; they were retried later, but some customers received invoices hours late. The container logs before each restart ended with:

node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
    ^

Error: Request failed with status code 503
    at settle (/app/node_modules/axios/lib/core/settle.js:19:12)
    at IncomingMessage.handleStreamEnd (/app/node_modules/axios/lib/adapters/http.js:599:11)
    ... {
  code: 'ERR_BAD_RESPONSE',
  config: { url: 'https://analytics.internal.example.com/events', method: 'post', ... }
}

Node.js v22.17.0

The process exited with code 1. When the rejection reason is an Error, Node.js rethrows it as is; for non-error reasons it wraps them in an ERR_UNHANDLED_REJECTION error instead. The failing request went to the internal analytics service, not to anything the invoice job needed.

Finding the call

A search for the analytics URL led to one place in the job handler:

async function handleInvoiceJob(job) {
  const invoice = await buildInvoice(job.data.orderId);
  await storePdf(invoice);
  await sendEmail(invoice);

  // record a metric for the dashboard; not important enough to wait for
  analytics.track('invoice_sent', { orderId: job.data.orderId, total: invoice.total });

  return { invoiceId: invoice.id };
}

// analytics.js
export function track(event, props) {
  return axios.post(ANALYTICS_URL, { event, props, at: Date.now() });
}

track returned a promise from axios.post. The handler neither awaited it nor attached .catch(). When the analytics service returned 503 during its own deploys, the promise rejected with no handler.

What we ruled out

• Errors in invoice generation. Jobs completed successfully before the crash; the error came after.
• Memory or OOM. Exit code 1 with a stack trace, not a SIGKILL.
• The queue library. It handled errors thrown by handleInvoiceJob, but this rejection happened in a promise unrelated to the handler's return value.

Root cause

Since Node.js 15, the default --unhandled-rejections mode is throw: a promise rejection that has no handler by the time the microtask queue drains is raised as an uncaught exception, and without an uncaughtException handler, the process exits. The fire-and-forget analytics.track() call produced an unhandled rejection whenever the analytics service failed, and that crashed the worker, killing every in-flight job with it.

What counts as unhandled

A rejected promise is "handled" if, at some point, someone attaches a rejection handler: .catch(), the second argument to .then(), or await inside a try. If a promise rejects and no handler is attached by the end of the current turn of the event loop, Node.js emits unhandledRejection. The default behaviour then depends on the mode:

The change in Node.js 15 was deliberate: silently ignored rejections hide bugs and can leave applications in inconsistent states.

Why fire-and-forget is tempting

Not awaiting a non-critical call is a reasonable intent: the invoice job should not fail or slow down because analytics is unavailable. The mistake is not handling the failure at all. The intent "ignore errors from this call" must be written as code.

Who is affected

Any Node.js 15 or later process with promises that are created and not handled: fire-and-forget logging, analytics and cache writes, async callbacks passed to event emitters or Array.forEach, and promises stored in variables and awaited only on one branch. Processes that set --unhandled-rejections=warn survive, but hide the same bugs.

Environment

Handle the promise where it is created

The analytics helper now never rejects, because tracking failures are not the caller's problem:

export function track(event, props) {
  return axios
    .post(ANALYTICS_URL, { event, props, at: Date.now() }, { timeout: 2000 })
    .catch((err) => {
      logger.warn({ err: err.message, event }, 'analytics track failed');
    });
}

And the call site makes the intent explicit with void, which the lint rule below understands as "deliberately not awaited":

void analytics.track('invoice_sent', { orderId: job.data.orderId, total: invoice.total });

The timeout also prevents analytics latency from keeping sockets and memory busy.

Lint for floating promises

With TypeScript, @typescript-eslint/no-floating-promises reports any promise that is neither awaited, returned, nor handled:

// eslint.config.mjs (excerpt)
rules: {
  '@typescript-eslint/no-floating-promises': ['error', { ignoreVoid: true }],
  '@typescript-eslint/no-misused-promises': 'error',
}

Running it on the worker codebase found 23 more floating promises, including two inside event emitter callbacks that could crash the process the same way.

A last-resort handler, for visibility

We added a process-level listener that logs and reports any unhandled rejection with context. It does not swallow the error: after reporting, it lets the process exit, because an unknown failure may have left state inconsistent.

process.on('unhandledRejection', (reason, promise) => {
  logger.fatal({ err: reason }, 'unhandled promise rejection');
  errorReporter.capture(reason);
  // let the default behaviour continue: exit so the orchestrator restarts a clean process
  process.exitCode = 1;
  setTimeout(() => process.exit(1), 1000).unref();
});

Graceful handling of in-flight jobs

Crashes will still happen for other reasons, so the worker now handles SIGTERM by stopping intake and finishing in-flight jobs, and job retries use idempotency keys so a retried invoice is not sent twice.

Alternatives and trade-offs

• --unhandled-rejections=warn keeps the process alive, but hides real bugs and can leave the process in an unknown state. We did not use it.
• A queue for analytics events instead of direct HTTP calls, so delivery is retried and decoupled. We did this later, using the same queue system as invoices.

How to test the fix

A unit test confirms track resolves even when the HTTP call fails:

test('track never rejects', async () => {
  nock('https://analytics.internal.example.com').post('/events').reply(503);
  await expect(track('x', {})).resolves.toBeUndefined();
});

And a test runs Node with a deliberately rejecting floating promise to confirm the process-level handler logs and exits, so the behaviour is known rather than assumed.

Confirming in production

After the release, the worker ran for four weeks without an unexpected restart, through several analytics deploys. The warning log showed failed analytics calls during those deploys, which is the desired outcome. Late invoices dropped to zero.

Regression guard

• no-floating-promises and no-misused-promises run as errors in CI for all Node.js services.
• An alert on container restarts with exit code 1 for worker deployments.
• The fatal log line from the process handler creates an incident ticket automatically.

Security implications

Any code path where external input can cause a rejected, unhandled promise is a way to crash the process repeatedly, a denial of service. Lint rules and explicit handling remove that class of crash. Error reports include request context but are scrubbed of personal data before leaving the service.

Prevention checklist

• Every promise must be awaited, returned or given a rejection handler.
• Write "ignore failures" explicitly with .catch() and mark intentional fire-and-forget with void.
• Enable no-floating-promises and no-misused-promises.
• Add a last-resort unhandledRejection listener that reports and exits.
• Make workers survive restarts with graceful shutdown and idempotent jobs.

References

• Node.js CLI, --unhandled-rejections: https://nodejs.org/docs/latest-v22.x/api/cli.html#--unhandled-rejectionsmode
• Node.js process events, unhandledRejection: https://nodejs.org/docs/latest-v22.x/api/process.html#event-unhandledrejection
• typescript-eslint, no-floating-promises: https://typescript-eslint.io/rules/no-floating-promises/
• MDN, Using promises: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises