Cloud & Infrastructure Bug Report

Users randomly could not log in: our Cognito post-authentication Lambda exceeded the 5-second limit

A few percent of logins failed with a generic error after we added a Cognito post-authentication trigger that recorded login events in our database. The Lambda ran in a VPC, opened a database connection on cold start, and sometimes exceeded Cognito's 5-second trigger timeout. How we moved the work off the login path.

Tomás Herrera Tomás Herrera @tomasherrera · July 23, 2026 · 7 min read · 30 Views
Banner: users could not log in because a Cognito post-authentication Lambda exceeded the 5-second limit

The security team asked for a record of every successful login: user, time, IP address and device details, stored in our own database for investigations and for a "recent activity" page. Cognito user pools can call a Lambda function after authentication, so we added a post-authentication trigger that wrote a row into PostgreSQL. It shipped on a Tuesday. By Thursday, support had a growing number of tickets from users who "sometimes" could not log in. Trying again usually worked.

The web app showed a generic error. The Cognito API response behind it:

UnexpectedLambdaException: PostAuthentication invocation failed due to error Socket timeout while
invoking Lambda function.

Lambda metrics and logs

The trigger's duration graph had two clusters: most invocations took 40 to 80 ms, and a smaller group took 4 to 7 seconds. The slow ones were cold starts:

INIT_START Runtime Version: nodejs:22...
REPORT RequestId: 1c2d... Duration: 3912.44 ms Billed Duration: 3913 ms Memory Size: 256 MB
  Init Duration: 1480.21 ms

Init plus duration was over five seconds. Cognito had stopped waiting, so the login failed, even though the Lambda eventually finished and wrote its row.

Where the time went

Tracing a cold start showed:

  • about 1.5 s of initialisation: runtime start, loading the database driver, fetching database credentials from Secrets Manager;
  • about 1.2 s to establish the first PostgreSQL connection with TLS;
  • about 1 s waiting for an RDS Proxy connection during peaks;
  • the insert itself, a few milliseconds.

What we ruled out

  • Cognito throttling. The error named the trigger invocation, and throttling errors have a different message.
  • Lambda concurrency limits. No throttles on the function; it had plenty of concurrency headroom.
  • Errors in the function code. Logs showed no exceptions; the function completed successfully, just too late.

Root cause

Cognito waits for Lambda triggers synchronously and gives each invocation 5 seconds. If the trigger does not respond in time, Cognito retries it up to three times in total for some triggers, and if those fail too, the authentication request fails. Our post-authentication trigger performed slow, non-essential work (secret retrieval, database connection, insert) inline, and on cold starts it regularly exceeded the limit, which turned an audit log feature into a login outage.

Triggers are on the critical path

Every Cognito Lambda trigger runs synchronously inside the authentication flow:

Trigger When Failure effect
Pre sign-up, pre authentication Before Cognito accepts the request Request rejected
Pre token generation Before tokens are issued Login fails
Post authentication After successful authentication Login fails, although credentials were valid
Post confirmation After sign-up confirmation Confirmation fails

Whatever runs in a trigger adds to login latency and can break login. Treat trigger code like code in a request handler with a strict 5-second budget, including cold starts.

Why VPC and database access make it worse

The function was attached to the VPC to reach the database. VPC attachment no longer adds the large cold start penalty it once did, but every dependency inside the trigger still adds latency: fetching secrets, DNS, TLS handshakes, connection pools that start empty on each new execution environment.

Environment

Component Version / value
Identity Cognito user pool, hosted UI and API logins
Trigger Post authentication, Node.js 22, 256 MB
Data store Amazon RDS for PostgreSQL behind RDS Proxy
Secrets Database credentials from Secrets Manager at init
Logins ~25,000 per day, morning peak ~40 per second

Make the trigger do almost nothing

The trigger now only extracts the event details and sends them to an SQS queue, then returns the event to Cognito. A separate consumer writes to the database at its own pace.

// post-auth trigger
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({});
const QUEUE_URL = process.env.LOGIN_EVENTS_QUEUE_URL;

export const handler = async (event) => {
  const record = {
    userSub: event.request.userAttributes.sub,
    username: event.userName,
    clientId: event.callerContext.clientId,
    sourceIp: event.request.userContextData?.sourceIp ?? null,
    at: new Date().toISOString(),
  };
  try {
    await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify(record) }));
  } catch (err) {
    console.error("login event not queued", { userSub: record.userSub, error: err.message });
    // do not fail the login because the audit pipeline is down
  }
  return event;
};

The function no longer runs in a VPC, has no database driver or secrets, and its cold start is around 300 ms. A small SQS call keeps it well under the limit. Errors are logged and alarmed but do not block the login.

The consumer

// login-events consumer, SQS event source mapping, batch size 50
export const handler = async (event) => {
  const rows = event.Records.map((r) => JSON.parse(r.body));
  await db.query(
    `INSERT INTO login_events (user_sub, username, client_id, source_ip, at)
     SELECT * FROM jsonb_to_recordset($1::jsonb)
       AS t(user_sub text, username text, client_id text, source_ip inet, at timestamptz)`,
    [JSON.stringify(rows.map((r) => ({ user_sub: r.userSub, username: r.username,
      client_id: r.clientId, source_ip: r.sourceIp, at: r.at })))]
  );
};

Batching 50 events per insert also reduced database load compared with one insert per login.

Consider whether you need the trigger at all

Cognito user pools with advanced security features can record authentication events with risk information, and CloudTrail records some user pool API calls. For our requirement, IP and client per login in our own database, the trigger plus queue was the simplest option, but it is worth checking built-in logging before adding code to the login path.

Alternatives and trade-offs

  • Provisioned concurrency on the original trigger would have removed cold starts, at a cost, while keeping a database dependency on every login. We preferred removing the dependency.
  • EventBridge instead of SQS for the hand-off, if several consumers need login events.

How to test the fix

A load test logs in 2,000 test users through the API in five minutes while forcing new execution environments by updating an environment variable on the trigger mid-test:

aws lambda update-function-configuration --function-name post-auth \
  --environment "Variables={LOGIN_EVENTS_QUEUE_URL=$QUEUE_URL,FORCE_COLD=$(date +%s)}"

Expected: zero UnexpectedLambdaException responses and p99 trigger duration below 1 second.

Confirming in production

After deployment, login failures attributable to the trigger dropped to zero. Trigger p99 duration was 180 ms warm and 420 ms on cold starts. The login events table kept up within a few seconds of real time, with occasional backlogs during peaks absorbed by the queue.

Regression guard

  • An alarm on the trigger's Duration p99 above 2,000 ms and on any Errors.
  • An alarm on the login events queue age, so the audit pipeline cannot silently fall behind.
  • A design rule documented for the team: Cognito triggers may not call databases or third-party APIs synchronously.

Security implications

Login records are personal data: IP addresses and user identifiers. The queue and table are encrypted, retention is 13 months, and access is limited to the security team and the activity page's read path. Not failing the login when the audit queue is unavailable is a deliberate trade-off between availability and completeness of the audit trail; the error alarm makes gaps visible so they can be reconstructed from Cognito and CloudTrail data.

Prevention checklist

  • Keep Cognito triggers fast, well under 5 seconds including cold starts.
  • Avoid database connections, secret fetches and third-party calls inside triggers.
  • Hand off non-essential work to a queue and process it asynchronously.
  • Decide explicitly whether a trigger failure should block login.
  • Test triggers under load with forced cold starts.

References

Add a label

#authentication #vpc #sqs #lambda #aws-cognito
Written by
Tomás Herrera @tomasherrera

I build and operate event-driven systems on AWS: Lambda, Step Functions, SQS, SNS, EventBridge and ECS on Fargate, mostly for e-commerce and logistics workloads where a stuck message means a stuck order. Serverless removes a lot of work, but it also hides limits until the day you hit them: payload sizes, trigger timeouts, scale-in behaviour, resource policies that silently drop events. I write down each one we hit, with the exact error, the configuration that caused it and the change that fixed it, so you can find the limit in a blog post instead of an incident channel.

Advertisement

Comments

0

No comments yet. Be the first to comment!

Sign In to post a comment.