Passwordless RDS IAM authentication failed after 15 minutes: the token was used as a static password

As part of removing long-lived secrets, we switched the accounts service on EKS from a PostgreSQL password stored in a Kubernetes secret to RDS IAM database authentication: the pod's IAM role (through EKS Pod Identity) generates a short-lived token, and the database accepts it instead of a password. Deployment went fine. Fifteen to twenty minutes later, error rates climbed:

psycopg.OperationalError: connection failed: connection to server at "accounts.cluster-c1x2y3.eu-west-1.rds.amazonaws.com"
(10.20.40.12), port 5432 failed: FATAL:  PAM authentication failed for user "accounts_app"

Existing connections kept working; new connections failed. Restarting the pods fixed it for another fifteen minutes.

The connection code

import boto3
from psycopg_pool import ConnectionPool

rds = boto3.client("rds", region_name="eu-west-1")
token = rds.generate_db_auth_token(DBHostname=HOST, Port=5432, DBUsername="accounts_app")

pool = ConnectionPool(
    conninfo=f"host={HOST} port=5432 dbname=accounts user=accounts_app password={token} sslmode=verify-full "
             f"sslrootcert=/etc/ssl/rds-global-bundle.pem",
    min_size=4, max_size=20,
)

The token was generated once, at import time, and baked into the pool's connection string. Every new connection the pool opened later used the same token.

Token lifetime

An RDS IAM authentication token is a signed request, valid for 15 minutes. The database checks the signature and expiry when a connection is established. Connections that are already open stay open after the token expires. So the pool's first four connections lived on, but any connection opened after 15 minutes (pool growth under load, replacement of connections closed by max_lifetime, reconnects after a network blip) presented an expired token.

What we ruled out

• IAM permissions. The first connections succeeded, so rds-db:connect was granted correctly.
• The database user setup. GRANT rds_iam TO accounts_app was in place.
• Pod Identity credentials expiring. The SDK refreshes role credentials automatically; the problem was the token derived from them, not the credentials.

Root cause

The application generated a single IAM authentication token at startup and used it as a static password for the life of the process. Tokens are valid for 15 minutes, so every connection opened after that time failed authentication.

How RDS IAM authentication works

1. The application calls generate_db_auth_token. This happens locally: the SDK signs a request with the pod's IAM credentials (SigV4). No network call is required.
2. The application connects to the database over TLS, sending the token as the password.
3. RDS validates the signature with IAM and checks that the role has rds-db:connect for that database user and resource.
4. The token is valid for 15 minutes from generation, for establishing connections.

Important limits: connections must use TLS, and AWS documents a limit on the rate of new IAM-authenticated connections per instance, so connection storms are worth avoiding; a pool helps.

Why pooling interacts with token lifetime

Pools open connections at different times: at startup, when demand grows, and to replace connections that reached max_lifetime or failed. Each of those needs a valid token at that moment. A static connection string cannot provide that.

Who is affected

Any application using RDS or Aurora IAM authentication with a connection pool that receives the token once: connection strings built at startup, framework settings files, environment variables populated by an init container, and ORM configurations that do not support a password callback.

Environment

Generate a token for every new connection

psycopg_pool creates every connection through the connect() class method of its connection_class. Overriding it gives a connection factory that fetches a fresh token each time:

import boto3, psycopg
from psycopg_pool import ConnectionPool

rds = boto3.client("rds", region_name="eu-west-1")

class IamAuthConnection(psycopg.Connection):
    @classmethod
    def connect(cls, conninfo: str = "", **kwargs):
        kwargs["password"] = rds.generate_db_auth_token(
            DBHostname=HOST, Port=5432, DBUsername="accounts_app")
        return super().connect(conninfo, **kwargs)

pool = ConnectionPool(
    conninfo=f"host={HOST} port=5432 dbname=accounts user=accounts_app sslmode=verify-full "
             f"sslrootcert=/etc/ssl/rds-global-bundle.pem",
    connection_class=IamAuthConnection,
    min_size=4, max_size=20,
    max_lifetime=10 * 60,       # recycle connections well within typical maintenance windows
)

Token generation is a local signing operation that takes well under a millisecond, so doing it per new connection is cheap.

Equivalent patterns elsewhere

• SQLAlchemy: listen to the do_connect event and set cparams["password"] to a fresh token.
• Java with HikariCP: the AWS Advanced JDBC Wrapper's IAM plugin, or a custom DataSource that generates the token in getConnection().
• Node.js with pg: the password option of pg.Pool can be an async function that returns a fresh token.

const pool = new Pool({
  host: HOST, user: 'accounts_app', database: 'accounts',
  ssl: { ca: rdsCaBundle },
  password: async () => signer.getAuthToken(),   // @aws-sdk/rds-signer
});

Keep the pool small and stable

With six pods and a maximum of 20 connections each, the service could open 120 connections. We reduced the maximum to 10 per pod after measuring actual concurrency, and use RDS Proxy for the services with the most pods, which also supports IAM authentication between clients and the proxy.

Failover behaviour

Aurora failovers close every connection at once, so every pod reconnects within seconds. With per-connection tokens, that works, but it creates a burst of new IAM-authenticated connections. Spreading reconnects with the pool's built-in backoff and keeping pools small kept the burst well under the documented connection rate limit in our failover tests.

Alternatives and trade-offs

• Secrets Manager with rotation instead of IAM auth: longer-lived passwords with automatic rotation. Simpler for some drivers, but a secret still exists.
• RDS Proxy in front of the database, with IAM authentication from the application to the proxy. Adds a component, reduces connection churn on the database.

How to test the fix

# staging: force new connections after the 15-minute window
kubectl exec -n accounts deploy/accounts -- python -c "
import time; from app.db import pool
time.sleep(17*60)
with pool.connection() as c: print(c.execute('select current_user').fetchone())"

A unit test asserts that two connections created a few seconds apart use different tokens, by mocking generate_db_auth_token.

Confirming in production

Over the next week, no authentication failures appeared, including during traffic peaks when pools grew, and after Aurora failover tests that dropped every connection at once and forced reconnects. CloudWatch showed the expected rate of new IAM-authenticated connections, well under the documented limit.

Regression guard

• An integration test in CI runs the service against a test database with IAM auth and a token lifetime check, opening connections after an artificial clock advance.
• An alert on FATAL: PAM authentication failed in database logs.

Security implications

IAM authentication removes long-lived database passwords from Kubernetes secrets and environment variables, and access can be revoked by changing the IAM policy. Tokens are short-lived bearer credentials, so they must never be logged; our logging redacts the password field of connection parameters. TLS with verify-full ensures tokens are only sent to the real database endpoint.

Prevention checklist

• Treat IAM auth tokens as expiring credentials, never as static passwords.
• Generate a token in the pool's connection factory for every new connection.
• Use TLS with certificate verification for IAM-authenticated connections.
• Keep pools small, or use RDS Proxy, to limit connection churn.
• Test connections opened after the 15-minute window.

References

• Amazon RDS, IAM database authentication: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html
• Amazon RDS, Connecting with IAM authentication and the AWS SDK for Python: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.Python.html
• psycopg 3, Connection pools: https://www.psycopg.org/psycopg3/docs/advanced/pool.html
• Amazon RDS Proxy, IAM authentication: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-setup.html