DuckDB read S3 Parquet files locally but got 403 in a container: secrets and the credential chain

We replaced a heavy Spark job with DuckDB for a daily revenue report: a single SQL query over a few hundred Parquet files in S3, written by our event pipeline. On a developer laptop, it ran in under a minute. Deployed as an ECS Fargate scheduled task, it failed immediately:

duckdb.duckdb.HTTPException: HTTP Error: HTTP GET error on
'https://acme-events.s3.amazonaws.com/events/date=2026-09-01/part-0001.parquet' (HTTP 403)

The task had an IAM role with read access to the bucket. The AWS CLI, run in the same container image as the same task role, could list and download the files.

What was different locally

The job's code:

import duckdb

con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs;")
con.execute("SET s3_region = 'eu-west-1';")
df = con.execute("""
    SELECT date_trunc('day', ts) AS day, sum(amount) AS revenue
    FROM read_parquet('s3://acme-events/events/date=2026-09-*/*.parquet', hive_partitioning = true)
    GROUP BY 1 ORDER BY 1
""").df()

No credentials anywhere. On the laptop, a teammate had added an S3 secret to their DuckDB setup months earlier, created from their personal SSO profile and stored persistently in DuckDB's local secrets directory:

ls ~/.duckdb/stored_secrets/
# my_s3.duckdb_secret

The laptop used that. In the container, there was no secret, and DuckDB's httpfs did not automatically use the ECS task role, so requests went out unsigned or with no valid credentials. S3 answered 403.

Confirming

print(con.execute("FROM duckdb_secrets()").fetchall())
# [] in the container

An empty list: DuckDB had no credentials to use.

What we ruled out

• Bucket policy restrictions. The AWS CLI with the same role worked.
• VPC endpoint policies. Same; the CLI used the same network path.
• Wrong region. The region was set explicitly; a wrong region usually produces a redirect or a different error.

Root cause

DuckDB does not read the AWS default credential chain automatically for s3:// URLs. It uses secrets created with CREATE SECRET, or legacy SET s3_access_key_id settings. Locally, a persistent secret from a personal profile made the query work; in the container, no secret existed, so DuckDB sent requests without credentials for the task role, and S3 denied them.

DuckDB secrets

Since DuckDB 0.10, credentials for remote storage are managed as secrets:

CREATE SECRET name (TYPE s3, PROVIDER config, KEY_ID '...', SECRET '...', REGION '...');
CREATE SECRET name (TYPE s3, PROVIDER credential_chain);

• PROVIDER config takes explicit keys.
• PROVIDER credential_chain uses the AWS SDK's credential chain through the aws extension: environment variables, shared config and SSO profiles, web identity, ECS container credentials, EC2 instance metadata.

Secrets can be temporary (for the connection) or PERSISTENT, stored on disk in the DuckDB secrets directory. Persistent secrets explain "it works on my machine" surprises like ours.

Secrets can also be scoped to a path prefix, so different buckets use different credentials.

Who is affected

DuckDB jobs that move from laptops to containers, serverless functions or CI, and anyone relying on persistent secrets or environment variables set up interactively. Python data tools that wrap DuckDB inherit the same behaviour.

Environment

Use the credential chain explicitly

con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs; INSTALL aws; LOAD aws;")
con.execute("""
    CREATE OR REPLACE SECRET events_s3 (
        TYPE s3,
        PROVIDER credential_chain,
        REGION 'eu-west-1',
        SCOPE 's3://acme-events'
    );
""")

In the ECS task, the credential chain finds the container credentials endpoint provided by ECS and uses the task role. On laptops, it uses the developer's current SSO session, so nobody needs a persistent secret. After this change, the job ran in the container, then failed with a different error:

HTTP Error: HTTP GET error on '.../part-0001.parquet' (HTTP 403) ... KMS.AccessDeniedException

Allow KMS decrypt for the task role

The Parquet files were encrypted with SSE-KMS using a customer managed key. Reading them requires kms:Decrypt on that key, which the CLI test had not exercised because it only listed objects. We added it to the task role:

{
  "Effect": "Allow",
  "Action": "kms:Decrypt",
  "Resource": "arn:aws:kms:eu-west-1:111122223333:key/3e1f...",
  "Condition": { "StringEquals": { "kms:ViaService": "s3.eu-west-1.amazonaws.com" } }
}

The kms:ViaService condition limits use of the key to requests made through S3.

Remove persistent personal secrets

Developers deleted their persistent secrets and use the credential chain with their SSO profile. A persistent secret with personal credentials on a laptop is a long-lived credential in a file; the SSO session expires on its own.

DROP PERSISTENT SECRET my_s3;

Partition pruning while we were there

With credentials working, the query read only the files it needed, because hive_partitioning = true lets DuckDB skip partitions that do not match a filter on the partition column. We made the date filter explicit on the partition column rather than relying on the glob alone:

SELECT date_trunc('day', ts) AS day, sum(amount) AS revenue
FROM read_parquet('s3://acme-events/events/*/*.parquet', hive_partitioning = true)
WHERE date BETWEEN '2026-09-01' AND '2026-09-30'
GROUP BY 1 ORDER BY 1;

EXPLAIN shows the filter pushed into the Parquet scan, and S3 request counts dropped accordingly.

Alternatives and trade-offs

• Pass credentials as environment variables and use PROVIDER config with values from the environment. Works, but means fetching and rotating keys yourself; the credential chain does that.
• Download files first with the AWS SDK, then query locally. Simpler permissions, more storage and time.

How to test the fix

Run the job locally with the same role the task uses, through aws sts assume-role or a profile:

AWS_PROFILE=report-task-role python report.py --date 2026-09-01

And in the container, print the secret and run a minimal query first:

print(con.execute("SELECT name, provider, scope FROM duckdb_secrets()").fetchall())
con.execute("SELECT count(*) FROM read_parquet('s3://acme-events/events/date=2026-09-01/*.parquet')").fetchone()

Confirming in production

The scheduled task completed in 48 seconds, reading 312 Parquet files (about 9 GB) with column pruning and partition filtering. CloudTrail showed GetObject calls from the task role and Decrypt calls on the KMS key through S3.

Regression guard

• The report job runs in CI inside the production container image, with a role that has the same permissions, against a small sample prefix.
• A lint check on the repository rejects PERSISTENT secrets and hard-coded KEY_ID values in job code.

Security implications

Personal credentials made a production job's code appear correct and could have been copied into an image by accident. The credential chain uses short-lived role credentials everywhere. The task role is limited to s3:GetObject and s3:ListBucket on one prefix and kms:Decrypt via S3, and the report contains only aggregates.

Prevention checklist

• Create DuckDB S3 secrets explicitly with PROVIDER credential_chain in job code.
• Do not rely on persistent secrets or interactive setup on laptops.
• Grant kms:Decrypt for SSE-KMS objects, limited with kms:ViaService.
• Test jobs in the production image with the production role before deploying.
• Scope secrets to the bucket or prefix they are for.

References

• DuckDB, Secrets manager: https://duckdb.org/docs/stable/configuration/secrets_manager
• DuckDB, S3 API support in httpfs: https://duckdb.org/docs/stable/core_extensions/httpfs/s3api
• DuckDB, AWS extension: https://duckdb.org/docs/stable/core_extensions/aws
• Amazon S3, Protecting data with SSE-KMS: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html