Lambda "No space left on device" on warm invocations: /tmp is reused between requests

The media service has a Lambda function triggered when a user uploads a video to S3. It downloads the video, uses ffmpeg (from a Lambda layer) to extract three thumbnail frames, and uploads them. In testing it was flawless. In production, error rates rose with traffic, and failures followed a pattern: an execution environment handled a few uploads successfully and then failed on every subsequent one.

[ERROR] OSError: [Errno 28] No space left on device: '/tmp/9f2c1e7a-upload.mp4'
Traceback (most recent call last):
  File "/var/task/handler.py", line 31, in handler
    s3.download_file(bucket, key, local_path)

Measuring /tmp

We added a log line at the start of each invocation with the disk usage of /tmp and the request id:

import shutil
usage = shutil.disk_usage("/tmp")
print(f"tmp_used_mb={usage.used // 2**20} tmp_free_mb={usage.free // 2**20}")

The logs for one log stream, which corresponds to one execution environment, told the story:

START RequestId: 0a1b...   tmp_used_mb=0    tmp_free_mb=512
START RequestId: 3c4d...   tmp_used_mb=118  tmp_free_mb=394
START RequestId: 5e6f...   tmp_used_mb=241  tmp_free_mb=271
START RequestId: 7a8b...   tmp_used_mb=402  tmp_free_mb=110
START RequestId: 9c0d...   tmp_used_mb=402  tmp_free_mb=110
[ERROR] OSError: [Errno 28] No space left on device

Each invocation left its video and thumbnails behind. Usage only grew.

The code

def handler(event, context):
    record = event["Records"][0]["s3"]
    bucket, key = record["bucket"]["name"], record["object"]["key"]
    local_path = f"/tmp/{uuid.uuid4()}-upload.mp4"
    s3.download_file(bucket, key, local_path)

    thumbs = []
    for i, ts in enumerate(["00:00:01", "00:00:05", "00:00:10"]):
        out = f"/tmp/{uuid.uuid4()}-thumb-{i}.jpg"
        subprocess.run(["/opt/bin/ffmpeg", "-ss", ts, "-i", local_path, "-frames:v", "1", out], check=True)
        thumbs.append(out)

    for i, path in enumerate(thumbs):
        s3.upload_file(path, OUTPUT_BUCKET, f"{key}/thumb-{i}.jpg")
    return {"thumbnails": len(thumbs)}

No file was ever deleted, and unique names guaranteed that nothing was overwritten.

What we ruled out

• One huge video. The failing invocations were for small files too; the disk was already nearly full when they started.
• The layer or package size. Code and layers are not stored in /tmp.
• Memory. /tmp is separate from the function's memory; the functions used 400 MB of their 1,536 MB.

Root cause

Lambda reuses execution environments for subsequent invocations, and the contents of /tmp persist with the environment. The function wrote every input and output file to /tmp with unique names and never removed them. With the default 512 MB of ephemeral storage, a warm environment filled up after a handful of videos and then failed every request until Lambda recycled it.

The execution environment lifecycle

When Lambda handles a request, it either creates a new execution environment (a cold start) or reuses an existing, idle one (a warm start). Between invocations, the environment is frozen, not reset. Global variables, open connections and files in /tmp are all still there for the next invocation. This is a feature: it lets you cache data such as models or reference files across invocations. It also means anything you write is your responsibility to clean up.

Environments are eventually recycled, but there is no guaranteed lifetime, and under steady traffic an environment can live for hours.

Ephemeral storage size

/tmp defaults to 512 MB and can be configured from 512 MB up to 10,240 MB per function. Extra storage above 512 MB is billed by GB-second, like memory.

Environment

Step 1: always clean up

Use a temporary directory per invocation and remove it whatever happens:

import tempfile

def handler(event, context):
    record = event["Records"][0]["s3"]
    bucket, key = record["bucket"]["name"], record["object"]["key"]

    with tempfile.TemporaryDirectory(dir="/tmp") as workdir:
        local_path = os.path.join(workdir, "upload.mp4")
        s3.download_file(bucket, key, local_path)

        for i, ts in enumerate(["00:00:01", "00:00:05", "00:00:10"]):
            out = os.path.join(workdir, f"thumb-{i}.jpg")
            subprocess.run(["/opt/bin/ffmpeg", "-ss", ts, "-i", local_path, "-frames:v", "1", out],
                           check=True, timeout=60)
            s3.upload_file(out, OUTPUT_BUCKET, f"{key}/thumb-{i}.jpg")

    return {"thumbnails": 3}

TemporaryDirectory deletes the directory and its contents when the block exits, including on exceptions. That alone fixed the failures.

As a safety net against a crash that skips cleanup, such as a timeout that kills the invocation mid-block, the handler also removes stale directories older than ten minutes at the start of each invocation.

Step 2: size ephemeral storage for the worst case

Our largest allowed upload is 2 GB. With cleanup, an invocation needs the input file plus small thumbnails, so we set ephemeral storage to 3 GB to leave headroom:

# AWS SAM
ThumbnailFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: handler.handler
    Runtime: python3.12
    MemorySize: 1536
    EphemeralStorage:
      Size: 3072
    Timeout: 120

Step 3: avoid downloading when you can stream

ffmpeg can read from a URL. A pre-signed S3 URL lets it fetch only the parts of the file it needs for the requested timestamps, instead of downloading the whole video:

url = s3.generate_presigned_url("get_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
subprocess.run(["/opt/bin/ffmpeg", "-ss", ts, "-i", url, "-frames:v", "1", out], check=True, timeout=60)

For MP4 files with the index at the start, this reduced both storage and duration. Files with the index at the end still required more reading, so we kept the download path as a fallback.

Alternatives and trade-offs

• Lambda with an EFS mount for very large working sets shared across invocations. More setup, more latency, and EFS costs.
• ECS Fargate tasks or AWS Batch for long video processing, where Lambda's 15-minute limit and storage sizes become constraints.
• Mountpoint-style S3 access from compute that supports it, to read objects as files without copying.

How to test the fix

Invoke the function repeatedly against the same warm environment and check free space:

for i in $(seq 1 30); do
  aws lambda invoke --function-name thumbnail --payload fileb://events/s3-put-150mb.json /dev/null > /dev/null
done
aws logs filter-log-events --log-group-name /aws/lambda/thumbnail --filter-pattern tmp_used_mb --limit 30 \
  --query 'events[].message' --output text | tr '\t' '\n' | sort | uniq -c
# every invocation: tmp_used_mb=0 at start

Confirming in production

After the deploy, tmp_used_mb at invocation start was 0 for every log line, and the error rate dropped to zero. Average duration fell by 30% for files that could use the streaming path.

Regression guard

• The tmp_used_mb log line became a CloudWatch metric via a metric filter, with an alarm above 50 MB at invocation start.
• A code review checklist item for Lambda: every file written to /tmp must be inside a per-invocation temporary directory or a deliberate cache with a size limit.

Security implications

Files left in /tmp are visible to later invocations in the same environment. For a single-tenant function processing the same kind of data that is mostly a reliability problem, but for functions that handle different customers' data, leftover files could be read or mixed up by code processing another customer's request. Per-invocation directories and cleanup prevent that. Pre-signed URLs used for streaming expire after five minutes and are never logged.

Prevention checklist

• Treat /tmp as shared across warm invocations; clean up every file you create.
• Use tempfile.TemporaryDirectory or equivalent per invocation.
• Size EphemeralStorage for the largest input you accept.
• Stream from S3 with pre-signed URLs where tools support it.
• Log and alarm on /tmp usage at invocation start.

References

• AWS Lambda execution environment lifecycle: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html
• Configuring ephemeral storage for Lambda functions: https://docs.aws.amazon.com/lambda/latest/dg/configuration-ephemeral-storage.html
• Python tempfile module: https://docs.python.org/3/library/tempfile.html
• Amazon S3 presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html