Overlapping cron runs corrupted our export file: fixing it with flock and atomic rename

Every five minutes a cron job on one of our Linux hosts exports open orders to a CSV file that a logistics partner picks up over SFTP. It had run unattended for two years. On a Thursday in July the partner's import started failing several times an hour with:

IMPORT FAILED partners.csv line 48213: expected 9 fields, got 14
IMPORT FAILED partners.csv: unexpected end of file inside quoted field at line 51077

Looking at the file the partner rejected, line 48213 was half of one order row glued to the start of another. The file was also shorter than it should have been, and the header row appeared twice: once at the top and once in the middle.

Reproducing it

The cron entry and script were simple:

*/5 * * * * deploy /opt/exports/export-orders.sh >> /var/log/exports/orders.log 2>&1

#!/bin/bash
# export-orders.sh (before)
psql "$EXPORT_DSN" -c "\copy (SELECT * FROM partner_open_orders) TO STDOUT WITH CSV HEADER" \
  > /srv/sftp/partner/partners.csv

We confirmed the overlap from the job log, which prints start and end lines from a wrapper:

2025-07-17T10:05:01Z start pid=40112
2025-07-17T10:10:01Z start pid=40877
2025-07-17T10:11:12Z end   pid=40112 rows=51203
2025-07-17T10:16:20Z end   pid=40877 rows=51244

Runs had grown from about 90 seconds to over six minutes as the order table grew and the view behind the export picked up two new joins. Locally, running the script twice in parallel against a copy of production data produced the same broken file almost every time.

What we ruled out

• Partner download during write. That was a real second problem (more below), but it does not explain duplicate headers inside the file.
• Database returning bad rows. Running the query once in isolation produced a clean file.
• Disk errors. dmesg and SMART data were clean, and the corruption appeared at different offsets every time.

Root cause

cron starts a job on schedule whether or not the previous run has finished. When a run took longer than five minutes, two psql processes wrote to the same path. The second run's > truncated the file to zero while the first was still writing, then both processes wrote at their own file offsets. The first process continued at its large offset, leaving the region before it to be overwritten by the second, so the file ended up as a mixture of two exports with a header in the middle. Separately, because the script wrote the final file in place, the partner sometimes downloaded it while it was still being written and got a truncated copy.

What is happening under the hood

> in the shell opens the file with O_TRUNC. Each process gets its own open file description with its own offset. Truncation by one process does not reset the other's offset, so the first writer keeps writing at, say, byte 4,000,000 in a file that was just truncated to zero. The kernel fills the gap with zeros or with whatever the second writer puts there. There is no error, no warning, and the final size depends on which writer finishes last.

cron has no notion of job state. Vixie cron and cronie both fork a new process at each matching minute. Nothing in the crontab format prevents overlap, so serialisation must come from the job itself.

flock(1) from util-linux takes an advisory lock on a file descriptor using the flock(2) system call. The lock belongs to the open file description and is released automatically when the process holding it exits, even after kill -9 or a crash. That is the key advantage over PID files, which go stale when a process dies without cleaning up.

Environment

Minimal reproduction

#!/bin/bash
# overlap.sh - shows two writers interleaving into one file
out=/tmp/overlap.csv
writer() {
  { echo "id,label"; for i in $(seq 1 200000); do echo "$i,$1"; done; } > "$out"
}
writer A & sleep 0.2; writer B & wait
grep -c '^id,label' "$out"     # often 2
awk -F, 'NF != 2' "$out" | head

Who is affected

Any scheduled job whose runtime can exceed its interval and whose output is shared: exports, report generation, rsync jobs, backup scripts, log rotation helpers. It hits Kubernetes CronJobs too unless concurrencyPolicy: Forbid is set.

The fix

Three changes: only one run at a time, a hard bound on runtime, and an atomic publish so readers only ever see complete files.

*/5 * * * * deploy /usr/bin/flock -n /run/lock/export-orders.lock /usr/bin/timeout --kill-after=30s 280s /opt/exports/export-orders.sh >> /var/log/exports/orders.log 2>&1

• flock -n tries to take the lock and exits immediately with status 1 if another run holds it. The skipped run is logged by the wrapper and the next scheduled run tries again.
• timeout 280s sends SIGTERM if the export runs more than 280 seconds, and --kill-after=30s follows with SIGKILL if it ignores SIGTERM. A hung run can no longer block every future run.
• /run/lock is a tmpfs on Ubuntu, so the lock file is recreated on boot and never accumulates stale state. The lock is held by flock for as long as its child runs, and released when the child exits.

The script itself now writes to a temporary file in the same directory tree and renames it into place:

#!/bin/bash
# export-orders.sh (after)
set -euo pipefail

dest_dir=/srv/sftp/partner
staging=/srv/sftp/.staging
mkdir -p "$staging"
tmp=$(mktemp "$staging/partners.XXXXXX.csv")
trap 'rm -f "$tmp"' EXIT

echo "$(date -u +%FT%TZ) start pid=$$"
psql "$EXPORT_DSN" -v ON_ERROR_STOP=1 \
  -c "\copy (SELECT * FROM partner_open_orders ORDER BY order_id) TO STDOUT WITH CSV HEADER" \
  > "$tmp"

rows=$(( $(wc -l < "$tmp") - 1 ))
if (( rows < 1000 )); then
  echo "refusing to publish: only $rows rows" >&2
  exit 1
fi

chmod 0640 "$tmp"
sync -f "$tmp"
mv -f "$tmp" "$dest_dir/partners.csv"
trap - EXIT
echo "$(date -u +%FT%TZ) end pid=$$ rows=$rows"

mv within one filesystem is a rename(2), which atomically replaces the directory entry. A reader that already opened the old file keeps reading the old, complete content; any new open sees the new, complete file. The staging directory sits outside the partner's chroot so they never see temp files, but on the same filesystem so the rename stays atomic.

set -euo pipefail and ON_ERROR_STOP=1 make a failed query fail the script, so a half-written temp file is deleted instead of published. The row-count floor catches the case where the query succeeds but returns nothing because of an upstream problem.

Alternatives and trade-offs

• Locking inside the script. exec 9>/run/lock/export-orders.lock; flock -n 9 || exit 0 at the top of the script keeps the crontab clean. We prefer the crontab form because the lock then covers the whole command, including anything a future edit adds before the lock line.
• Waiting instead of skipping. flock -w 60 waits up to a minute for the lock. Use it when every run must happen; for an export of current state, skipping is correct.
• systemd timers. A timer triggering a Type=oneshot service will not start a second instance while the first is active, and RuntimeMaxSec= bounds it. Logs go to the journal. We moved most new jobs to timers; this one stayed on cron for now.
• Making the job faster. We also added an index that brought the export back to 70 seconds. That alone would have hidden the bug until the next slowdown.

How to test the fix

# two concurrent runs: second must exit 1 immediately
/usr/bin/flock -n /run/lock/export-orders.lock sleep 30 & sleep 1
/usr/bin/flock -n /run/lock/export-orders.lock true; echo "exit=$?"   # exit=1

# file is always complete and has one header
bash -c 'for i in 1 2 3; do /opt/exports/export-orders.sh & done; wait'
grep -c '^order_id,' /srv/sftp/partner/partners.csv                  # 1
python3 -c "import csv,sys; r=list(csv.reader(open(sys.argv[1]))); assert len({len(x) for x in r})==1" /srv/sftp/partner/partners.csv

The second test intentionally runs the script without flock three times at once: the atomic rename alone guarantees a complete file, with the last finisher winning.

Confirming in production

Since the change, the job log shows a clear pattern on slow days: occasional flock skips (exit status 1 recorded by our wrapper) and never two start lines without an end between them. The partner has reported zero import failures in the four weeks since. We check overlap with:

awk '/ start /{s++} / end /{e++} END{print "starts="s, "ends="e}' /var/log/exports/orders.log
grep -c 'refusing to publish' /var/log/exports/orders.log

Regression guard

• A lint script in the config repo flags any crontab line without flock for jobs that write under /srv.
• An alert fires if partners.csv is older than 15 minutes, which catches both repeated skips and timeouts.
• The concurrency test above runs in CI against a container with a seeded database.

Security implications

None beyond integrity and availability of the exported data, but two details matter. Lock files belong in /run/lock or another directory not writable by other users; a lock in a world-writable path like /tmp can be pre-created or held by another user to block the job. mktemp creates the staging file with a random name and mode 0600, so other local users cannot predict or read it before the chmod and rename. The partner's SFTP chroot does not include the staging directory.

Prevention checklist

• Wrap every cron job that can outlive its interval in flock -n.
• Bound runtime with timeout so one hung run cannot block all others.
• Write to a temp file on the same filesystem and publish with mv.
• Fail loudly on partial results with set -euo pipefail and sanity checks.
• Alert on output freshness, not only on job errors.

References

• util-linux, flock(1) manual: https://man7.org/linux/man-pages/man1/flock.1.html
• Linux man-pages, flock(2): https://man7.org/linux/man-pages/man2/flock.2.html
• Linux man-pages, rename(2): https://man7.org/linux/man-pages/man2/rename.2.html
• GNU coreutils, timeout invocation: https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html
• systemd.timer(5) manual: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html