Bash deploy script kept going after a failed backup: set -euo pipefail and ERR traps

A Sunday release of our invoicing service ended with half the API returning 500s. The deploy log showed the problem in plain sight, about 40 lines before the end:

==> Backing up database
pg_dump: error: connection to server at "db.example.com" (198.51.100.20), port 5432 failed: FATAL:  password authentication failed for user "deploy"
==> Backup written: /var/backups/invoicing/pre-2025-04-20T0210.sql.gz
==> Running migrations
ERROR:  column "tax_region" of relation "invoices" already exists
==> Switching current -> releases/2025-04-20T0210
==> Restarting invoicing-api
==> Deploy finished

The backup failed. The script announced that the backup was written anyway. A migration then failed half-way, and the script still switched the symlink to the new release and restarted the service. The new code expected a schema that only half existed. The "backup" file was 20 bytes: an empty gzip stream.

Reproducing it

A cut-down version of the script reproduces both failures on any machine:

#!/usr/bin/env bash
echo "==> Backing up database"
false | gzip > /tmp/backup.sql.gz
echo "==> Backup written ($(stat -c %s /tmp/backup.sql.gz) bytes)"
echo "==> Running migrations"
ls /does/not/exist
echo "==> Deploy finished"

Every line runs, and the script exits 0 because its last command succeeded.

What we ruled out

• CI swallowing the exit code. The CI step ran the script directly; the script really did exit 0.
• The password rotation alone. The rotated database password explained the backup failure, but not why the script continued. A failing step should have stopped the deploy.
• "We already use set -e". The old single-file script did have set -e. When it was split into a wrapper and a main script, the option stayed in the wrapper, which calls bash main.sh. That starts a new process, and shell options are not inherited by it. Even where it applied, it would not have caught the pipe.

Root cause

Two Bash defaults combined:

1. Without errexit (set -e), Bash runs the next command no matter how the previous one exited. Our main script never enabled it.
2. A pipeline's exit status is the status of its last command. pg_dump ... | gzip > file returns gzip's status, which was 0 because gzip happily compressed empty input. Even with set -e, this line would have passed. That needs pipefail.

What is happening under the hood

Bash is designed for interactive use, where continuing after an error is the right behaviour. Scripts inherit that. The three options people call "strict mode" change it:

• set -e (errexit): exit when a simple command, pipeline or list exits non-zero, with a long list of exceptions.
• set -u (nounset): treat expansion of an unset variable as an error.
• set -o pipefail: a pipeline's status is that of the last command that failed, or zero if all succeeded.

The exceptions to set -e are where people get hurt. It is ignored:

• in the condition of if, while, until, and in any command before && or || except the last;
• for any command negated with !;
• inside a function, if that function was called from one of those contexts, for the whole function body;
• in command substitution $(...), unless shopt -s inherit_errexit is set (Bash 4.4 and later);
• and the status of $(...) is lost when combined with local or export on the same line, because those builtins return their own status.

Environment

Minimal reproduction

The subtle cases, each of which prints "still here" even with set -e enabled:

#!/usr/bin/env bash
set -e

check() { false; echo "still here (function in if)"; }
if check; then :; fi

f() { local out=$(false); echo "still here (local masks status)"; }
f

false | true
echo "still here (pipe without pipefail)"

out=$(false; echo "still here (subshell without inherit_errexit)")
echo "$out"

Who is affected

Every Bash script that does more than one thing and assumes it stops on errors. Deploy, backup and migration scripts are the dangerous ones, because continuing after a failed step turns an error into data loss or an outage.

The fix

We rewrote the deploy script's header and structure. The full skeleton:

#!/usr/bin/env bash
set -Eeuo pipefail
shopt -s inherit_errexit

readonly RELEASE="${1:?usage: deploy.sh <release-dir>}"
readonly APP_ROOT=/srv/invoicing
readonly BACKUP="/var/backups/invoicing/pre-$(date -u +%Y-%m-%dT%H%M).sql.gz"
previous=""

on_error() {
  local status=$? line=$1
  echo "!! deploy failed at line ${line} (exit ${status}): ${BASH_COMMAND}" >&2
  if [[ -n "$previous" ]]; then
    ln -sfn "$previous" "$APP_ROOT/current"
    systemctl restart invoicing-api || true
    echo "!! rolled back to ${previous}" >&2
  fi
  exit "$status"
}
trap 'on_error $LINENO' ERR

cleanup() { rm -f "${tmpfile:-}"; }
trap cleanup EXIT

backup() {
  pg_dump --format=plain --no-owner invoicing | gzip > "$BACKUP"
  # Do not trust size alone: verify the archive and the dump footer
  gzip -t "$BACKUP"
  local footer
  footer=$(zcat "$BACKUP" | tail -n 5)
  [[ "$footer" == *"PostgreSQL database dump complete"* ]]
}

main() {
  echo "==> Backing up database"
  backup
  echo "==> Running migrations"
  psql --set ON_ERROR_STOP=1 --single-transaction -f "$APP_ROOT/releases/$RELEASE/migrate.sql" invoicing
  previous=$(readlink -f "$APP_ROOT/current")
  echo "==> Switching current -> releases/$RELEASE"
  ln -sfn "$APP_ROOT/releases/$RELEASE" "$APP_ROOT/current"
  systemctl restart invoicing-api
  curl --fail --silent --retry 5 --retry-delay 2 --retry-all-errors http://127.0.0.1:8080/healthz > /dev/null
  echo "==> Deploy finished"
}

main "$@"

What each part does:

• set -Eeuo pipefail: errexit, nounset, pipefail, and -E (errtrace) so the ERR trap also fires inside functions and subshells.
• shopt -s inherit_errexit: command substitutions keep set -e.
• ${1:?usage...}: fails with a message if the argument is missing instead of deploying an empty path.
• The ERR trap reports the failing line and command, and rolls back the symlink only if we already switched it.
• The EXIT trap cleans up temporary files on any exit, success or failure.
• psql --set ON_ERROR_STOP=1 --single-transaction makes psql itself exit non-zero on the first SQL error and rolls back the whole migration file, so a migration cannot be half applied.
• The backup is verified for content, not just existence.

Patterns we changed throughout the codebase

- local version=$(cat VERSION)
+ local version
+ version=$(cat VERSION)

- cd "$dir"
+ cd "$dir" || exit 1          # explicit, also clear to readers

- ((retries++))
+ retries=$((retries + 1))     # ((x++)) returns 1 when x was 0 and trips set -e

- rm -rf "$BUILD_DIR/"*
+ rm -rf "${BUILD_DIR:?}/"*    # never expands to rm -rf /*

And where a non-zero exit is expected, we say so explicitly instead of relying on exceptions:

if ! grep -q "^${user}:" /etc/passwd; then
  useradd --system "$user"
fi
matches=$(grep -c ERROR app.log || true)

Alternatives and trade-offs

• Checking $? after every command. Explicit, but verbose and easy to forget once.
• Rewriting in Python or Go. Reasonable for complex logic. We kept Bash for a 60-line orchestrator, with the rules above and linting.
• Relying only on set -e. Insufficient on its own, as the exceptions show. Use it as a safety net, not as the error-handling design.

How to test the fix

We added Bats tests that run the script with stubbed commands on PATH:

# test/deploy.bats
setup() {
  export PATH="$BATS_TEST_DIRNAME/stubs:$PATH"
}

@test "aborts before migrations when pg_dump fails" {
  export STUB_PG_DUMP_EXIT=1
  run bash deploy.sh 2025-04-20T0210
  [ "$status" -ne 0 ]
  [[ "$output" != *"Running migrations"* ]]
}

@test "rolls back symlink when health check fails" {
  export STUB_CURL_EXIT=22
  run bash deploy.sh 2025-04-20T0210
  [ "$status" -ne 0 ]
  [[ "$output" == *"rolled back"* ]]
}

bats test/

Confirming in production

We ran the next release with the database password deliberately wrong on a staging host. The deploy stopped at the backup step with the ERR trap message, exit 1, and nothing after it ran. A second staging run with a broken migration rolled back inside the transaction and never touched the symlink. In production, the first real release after the change completed normally, and the backup verification step has since caught one truncated dump caused by a full backup disk.

Regression guard

ShellCheck 0.10 runs in CI on every *.sh file. Several of its checks map directly to the traps above: SC2155 (declare and assign separately to avoid masking return values), SC2164 (use cd ... || exit), SC2115 (use "${var:?}" to ensure a path is never empty in rm -rf). A small CI check also fails if a script in deploy/ lacks set -Eeuo pipefail in its first lines.

Security implications

Strict mode is a safety feature as much as a reliability one. set -u and ${var:?} prevent the classic rm -rf "$DIR"/* with an unset variable, which deletes from the filesystem root. Continuing after a failed backup removed our ability to recover from a bad migration, which turns a routine error into potential data loss. The ERR trap prints BASH_COMMAND, so we made sure no command in the script carries a secret on its command line; pg_dump and psql read credentials from PGPASSFILE instead. Finally, curl --fail is needed for the health check because by default curl exits 0 even on an HTTP 500.

Prevention checklist

• Start every non-trivial script with set -Eeuo pipefail and shopt -s inherit_errexit.
• Never combine local or export with a command substitution on the same line.
• Make expected failures explicit with if ! or || true.
• Verify artifacts (backups, downloads) by content, not by existence.
• Run ShellCheck and Bats tests on deploy scripts in CI.

References

• GNU Bash manual, the set builtin: https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
• GNU Bash manual, the shopt builtin: https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html
• GNU Bash manual, pipelines: https://www.gnu.org/software/bash/manual/html_node/Pipelines.html
• ShellCheck wiki, SC2155: https://www.shellcheck.net/wiki/SC2155
• psql documentation, ON_ERROR_STOP: https://www.postgresql.org/docs/16/app-psql.html
• Bats-core documentation: https://bats-core.readthedocs.io/