systemd service dead with start-limit-hit: why Restart=always gave up and how to fix it

Monday morning, the support team noticed that no order confirmation emails had gone out since Saturday night. The email sender was a queue worker running as a systemd service on two hosts, with Restart=always, which everyone assumed meant "it can never stay down".

On both hosts:

$ systemctl status mail-worker
× mail-worker.service - Order mail queue worker
     Loaded: loaded (/etc/systemd/system/mail-worker.service; enabled; preset: enabled)
     Active: failed (Result: start-limit-hit) since Sat 2025-02-08 23:41:17 UTC; 1 day 9h ago
   Duration: 1.204s
    Process: 22918 ExecStart=/usr/bin/php /srv/app/artisan queue:work redis --queue=mail (code=exited, status=1/FAILURE)
   Main PID: 22918 (code=exited, status=1/FAILURE)

Result: start-limit-hit means systemd itself decided to stop trying.

Reading the journal around the failure

journalctl -u mail-worker -b --since "2025-02-08 23:40" --until "2025-02-08 23:42" --no-pager

23:41:11 mail-worker[22880]: In PhpRedisConnector.php line 159: Connection refused
23:41:11 systemd[1]: mail-worker.service: Main process exited, code=exited, status=1/FAILURE
23:41:11 systemd[1]: mail-worker.service: Scheduled restart job, restart counter is at 1.
23:41:12 mail-worker[22891]: In PhpRedisConnector.php line 159: Connection refused
...
23:41:16 systemd[1]: mail-worker.service: Scheduled restart job, restart counter is at 5.
23:41:17 systemd[1]: mail-worker.service: Start request repeated too quickly.
23:41:17 systemd[1]: mail-worker.service: Failed with result 'start-limit-hit'.

Five starts in about six seconds, each failing immediately because Redis refused connections. Then systemd gave up.

What happened to Redis

Redis had been restarted by an automatic minor version update at 23:41 and was back within roughly 40 seconds. Everything else that used Redis recovered on its own. The worker did not, because by the time Redis was back, systemd had already stopped restarting it.

What we ruled out

• A bug in the worker. It had run for weeks; it exited only because it could not connect.
• The service being disabled. It was enabled; start-limit-hit is a runtime failure state, not a configuration change.
• OOM kills. No kernel OOM messages, and exit code 1 came from the application.

Root cause

The unit had Restart=always with the default RestartSec=100ms and the default start rate limit of 5 starts per 10 seconds. A dependency outage of 40 seconds caused five fast failures within the window, and systemd's rate limiting marked the unit as failed. Nothing restarted it afterwards.

Two separate mechanisms

systemd has two settings groups that people often treat as one:

• Restart policy ([Service] section): Restart= decides when to restart, RestartSec= how long to wait before each restart.
• Start rate limiting ([Unit] section): StartLimitIntervalSec= and StartLimitBurst= cap how many starts are allowed within an interval. Restarts count as starts.

With the defaults, DefaultStartLimitIntervalSec=10s and DefaultStartLimitBurst=5, and a 100 ms restart delay, a service that fails instantly exhausts its five starts in well under 10 seconds. When the limit is hit, the unit enters the failed state with start-limit-hit, and Restart=always no longer applies. It stays down until someone runs systemctl reset-failed and starts it, or the system reboots.

The limit exists for good reasons: a broken service restarting in a tight loop wastes CPU, fills logs and can hammer whatever it depends on. The problem is the combination of a short restart delay with a dependency that needs longer than a few seconds to come back.

Environment

Immediate recovery

sudo systemctl reset-failed mail-worker
sudo systemctl start mail-worker
systemctl is-active mail-worker

The queue had kept the jobs, so emails went out within minutes, with an apology for the delay.

A unit that survives dependency outages

[Unit]
Description=Order mail queue worker
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=600
StartLimitBurst=20

[Service]
User=app
WorkingDirectory=/srv/app
ExecStart=/usr/bin/php artisan queue:work redis --queue=mail --sleep=3 --tries=3 --max-time=3600
Restart=always
RestartSec=5s
RestartSteps=6
RestartMaxDelaySec=120s
TimeoutStopSec=60
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

What changed:

• RestartSec=5s spaces restarts out, so a short outage uses only a few attempts.
• RestartSteps and RestartMaxDelaySec (systemd 254 and later) make the delay grow from 5 seconds to 2 minutes across six steps, an exponential-style backoff. On systemd 252 we kept a flat RestartSec=10s, since these two options are ignored by older versions with a warning.
• StartLimitIntervalSec=600 and StartLimitBurst=20 allow about 20 attempts in 10 minutes before giving up, which covers realistic dependency restarts while still stopping a truly broken service.

If you prefer that a service never gives up, StartLimitIntervalSec=0 disables rate limiting. Do that only together with a meaningful RestartSec, or a broken service will spin forever.

Make the application tolerate the dependency

Exiting on the first connection error is what turned a 40-second Redis restart into a dead worker. We added a retry with backoff at startup, so a short outage does not cause the process to exit at all:

// app/Console/Commands/WaitForRedis.php, called by the worker bootstrap
$attempt = 0;
while (true) {
    try {
        Redis::connection('queue')->ping();
        break;
    } catch (\Throwable $e) {
        $delay = min(30, 2 ** $attempt++);
        Log::warning('Redis unavailable, retrying', ['in_seconds' => $delay, 'error' => $e->getMessage()]);
        sleep($delay);
    }
}

Get told when it gives up

A unit can trigger another unit when it fails. We use it to send an alert:

[Unit]
OnFailure=notify-failure@%n.service

# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Alert on failure of %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/alert-oncall "systemd unit %i entered failed state on %H"

Alternatives and trade-offs

• A supervisor in the application (for example Supervisor or a process manager) instead of systemd restarts. It adds another layer to configure and monitor; systemd can do this job well once tuned.
• Container orchestration with back-off restarts, such as Kubernetes CrashLoopBackOff, which never gives up permanently. If you run on Kubernetes, rely on that instead.

How to test the fix

On staging, stop Redis for two minutes and watch the worker:

sudo systemctl stop redis-server      # on the staging Redis host
journalctl -fu mail-worker            # on the worker host: retries with growing delays
sudo systemctl start redis-server
systemctl is-active mail-worker       # active

Confirming in production

Two weeks later Redis was restarted again for a configuration change. The worker logged retries for about 35 seconds, reconnected and continued, without systemd restarting it at all. No unit entered the failed state.

Regression guard

• Monitoring checks systemctl is-failed for all application units every minute, plus the OnFailure alert.
• Queue depth for the mail queue alerts when older than five minutes.
• A unit template in our configuration repository sets RestartSec, StartLimit* and OnFailure for every worker.

Security implications

None directly. Indirectly, a service that silently stays dead can disable security-relevant work, such as sending login alerts or processing account deletion requests. Alerting on failed units covers those cases too.

Prevention checklist

• Never rely on Restart=always alone; set RestartSec to a few seconds at least.
• Size StartLimitIntervalSec and StartLimitBurst for the longest dependency outage you expect.
• Make applications retry dependencies with backoff instead of exiting.
• Alert on units in the failed state with OnFailure= or monitoring.
• Monitor the work itself, such as queue age, not just the process.

References

• systemd.service, Restart and RestartSec: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html
• systemd.unit, StartLimitIntervalSec and OnFailure: https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html
• systemctl, reset-failed: https://www.freedesktop.org/software/systemd/man/latest/systemctl.html
• Laravel queues, running the queue worker: https://laravel.com/docs/11.x/queues#running-the-queue-worker