Some users' two-factor codes always failed: server clock drift in one container broke TOTP

Support tickets about two-factor authentication arrived at a steady trickle: "my authenticator code is always rejected", "I have to try three or four times". Most users never had a problem. The affected users were not on a specific phone type or authenticator app, and their phones' clocks were correct. Resetting their 2FA did not help.

The login endpoint logged a failure reason without the code itself:

{"event":"2fa.failed","user_id":48211,"reason":"invalid_totp","server":"app-03","ip_hash":"9f1c..."}
{"event":"2fa.failed","user_id":48211,"reason":"invalid_totp","server":"app-03","ip_hash":"9f1c..."}
{"event":"2fa.success","user_id":48211,"server":"app-01","ip_hash":"9f1c..."}

A pattern appeared once we grouped by server: almost all failures came from app-03.

Checking time

for h in app-01 app-02 app-03 app-04; do
  printf "%s " "$h"; ssh "$h" 'date -u +%H:%M:%S'
done

app-01 09:14:02
app-02 09:14:02
app-03 09:12:52
app-04 09:14:03

app-03 was 70 seconds behind. Its time synchronisation status:

ssh app-03 'timedatectl; chronyc tracking'

System clock synchronized: no
              NTP service: inactive
506 Cannot talk to daemon

The NTP service was not running. A configuration management change two months earlier had replaced chrony with a minimal base image for that server group, and time sync was never re-enabled. The clock had drifted slowly since.

What we ruled out

• Users' phones. Their clocks matched a time server within a second.
• The secret encoding. The same secrets worked on other servers.
• Code reuse protection rejecting codes. The rejection reason was invalid_totp, not reused.

Root cause

TOTP codes are derived from the current time in 30-second steps. The server validating the code computes the expected code from its own clock. app-03 was 70 seconds behind, more than two steps, so for any login routed to it, the code on the user's phone corresponded to a time step the server considered to be in the future, outside the allowed window. Logins succeeded on retry when the load balancer happened to route to another server.

How TOTP works

TOTP (RFC 6238) computes a one-time password from a shared secret and a counter derived from time:

T = floor((unix_time - T0) / 30)
code = truncate(HMAC-SHA1(secret, T)) mod 10^6

Both the phone and the server compute it independently. They agree only if their clocks place them in the same 30-second step, or within the window the server accepts.

The verification window

Servers usually accept the current step plus one step before and after, to tolerate small clock differences and the time a user takes to type the code. That is a window of about plus or minus 30 seconds. A server clock 70 seconds off falls outside it.

Widening the window would have hidden the problem, at the cost of security: every extra step accepted gives attackers more valid codes to guess or replay.

Who is affected

Any service validating TOTP across several servers or containers, especially ones built from minimal images without time synchronisation, VMs restored from snapshots, and hosts whose NTP traffic is blocked by firewall rules. Containers use the host's kernel clock, so fixing time on the host fixes all containers on it.

Environment

Restore time synchronisation

sudo apt-get install -y chrony
sudo systemctl enable --now chrony
chronyc makestep          # step the clock immediately instead of slewing slowly
chronyc tracking

System time     : 0.000012310 seconds fast of NTP time
Leap status     : Normal

On AWS we use the Amazon Time Sync Service, reachable on a link-local address from every instance without internet access; the address is in the AWS documentation linked below. makestep corrected the 70-second offset at once; without it, chrony would gradually slew the clock, which takes hours for large offsets.

Monitor clock offset on every host

node_exporter exposes the clock offset and synchronisation state:

node_timex_offset_seconds
node_timex_sync_status

An alert fires if any host's absolute offset exceeds one second or sync_status is 0 for five minutes.

Keep the window tight, add replay protection

We kept the verification window at one step and made sure a code cannot be used twice. With google2fa, verifyKeyNewer checks that the time step is newer than the last one used by that user:

$timestamp = $google2fa->verifyKeyNewer(
    $user->two_factor_secret_decrypted,
    $request->input('code'),
    $user->two_factor_last_timestamp,   // last accepted time step
    1                                    // window: one step either side
);

if ($timestamp === false) {
    throw ValidationException::withMessages(['code' => __('The code is invalid or has already been used.')]);
}

$user->forceFill(['two_factor_last_timestamp' => $timestamp])->save();

Storing the last accepted step means an intercepted code cannot be replayed within its validity window.

Rate limit attempts

Six-digit codes have a million possibilities. With a window of three steps, an attacker guessing at unlimited speed could succeed eventually. Login attempts for the 2FA step are limited to five per user per 15 minutes, after which the account requires a recovery path.

Why the drift went unnoticed for two months

Nothing else in the application depended on second-level accuracy: logs on that server were a minute off, but log timestamps across servers were rarely compared, and TLS certificates tolerate far larger skew. TOTP was simply the first feature sensitive enough to notice. Any time-based security mechanism (signed URLs with short expiry, JWT nbf and exp checks, webhook timestamp tolerance) would eventually have failed in similar ways.

Alternatives and trade-offs

• Widen the window to two or three steps to tolerate drift. Easier, weaker, and still breaks for larger drift. Fix the clock instead.
• Passkeys or security keys remove time dependence entirely and resist phishing. We offer them alongside TOTP and encourage them for admins.

How to test the fix

A test freezes time and checks acceptance at the window edges:

public function test_codes_from_adjacent_steps_are_accepted_and_reuse_is_rejected(): void
{
    $this->travelTo(now()->setTimestamp(1750660800));
    $code = $this->google2fa->getCurrentOtp($this->secret);

    $this->travel(29)->seconds();
    $this->postJson('/two-factor', ['code' => $code])->assertOk();
    $this->postJson('/two-factor', ['code' => $code])->assertStatus(422);   // replay
}

Confirming in production

After the clock was corrected, invalid_totp failures on app-03 dropped to the same low rate as the other servers. The support tickets stopped within a day. The offset alert has since caught one other host whose NTP traffic was blocked after a security group change.

Regression guard

• Base images include chrony with the time source configured; a CI test boots the image and checks timedatectl reports synchronisation.
• Clock offset alerts on all hosts.
• Login failure metrics are broken down by server, so a single bad host stands out.

Security implications

Clock drift weakened usability, not security, because codes were rejected rather than wrongly accepted. A clock running fast or slow by a lot could, in theory, accept codes for the wrong time steps, which is another reason to keep both time sync and a tight window. Replay protection and rate limiting close the remaining gaps. TOTP secrets are encrypted at rest with the application key.

Prevention checklist

• Run and monitor time synchronisation on every host that validates TOTP.
• Alert on clock offset above one second.
• Keep the TOTP window at one step; do not widen it to hide drift.
• Reject reused codes by storing the last accepted time step.
• Rate limit 2FA attempts and offer passkeys.

References

• RFC 6238, TOTP: Time-Based One-Time Password Algorithm: https://www.rfc-editor.org/rfc/rfc6238
• chrony documentation: https://chrony-project.org/documentation.html
• Amazon Time Sync Service: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html
• OWASP Multifactor Authentication Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Multifactor_Authentication_Cheat_Sheet.html