CloudWatch alarm stayed in INSUFFICIENT_DATA during an outage and never paged anyone

On a Tuesday morning our order processing worker stopped consuming from its queue for 47 minutes. Customers noticed before we did: support tickets arrived about confirmation emails that never came. We had an alarm for exactly this case, "orders processed per minute below 5", wired to our on-call SNS topic. It did not page.

The alarm history showed what it had been doing during the outage:

aws cloudwatch describe-alarm-history \
  --alarm-name orders-processed-low \
  --history-item-type StateUpdate \
  --query 'AlarmHistoryItems[].[Timestamp,HistorySummary]' --output text

2026-07-14T09:52:31Z  Alarm updated from INSUFFICIENT_DATA to OK
2026-07-14T09:05:31Z  Alarm updated from OK to INSUFFICIENT_DATA

And the state reason at the time it went grey:

"StateValue": "INSUFFICIENT_DATA",
"StateReason": "Insufficient Data: 3 datapoints were unknown."

Reproducing it

In staging we stopped the worker and watched the same alarm configuration. Within three evaluation periods it went from OK to INSUFFICIENT_DATA, never to ALARM. Restart the worker and it goes back to OK. Nothing was sent to SNS in either direction, because the alarm only had alarm_actions and ok_actions were not set.

What we ruled out

• SNS delivery. aws cloudwatch set-alarm-state --state-value ALARM on the alarm produced a page within a minute, so the topic, subscription and PagerDuty integration worked.
• Wrong metric or dimensions. The alarm graph matched the dashboard graph exactly, including the gap.
• Delayed metrics. The worker publishes through the CloudWatch embedded metric format in its logs. We checked that metrics were not simply late: the gap never filled in afterwards, because a stopped process emits nothing.

Root cause

The metric only exists when the worker is alive and processing. When the worker died, there were no datapoints at all, not datapoints with the value zero. The alarm used the default TreatMissingData of missing, and when every datapoint in the evaluation range is missing, CloudWatch cannot decide between OK and ALARM, so it reports INSUFFICIENT_DATA. We had no action on that state. The alarm did exactly what it was configured to do; the configuration assumed "no orders" would look like a low number.

What is happening under the hood

A metric alarm evaluates a window of datapoints: EvaluationPeriods periods of Period seconds each, and it goes to ALARM when DatapointsToAlarm of them breach the threshold. Each period's datapoint can be breaching, not breaching, or missing. TreatMissingData decides what missing means:

Two common metric shapes behave in opposite ways. Error counters such as ALB HTTPCode_Target_5XX_Count are only published in periods where the count is non-zero, so missing data should usually be notBreaching. Throughput and heartbeat metrics stop when the producer dies, which is precisely the condition you want to alert on, so missing data should be breaching. Using the default on both means the throughput alarm goes grey during an outage and the error alarm goes grey during a quiet night.

Environment

Minimal reproduction

resource "aws_cloudwatch_metric_alarm" "orders_low" {
  alarm_name          = "orders-processed-low"
  namespace           = "Orders"
  metric_name         = "ProcessedCount"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 3
  threshold           = 5
  comparison_operator = "LessThanThreshold"
  alarm_actions       = [aws_sns_topic.oncall.arn]
  # treat_missing_data not set -> "missing"
}

Stop the producer and wait three minutes. The alarm reports INSUFFICIENT_DATA and no action runs.

Who is affected

Any alarm whose metric disappears when the thing it measures fails: custom application metrics, queue consumer throughput, Lambda Invocations for a scheduled function whose trigger was disabled, and any metric emitted from logs by a process that can crash. AWS service metrics that are always published, such as EC2 CPUUtilization for a running instance, are less exposed, though they also go missing when the resource is terminated.

The fix

For the throughput alarm, silence is the failure, so missing data must count as breaching:

 resource "aws_cloudwatch_metric_alarm" "orders_low" {
   alarm_name          = "orders-processed-low"
   namespace           = "Orders"
   metric_name         = "ProcessedCount"
   statistic           = "Sum"
   period              = 60
   evaluation_periods  = 3
+  datapoints_to_alarm = 3
   threshold           = 5
   comparison_operator = "LessThanThreshold"
+  treat_missing_data  = "breaching"
   alarm_actions       = [aws_sns_topic.oncall.arn]
+  ok_actions          = [aws_sns_topic.oncall.arn]
 }

Adding ok_actions means on-call also gets the recovery message, which closes the incident automatically in our paging tool.

We did not want this alarm firing every night, because order volume genuinely drops below 5 per minute between 02:00 and 05:00. Rather than lower the threshold, we switched to metric math that compares against the same hour last week. A simpler intermediate step that worked well for us was FILL(), which turns missing periods into an explicit zero so the graph and the alarm agree:

resource "aws_cloudwatch_metric_alarm" "orders_low" {
  alarm_name          = "orders-processed-low"
  evaluation_periods  = 3
  datapoints_to_alarm = 3
  threshold           = 5
  comparison_operator = "LessThanThreshold"
  treat_missing_data  = "breaching"
  alarm_actions       = [aws_sns_topic.oncall.arn]
  ok_actions          = [aws_sns_topic.oncall.arn]

  metric_query {
    id          = "processed"
    return_data = false
    metric {
      namespace   = "Orders"
      metric_name = "ProcessedCount"
      stat        = "Sum"
      period      = 60
    }
  }

  metric_query {
    id          = "filled"
    expression  = "FILL(processed, 0)"
    label       = "Orders processed (gaps as 0)"
    return_data = true
  }
}

With FILL, the alarm sees a real zero, so it goes to ALARM even without the missing data setting. We kept breaching anyway as a second line: if the expression itself returns nothing, for example because the namespace was renamed, we still want a page.

For our 5XX alarms we did the opposite and set treat_missing_data = "notBreaching", because no datapoint there means no errors.

Alternatives and trade-offs

• Alert on INSUFFICIENT_DATA. Setting insufficient_data_actions to the on-call topic catches the same outage, but it also pages when a metric is renamed or a deployment changes dimensions. We route that state to a low-urgency channel instead.
• A separate heartbeat. Emitting a WorkerAlive metric every minute regardless of work, with breaching on missing data, distinguishes "dead" from "idle". It costs one custom metric per worker.
• Queue-side alarm. ApproximateAgeOfOldestMessage on the SQS queue grows when consumers stop and is published by AWS whether or not the worker is alive. We added it, and it would have caught this outage on its own.

How to test the fix

We test alarms the way we test code. First the notification path:

aws cloudwatch set-alarm-state --alarm-name orders-processed-low \
  --state-value ALARM --state-reason "manual test of paging path"

The alarm returns to its real state at the next evaluation. Then the evaluation logic in staging: scale the worker to zero and time how long until the alarm changes state.

aws ecs update-service --cluster staging --service order-worker --desired-count 0
watch -n 30 "aws cloudwatch describe-alarms --alarm-names orders-processed-low \
  --query 'MetricAlarms[0].[StateValue,StateReason]' --output text"

Expected after the fix, within about three minutes:

ALARM  Threshold Crossed: 3 out of the last 3 datapoints [0.0 (14/07/26 10:31:00), 0.0 (14/07/26 10:30:00), 0.0 (14/07/26 10:29:00)] were less than the threshold (5.0) (minimum 3 datapoints for OK -> ALARM transition).

Confirming in production

We could not stop production workers to test, so we confirmed in two ways. Over the following week, describe-alarm-history showed no INSUFFICIENT_DATA transitions for the alarm. And a planned 10-minute maintenance window, during which the worker was intentionally stopped, produced the expected page at minute three and the OK message on restart.

Regression guard

We audit every alarm in the account weekly for the default missing data setting and for missing actions:

aws cloudwatch describe-alarms --alarm-types MetricAlarm \
  --query 'MetricAlarms[?TreatMissingData==`missing` || length(OKActions)==`0`].AlarmName' \
  --output text

Each alarm in that list must either be fixed or carry a tag explaining the choice. Our Terraform module for alarms now makes treat_missing_data a required variable with no default, so nobody gets the default by accident.

Security implications

None beyond monitoring coverage. The gap meant security-relevant alarms with the same pattern, such as a log-derived metric for failed logins, could also go grey if the log pipeline stopped. We reviewed those alarms with the same audit query. Alarms that page on silence should not reveal anything sensitive in their descriptions, since SNS messages may land in third-party tools.

Prevention checklist

• Decide missing data treatment per alarm: breaching for throughput and heartbeats, notBreaching for sparse error counts.
• Pair every application metric alarm with an alarm on a metric AWS publishes independently, such as queue age.
• Route INSUFFICIENT_DATA somewhere a human will see it.
• Test both the paging path and the evaluation logic before trusting an alarm.
• Make treat_missing_data mandatory in shared Terraform modules.

References

• Configuring how alarms treat missing data: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
• Metric math functions, including FILL: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html
• SetAlarmState API: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_SetAlarmState.html
• Terraform aws_cloudwatch_metric_alarm: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_metric_alarm