Splitting a 900-resource Terraform state into layers: network, data and app

Our production environment lived in a single Terraform root module with about 900 resources: VPC and subnets, transit gateway attachments, RDS clusters, ElastiCache, S3 buckets, IAM, ECS services, alarms and DNS records. It had grown that way naturally, and for a while it was fine. Then plans took eleven minutes, every application change ran a refresh of the entire network, and one afternoon a typo in an ECS module change produced a plan that wanted to replace a NAT gateway.

Nothing was destroyed, because a reviewer caught it. But the lesson was clear: the state boundary was also our blast radius boundary, and it was the whole environment. This post describes how we split it.

Choosing the boundaries

We used three questions to decide what belongs together:

1. How often does it change? Networking changes a few times a year; application services change daily.
2. What is the blast radius of a mistake? Deleting a subnet affects everything; deleting an alarm affects one service.
3. Who owns it? The platform team owns the network and shared data stores; product teams own their services.

That led to layers, each with its own state and pipeline:

Numbered prefixes make the dependency direction obvious: a layer may read from lower numbers, never higher ones.

Passing values between layers

Higher layers need IDs from lower ones: subnet IDs, security group IDs, database endpoints. There are two common ways to get them.

Remote state

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tf-state"
    key    = "prod/10-network/terraform.tfstate"
    region = "eu-west-1"
  }
}

locals {
  private_subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
}

Simple, but it couples layers to each other's state location and requires read access to the whole state file of the lower layer, including any sensitive values in it.

Provider data sources or a parameter store

data "aws_subnets" "private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.main.id]
  }
  tags = { Tier = "private" }
}

Or the lower layer writes selected values to SSM Parameter Store, and higher layers read them:

# in 10-network
resource "aws_ssm_parameter" "private_subnets" {
  name  = "/infra/prod/network/private_subnet_ids"
  type  = "StringList"
  value = join(",", aws_subnet.private[*].id)
}

# in 40-app-orders
data "aws_ssm_parameter" "private_subnets" {
  name = "/infra/prod/network/private_subnet_ids"
}

We chose the parameter store approach for most values. It gives each layer an explicit, minimal interface, works across tools, and does not grant access to the full state of another layer.

Moving resources without recreating them

The split is a refactor: resources must move from the old state to new ones without being destroyed and recreated. Terraform 1.7 and later make this possible entirely in code, with removed and import blocks, so every move is a reviewed plan.

In the old monolith, tell Terraform to forget the resource without destroying it:

removed {
  from = module.network.aws_vpc.main
  lifecycle {
    destroy = false
  }
}

In the new layer, adopt it:

import {
  to = aws_vpc.main
  id = "vpc-0f12ab34cd56ef789"
}

resource "aws_vpc" "main" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  # ...
}

The order for each group of resources:

1. Add resources and import blocks to the new layer; plan must show only imports.
2. Apply the new layer. Both states now track the resources briefly.
3. Add removed blocks to the monolith; plan must show only "will no longer be managed".
4. Apply the monolith.

Between steps 2 and 4, nobody applies the monolith for anything else. We froze it for the duration of each move, typically under an hour.

For large groups, writing hundreds of import blocks by hand is tedious. We generated them from terraform state list and terraform state show output with a small script, then reviewed the result.

Apply order and pipelines

Each layer has its own pipeline, triggered by changes in its directory. Lower layers apply first when a change spans layers, which is rare by design. When it happens, for example a new subnet that a new service needs, it becomes two pull requests, merged in order.

To catch accidental cross-layer impact, the pipeline for a lower layer runs terraform plan for the layers above it after applying, and reports any unexpected changes.

A cross-layer change, worked through

Three months after the split, the orders team needed a new private subnet range for a service with many tasks. Under the monolith this would have been one pull request touching network and app code together. Under layers it was:

1. Network PR. Add the subnets in 10-network and publish their IDs to /infra/prod/network/orders_subnet_ids. The platform team reviewed and applied it. Nothing else used the parameter yet, so the change had no effect on running services.
2. App PR. In 40-app-orders, read the new parameter and move the ECS service to the new subnets. The orders team reviewed and applied it, with a plan that touched only their service.

Each change had a small, readable plan and the right reviewers. It took a day longer than a single pull request would have, which everyone agreed was a fair price for not putting the network at risk in an application deploy.

Drift and the old monolith

After the last move, the monolith directory contained only removed blocks and an empty state. We archived it rather than deleting it immediately, ran a final terraform plan to confirm it managed nothing, and removed the directory a month later.

Results

Common mistakes

Key takeaways

• A state file is a blast radius; size it to what a single change should be allowed to touch.
• Choose layers by change rate, blast radius and ownership, with dependencies flowing one way.
• Share minimal values through a parameter store or data sources rather than whole remote states.
• Move resources with removed and import blocks so every step is a reviewed plan.
• Freeze the old state during each move and verify zero-change plans on both sides.

References

• Terraform state: https://developer.hashicorp.com/terraform/language/state
• removed block: https://developer.hashicorp.com/terraform/language/resources/syntax#removing-resources
• import block: https://developer.hashicorp.com/terraform/language/import
• terraform_remote_state data source: https://developer.hashicorp.com/terraform/language/state/remote-state-data
• AWS Systems Manager Parameter Store: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html