When we had one Kubernetes cluster, Argo CD was easy: one installation, one Git repository, a folder of Application manifests. Two years later we had nine clusters across three environments and two regions, about forty services and a dozen shared platform components. The folder had grown to over three hundred Application manifests that differed only in cluster name and values file, and an Argo CD outage blocked deployments to every cluster at once.

This post compares the two big design decisions we had to make again at that point: how to generate Applications (app of apps versus ApplicationSet), and where Argo CD runs (one central instance versus one per cluster). Neither has a universally correct answer, but the trade-offs are clear once laid out.

Decision 1: how Applications are created

App of apps

An "app of apps" is an Argo CD Application whose source is a folder of other Application manifests. Argo CD syncs the parent, which creates the children, which deploy the workloads.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-prod-eu
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://git.example.com/platform/gitops.git
    targetRevision: main
    path: clusters/prod-eu/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Strengths: simple, explicit, every child is a file you can read. Sync waves on the children control order, for example installing cert-manager before anything that needs certificates.

Weakness at scale: the child manifests are near-duplicates. Adding a cluster means copying a folder of thirty files and editing the cluster name in each. Small inconsistencies creep in.

ApplicationSet

An ApplicationSet generates Applications from a template and one or more generators. The generators produce parameters, such as cluster names or folder paths, and the template is rendered once per parameter set.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cert-manager
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - clusters:
        selector:
          matchLabels:
            platform.example.com/cert-manager: "enabled"
  template:
    metadata:
      name: "cert-manager-{{ .name }}"
    spec:
      project: platform
      source:
        repoURL: https://git.example.com/platform/gitops.git
        targetRevision: main
        path: platform/cert-manager
        helm:
          valueFiles:
            - values.yaml
            - "values-{{ index .metadata.labels \"platform.example.com/env\" }}.yaml"
      destination:
        server: "{{ .server }}"
        namespace: cert-manager
      syncPolicy:
        automated: { prune: true, selfHeal: true }

The clusters generator iterates over clusters registered in Argo CD whose labels match. Label a new cluster and it gets cert-manager automatically. Other useful generators:

Generator Produces parameters from Typical use
clusters Registered clusters and their labels Platform components on every cluster
git (directories or files) Folders or config files in a repository One Application per service folder
list An inline list Small, explicit sets
matrix Combination of two generators Every service on every cluster
pullRequest Open pull requests Preview environments

Strengths: one definition, consistent across clusters, adding a cluster is a label. Weaknesses: the generated Applications are less visible in Git, and a mistake in one ApplicationSet template affects every generated Application at once. Templating also has its own learning curve.

What we chose

ApplicationSets for platform components and for services that run on many clusters, with the clusters generator for platform and a matrix of git directories and clusters for services. App of apps remains at the top: each cluster has one bootstrap Application that installs the ApplicationSets relevant to it. That gives a single, readable entry point per cluster with generated content below.

Decision 2: where Argo CD runs

Hub and spoke

One Argo CD instance, usually in a management cluster, deploys to all other clusters through their API servers.

  • One UI and one place to see the state of everything.
  • One set of SSO, RBAC and notification configuration.
  • Needs network access and credentials from the hub to every cluster's API server.
  • The hub is a single point of failure for deployments and a high-value target: it holds credentials for every cluster.
  • The application controller must handle all Applications; at hundreds or thousands, it needs sharding.

One Argo CD per cluster

Each cluster runs its own Argo CD, deploying only to itself.

  • Failure and compromise are contained to one cluster.
  • No cross-cluster credentials or network paths.
  • More instances to upgrade and configure, and no single view without extra tooling.

Comparison

Concern Hub and spoke Per cluster
Operational overhead Low (one instance) Higher (n instances)
Blast radius of an Argo CD problem All clusters One cluster
Credentials Hub holds admin access to all clusters Each instance local only
Network requirements Hub to every API server None across clusters
Global visibility Built in Needs aggregation
Scaling the controller Sharding needed at large scale Naturally split

What we chose

A split by environment: one hub for non-production clusters, where convenience matters most, and one Argo CD per production cluster, so production deployments never depend on another cluster and no single instance holds credentials for all production clusters. All instances are installed and configured from the same Git repository with the same ApplicationSets, so they stay consistent.

Repository layout

gitops/
  bootstrap/
    prod-eu.yaml            # app of apps entry point per cluster
    prod-us.yaml
    nonprod-hub.yaml
  applicationsets/
    platform/               # cert-manager, ingress, external-secrets, monitoring...
    services/               # matrix: services x clusters
  platform/
    cert-manager/
      values.yaml
      values-prod.yaml
  services/
    payments/
      values.yaml
      values-prod-eu.yaml

Cluster-specific behaviour comes from cluster labels and values files named after them, not from per-cluster copies of manifests.

Ordering across components

Sync waves still matter with ApplicationSets. We annotate generated Applications with argocd.argoproj.io/sync-wave in the template: CRDs and operators at wave -2, platform services at -1, applications at 0. For ordering within an Application, such as running a database migration before the new Deployment, sync waves and hooks on the resources themselves apply, as described in an earlier post.

Rolling out ApplicationSet changes safely

The power of an ApplicationSet is also its danger: one edit regenerates every Application it owns. Three practices keep that under control:

  • Per-environment ApplicationSets for anything risky, so a change reaches non-production clusters first and production only in a separate pull request.
  • Progressive syncs, an ApplicationSet feature that updates generated Applications in steps defined by label selectors, for example one canary cluster, then the rest.
  • Previewing the generated output in CI, rendering the ApplicationSet with its generators against a fixture list of clusters, so reviewers see which Applications change.

Common mistakes

Mistake Effect Do this instead
Copying Application folders per cluster Drift between clusters ApplicationSets with cluster labels
One ApplicationSet change rolled to all clusters at once A bad template breaks everything Progressive syncs or per-environment ApplicationSets
Hub with admin credentials to production One compromise reaches every cluster Separate production instances or scoped credentials
prune: true without care on generated apps Removing a label deletes workloads Use preserveResourcesOnDeletion where appropriate
No controller sharding on a busy hub Slow syncs and reconciliation Shard the application controller

Key takeaways

  • App of apps is explicit and simple; ApplicationSets remove duplication with generators.
  • Combining them, a bootstrap app per cluster with ApplicationSets below, gives a readable entry point and consistent content.
  • Hub and spoke is convenient but concentrates risk; per-cluster Argo CD contains it.
  • Splitting by environment, hub for non-production and per-cluster for production, is a pragmatic middle.
  • Drive cluster differences through labels and values files, not copied manifests.

References