Ansible playbook reports changed on every run: shell tasks, creates and changed_when

Our base playbook configures about 40 Ubuntu hosts: kernel settings, a monitoring agent, log shipping, users. We run it nightly in check mode as a drift report, and on demand for real. The nightly report had been useless for months because every host, every night, reported changes:

PLAY RECAP *********************************************************************
app-07                     : ok=24   changed=4    unreachable=0    failed=0    skipped=3    rescued=0    ignored=1
app-08                     : ok=24   changed=4    unreachable=0    failed=0    skipped=3    rescued=0    ignored=1
db-02                      : ok=26   changed=4    unreachable=0    failed=0    skipped=3    rescued=0    ignored=1

The thing that finally made us look: while debugging an Elasticsearch node, someone opened /etc/sysctl.conf and found the same line 212 times.

vm.max_map_count=262144
vm.max_map_count=262144
vm.max_map_count=262144
...

Reproducing it

Running the playbook twice in a row against a freshly built host was enough. The second run should report changed=0. It reported 4, plus one ignored failure, and the task names told us which ones:

ansible-playbook -i inventory/staging site.yml --limit app-07
ansible-playbook -i inventory/staging site.yml --limit app-07 | grep -B1 '^changed:'

TASK [Raise vm.max_map_count] **************************************************
changed: [app-07]
--
TASK [Install node_exporter] ***************************************************
changed: [app-07]
--
TASK [Reload systemd and restart node_exporter] ********************************
changed: [app-07]
--
TASK [Set timezone] ************************************************************
changed: [app-07]

The fifth offender, the user creation task, did not show up as changed because it failed on every run after the first with useradd: user 'node_exporter' already exists, and ignore_errors: true turned that into the ignored=1 in the recap.

What we ruled out

• Real drift between runs. Nothing else touched these hosts in the minutes between the two runs.
• A templated file with a timestamp. A classic cause of perpetual changes, but our templates had no ansible_date_time or random values.
• Facts changing between runs. We diffed ansible -m setup output from both runs; nothing relevant changed.

Root cause

All of these tasks used ansible.builtin.shell or ansible.builtin.command. Those modules run whatever you give them and have no idea whether the command changed the system, so by default they always report changed. Worse, the sysctl task appended a line with echo ... >>, which is not idempotent at all, so each run added another copy. In check mode, command and shell tasks are skipped, so the drift report could neither predict nor detect anything about them.

What is happening under the hood

Ansible modules are expected to compare desired state with actual state and report changed: true only when they modify something. ansible.builtin.file, ansible.builtin.user, ansible.posix.sysctl and friends all do this. The command and shell modules cannot: from Ansible's point of view, the command is opaque. They return changed: true unconditionally, unless you tell them otherwise with:

• creates: /path skips the task if the path exists.
• removes: /path skips the task if the path does not exist.
• changed_when: <expression> decides changed status from the registered result.
• failed_when: <expression> decides failure the same way.

In check mode, command and shell do not run (they cannot know what the command would do), except that creates and removes are evaluated, so a task that would be skipped is reported correctly.

Environment

Minimal reproduction

- hosts: all
  become: true
  tasks:
    - name: Raise vm.max_map_count
      ansible.builtin.shell: echo 'vm.max_map_count=262144' >> /etc/sysctl.conf && sysctl -p

    - name: Install node_exporter
      ansible.builtin.shell: >
        curl -sL https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz
        | tar xz -C /opt

    - name: Create node_exporter user
      ansible.builtin.command: useradd --system --no-create-home node_exporter
      ignore_errors: true

    - name: Reload systemd and restart node_exporter
      ansible.builtin.shell: systemctl daemon-reload && systemctl restart node_exporter

    - name: Set timezone
      ansible.builtin.command: timedatectl set-timezone UTC

Run it twice; the second recap still says changed=4 and ignored=1.

Who is affected

Any playbook that leans on shell or command for things a module could do, which is very common in playbooks translated from a bash setup script. The visible symptom is noisy recaps; the real cost is that "changed" stops meaning anything, handlers fire every run, services restart for no reason, and appends pile up.

The fix

The rule we adopted: use a module if one exists; if not, make the command honest with creates, removes or changed_when. Here is the rewritten play:

- hosts: all
  become: true
  vars:
    node_exporter_version: "1.9.0"
  tasks:
    - name: Raise vm.max_map_count
      ansible.posix.sysctl:
        name: vm.max_map_count
        value: "262144"
        sysctl_file: /etc/sysctl.d/60-elasticsearch.conf
        reload: true

    - name: Create node_exporter user
      ansible.builtin.user:
        name: node_exporter
        system: true
        create_home: false
        shell: /usr/sbin/nologin

    - name: Download node_exporter
      ansible.builtin.get_url:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
        dest: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        checksum: "sha256:https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/sha256sums.txt"
        mode: "0644"

    - name: Unpack node_exporter
      ansible.builtin.unarchive:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        dest: /opt
        remote_src: true
        creates: "/opt/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
      notify: Restart node_exporter

    - name: Install unit file
      ansible.builtin.template:
        src: node_exporter.service.j2
        dest: /etc/systemd/system/node_exporter.service
        mode: "0644"
      notify: Restart node_exporter

    - name: Set timezone
      community.general.timezone:
        name: UTC

  handlers:
    - name: Restart node_exporter
      ansible.builtin.systemd_service:
        name: node_exporter
        state: restarted
        enabled: true
        daemon_reload: true

The restart now happens only when the binary or unit file actually changes, through a handler. The sysctl value lives in its own drop-in file, so we also removed the 212 duplicate lines from /etc/sysctl.conf with a one-off lineinfile task using state: absent.

When you really need a command

Some steps have no module. For those, we tell Ansible how to judge the result. A one-time bootstrap that leaves a marker file uses creates:

- name: Initialise the control plane once
  ansible.builtin.command: kubeadm init --config /etc/kubeadm/config.yaml
  args:
    creates: /etc/kubernetes/admin.conf

A tool that reports what it did gets changed_when based on its output:

- name: Apply database migrations
  ansible.builtin.command: /opt/app/bin/migrate up
  register: migrate
  changed_when: "'no pending migrations' not in migrate.stdout"

A read-only command used to gather information should never count as a change, and should run even in check mode so later tasks have the data:

- name: Read current kernel release
  ansible.builtin.command: uname -r
  register: kernel
  changed_when: false
  check_mode: false

Alternatives and trade-offs

• changed_when: false everywhere. Silences the noise and hides real changes. Only use it for genuinely read-only commands.
• Wrapping shell in a script that checks state first. Works, but you rebuild what modules already do, and Ansible still cannot report diffs.
• Writing a small custom module. Worth it for a repeated, complex operation; overkill for one task.

How to test the fix

The simplest test is to run twice and require zero changes on the second run:

ansible-playbook -i inventory/staging site.yml --limit app-07
ansible-playbook -i inventory/staging site.yml --limit app-07 \
  | tee second-run.log | tail -3
grep -q 'changed=0 ' second-run.log && echo "idempotent"

We moved that into Molecule, whose default test sequence includes an idempotence step that fails if the second converge reports any change:

molecule test -s default

Confirming in production

After rolling the new playbook out, the next nightly check run reported changed=0 on 38 of 40 hosts. The two exceptions were real drift: one host had a manually edited unit file, the other a missing user. That was the first time in months the report found something true, and it is now wired to alert on any non-zero changed count.

We confirmed the kernel setting on every host and that the old duplicates were gone:

ansible all -i inventory/prod -b -m ansible.builtin.command -a "sysctl -n vm.max_map_count"
ansible all -i inventory/prod -b -m ansible.builtin.shell -a "grep -c max_map_count /etc/sysctl.conf || true"

Regression guard

ansible-lint runs in CI with the production profile. Its rules catch most of this pattern before review: command-instead-of-module flags commands like systemctl or useradd that have module equivalents, command-instead-of-shell flags unnecessary shell, no-changed-when flags command tasks without changed_when, and risky-shell-pipe flags pipes without pipefail. Molecule's idempotence step runs for every role on each PR.

Security implications

The old tasks had two real security problems. First, curl | tar downloaded a binary with no checksum verification; get_url with checksum now refuses a tampered or truncated file. Second, shell tasks that interpolate variables are exposed to shell injection if a variable ever comes from untrusted input (inventory from a CMDB, for example). Prefer command with arguments, or apply the quote filter to every interpolated value. The useradd task with ignore_errors: true also hid any other failure, including a broken user database. Removing ignore_errors means failures are visible again. Finally, needless daily restarts of the monitoring agent created gaps in metrics, which is a small but real blind spot.

Prevention checklist

• Use a module whenever one exists; treat shell as the last resort.
• Give every remaining command creates, removes or changed_when.
• Mark read-only commands changed_when: false and check_mode: false.
• Run Molecule's idempotence test and ansible-lint in CI.
• Alert on non-zero changed in scheduled check-mode runs.

References

• Ansible docs, ansible.builtin.command module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/command_module.html
• Ansible docs, defining failure and changed conditions: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_error_handling.html
• Ansible docs, check mode and diff mode: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_checkmode.html
• Ansible docs, ansible.posix.sysctl: https://docs.ansible.com/ansible/latest/collections/ansible/posix/sysctl_module.html
• ansible-lint rules: https://ansible.readthedocs.io/projects/lint/rules/
• Molecule documentation: https://ansible.readthedocs.io/projects/molecule/