An operations runbook we inherited said: "Before deleting exported customer files, run shred -u on them." It is common advice, and on the spinning hard disks it was written for, it mostly worked. On the systems we run today, laptops with NVMe SSDs, servers with ext4 or XFS on SSDs, cloud block volumes and copy-on-write filesystems, it gives a false sense of security.

This post explains why overwriting a file in place does not reliably remove its data on modern storage, and the approaches that do, organised by the situation you are in.

What shred does

shred overwrites a file's contents several times with patterns and random data, optionally truncating and deleting it afterwards:

shred -v -n 3 -z -u customer-export.csv

It assumes that writing to the file's blocks writes to the same physical location on the storage device. The shred manual page itself warns that this assumption does not hold for many filesystems and devices.

Why the assumption fails

SSDs remap writes

Flash memory cannot overwrite a cell in place; it must erase whole blocks before writing. SSD controllers use a flash translation layer: when you "overwrite" a logical block, the controller writes the new data to a different physical location and marks the old one as stale. The old data stays in flash until garbage collection erases that block, which may be much later. SSDs also keep spare capacity (over-provisioning) that the operating system cannot address at all.

So after shred, the original data may still exist on the flash chips, inaccessible through the normal interface but potentially recoverable with specialised tools that read the chips directly.

Filesystems do not always write in place

  • Copy-on-write filesystems such as Btrfs and ZFS never overwrite data in place; every write goes to a new location, and snapshots may keep old versions indefinitely.
  • Journaling filesystems may write data to a journal first, depending on mount options (ext4 with data=journal).
  • Compression, deduplication and encryption layers change where and how data is stored.

Copies you did not make

Even if the file itself were overwritten perfectly, the data often exists elsewhere: editor swap files, backups, snapshots, previous versions in object storage, logs, temporary files written by the tool that created the export, and caches.

Cloud volumes

On cloud block storage, you do not control the physical media at all. Providers handle media sanitisation when storage is decommissioned and document their processes; what you control is encryption and access to your volumes and snapshots.

What works, by situation

For data at rest in general: encrypt everything

The most effective approach is to make deletion unnecessary by ensuring data is always encrypted at rest:

  • full-disk encryption on laptops (FileVault, BitLocker, LUKS);
  • encrypted volumes and storage in the cloud, with keys in a key management service;
  • application-level encryption for especially sensitive fields.

When data is encrypted with a key you control, destroying the key makes the data unreadable everywhere it exists, including stale flash blocks, snapshots and backups encrypted with that key. This is cryptographic erasure.

For a single sensitive file: avoid needing to erase it

For files like exports, the practical controls are:

  • generate them on encrypted storage, never on unencrypted temporary disks;
  • give them short lifetimes with automatic deletion (object storage lifecycle rules, tmpfs for transient files);
  • avoid copies: stream exports directly to their destination instead of writing intermediate files;
  • for highly sensitive cases, encrypt each file with its own key and destroy the key when the file is no longer needed.

tmpfs keeps files in memory, so they never reach disk unless memory is swapped; with encrypted swap or no swap, that is a good place for short-lived secrets.

For a whole drive you are reusing or disposing of

Use the device's built-in sanitisation, which instructs the controller to erase all cells including spare areas:

# NVMe: check supported sanitize operations, then run one (destroys all data on the device)
sudo nvme id-ctrl /dev/nvme0 -H | grep -i -A3 "sanitize"
sudo nvme sanitize /dev/nvme0 --sanact=4      # crypto erase, if supported
sudo nvme sanitize-log /dev/nvme0             # check progress and result

# NVMe format with cryptographic erase (secure erase setting 2)
sudo nvme format /dev/nvme0n1 --ses=2

# SATA SSDs: ATA Secure Erase via hdparm (requires the drive to not be frozen)
sudo hdparm -I /dev/sda | grep -i -A8 security

These commands are destructive by design and operate on the entire device. Run them only on the drive you intend to wipe, after confirming the device name, and never on a system disk you still need.

blkdiscard sends TRIM/discard for the whole device, telling the SSD that all blocks are unused. Many SSDs then return zeros for discarded blocks, but discard is a hint, not a guarantee of physical erasure, so it is weaker than sanitize or crypto erase.

For drives you do not trust to erase themselves

When a device does not support sanitize, its firmware is not trusted, or the data is highly sensitive, physical destruction by a certified service is the remaining option.

For cloud volumes and object storage

  • Delete volumes, snapshots and object versions you no longer need, including in backup vaults.
  • Use encryption with customer managed keys, and schedule key deletion when a dataset must be destroyed, which provides cryptographic erasure of every copy encrypted with that key.
  • Rely on the provider's documented media sanitisation for physical disposal.

Verifying erasure

After a sanitize command, read back samples from the device to confirm it returns zeros or random data, and record the device serial number, method, date and operator. For crypto-erase in the cloud, keep the KMS key deletion record. Auditors increasingly ask for this evidence, and having it in a log is much easier than reconstructing it later.

The standard to follow

NIST SP 800-88, Guidelines for Media Sanitization, defines three levels:

Level Meaning Examples
Clear Protects against simple, non-invasive recovery Overwrite through the normal interface (for hard disks)
Purge Protects against laboratory recovery Sanitize commands, crypto erase, degaussing hard disks
Destroy Media cannot be reused Shredding, disintegration, incineration

For SSDs, NIST recommends purge methods such as the device's sanitize command or cryptographic erase rather than overwriting.

Updating the runbook

Our runbook now says:

  • Exports are generated in an encrypted bucket with a 7-day lifecycle rule; nobody downloads them to laptops.
  • Laptops have full-disk encryption enforced by device management; lost devices are an encryption question, not an erasure question.
  • Decommissioned server drives are sanitized with the vendor's sanitize command, verified, and recorded; drives that fail sanitize are physically destroyed.
  • shred is not used.

Key takeaways

  • shred assumes in-place overwrites, which SSDs, copy-on-write filesystems and cloud storage do not provide.
  • Copies in snapshots, backups, caches and swap make single-file erasure unreliable anyway.
  • Encrypt data at rest, and use key destruction (crypto-erase) to make data unrecoverable.
  • For whole drives, use the device's sanitize or secure erase commands, or physical destruction.
  • Follow NIST SP 800-88 for choosing between clear, purge and destroy.

References