AWS VM Import fails on a VMDK: disk format, GRUB and missing driver checklist

We were retiring an old VMware cluster and moving a handful of line-of-business VMs to EC2. Rebuilding them was not an option for this phase; the vendor software on them had a licence tied to its installation. AWS VM Import/Export looked like the shortest path: export the disk, upload to S3, run import-image, get an AMI.

The first import task failed after about twenty minutes:

aws ec2 describe-import-image-tasks --import-task-ids import-ami-0a1b2c3d4e5f60718 \
  --query 'ImportImageTasks[0].[Status,StatusMessage]' --output text

deleted  ClientError: Disk validation failed [Unsupported VMDK File Format]

After fixing that, the second attempt got further and failed with:

deleted  ClientError: BLSC-style GRUB found, but unable to detect default kernel

And after fixing that, the import succeeded, the instance launched, both status checks stayed at "initializing" for a while, and the instance never became reachable. The console output showed the boot hanging while waiting for the root device.

Reproducing it

Export a VM from vSphere by downloading only the .vmdk from the datastore browser, upload it to S3 and import it:

aws s3 cp app01-flat.vmdk s3://vmimport-staging-example/app01.vmdk
aws ec2 import-image --description "app01" \
  --disk-containers "Format=vmdk,UserBucket={S3Bucket=vmimport-staging-example,S3Key=app01.vmdk}"

That reliably triggers the first error. The second and third depend on the guest, which is why we ended up with a checklist.

What we ruled out

• IAM. A missing or wrong vmimport service role fails immediately with a message about the role, not after disk validation. Our role had the documented trust policy for vmie.amazonaws.com with the sts:ExternalId condition set to vmimport.
• Unsupported OS. RHEL 9 is on the supported list for VM Import. The failures were about how this particular guest was set up.
• Disk size and partitioning. A single 80 GiB disk, MBR, BIOS boot, well within limits.

Root cause

Three separate problems, one per stage:

1. The file we uploaded was the raw "flat" extent copied from the datastore. VM Import accepts VMDK files produced by an OVF/OVA export, which use the streamOptimized subformat. A flat extent or a split sparse disk is rejected at validation.
2. The guest's GRUB used Boot Loader Specification entries in /boot/loader/entries. Someone had removed old kernels by deleting files by hand, leaving the default entry pointing at a kernel that no longer existed. VM Import reads the boot configuration to find the kernel it needs to modify and could not resolve the default.
3. The guest's initramfs was built with hostonly mode on VMware, so it contained only the drivers that hardware needed (vmw_pvscsi, vmxnet3). On a Nitro instance, the root disk is NVMe and the NIC is ENA. Without the nvme module in the initramfs, the kernel could not find the root filesystem.

What is happening under the hood

VM Import is not a simple copy. It validates the disk image, mounts it, detects the operating system, and modifies the guest so it can boot on EC2: it adjusts the boot loader and installs or enables drivers where it can. Each of those steps can fail, and the error messages, all prefixed ClientError, tell you which step failed. The task status moves through pending, active with progress percentages and StatusMessage values such as validating, converting and booting, and on failure it goes to deleted.

For Linux guests the driver part is the fragile one. Modern RHEL kernels ship the nvme and ena modules, but the initramfs only includes what dracut decided was needed on the original host. If root is on a device the initramfs cannot drive, the boot stops before the real root filesystem is mounted.

Environment

Minimal reproduction

On the guest before export, the state that breaks the import and the boot:

grubby --default-kernel
# /boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64   <- file deleted, entry still default
ls /boot/vmlinuz-*
# /boot/vmlinuz-5.14.0-427.13.1.el9_4.x86_64
lsinitrd /boot/initramfs-$(uname -r).img | grep -E 'nvme|ena\.ko' || echo "no nvme/ena in initramfs"
# no nvme/ena in initramfs

Who is affected

Linux guests imported from VMware or Hyper-V that were built for that hypervisor's virtual hardware, guests with hand-edited boot configuration, and anyone uploading raw or split VMDK extents instead of an OVF export. Windows guests have their own driver story (VM Import installs drivers for them) and are not covered here.

The fix

We now run this preparation on every Linux guest, in order, before shutting it down for export.

1. Repair the boot configuration. Make sure the default entry points at an installed kernel and remove orphaned entries:

sudo grubby --set-default /boot/vmlinuz-$(uname -r)
for e in /boot/loader/entries/*.conf; do
  k=$(awk '/^linux /{print $2}' "$e")
  [ -e "/boot${k#/boot}" ] || { echo "orphan: $e"; sudo rm -i "$e"; }
done
grubby --default-kernel

2. Put the cloud drivers in the initramfs. Tell dracut to always include them, then rebuild for the installed kernel:

echo 'add_drivers+=" nvme ena xen-blkfront xen-netfront "' | sudo tee /etc/dracut.conf.d/ec2.conf
sudo dracut -f --kver "$(uname -r)"
lsinitrd /boot/initramfs-$(uname -r).img | grep -E 'nvme|ena'

The Xen drivers are only needed if the AMI might run on older Xen-based instance types; they are harmless otherwise.

3. Make networking and mounts hardware-independent.

• Root and other mounts in /etc/fstab by UUID=, not /dev/sda1, because the device name changes to /dev/nvme0n1p1.
• NetworkManager connection for the primary interface set to DHCP, without a hard-coded MAC address or interface name.
• Remove any static IP configuration from the old network.

4. Export in a format VM Import accepts. Either export the VM as OVF from vSphere, which produces a streamOptimized VMDK, or convert what you have:

qemu-img info app01-flat.vmdk
qemu-img convert -p -O vmdk -o subformat=streamOptimized app01.vmdk app01-stream.vmdk
qemu-img info app01-stream.vmdk | grep -i 'create type'
#     create type: streamOptimized

Converting to raw is also accepted by VM Import (Format=raw) and avoids VMDK subformat questions, at the cost of uploading the full disk size.

5. Import with an explicit platform and boot mode:

aws ec2 import-image --description "app01 rhel9" \
  --platform Linux --boot-mode legacy-bios --license-type BYOL \
  --disk-containers "Format=vmdk,UserBucket={S3Bucket=vmimport-staging-example,S3Key=app01-stream.vmdk}"

Alternatives and trade-offs

• AWS Application Migration Service. It replicates block-level from the running source and handles driver injection during cutover. Better for many servers or low-downtime moves; more setup than a one-off import.
• Rebuild from a fresh AMI. Always the cleanest result. Not possible for us because of the vendor licence, and the right choice where it is possible.
• Import as a snapshot. import-snapshot skips OS detection and gives you an EBS snapshot to register yourself. Useful when VM Import refuses an unusual guest, but you then own every boot fix.

How to test the fix

We test the prepared disk locally before spending twenty minutes on an import. Booting it under QEMU with NVMe emulation catches the initramfs problem in seconds:

qemu-system-x86_64 -m 2048 -enable-kvm -nographic \
  -drive file=app01-stream.vmdk,if=none,id=d0,format=vmdk \
  -device nvme,drive=d0,serial=test01

If the guest reaches a login prompt with root on /dev/nvme0n1p1, the import's boot stage will not fail on drivers. Then the import itself:

aws ec2 describe-import-image-tasks --import-task-ids import-ami-0f9e8d7c6b5a40312 \
  --query 'ImportImageTasks[0].[Status,StatusMessage,Progress,ImageId]' --output text
# completed  None  None  ami-0123456789abcdef0

Confirming in production

We launched the AMI into a private subnet and checked from the console output first, since it works before networking does:

aws ec2 get-console-output --instance-id i-0abc0abc0abc0abc0 --latest --output text | tail -n 20

On the instance, through Session Manager after installing the SSM Agent:

findmnt /          # SOURCE /dev/nvme0n1p1
ethtool -i eth0    # driver: ena
lsmod | grep -E '^(ena|nvme) '

Both status checks passed, the vendor application started with its licence intact, and we ran the application's own smoke test suite against the instance before switching DNS.

Regression guard

The preparation steps live in a script that runs on each guest and fails if any check is not met: default kernel exists, nvme and ena in the initramfs, fstab uses UUIDs, no static MAC in network config. The import step in our runbook refuses a VMDK whose qemu-img info does not report streamOptimized.

Security implications

An imported VM brings its whole history with it. Before first boot on EC2 we regenerate SSH host keys, remove local user accounts and authorised keys that belonged to the old environment, and rotate any credentials stored on disk, because the image sat in S3 and in a snapshot. The staging bucket uses SSE-KMS and blocks public access; we delete the uploaded disk and the intermediate snapshots after verification. Keep the sts:ExternalId condition on the vmimport role trust policy and scope its S3 permissions to the one staging bucket so the service role cannot read other data.

Prevention checklist

• Export as OVF or convert to streamOptimized before upload.
• Verify the default GRUB entry points at an installed kernel.
• Add nvme and ena to the initramfs with a dracut config file.
• Use UUIDs in fstab and DHCP without MAC binding.
• Boot-test the disk under QEMU with an NVMe device before importing.

References

• VM Import/Export user guide: https://docs.aws.amazon.com/vm-import/latest/userguide/what-is-vmimport.html
• VM Import requirements and supported formats: https://docs.aws.amazon.com/vm-import/latest/userguide/prerequisites.html
• Troubleshooting VM Import/Export: https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-troubleshooting.html
• Amazon EBS and NVMe: https://docs.aws.amazon.com/ebs/latest/userguide/nvme-ebs-volumes.html
• dracut.conf manual: https://man7.org/linux/man-pages/man5/dracut.conf.5.html