Live Migrate KVM VMs Between Hosts

Live migration of Kernel-based Virtual Machines (KVM) allows for the seamless transfer of a running VM from one physical host to another without any…

Live migration of Kernel-based Virtual Machines (KVM) allows for the seamless transfer of a running VM from one physical host to another without any perceptible downtime. This capability is critical for maintaining high availability, performing host maintenance, and balancing workloads across a virtualization cluster. This article details the procedures and considerations for performing KVM live migrations using virsh, covering scenarios with both shared and local storage, and addressing common challenges.

The core mechanism behind KVM live migration involves transferring the VM's memory state, CPU state, and device state over the network to the destination host, followed by a cutover where the VM's execution switches to the new host. For storage, two primary approaches exist: shared storage, where both hosts access the same underlying disk image, and non-shared storage, where the disk image itself must also be migrated.

Prerequisites for KVM Live Migration

Successful live migration hinges on a well-configured environment. Ensure the following prerequisites are met on both source and destination KVM hosts:

  • Identical CPU Architectures: Both hosts must have CPUs from the same vendor (e.g., Intel to Intel, AMD to AMD). While minor differences are often handled by KVM, significant disparities can lead to migration failure. For best compatibility, aim for identical CPU models or use KVM's CPU masking features effectively.
  • Matching KVM/QEMU Versions: Ideally, the KVM and QEMU versions should be identical on both hosts. While minor version differences (e.g., QEMU 8.0.0 to 8.0.1) might work, significant discrepancies can cause issues with guest state compatibility. Always test migrations after host upgrades.
  • Network Connectivity: High-bandwidth, low-latency network connectivity between hosts is crucial. A dedicated migration network is highly recommended to prevent contention with guest traffic. Standard Gigabit Ethernet is a minimum; 10GbE or faster is preferred for production environments, especially with non-shared storage migrations. Ensure SSH access is configured for virsh, and firewall rules permit traffic on relevant ports (e.g., 16509 for libvirtd, 49152-49215 for QEMU migration, though these can vary).
  • Shared Storage (Recommended): For optimal performance and simplicity, a shared storage solution (e.g., NFS, iSCSI, Fibre Channel, Ceph RBD, GlusterFS) accessible by both source and destination hosts is highly recommended. The VM's disk image path in the domain XML must point to this shared location.
  • CPU Compatibility Mode: To minimize CPU incompatibility issues, it's often best to configure VMs with cpu mode='host-model' or cpu mode='host-passthrough' if the CPUs are identical. If CPUs differ slightly, host-model attempts to expose the common features of the source host's CPU, which often allows migration to a host with a newer CPU, but may fail if the destination has an older CPU lacking features used by the guest. Avoid host-passthrough unless both hosts have truly identical CPUs, including microcode versions.

Checking CPU Compatibility

Before attempting migration, verify CPU capabilities. On both hosts, execute:

virsh capabilities | xpath '/capabilities/host/cpu/arch/text()'
virsh capabilities | xpath '/capabilities/host/cpu/model/text()'
virsh capabilities | xpath '/capabilities/host/cpu/feature/@name'

To check a specific VM's CPU configuration:

virsh dumpxml myvm | grep -A5 '<cpu>'

Look for the <model> and <feature> elements. If your VM uses host-model, its effective CPU definition will be derived from the source host. If you explicitly define a CPU model, ensure that model is supported by the destination host.

Live Migration with Shared Storage

This is the preferred and most efficient method. With shared storage, the VM's disk image does not need to be transferred over the network during migration, significantly reducing migration time and network load. Both source and destination hosts must have read/write access to the same shared storage volume where the VM's disk image resides.

Configuration Example (NFS)

Assuming an NFS share /mnt/nfs_kvm is exported from a storage server and mounted on both KVM hosts:

# On both KVM hosts (e.g., kvmhost1, kvmhost2)
sudo mkdir -p /var/lib/libvirt/images/shared
sudo mount -t nfs 192.168.1.100:/mnt/nfs_kvm /var/lib/libvirt/images/shared
sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/shared # Ensure correct permissions

The VM's domain XML should reference this shared path:

<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2'/>
  <source file='/var/lib/libvirt/images/shared/myvm.qcow2'/>
  <target dev='vda' bus='virtio'/>
</disk>

Performing the Migration

From the source host (or a management host with virsh configured to connect to both), execute the following:

virsh migrate --live myvm qemu+ssh://kvmhost2/system
  • --live: Specifies a live migration, meaning the VM remains running throughout the process.
  • myvm: The name of the VM to migrate.
  • qemu+ssh://kvmhost2/system: The connection URI for the destination host.
    • qemu: The hypervisor type.
    • ssh: The transport mechanism. SSH keys should be set up for passwordless access between hosts for convenience.
    • kvmhost2: The hostname or IP address of the destination host.
    • /system: Connects to the system libvirt daemon (root privileges needed).

You can add --persistent to ensure the VM's XML definition is saved on the destination host after migration, so it automatically starts there after a reboot.

virsh migrate --live --persistent myvm qemu+ssh://kvmhost2/system

For verbose output to monitor progress:

virsh migrate --live --verbose myvm qemu+ssh://kvmhost2/system

Live Migration Without Shared Storage (Block Migration)

When shared storage is not available, KVM can perform a "block migration." In this scenario, the VM's disk image is copied from the source host's local storage to the destination host's local storage during the migration process, in addition to the memory and CPU state.

This method is significantly slower, especially for large disk images or slower network links, and consumes considerable network bandwidth. It's generally reserved for smaller VMs or lab environments where shared storage is not feasible.

Performing the Migration

Use the --copy-storage-all flag:

virsh migrate --live --copy-storage-all myvm qemu+ssh://kvmhost2/system
  • --copy-storage-all: Instructs KVM to migrate all disk images associated with the VM.

If you only need to migrate the active disk chain (e.g., a COW image on top of a base image), you can use --copy-storage-inc. This is useful when the base image already exists on the destination, and only the changes need to be transferred.

virsh migrate --live --copy-storage-inc myvm qemu+ssh://kvmhost2/system

It's crucial that the destination host has sufficient free disk space to accommodate the VM's disk images.

Advanced Migration Options

Post-copy Migration

For VMs with very large memory footprints or applications that write to memory extensively, a live migration might get stuck in an iterative pre-copy phase, failing to converge. Post-copy migration attempts to resolve this by switching the VM to the destination earlier, and fetching remaining memory pages on demand. This might introduce a brief pause or performance degradation right after the switch, but can enable migration for otherwise problematic VMs.

virsh migrate --live --unsafe --postcopy myvm qemu+ssh://kvmhost2/system

--unsafe is required with --postcopy because if the destination host fails during the post-copy phase, the VM might be irretrievably lost. Use with caution.

Migration with Downtime (Non-Live)

If a short downtime is acceptable, a non-live migration can be performed. This involves pausing or shutting down the VM, transferring its state and/or storage, and then starting it on the destination. This is simpler and more reliable for problematic live migrations or when troubleshooting.

# Pause the VM
virsh suspend myvm

# Migrate (no --live flag, --copy-storage-all if local storage)
virsh migrate --copy-storage-all --persistent myvm qemu+ssh://kvmhost2/system

# Resume the VM on the destination (if suspended state was migrated)
# Or virsh start myvm on destination if it was shut down.

Or, more commonly, shut down the VM, move its configuration and storage, then redefine and start it on the new host.

# On source
virsh shutdown myvm
# Wait for VM to shut down
virsh undefine myvm # Remove VM definition from source

# Transfer disk image (e.g., using scp, rsync) if not using shared storage
scp /var/lib/libvirt/images/myvm.qcow2 kvmhost2:/var/lib/libvirt/images/myvm.qcow2

# Get VM XML definition from source (if undefine hasn't happened yet)
virsh dumpxml myvm > myvm.xml

# On destination
virsh define myvm.xml
virsh start myvm

Common Pitfalls and Troubleshooting

  • Firewall Issues: Ensure TCP ports 16509 (libvirtd) and the QEMU migration ports (typically in the 49152-49215 range, or dynamically allocated) are open between hosts. Test connectivity with nc -zv kvmhost2 16509.
  • SSH Key Authentication: Set up passwordless SSH access between hosts for the user running virsh (usually root or a user with sudo privileges for libvirtd connections).
  • CPU Incompatibility: The most frequent cause of failure. Error messages like "migration failed: Guest CPU is not compatible with host CPU" indicate this. Review virsh capabilities and the VM's XML CPU configuration. Consider using host-model or a carefully chosen generic CPU model.
  • Network Saturation/Latency: Slow migration or timeouts can occur if the network link is saturated or has high latency. Check network utilization during migration. Using a dedicated migration network is highly effective.
  • Insufficient Disk Space (Block Migration): Ensure the destination host has enough free space for the VM's disk image(s) when using --copy-storage-all.
  • Libvirt/QEMU Version Mismatch: Significant differences can lead to "internal error: unknown QEMU migration capability" or similar messages. Try to keep versions synchronized.
  • SELinux/AppArmor: These security modules can interfere with file access or network connections. Temporarily disable them (setenforce 0 for SELinux) for testing to rule them out, then configure appropriate policies.
  • migration_timeout: For very large VMs or slower networks, the default migration timeout might be too short. You can adjust this in /etc/libvirt/qemu.conf on both hosts: 
    # Example: Set migration timeout to 10 minutes (600 seconds)
migration_timeout = 600
    Restart libvirtd after changing this.
  • Disk I/O During Migration: High disk I/O from the guest VM during live migration can significantly slow down the process, especially during the "pre-copy" phase, and might even lead to migration failure if the dirty page rate exceeds the network transfer rate.

Back to the knowledge base · Ask the AI assistant