Cold vs Hot vMotion: Choose the Right Migration

Virtual machine migration is a cornerstone of modern data center management, enabling load balancing, hardware maintenance, and disaster recovery. VMware…

Virtual machine migration is a cornerstone of modern data center management, enabling load balancing, hardware maintenance, and disaster recovery. VMware vSphere provides several mechanisms for moving VMs, primarily categorized as "hot" (live) and "cold" migrations. Understanding the distinctions, prerequisites, and use cases for each is critical for maintaining application uptime and operational efficiency.

This article delves into the specifics of vMotion (hot migration) and cold migration, including their underlying technologies, practical implementations, and scenarios where one is preferable over the other. We will also touch upon Storage vMotion and Enhanced vMotion Compatibility (EVC) as complementary technologies that expand migration capabilities.

vMotion: Live Migration for Zero Downtime

vMotion allows for the live migration of a running virtual machine from one physical host to another without any downtime or service interruption. The VM's memory, CPU state, and network connectivity are transferred seamlessly. This capability is foundational for performing host maintenance (e.g., patching, hardware upgrades), rebalancing workloads across a cluster, and optimizing resource utilization without impacting users.

Prerequisites for vMotion

• Shared Storage: Both source and destination ESXi hosts must have access to the same datastores where the VM's configuration files and virtual disks reside. This is typically achieved with SAN (Fibre Channel, iSCSI), NAS (NFS), or vSAN.
• Compatible CPUs: The CPU feature sets of the source and destination hosts must be compatible. This means they must be from the same vendor (Intel or AMD) and, ideally, from the same CPU family or within an EVC-enabled cluster.
• Dedicated vMotion Network: While not strictly mandatory, a dedicated network interface (or a pair for redundancy) configured for vMotion traffic is highly recommended. This prevents vMotion traffic from contending with production VM traffic and provides higher bandwidth, speeding up migrations. Configure this on a separate VMkernel adapter.
• Network Connectivity: The VM's network adapters must be connected to network ports (vSwitch or Distributed Switch) on the destination host that provide equivalent network connectivity. With vSphere 6.0 and later, cross-vSwitch vMotion is possible, removing the need for identical network labels or port groups if the destination VDS is appropriately configured.
• vCenter Server: vCenter Server is required to orchestrate vMotion operations.

How vMotion Works

The vMotion process involves several key steps, all managed by vCenter Server:

1. Pre-Migration Checks: vCenter verifies prerequisites like host compatibility, shared storage access, and network configuration.
2. Source Host Pre-copy: The VM's memory state is copied from the source ESXi host to the destination ESXi host iteratively. Dirty (changed) memory pages are tracked and re-copied.
3. Stun and Copy Final Pages: Once the memory page divergence is small enough (typically milliseconds), the VM is briefly "stunned" on the source host. The final changed memory pages, CPU state, and device state are transferred.
4. Resume on Destination: The VM is resumed on the destination host, and network connectivity is seamlessly transferred by updating ARP tables on physical switches.
5. Cleanup: The VM's state is removed from the source host.

vMotion Configuration Example (VMkernel adapter)

To configure a dedicated vMotion interface, for instance, on vmk1:

# On ESXi host:
esxcli network ip interface add --interface-name=vmk1 --portgroup-name="vMotion_PG" --mtu=9000
esxcli network ip interface ipv4 set --interface-name=vmk1 --ipv4=10.10.10.101 --netmask=255.255.255.0 --type=static
esxcli network ip interface set --interface-name=vmk1 --enabled=true
esxcli network ip interface tag add --interface-name=vmk1 --tag=vMotion
esxcli network nic set --nic-name=vmnic2 --speed=10000 --duplex=full # If using dedicated physical NICs

Repeat this on all ESXi hosts, ensuring IP addresses are unique within the vMotion subnet.

Storage vMotion: Live Storage Migration

Storage vMotion allows for the live migration of a virtual machine's virtual disks and configuration files from one datastore to another without any downtime. This is invaluable for rebalancing storage workloads, upgrading storage arrays, or migrating VMs from slower to faster storage tiers.

Prerequisites for Storage vMotion

• vCenter Server: Required for orchestration.
• vSphere Licensing: Enterprise Plus or equivalent.
• Sufficient Free Space: Both source and destination datastores need enough free space for the migration.

How Storage vMotion Works

1. Shadow VM Creation: A shadow VM is created on the destination datastore, containing a copy of the VM's configuration files.
2. Disk Block Copy: The VM's virtual disk blocks are copied from the source datastore to the destination datastore in the background.
3. Changed Block Tracking: During the block copy, any changes made to the source disk by the running VM are tracked.
4. Iterative Copy: After the initial copy, subsequent iterations copy only the changed blocks. This process continues until the number of changed blocks is minimal.
5. Final Cutover: The VM is briefly "stunned," and the final changed blocks are copied. The VM is then re-pointed to the disks on the destination datastore and resumed.

Storage vMotion can be combined with vMotion (often called "Enhanced vMotion" or "Migration with change compute resource and storage") to move a VM to a different host and a different datastore simultaneously, all while the VM remains powered on.

Cold Migration: Offline VM Movement

Cold migration involves moving a powered-off or suspended virtual machine from one host to another, and optionally, from one datastore to another. Because the VM is not running during the migration, there is no risk of data corruption or service interruption during the move, but it does incur downtime.

Use Cases for Cold Migration

• CPU Incompatibility: When vMotion is not possible due to significant CPU incompatibility between hosts (e.g., Intel to AMD, or very old to very new CPU generations without EVC).
• Moving to Incompatible Networks: If the destination host's network configuration cannot be made compatible for a live vMotion.
• Moving between vCenter Servers: While vSphere 6.0 introduced cross-vCenter vMotion, cold migration is still a viable option for moving VMs between entirely separate vCenter environments or different SSO domains if live migration isn't feasible or desired.
• Moving between Datacenters (without shared storage/stretch cluster): For long-distance migrations where shared storage isn't practical or available.
• Troubleshooting: In rare cases where a vMotion fails repeatedly due to complex underlying issues, a cold migration might be a more robust alternative.

How Cold Migration Works

When a cold migration is initiated, vCenter Server copies the VM's configuration files and virtual disk files from the source location to the destination location. If only the host is changed, and shared storage is used, only the VM's registration is moved. If both host and datastore are changed, all files are copied. The time taken depends on the size of the VM's virtual disks and the network/storage bandwidth.

Enhanced vMotion Compatibility (EVC)

EVC is a cluster-level feature that allows vMotion between ESXi hosts with different CPU generations within the same vendor (Intel or AMD). EVC masks advanced CPU features, presenting a consistent, baseline CPU feature set to all VMs within the cluster. This ensures that a VM, once started on any host in the EVC cluster, can be vMotioned to any other host in that cluster without CPU compatibility errors.

EVC Configuration Example

To enable EVC:

1. Ensure all VMs in the cluster are powered off. (Alternatively, remove hosts, enable EVC, then add hosts back, vMotioning VMs to EVC-enabled hosts.)
2. In vCenter, navigate to the cluster > Configure > vSphere DRS > VMware EVC.
3. Click "Edit" and choose the lowest common denominator CPU mode compatible with all hosts in your cluster.
4. Click "OK." All future VMs powered on in this cluster will use the EVC baseline.

Note: EVC cannot bridge between Intel and AMD CPUs. All hosts must be from the same vendor.

Choosing the Right Migration Type

Common Pitfalls and Troubleshooting

• vMotion Network Misconfiguration: Incorrect MTU settings, firewall rules blocking port 8000, or insufficient bandwidth can lead to slow or failed vMotions. Always verify connectivity with vmkping -S vmotion_vmk -d <destination_ip>.
• Shared Storage Access Issues: If the destination host cannot see the VM's datastore, vMotion will fail with storage access errors. Verify LUN masking, iSCSI/NFS configuration, and zoning.
• CPU Incompatibility: The most common reason for vMotion failure. Ensure EVC is enabled and properly configured, or perform a cold migration if EVC isn't an option. Look for messages like "The VM requires CPU capabilities not present on the host."
• Insufficient Resources: The destination host must have enough CPU, memory, and network resources available to accommodate the incoming VM.
• Migration Timeouts: Large VMs with high memory change rates might exceed the default vMotion timeout. While not recommended for regular practice, this timeout can be adjusted in advanced settings (vpxd.vMotion.timeout in vCenter).
• Locked Files: Sometimes, third-party backup solutions or ungraceful VM shutdowns can leave virtual disk files locked, preventing migration. Check for stale locks using vmfsfilelockinfo -p <vmx_path> on the ESXi host.