Migrate a VM Between Proxmox Nodes Without Shared Storage

Migrating a Virtual Machine (VM) between Proxmox VE nodes is a common administrative task for load balancing, hardware maintenance, or disaster recovery.…

Migrating a Virtual Machine (VM) between Proxmox VE nodes is a common administrative task for load balancing, hardware maintenance, or disaster recovery. While shared storage (like NFS, iSCSI, or Ceph RBD) simplifies live migration by detaching and reattaching the VM to a new host, many smaller Proxmox clusters operate with local storage only. This article details the process of migrating VMs in such environments, focusing on the techniques for both offline and online migrations without a clustered filesystem.

Understanding Proxmox Migration Types with Local Storage

Proxmox VE differentiates between two primary migration methods when shared storage is not available: offline and online (live) migrations. Each has distinct implications for VM downtime and resource requirements.

Offline Migration (Cold Migration)

An offline migration requires the VM to be shut down. During this process, Proxmox copies all VM configuration files and disk images from the source node's local storage to the target node's local storage. Once all data is transferred, the VM can be started on the new node. This method guarantees data consistency as no writes occur during the transfer, but it incurs downtime equal to the transfer duration.

Online Migration (Live Migration)

Live migration, also known as hot migration, allows a running VM to be moved to another node with minimal interruption. Proxmox achieves this by an iterative process: it copies the VM's memory and disk state in chunks. While memory pages are being copied, any pages modified on the source are tracked and re-sent. Disk blocks are also copied. For local storage, the disk data is transferred directly over the network to the target node's local storage. This process repeats until the amount of data to be re-sent becomes small enough that a final "stop-and-copy" phase can complete quickly, often resulting in only a few seconds of downtime for the VM's network connectivity.

Prerequisites for Local Storage Migration

Before attempting any migration, ensure the following conditions are met:

Proxmox Cluster Configuration: Both source and target nodes must be part of the same Proxmox VE cluster. Node communication relies on the Corosync cluster network.
Network Connectivity: Reliable network connectivity is essential between source and target nodes. For large VMs, a dedicated 10Gbps or faster network interface configured for migration traffic is highly recommended to minimize transfer times and avoid impacting other cluster services. This is typically configured in /etc/pve/datacenter.cfg under the migration setting.
Sufficient Disk Space: The target node must have enough available local storage on the specified storage pool to accommodate the VM's disk images.
Compatible CPU Features: The target node's CPU must support the same or a compatible set of CPU features as the source node. Proxmox usually handles basic compatibility, but significant differences (e.g., Intel to AMD without careful CPU type selection like host) can cause issues.
QEMU Agent: For online migrations, the QEMU Guest Agent should ideally be installed and running inside the guest OS. While not strictly mandatory for the migration itself, it significantly improves the reliability of certain operations (like graceful shutdown or obtaining IP addresses).
Proxmox VE Version: Ensure both nodes are running compatible Proxmox VE versions. While minor version differences are often fine, significant jumps might require cluster upgrades first. For example, Proxmox VE 7.x to Proxmox VE 8.x migrations within the same cluster are generally supported but review release notes for any specific considerations.

Performing an Offline Migration (GUI and CLI)

Offline migrations are simpler and more robust, especially for VMs with high I/O workloads, but require downtime.

From the Proxmox Web GUI

Navigate to the VM you wish to migrate in the left-hand tree view.
Ensure the VM is shut down. If it's running, select "Shutdown" or "Stop" from the "More" dropdown.
With the VM selected, click the "Migrate" button in the top menu bar.
In the "Migrate" dialog:
- Target Node: Select the destination Proxmox node from the dropdown.
- Storage (for disks): For each disk listed, choose a local storage pool on the target node. You can select the same storage ID (e.g., local-lvm) if it exists and has space, or a different one.
- Advanced Options: Typically, the defaults are fine. Ensure "Delete Source" is NOT checked unless you want to automatically remove the VM and its disks from the source node after a successful migration (use with caution).
Click "Migrate". A task window will appear, showing the progress of the disk transfers.
Once the task completes successfully, the VM will be available on the target node. You can then start it.

From the Proxmox CLI

The command-line interface provides more direct control. First, shut down the VM:

qm stop <VMID>

Then, initiate the migration. The key option here is --with-local-disks to explicitly tell Proxmox to transfer local storage:

qm migrate <VMID> <target_node_name> --online 0 --with-local-disks 1

For example, to migrate VM 100 from pve01 to pve02:

qm stop 100
qm migrate 100 pve02 --online 0 --with-local-disks 1

You can monitor the progress of the migration using tail -f /var/log/syslog or by watching the task output in the GUI.

Performing an Online Migration (GUI and CLI)

Online migration is more complex but minimizes downtime, making it suitable for production environments.

From the Proxmox Web GUI

Navigate to the VM you wish to migrate.
Click the "Migrate" button.
In the "Migrate" dialog:
- Target Node: Select the destination Proxmox node.
- Storage (for disks): For each disk, select a local storage pool on the target node.
- Online: Ensure this checkbox is checked (it's typically the default when the VM is running).
- Advanced Options:
  - --with-local-disks: This option is automatically implied by the GUI when you select local storage on the target.
  - Bandwidth Limit: (Optional) You can limit the network bandwidth used for migration. This can prevent saturation of a shared network but will prolong migration time.
  - Live Storage Migration Timeout: Adjust if you expect very slow transfers, but default (3600 seconds) is usually sufficient.
Click "Migrate". A task window will appear, displaying the migration status. The VM will briefly pause during the final cut-over phase.

From the Proxmox CLI

The CLI command for online migration is very similar to offline, but with --online 1:

qm migrate <VMID> <target_node_name> --online 1 --with-local-disks 1

Example for VM 100 to pve02:

qm migrate 100 pve02 --online 1 --with-local-disks 1

To specify a different target storage for a particular disk, you can use the --target-storage option. This is especially useful if your target node has multiple local storage pools and you want to place disks on specific ones:

qm migrate 100 pve02 --online 1 --with-local-disks 1 --target-storage local-lvm:vm-100-disk-0,ssd-storage:vm-100-disk-1

This example assumes VM 100 has two disks (vm-100-disk-0 and vm-100-disk-1) and places them on local-lvm and ssd-storage respectively on the target node.

Optimizing Migration Performance

Dedicated Migration Network

For clusters with significant migration activity, isolating migration traffic to a dedicated high-speed network is crucial. Edit /etc/pve/datacenter.cfg on all nodes:

# /etc/pve/datacenter.cfg
...
migration: type=secure,network=10.0.0.0/24 # Example: use 10.0.0.x for migration network
...

Replace 10.0.0.0/24 with your actual migration network CIDR. This tells Proxmox to use network interfaces configured with IPs in this range for migration traffic. Ensure these interfaces are 10GbE or faster, and ideally, directly connected or on a dedicated switch fabric.

ZFS Send/Receive (Advanced)

If your local storage is ZFS, an alternative migration method involves ZFS send/receive. While Proxmox's built-in migration uses QEMU disk transfer, you can manually snapshot a VM's ZFS dataset and send it incrementally to the target. This is not integrated into the standard Proxmox migration commands but can be faster for very large ZFS volumes or specific recovery scenarios. This method typically requires manual VM recreation on the target and attaching the received ZFS volume, making it more complex than the built-in tools.

Common Pitfalls and Troubleshooting

Insufficient Disk Space: The most common error. Always verify the target storage has enough free space. Error message often looks like: ERROR: storage 'local-lvm' does not have enough free space (required 100G but only 50G available).
Network Issues: Dropped packets, high latency, or low bandwidth on the cluster network can cause migrations to fail or time out. Check network cables, switch health, and interface statistics (ip -s link show <interface>).
Firewall Blocking: Proxmox uses specific ports for cluster communication and migration (e.g., 6000-6999 for migration, 8007 for API, 5404/5405 for Corosync). Ensure these are open between nodes if a firewall is active.
QEMU Agent Not Running: While not a blocker, a missing QEMU agent can lead to less graceful shutdowns during offline migration (hard stop instead of ACPI shutdown) or inability to get guest IP for some Proxmox features.
CPU Compatibility: Migrating VMs with host CPU type between physically different CPUs (e.g., Intel to AMD) will fail during online migration. For such scenarios, use a more generic CPU type like kvm64 or host-passthrough (Proxmox VE 7.x+) and perform an offline migration. Always shut down the VM before changing its CPU type.
Migration Timeout: For very large VMs or slow networks, the default migration timeout (3600 seconds) might be insufficient. You can increase this via the GUI (Advanced Options) or CLI (--timeout <seconds>).
Zombie Processes/Lock Files: Occasionally, a failed migration can leave behind stale QEMU processes or lock files. These might prevent subsequent attempts. Check qm status <VMID> and ls -l /var/lock/qemu-server/ on the source node. Restarting the pve-cluster service or the node itself can resolve this, but investigate carefully before restarting services in a production environment.