Snapshot Hygiene: Why You Should Delete Them Quickly

VMware vSphere snapshots are invaluable tools for creating temporary, reversible checkpoints of virtual machines. They are frequently used before applying…

VMware vSphere snapshots are invaluable tools for creating temporary, reversible checkpoints of virtual machines. They are frequently used before applying patches, performing risky software installations, or making configuration changes. However, their convenience often leads to misuse and neglect, resulting in significant performance degradation, storage consumption, and operational headaches. This article delves into the technical underpinnings of vSphere snapshots, detailing why timely deletion is critical and outlining best practices for their management.

The core principle to understand is that snapshots are not backups. They are mechanisms for state reversion. Relying on them for disaster recovery or long-term data retention will inevitably lead to problems.

Understanding vSphere Snapshot Mechanics

When a snapshot is created on a running VM, vSphere performs several key actions:

• The original VMDK (virtual machine disk) becomes read-only to the virtual machine.
• A new delta disk (.vmdk) and an associated descriptor file (-delta.vmdk) are created for each virtual disk attached to the VM. All subsequent writes from the VM are directed to these delta disks.
• A .vmsn file is created to store the VM's power state (if "quiesce virtual machine memory" is selected) and the virtual machine's settings at the time of the snapshot.
• A .vmsd file is updated to record the snapshot hierarchy and metadata.

Each subsequent snapshot creates another layer of delta disks. This forms a chain, where each delta disk depends on the previous one, and ultimately on the base disk. For example, a VM with two snapshots would have a chain like: base.vmdk -> base-000001.vmdk -> base-000002.vmdk.

Delta Disk Growth and Performance Impact

The primary reason for rapid snapshot deletion is the continuous growth of these delta disks. Every write operation by the guest OS is directed to the current active delta disk. On a busy server, such as a database server or an ERP system, these delta disks can grow rapidly, sometimes by tens or even hundreds of gigabytes within days. Consider a 100GB VMDK on a busy I/O system; a single snapshot left for a week could easily result in a 50GB-100GB delta disk, consuming valuable datastore space.

The performance degradation stems from the I/O path. When a VM runs on a snapshot, every read operation that requests data not present in the active delta disk must traverse the entire snapshot chain until the data is found. This means reading through multiple delta files and potentially the base disk. Write operations are simpler as they only go to the active delta, but the overall I/O overhead increases significantly with chain length due to the lookup process for reads.

Each additional snapshot in the chain adds another layer of indirection, increasing latency and reducing throughput. This is particularly noticeable in I/O-intensive workloads where the difference between 1-2 snapshots and 5-6 snapshots can be a 50% or more drop in performance. VMware officially recommends a maximum of 32 snapshots in a chain, but practical performance limits are far lower, often 1-2 snapshots for production systems.

Snapshot Creation and Deletion Processes

Understanding what happens during these operations helps to appreciate their impact.

Creating a Snapshot

The process of creating a snapshot is typically fast, especially if memory quiescing is not selected. The creation of the delta file pointers and the .vmsn file (if applicable) is quick. Quiescing, which involves coordinating with the guest OS (via VMware Tools) to flush I/O buffers and bring the filesystem to a consistent state, adds a short delay but is crucial for application-consistent snapshots.

# Example using PowerCLI to create a snapshot
Get-VM -Name "MyProdWebServer" | New-Snapshot -Name "PrePatch_20231026" -Memory -Quiesce

This command creates a memory-quiesced snapshot. Omitting -Memory and -Quiesce results in a crash-consistent snapshot, which is faster to create but less reliable for applications.

Deleting a Snapshot (Consolidation)

Deleting a snapshot is known as "consolidating" it. This process involves merging the data from the delta disk(s) back into the parent disk (either the base disk or an earlier delta disk in the chain). This is an I/O-intensive operation and can take a significant amount of time and consume substantial datastore resources, particularly if the delta files are large.

• The VM continues to run, but all new writes still go to the active delta disk.
• vSphere reads data blocks from the delta disk(s) and writes them into the parent disk.
• Once all data is merged, the delta disk(s) and associated metadata files (.vmsn, .vmsd entries) are removed.

During consolidation, the storage array experiences a high I/O load due to the read/write operations. This can impact the performance of other VMs on the same datastore. If the datastore lacks sufficient free space during consolidation, it can fail, leaving the snapshot chain intact and potentially leading to disk full conditions later.

# Example using PowerCLI to remove all snapshots for a VM
Get-VM -Name "MyProdWebServer" | Get-Snapshot | Remove-Snapshot -Confirm:$false

# Or directly from vCenter UI: Right-click VM -> Snapshots -> Delete All Snapshots

For large, long-lived snapshots, the consolidation process can take hours, even on fast storage. During this time, datastore I/O latency often increases. If a consolidation task fails or is interrupted (e.g., due to host failure), it might need to be retried, or in extreme cases, manual intervention might be required to consolidate disks.

Best Practices for Snapshot Hygiene

Adhering to strict snapshot hygiene is critical for maintaining a healthy vSphere environment.

1. Keep Snapshots Short-Lived

Snapshots should be temporary. The general recommendation is to keep them for no longer than 24-72 hours. For critical production systems, even shorter durations (e.g., a few hours) are preferable. Never run a production VM indefinitely on a snapshot.

2. Limit Snapshot Count

Avoid creating multiple, nested snapshots. A maximum of 1-2 snapshots in a chain is a good rule of thumb for most production environments to minimize performance impact and simplify consolidation. Each additional snapshot increases the chain length and overhead.

3. Monitor Datastore Space

Regularly monitor datastore free space. Large delta files can quickly consume available capacity, leading to VMs pausing or host instability. Ensure sufficient headroom for delta file growth and consolidation operations.

4. Leverage Automated Monitoring and Alerts

Utilize vCenter alarms to detect old or numerous snapshots. Configure alerts for:

• VM Has Old Snapshots: Trigger if a snapshot exists for more than, say, 48 hours.
• VM Snapshot Size: Alert if a snapshot's delta disk exceeds a certain size (e.g., 20% of the base disk size).
• Datastore Usage: Trigger warnings when datastore utilization exceeds 80-85%.

5. Use Proper Backup Solutions, Not Snapshots

For actual backups and disaster recovery, use dedicated backup solutions like Veeam Backup & Replication, Nakivo Backup & Recovery, Rubrik, Cohesity, or VMware's built-in vSphere Replication. These solutions leverage snapshots during their backup window but delete them immediately after the backup job completes. They provide features like deduplication, compression, long-term retention, and granular recovery that snapshots do not.

6. Schedule Consolidations Appropriately

If you must keep a snapshot for an extended period (which should be rare and exceptional), plan its consolidation during maintenance windows or periods of low I/O activity to minimize impact on other systems. Be aware that consolidation cannot be paused once started.

The following table summarizes snapshot usage guidelines:

Common Pitfalls and Troubleshooting

• "Disk consolidation is needed" Alarm: This alarm indicates that snapshot delta files were not properly committed and deleted. This often happens if a consolidation task fails, the host loses power during consolidation, or if a third-party backup tool crashed. To resolve, right-click the VM and select "Snapshot" -> "Consolidate". Monitor the task for completion.
• Datastore Full during Consolidation: If a datastore runs out of space during consolidation, the task will fail. You may need to migrate other VMs off the datastore or expand the datastore if possible. In extreme cases, you might need to manually delete the delta files (not recommended without VMware support guidance) or clone the VM to a new datastore.
• VM Performance Degradation: If a VM suddenly experiences slow performance, check for active snapshots, especially old or numerous ones. Use the vSphere Client to inspect the "Snapshots" tab for the VM.
• VMDK Locked Errors: Sometimes, orphaned snapshots or failed consolidation processes can leave VMDK files locked, preventing operations like Storage vMotion. A host reboot might clear some locks, but often deeper investigation is needed, potentially involving command-line tools on the ESXi host (e.g., vmkfstools -L getlock /vmfs/volumes/<datastore_name>/<vm_name>/<vm_disk.vmdk>) or VMware support.
• Backup Job Failures: Many backup solutions use snapshots. If snapshots are already present, or if there's insufficient datastore space for the temporary snapshot a backup solution creates, backup jobs will fail. Ensure snapshots are clean before backup windows.