When Hyper-V Disk Merging Fails, PowerShell Comes to the Rescue

Disk merging in Hyper-V is probably one of the most frustrating features of backup management. When it works properly, you should never know it is happening, assuming you have proactive monitoring policies for disk space and regularly manage your snapshots.  When disk merging in Hyper-V doesn’t work on the other hand, you can be in for some tangible and potentially damaging downtime of your virtual machine. Alerts are handled clumsily at best for virtual machines that cannot reconcile or accumulate a significant number of differencing disk files (commonly known as AVHDX extensions).  A diligent Hyper-V system administrator is one of your only viable options to these shortcomings, but of course this is not necessarily a reality for most organizations.

We recently ran into a situation where a Hyper-V host was restarted with a virtual machine still running. The virtual machine had accumulated differencing disk files over the past 3 months.  It appeared that daily backups, while running fine, were leaving differencing files behind afterward. Needless to say, these issues were not alerted on, and therefore not acted on before it could cause an issue. The host rebooted, and the virtual machine was forced down before disk merging could be completed. After the Hyper-V host complete the reboot, the virtual machine configuration file was missing, and the virtual machine would not start.  This led to a series of troubleshooting steps to focus in on the issue.  Regardless of efforts, the VM would not start and would not merge the disks. A common fix to trigger the merge manually is to attach the most current AVHDX file to the virtual machine. There are also a few other tricks to get Hyper-V to merge disk files – none of these seemed to work.

After some effort and diligent troubleshooting, the only solution was the PowerShell merge-VHD command, which merges the hard disks by specifying a source and destination child disk. The virtual machine had 5 virtual disks and each virtual disk had 129 differencing disk files. With each execution of the merge-VHD command, the resulting AVHDX (a combination of the file specified in the command and the next file in sequence) had a new name! Meaning the process needed to be restarted each time with the using the new AVHDX file name, needless to say, once we discovered this we knew the process was going to take a very long time.

That is where PowerShell really came in handy. Upon further research through PowerShell forums and then several Internet searches later, it was clear we were dealing with a situation that had never been attempted to resolve by a decent script writer, we took this challenge into our own hands and wrote this post to share our success to a very odd, but critical issue.

Essentially, our script requires you to run the first merge manually; a single execution of the handy merge-VHD command. Then, you can update the variables at the top of the script to reflect your environment and let the script loop through all your AVHDX files until only the original VHDX is remaining! This while-loop method proved successful and was a complete time saver. Something that would’ve taken a weeks’ time of manual merging took only hours, and several runs until all the disks were merged. Obviously run time will vary depending on the size and amount of your AVHDX files, and we had a lot, but we built a counter into the loop to keep track of our progress and verify the end total.

This solution just goes to show that with a little bit of scripting knowledge, a lot of commands at your disposal can be used to automate and improve your processes, leading to less downtime and more problem solving.

 
Clear-Host

    # Written by Steve Williams, Infotect Design Solutions, 2-11-2019
    #
    # Set the folder to search for newest file
   # FolderPath is the location of the main VHDX and AVHDX files
    $FolderPath = "C:\ClusterStorage\Volume1\VM1"
   # ActiveVM is the first few letters of the VHDX name
    $ActiveVM = "VM1-C_"
    $LoopCtr = 0
   # PrefixLen is the length of ActiveVM variable (character length)
    $PrefixLen = 6

    # use Get-ChildItem to search folder. Note the Sort and Select
    $currfile1 = gci -Path $FolderPath -File | Sort-Object -Property LastWriteTime -Descending | Select Name -First 1 

    # Write to screen       
    $currfile1.Name
    $gofile = $currfile1.Name
    if($currfile1.Name.Substring(0,PrefixLen) -match $ActiveVM) {

        Do {
            write-host "Running disk merge..."
            Write-host ""
            write-host "Attempting merge on" $gofile
            Write-host ""
            gci -path $FolderPath -File | Sort-Object -Property LastWriteTime -Descending | Select FullName, LastWriteTime -First 1
            # CHANGE THE PATH BELOW TO YOUR VM PATH
            Merge-VHD -Path C:\ClusterStorage\Volume1\VM1\$gofile -Force
            Write-Host "Pausing for 20 seconds..."; start-sleep (20); Write-Host "Resuming..."
            $currfile1 = gci -Path $FolderPath -File | Sort-Object -Property LastWriteTime -Descending | Select Name -First 1
            $gofile = $currfile1.Name
            $LoopCtr++
            write-host "Loops Completed:" $LoopCtr 
        } while($currfile1.Name.Substring(0,PrefixLen) -match $ActiveVM)
    } else {
        write-host "Stop running the script, the latest drive file is not matching."
    }
    write-host "DONE!"
 
Infotect Web Admin