Recover From Expanding VHD or VDHX Files On VMs With Checkpoints

So you’ve expanded the virtual disk (VHD/VHDX) of a virtual machine that has checkpoints (or snapshots as they used to be called) on it. Did you forget about them?  Did you really leave them lingering around for that long?  Bad practice and not supported (we don’t have production snapshots yet, that’s for Windows Server 2016). Anyway your virtual machine won’t boot. Depending on the importance of that VM you might be chewed out big time or ridiculed. But what if you don’t have a restore that works? Suddenly it’s might have become a resume generating event.

All does not have to be lost. Their might be hope if you didn’t panic and made even more bad decisions. Please, if you’re unsure what to do, call an expert, a real one, or at least some one who knows real experts. It also helps if you have spare disk space, the fast sort if possible and a Hyper-V node where you can work without risk. We’ll walk you through the scenarios for both a VHDX and a VHD.

How did you get into this pickle?

If you go to the Edit Virtual Hard Disk Wizard via the VM settings it won’t allow for that if the VM has checkpoints, whether the VM is online or not.

image

VHDs cannot be expanded on line. If the VM had checkpoints it must have been shut down when you expanded the VHD. If you went to the Edit Disk tool in Hyper-V Manager directly to open up the disk you don’t get a warning. It’s treated as a virtual disk that’s not in use. Same deal if you do it in PowerShell

Resize-VHD -Path “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd” -SizeBytes 15GB

That just works.

VHDXs can be expanded on online if they’re attached to a vSCSI controller. But if the VM has checkpoints it will not allow for expanding.

image

So yes, you deliberately shut it down to be able to do it with the the Edit Disk tool in Hyper-V Manager. I know, the warning message was not specific enough but consider this. The Edit disk tool when launched directly has no idea of what the disk you’re opening is used for, only if it’s online / locked.

Anyway the result is the same for the VM whether it was a VHD or a VHDX. An error when you start it up.

[Window Title]
Hyper-V Manager

[Main Instruction]
An error occurred while attempting to start the selected virtual machine(s).

[Content]
‘DidierTest06’ failed to start.

Synthetic SCSI Controller (Instance ID 92ABA591-75A7-47B3-A078-050E757B769A): Failed to Power on with Error ‘The chain of virtual hard disks is corrupted. There is a mismatch in the virtual sizes of the parent virtual hard disk and differencing disk.’.

Virtual disk ‘C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD_8DFF476F-7A41-4E4D-B41F-C639478E3537.avhd’ failed to open because a problem occurred when attempting to open a virtual disk in the differencing chain, ‘C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd’: ‘The size of the virtual hard disk is not valid.’.

image

You might want to delete the checkpoint but the merge will only succeed for the virtual disk that have not been expanded.  You actually don’t need to do this now, it’s better if you don’t, it saves you some stress and extra work. You could remove the expanded virtual disks from the VM. It will boot but in many cased the missing data on those disks are very bad news. But al least you’ve proven the root cause of your problems.

If you inspect the AVVHD/AVHDX file you’ll get an error that states

The differencing virtual disk chain is broken. Please reconnect the child to the correct parent virtual hard disk.

image

However attempting to do so will fail in this case.

Failed to set new parent for the virtual disk.

The Hyper-V Virtual Machine Management service encountered an unexpected error: The chain of virtual hard disks is corrupted. There is a mismatch in the virtual sizes of the parent virtual hard disk and differencing disk. (0xC03A0017).

image

Is there a fix?

Let’s say you don’t have a backup (shame on you). So now what? Make copies of the VHDX/AVHDX or VHD/AVHD and save guard those. You can also work on copies or on the original files.I’ll just the originals as this blog post is already way too long. If you. Note that some extra disk space and speed come in very handy now. You might even copy them of to a lab server. Takes more time but at least you’re not working on a production host than.

Working on the original virtual disk files (VHD/AVHD and / or VHDX/AVHDX)

If you know the original size of the VHDX before you expanded it you can shrink it to exactly that. If you don’t there’s PowerShell to the rescue if you want to find out the minimum size.

image

But even better you can shrink it to it’s minimum size, it’s a parameter!

Resize-VHD -Path “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd” -ToMinimumSize

Now you not home yet. If you restart the VM right now it will fail … with the following error:

‘DidierTest06’ failed to start. (Virtual machine ID 7A54E4DB-7CCB-42A6-8917-50A05354634F)

‘DidierTest06’ Synthetic SCSI Controller (Instance ID 92ABA591-75A7-47B3-A078-050E757B769A): Failed to Power on with Error ‘The chain of virtual hard disks is corrupted. There is a mismatch in the identifiers of the parent virtual hard disk and differencing disk.’ (0xC03A000E). (Virtual machine ID 7A54E4DB-7CCB-42A6-8917-50A05354634F)

image

What you need to do is reconnect the AVHDX to it’s parent and choose to ignore the ID mismatch. You can do this via Edit Disk in Hyper-V Manager of in PowerShell. For more information on manually merging & repairing checkpoints see my blogs on this subject here. In this post I’ll just show the screenshots as walk through.

image

image

image

image

image

Once that’s done you’re VHDX is good to go.

For a VHD you can’t shrink that with the inbox tools. There is however a free command line tool that can do that names VHDTool.exe. The original is hard to find on the web so here is the installer if you need it. You only need the executable, which is portable actually, don’t install this on a production server. It has a repair switch to deal with just this occurrence!

Here’s an example of my lab …

D:\SysAdmin>VhdTool.exe /repair “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd” “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD_8DFF476F-7A41-4E4D-B41F-C639478E3537.avhd”

image

That’s it for the VHD …

You’re back in business!  All that’s left to do is get rid of the checkpoints. So you delete them. If you wanted to apply them an get rid of the delta, you could have just removed the disks, re-added the VHD/VHDX and be done with it actually. But in most of these scenarios you want to keep the delta as you most probably didn’t even realize you still had checkpoints around. Zero data loss Winking smile.

Conclusion

Save your self the stress, hassle and possibly expense of hiring an expert.  How? Please do not expand a VHD or VHDX of a virtual machine that has checkpoints. It will cause boot issues with the expanded virtual disk or disks! You will be in a stressful, painful pickle where you might not get out of if you make the wrong decisions and choices!

As a closing note, you must have have backups and restores that you have tested. Do not rely on your smarts and creativity or that others, let alone luck. Luck runs out. Otions run out. Even for the best and luckiest of us. VEEAM has save my proverbial behind a few times already.

Update that enables Windows 7 and Windows Server 2008 R2 KMS hosts to activate Windows 10

Today, a blog reader notified me that Microsoft released an update that extends KMS for Windows 7 Service Pack 1 (SP1) and Windows Server 2008 R2 SP1 to enable the activation of Windows 10-based clients. Find it here.

image

I blogged about the update for Windows Server 2012 (R2) and Windows 8/8.1 before but now these older operating systems can also serve as KMS servers for Windows 10. All you need is the update that enables Windows 7 and Windows Server 2008 R2 KMS hosts to activate Windows 10 as discussed in KB3079821

Until now the workaround was to leverage Active Directory based activation for Windows 10 whilst you left your KMS server in place for the older clients. That works well and I have used it before. But with this update you can use your older KMS servers for all activations of windows clients.

Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 2

Introduction

In this blog series we’ll walk you through the process of migrating a Windows Server 2008 R2 Hyper-V Cluster to a Windows Server 2012 R2 Hyper-V Cluster in another Active Directory domain. You are now reading part 2.

  1. Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 1
  2. Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 2

The source W2K8R2 Hyper-V cluster is a production environment. To test the procedure for the migration we created a new CSV on the source cluster with some highly available test virtual machines with production like network configurations (multi homed virtual machined). This allows us to demonstrate the soundness of the process on one CSV before we tackle the 4 production CSVs.

We left off in part 1 with the virtual machines on the CSV LUN we are going to migrate shutdown. We’ll now continue the process of moving the CSV LUN from the old Windows Server 2008 R2 SP1 cluster to the new Windows Server 2012 R2 cluster. After that we can import them and, start them up, test that all is well and finally make them highly available in the cluster. Don’t forget the upgrade the integration components when all is done.

Removing the CSV LUN from the the source W2K8R2 Hyper-V Cluster

Just leave the VMs where they are on the LUN, un-present that LUN from the old source W2K8R2 Hyper-V cluster and present it to the new W2K12R2 Hyper-V Cluster. In our case, with a dealing with a cluster so we use a CSV. So when the LUN is presented and added to the cluster don’t forget to add it to the CSVs. Well

In Failover Cluster Manager bring the CSV that you are migrating off line. Make sure you have the correct one (green circles/arrow) to avoid down time in production.

imageWhen asked if you’re sure, confirm this

image

The CSV will be brought of line, which you can verify in Disk Management

image

We’re going to do our clean up already. You could wait until after the migration but we want the old cluster to look as clean and healthy for the operations people as possible so they don’t worry. So we go and remove this LUN from Cluster Shared Volumes.

image

Which you’ll need to confirm

image

after which your disk will be move to available storage

image

Do note that if you do this it brings the LUN back on line. As it’s still a clustered diskand  there is no IO (all VMS are shut down) that’s OK. We’ll remove it form available cluster storage (“Delete” isn’t a bad as it sounds in this context)

image

The storage will be gone form the cluster and off line in disk manager.

On the SAN / Shared Storage

We create a SAN snapshot for fall back purposes (we throw it away after all has gone well). If you have this option I highly advise you to do so. It’s not easy to move back form Windows Server 2012 R2 to W2K8R2 in the unlikely event you would need to do so. It also protects the VM against any errors & mishaps that might occur, if you understand how to use the snapshot to recover.

On the SAN we un map the CSV LUN from the old cluster. We could wait but this is an extra protection against two clusters seeing the same storage.

On the SAN we map that CSV LUN to the new cluster. It will appear in disk manager.

image

We add this disk to the new cluster

image

image

We add it to the CSV on the new cluster, which brings it on line.

image

It uses the default naming convention of clustered disks. So this is the moment to change the name if you need or want to do so.

image

So now it’s time to go Hyper-V Manager and do the actual import.

image

Navigate to the folder where you Hyper-V Virtual Machine Configuration lives. This location can be central for all VM or individual per VM, depending on how the virtual machines were organized on the old source cluster. In our example it is the latter. Also note that we only have one CSV involved per VM here, so it easy. Otherwise you will need to move multiple CSVs across together, all the ones the VM or VMs depend on.

image

It has found a virtual machine to import.

image

This is important, select “Register the virtual machine in-place (use the existing unique ID)”

image

Click “Next” to confirm the your actions

If anything about your virtual machine is not compatible with your host, the GUI allows you to make fix this. Here we have to change the correct virtual switch as they are different from the source host.

image

When done, just click next and in a blink of the eye your machine will be imported. You can start it up right now to see if all went well.

image

As in Windows Server 2012 (R2) we can add running virtual machines to the cluster for high availability that’s the final step.

image

We can import all virtual machines on the demo CSV in the same manner. Congrats, if you set up network connectivity right and done this manual migration procedure correctly you have now migrated a first CSV with VMs to the new cluster in another AD domain that can talk to to VMs that are still on the old cluster.  Cool huh! What scenarios? Well, a hoster that has clusters in a management domain that runs different workloads for different customers (multiple ADs) or a company consolidating multiple environments on a common Hyper-V Cluster or clusters in a management domain, etc.

You need to update the integration components of the virtual machines now running but other than that, you’re all set. Just move along with the next CSVs / Virtual machines until you’re done.

Closing comments

Note, what to do if you don’t have shared storage. Move the disks to the new host/cluster, copy the data over (do NOT export the VMs, as that will not work in this scenario, see part 1) or … use VEEAM Replica. It will do the heavy lifting for you and help minimize down time.. Read this blog post by our fellow MVP Silvio Di Benedetto  and for more information Veeam Backup & Replication: Migrate VM from Hyper-V 2008 R2 to 2012 R2.

Good luck. And remember if you need any assistance, there are many highly experienced Hyper-V MVPs /consultants out there. They can always help you with your migration plans if you need it.

Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 1

Introduction

In this blog we’ll walk you through the process of migrating a Windows Server 2008 R2 Hyper-V Cluster to a Windows Server 2012 R2 Hyper-V Cluster in another Active Directory domain. You are reading part 1.

  1. Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 1
  2. Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 2

The source W2K8R2 Hyper-V cluster is a production environment. To test the procedure for the migration we created a new CSV on the source cluster with some highly available test virtual machines with production like network configurations (multi homed virtual machined). This allows us to demonstrate the soundness of the process on one CSV before we tackle the 4 production CSVs. Do note that in this case the two clusters do share the same SAN. If not we can move the storage, copy the data, replicate between SANs or use VEEAM Replica (see part 2 for more info).

Preparing the source W2K8R2 Hyper-V Cluster virtual machines & Cluster

Before we begin, I always make sure I have no Hyper-V snapshots  anymore on virtual machines I migrate. It prevents any issues on that front an while Windows Server 2012 R2 is better than before dealing with snapshots I prefer to have a little possible points of concern before I start such an operation.

Go to Failover Cluster Manager

image

and shut down the virtual machines on the CSV you want to migrate.

image

You’ll see them pending whilst they are shutting down …

image

And when they are fully stopped we’ll removed the form the cluster.

image

To do so, delete (scary word) the virtual machines on our CSV that’s going to be migrated from the cluster, which makes them no longer high available

image

To do so you’ll need to confirm that this is what you want to do.

image

In Hyper-V Manager we see that the virtual machines are indeed of line. As the virtual machines reside on cluster / CSV the path to the hard disk, config files etc is indeed under C:ClusterStorage.

image

We just close the Hyper-V Manager GUI. We will NOT export the VMs to import them on the new cluster. Why?

  1. This is not necessary as since Windows Server 2012 and as such also in R2 we can import them with the option to register them in place. No export is needed for this.
  2. Due to the fact the the is no longer there you cannot import virtual machines that have been exported from Windows 2008 R2 directly into Windows Server 2012 R2. This is due to the fact that the WMI v1 namespace was deprecated in Windows Server 2012, and then removed in Windows Server 2012 R2.  When exporting a VM from Windows 2008 R2, the WMI v1 namespace was used that resulted in an .exp file to represent the exported virtual machine. In Windows Server 2012 (R2) a new WMI namespace (version 2 or rootvirtualizationv2) leverages an improved import/export model. This allows for registering the VMs in place as said in point 1. In Windows Server 2012 the version 1 WMI namespace was still there which allowed for importing of Windows Server 2008/R2 VM’s. In Windows Server 2012 R2 the version 1 namespace has been removed. So YOU CANNOT import virtual machines that where exported from Windows Server 2008/R2 into Windows Server 2012 R2. The workarounds are described here: http://blogs.technet.com/b/rmilne/archive/2013/10/22/windows-hyper-v-2012-amp-8-1-hyper-v-did-not-find-virtual-machine-to-import.aspx.

Now the combination of point 1 and 2 is what is used by the Copy cluster roles wizard in Windows Server 2012 R2. That works within a domain but not across separate AD Domains as in our case. But don’t worry. All this means is that we need to do some work manually and that’s it. That’s what we’ll describe in part 2 of this blog. Do realize you want to do this in one go as that ensures you have the least possible down time. In production don’t do part 1 of the blog on Monday and part 2 on Thursday or so Winking smile.

Read on here Migrating A Windows Server 2008 R2 Hyper-V Cluster To Windows Server 2012 R2 Hyper-V Cluster In Another Active Directory Domain – PART 2