DELL Compellent Hardware VSS Provider & Commvault on Windows Server 2012 Hyper-V nodes – Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.

As you know by now I’ve been building a high throughput, large volume Disk2Disk backup solution and that has been rather successful. At optimal speed we get 2TB/Hour per backup media server. As we currently have two, we can get to 4TB/hour at maximum throughput Smile. Currently that is, we’ll see if more is possible. The solution, which I’ll blog about later, is based on the Dell Compellent hardware VSS provider (Dell Compellent Replay Manager 6.2.0.9), Windows Server 2012, CommVault 9.0 R2 and PowerVault storage a the target disks and as it’s working now is saving us many hundred of thousands of Euros compared to dedicated D2D backup appliances or solutions.

The entire process has been a fairly smooth one, which was a relief as decent hardware VSS providers are not easy to come by, based on our experience and that of many colleagues. So, once again the DELL Compellent choice is turning out to be a good one. We did have to fix one small issue along the way.

Whilst using the DELL Compellent Hardware VSS Provider with Commvault Simpana 9.0 R2  to do host level back of the virtual machines on Windows Server 2012 Hyper-V cluster nodes. We ran into a small issue. The backup run fine but we saw this error being logged:

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {3f4965d8-10ac-411b-bf6d-6a607f237775}

image

The description found here was not helpful in resolving this. This error is found all over the internet with just about any backup product. The possible causes and solutions that are suggested are as follows:

Change the Language for non-Unicode programs to English (United States)

This is not a solution in our case, which would have surprised me if it had been.

image

Eliminate the error condition by adding the access permissions for the domain account that the Compellent Replay Manager Service for Microsoft Servers (VSS) runs under to COM Security of the affected server

As the domain account used for this service is a member of the local administrators group that already has the permissions this would also have surprised me but we tested it anyway. It turns out this is not the cause either.

image

But for your information this is how it’s done:

You can eliminate the error condition by adding the access permissions for the Network Service account to the COM Security of the affected server. To add the access permissions for the Network Service account, do the following:

  • From the Start Menu, select Run
    The Run dialog opens.In the Open field, input dcomcnfg and click OK.
    The Component Services dialog opens.
  • Expand Component Services, Computers, and My Computer.
    Right-click My Computer and click Properties on the pop-up menu.
  • The My Computer Properties dialog opens.
  • Click the COM Security tab.
    Under Access Permission click Edit Default.
  • The Access Permissions dialog opens.
  • From the Access Permissions dialog, add the DOMAINcompellentreplayservice account with Local Access & Remote access allowed (cluser => not just a local host).
  • Close all open dialogs.
  • Restart the computer.

The real cause & solution

According to Microsoft support this issue occurs when using a 3rd party backup program that utilizes Windows VSS (Volume Shadow Service) and has its own requestor. This is indeed the case here. “It looks like the requestor (the backup application) does not allow system writer to call back into their process and hence generates the error in the application log.” This sounds very plausible Winking smile

The fix:

  1. The following example grants access to the "DOMAINcompellentreplayservice" account.
  2. Click on Start, type regedit in the search box.
  3. On the Registry Editor window, navigate to: KEY_LOCAL_MACHINE>SYSTEM>CurrentControlSet>Services>VSS>VssAccessControl
  4. Add a DWORD value with the name: "DOMAINcompellentreplayservice" and set the value to “1”.

image

image

image

And voila: the error has gone when running backups Open-mouthed smile

image

I assume no responsibility if you do this in your environment but I can say that all this works perfectly in our CommVault Simpana 9.0 R2 setup whist backing up the virtual machines on our Hyper-V cluster nodes at the host level using the DELL Compellent hardware VSS provider. And yes this is Windows Server 2012, careful testing, planning helps when being an early adaptor.

9 thoughts on “DELL Compellent Hardware VSS Provider & Commvault on Windows Server 2012 Hyper-V nodes – Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.

  1. FYI Dell’s EqualLogic Host Integration Toolkit (HIT) suffers from this same issue. I’ve been looking everywhere for a solution. Thanks.

  2. I’ve been looking for a solution to this for a while and thought I was onto a winner here.
    Unfortunately I now get the following: (Yes this is using the domain Admin account)
    Volume Shadow Copy Service error: Failed resolving account Administrator with status 1376. Check connection to domain controller and VssAccessControl registry key.

    Operation:
    Gathering Writer Data

    Context:
    Writer Class Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
    Writer Name: Cluster Database
    Writer Instance ID: {ae23e849-6297-46ce-9011-79d92f993c14}

    Error-specific details:
    Error: NetLocalGroupGetMemebers(Administrator), 0x80070560, The specified local group does not exist.

    MS fix is the remove the key that I have added!

  3. Thanks for a great article. I am in the same boat, Win2012 Hyper-V, Compellent SAN, and CommVault 10. I got the hardware VSS working, however I cannot seem to perform backups of multiple nodes in the same cluster in parallel. I always get an error saying that another shadow is being created. I can backup the nodes serially, but thats not a preferred option. Were you able to perform parallel backups of multiple nodes/csv’s with this solution? Any tips or advice you can share on how you accomplished this would be appreciated. Thanks!

Leave a Reply, get the discussion going, share and learn with your peers.

This site uses Akismet to reduce spam. Learn how your comment data is processed.