Reconfigure HA - The object 'vim.Datastore:datastore-XXXX' has already been deleted or has not been completely created
Another lovely Friday starting off with strange issues, when taking a host out of standard maintenance we noticed that HA would not reconfigure, and kept saying The object ‘vim.Datastore:datastore-1131’ has already been deleted or has not been completely created. This was strange because no other hosts had this issue.
I proceeded to follow standard troubleshooting steps, reconfigure HA…same issue….disable HA for entire cluster and re-enable…..same issue…whoops thats not good!
First step, check what luns are available to the cluster.
1Get-Cluster $cluster | Get-Datastore | Select Name, ID | Sort ID
Seems that 1131 is missing..
Sometimes the more tools you have available to you, the easier it is to troubleshoot an issue. As part of my daily processes we have an automated script to execute and emails us an RVTools report so I can easily keep track of inventory objects in my vCenter, I went back a few days and was able to identify the lun, datastore and NAAID of this object.
Using vRealize Log Insight (Free for up to 25 OSI’s, in my case i only have my vCenter for logs) I was able to search for the NAAID and see exactly when and what happened to this lun….it was removed.
With this newfound information I went and spoke to our storage team, this lun was removed because it was no longer being used by Cluster1. Turns out This lun was zoned to two different clusters, but only used for VMs in Cluster1. Cluster2 in turn was using it for HA heartbeat. In my case to get HA to use a different lun, i chose option 3. Just in case lun’s ever get removed again. However after reconfiguring HA it still was trying to use the lun that was removed.
Resolution:
Use Option 2 - Once I selected two specific datastores, I was able to reconfigure HA and no longer had any errors. At this point you can reconfigure HA Datastore Heartbeat to your standard settings and reconfigure the cluster once again.
How to Prevent This?
Always remove luns the right way, its a lot more work, but can prevent issues like this in the future.
- Unmount Datastore from all hosts
- Detach lun from all hosts (if LUN is used for HA, it wont let you)
- Unmap from storage array.
You can also use the following script to detach lun’s if you have a bunch of lun’s and/or hosts. You will need to unmount the datastores first before running.
http://davidstamen.com/2015/09/14/using-powercli-to-detach-luns/
comments powered by Disqus