Resiliency is a key aspect of any infrastructure—it is even more important in infrastructure-as-a-service solutions. This session will include guidance, examples and demonstrations of the use of VMware vSphereTM PowerCLI (PowerCLI) to automate the recovery of a vCloud Director–based infrastructure. In particular the session will focus on the automation of the recovery steps for vCloud Director managed vApp workloads, that cannot be recovered with vSphere Site Recovery Manager (SRM). This session will offer a perfect complimentary follow on from the whitepaper regarding the high-level process released by Duncan Epping and Chris Colotti. The paper discussed the process and this session will describe the automation.

Presentation Materials

Whilst the presentation materials for this session should have already been released, I’m conscious that this is in the form of a .pdf and hence does not include the recorded demonstration videos.  Furthermore I recognise that the PowerCLI code included was in the form of an image and not text that can easily be leveraged.  In light of this I wanted to take opportunity to release these materials in a format that can be consumed more easily.

The original presentation, including the VMworld recording is available from the VMworld 2012 Virtual Pavilion.

Within the presentation we covered a selection of key functions used to automate the recovery of vCloud Director, including:

  • Mount-VmfsVolumes
  • Disable-VMwareHA
  • Exit-MaintenanceMode
  • Register-VirtualMachines
  • Start-vShieldEdge
  • Find-ProviderVdc
  • Find-OrganizationVdc
  • Find-VApps
  • Set-OptionVaules
  • PowerOn-VApp

In conjunction with the PowerCLI code for each of the above function I’ve added some brief explanation around the purpose of each function.


Following successful use of your storage management utility to break replication and make volumes read/write (if required by storage platform), virtual machines will appear ‘Inactive’ in the vCenter Server inventory. To rectify this, replicated VMFS volumes need to be force mounted by the recovery hosts.

function Mount-VmfsVolumes ($ClusterName, $EsxUser, $EsxPassword){
    $Cluster = Get-Cluster -Name $ClusterName
    $AvailVMHosts = $Cluster | Get-VMHost | where {($_.ConnectionState -eq "Maintenance")}
    $AvailVMHosts | Get-VMHostStorage -RescanAllHba | Out-Null
    foreach ($AvailVMHost in $AvailVMHosts){
        Connect-VIServer -Server $AvailVMHost.Name -User $EsxUser -Password $EsxPassword | Out-Null
        $VMHost = Get-VMHost -Server $AvailVMHost.Name
        $HstSys = Get-View $VMHost.id
        $HstStSys = Get-view $VMHost.StorageInfo
        $HstDsSys = Get-View $HstSys.ConfigManager.DatastoreSystem
        $UnresVols = $HstDsSys.QueryUnresolvedVmfsVolumes()
        foreach ($UnresVol in $UnresVols){
            $DevicePaths = New-Object System.Collections.ArrayList
            $Extents = $UnresVol.Extent
            $Extents | Foreach {
                $DevicePath = $_.DevicePath
                $DevicePaths += $DevicePath
            $VmfsResSpec = New-Object Vmware.Vim.HostUnresolvedVmfsResolutionSpec[](1)
            $VmfsResSpec[0] = New-Object Vmware.Vim.HostUnresolvedVmfsResolutionSpec
            $VmfsResSpec[0].ExtentDevicePath = $DevicePaths
            $VmfsResSpec[0].UuidResolution = “forceMount”
            $HstStSys.ResolveMultipleUnresolvedVmfsVolumes($VmfsResSpec) | Out-Null
        Disconnect-VIServer -Server $AvailVMHost.Name -Confirm:$false | Out-Null


Following the successful mounting of replicated VMFS volumes, there is a requirement to take the recovery ESXi Server hosts out of maintenance mode for the vCloud Director resource cluster. If there is an intention to use the vCloud Director API to initiate vApp PowerOn operations (in order to honour any defined power sequence) then we must first disable VMware HA in order to prevent virtual machines being powered on automatically.

function Disable-VMwareHA ($ClusterName){
    $Cluster = Get-Cluster -Name $ClusterName
    $Cluster | Set-Cluster –HARestartPriority Disabled –Confirm:$false | Out-Null


Having made the decision regarding the use of VMware HA the recovery ESXi Server hosts need to be removed from Maintenance Mode.

function Exit-MaintenceMode ($ClusterName){
    $Cluster = Get-Cluster -Name $ClusterName
    $Cluster | Get-VMHost -State Maintenance | Set-VMHost –State Connected  | Out-Null


In the event that VMware HA is reconfigured to prevent virtual machines being powered on automatically, the associated consequence is that the virtual machines will remain ‘Inactive’ and require registering.

function Register-VirtualMachines ($ClusterName, $EsxUser, $EsxPassword){
    $Cluster = Get-Cluster -Name $ClusterName
    $AvailVMHosts = $Cluster | Get-VMHost | where {$_.ConnectionState -eq "Connected"}
    $InActVms = $Cluster | Get-VM | where {$_.ExtensionData.OverallStatus -eq "gray"}
    $InActVmObjs = @()
    $InActVms | Foreach {
        $InActVmObj = New-Object -TypeName PSOBject -Property @{
            VmPathName = $_.ExtensionData.Config.Files.VmPathName
            VmName = $_.Name
            VmResourcePool = $_.ResourcePool
            VmMoRef = $_.Id
        $InActVmObjs += $InActVmObj
     foreach ($InActVmObj in $InActVmObjs){
         $RandVMHostIndex = Get-Random -Minimum 0 -Maximum ($AvailVMHosts | Measure-Object).Count
         $InActVmObj | Add-Member -MemberType NoteProperty -Name VmHostIndex -Value $RandVMHostIndex
     $VMHostIndex = 0
     foreach ($AvailVMHost in $AvailVMHosts){
         Connect-VIServer -Server $AvailVMHost.Name -User $EsxUser -Password $EsxPassword | Out-Null
         $HstResPools = Get-ResourcePool -Server $AvailVMHost.Name
         $HstResPoolsCount = ($HstResPools | Measure-Object).Count
         $HstVmFolder = Get-Folder –Name vm -Server $AvailVMHost.Name
         $HstVmFolderRef = Get-View $HstVmFolder.Id
         $HstInActVmObjs = $InActVmObjs | where {$_.VmHostIndex -eq $VMHostIndex}
         foreach ($HstInActVmObj in $HstInActVmObjs){
             $VmPath = $HstInActVmObj.VmPathName
             $VmName = $HstInActVmObj.VmName
             $ResourcePool = $HstInActVmObj.VmResourcePool
             if ($HstResPoolsCount -gt 1){
                 $VmResPool = $HstResPools | where {$_.Name -eq $ResourcePool}
                 $VmResPool = Get-ResourcePool -Server $AvailVMHost.Name
             $VmResPoolRef = (Get-View $VmResPool.id).MoRef
         $VMHostIndex ++
         Disconnect-VIServer -Server $AvailVMHost.Name -Confirm:$false


Even if a decision has been taken to use the vCloud Director API to power on virtual machines, this will not address vShield Edge appliances deployed on organization external routed networks. It is logical that the relatively low number of vShield Edge appliances supporting organization networks be started earlier, in order to support any connectivity to vApp external dependencies.

function Start-vShieldEdge ($ClusterName){
    $Cluster = Get-Cluster -Name $ClusterName
    $Cluster | Get-ResourcePool –Name “System vDC*” | Get-VM -Name "vse-*" | Start-VM | Out-Null


Automation of this activity can range from being relatively simple to more complex, depending on the approach taken. This could be achieved manually in the form of a simple ‘mapping table’, however this section will highlight how this can be achieved dynamically.

function Find-ProviderVdc ($ClusterName, $VcdServer, $VcdUser, $VcdPassword){
    $Cluster = Get-Cluster -Name $ClusterName
    $AvailVMHosts = $Cluster | Get-VMHost | where {$_.ConnectionState -eq "Connected"}
    $RefVMHost = $AvailVMHosts | Select -First 1
    Connect-CIServer -Server $VcdServer -User $VcdUser -Password $VcdPassword | Out-Null
    $Result = Search-Cloud -QueryType Host -Name $RefVMHost.Name
    $RefVMHostHref = (Get-CIView -SearchResult $Result).Href
    $Pvdcs = Get-ProviderVdc
    foreach ($Pvdc in $Pvdcs){
        foreach ($VMHost in $Pvdc.ExtensionData.HostReferences.HostReference){
            if ($VMHost.Href -eq $RefVMHostHref){
                $MatchedPvdc = $Pvdc
                Write-Host "Match found in Provider vDC" $MatchedPvdc.Name
    Disconnect-CIServer -Server $VcdServer -Confirm:$false


In order to power on vApps we first need to understand which organization virtual datacenters are present and later which vApps they contain. Having previously determined the provider virtual datacenter the process for determining associated organization virtual datacenters is relatively simple.

function Find-OrganisationVdc ($VcdServer, $VcdUser, $VcdPassword, $ProviderVdc){
    Connect-CIServer -Server $VcdServer -User $VcdUser -Password $VcdPassword | Out-Null
    $Ovdcs = Get-OrgVdc -ProviderVdc $ProviderVdc
    Disconnect-CIServer -Server $VcdServer -Confirm:$false


Having previously determined the organization virtual datacenters the process for determining associated vApps is relatively simple.

function Find-VApps ($VcdServer, $VcdUser, $VcdPassword, $OrganisationVdc){
    Connect-CIServer -Server $VcdServer -User $VcdUser -Password $VcdPassword | Out-Null
    $vApps = Get-CIVApp –OrgVdc $OrganisationVdc
    Disconnect-CIServer -Server $VcdServer -Confirm:$false


When created virtual machines are assigned a UUID, derived from the physical ESXi Server host UUID and the path to the virtual machine configuration file. Although the process to mount VMFS volumes does not alter the path to the configuration file, the virtual machine will have been recovered on a different ESXi Server hosts. The result being that the constituent virtual machines of a vApp may be detected as being moved or copied.

function Set-OptionValues ($ClusterName, $Key, $Value){
    $Cluster = Get-Cluster -Name $ClusterName
    $VMs = $Cluster | Get-VM
    Foreach ($VM in $VMs){
        $VirtualMachineConfigSpec = new-object VMware.Vim.VirtualMachineConfigSpec
        $VirtualMachineConfigSpec.ExtraConfig += new-object VMware.Vim.OptionValue
        $VirtualMachineConfigSpec.ExtraConfig[0].key = $Key
        $VirtualMachineConfigSpec.ExtraConfig[0].value = $Value


Lastly, having identified the vApps for a given organization virtual datacenter, it is relatively simple to initiate a power on operation.

function PowerOn-VApp ($OrganisationVdc, $VcdServer, $VcdUser, $VcdPassword){
    Connect-CIServer -Server $VcdServer -User $VcdUser -Password $VcdPassword | Out-Null
    $vApps = Get-CIVApp –OrgVdc $OrganisationVdc | where {$_.Status -ne "PoweredOn"}
    Foreach ($vApp in $vApps){
        ($vApp.Extensiondata).PowerOn() | Out-Null


Thank you to those of you that provided feedback. In the most part it seems like people enjoyed the sessions and found the content useful.  I wanted to take the opportunity to thank people for their feedback and make it clear I have read all of the comments provided and will take this in to consideration for future versions of this presentation or anything similar in nature.

Leave a Reply

Your email address will not be published. Required fields are marked *