Home > Uncategorized > Restart Hyper-V Guests Automatically When iSCSI Volume Goes Offline

Restart Hyper-V Guests Automatically When iSCSI Volume Goes Offline

I have a couple of Hyper-V Server 2016 hosts with guest VHDX’s stored on some Synology-based iSCSI volumes. I like this setup as I don’t have to worry about maintaining reliable storage on the hosts as I’ve already invested in that with my Synology NAS and my iSCSI volumes are more than fast enough for my usage.

When I first got the guests running on the iSCSI volume I would have bi-monthly issues where the guests would go “Critical” and would refuse to boot unless I restarted the Hyper-V host. A little while later I realized my update installations and reboots of the NAS was taking the iSCSI volumes offline which Hyper-V didn’t handle well as it would get stuck “Reconnecting.” Additionally, having the iSCSI volume go out underneath running guests was a reliable way to have the guests BSOD.

Fixes for this were:

1) Reconfigure the iSCSI volumes to auto-reconnect once the Synology NAS had rebooted.

2) Use WMI event binding to handle the iScsiPrt System events for the volume going offline and then reconnecting successfully:

  1. EventID=20, “Connection to the target was lost. The initiator will attempt to retry the connection.
  2. EventId=34, “A connection to the target was lost, but Initiator successfully reconnected to the target. Dump data contains the target name.

Part 1 was mostly just reconfiguring the iSCSI connection again and ensuring that my iSCSI target was marked as a “favorite”. I still can’t explain why it would get stuck on “Reconnecting”. As of now, with all updates to Hyper-V 2016 and Synology installed, everything is working well.

Part 2 was trickier. I knew from looking at the Event Log that the two iScsiPrt events reliably occurred during the iSCSI outages. Searching around online turned up WMI eventing as a route to trigger an action after an event was logged and doing so with “permanent” events would mean they would survive host reboots. Here’s what I ended up with including my path names for reference:

c:\bin\iSCSI_Monitor.ps1

# See existing event filters, consumers, and bindings
# Get-WmiObject -Namespace root\Subscription -Class __EventFilter
# Get-WmiObject -Namespace root\Subscription -Class __EventConsumer
# Get-WmiObject -Namespace root\Subscription -Class __FilterToConsumerBinding

#
# Configure Down Events
#

$FilterNameDown = "iSCSI_TargetConnection_EventFilter_Down"
$ExistingFilterDown = Get-WmiObject -Namespace root\Subscription -Class __EventFilter -Filter "name='$FilterNameDown'"
if ($ExistingFilterDown -ne $null)
{
    $ExistingFilterDown | Remove-WmiObject -Verbose
    $ExistingFilterDown = $null
    Write-Host "Deleted existing DOWN event filter."
}

$ConsumerNameDown = "iSCSI_TargetConnection_EventConsumer_Down"
$ExistingCommandDown = Get-WmiObject -Namespace root\Subscription -Class CommandLineEventConsumer -Filter "name='$ConsumerNameDown'"
if ($ExistingCommandDown -ne $null)
{
    $ExistingCommandDown | Remove-WmiObject -Verbose
    $ExistingCommandDown = $null
    Write-Host "Deleted existing DOWN event command."
}

$ExistingBindingDown = Get-WMIObject -Namespace root\Subscription -Class __FilterToConsumerBinding -Filter "__Path LIKE '%$ConsumerNameDown%'"
if ($ExistingBindingDown -ne $null)
{
    $ExistingBindingDown | Remove-WmiObject -Verbose
    $ExistingBindingDown = $null
    Write-Host "Deleted existing DOWN event binding."
}

$QueryDown = "SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA 'Win32_NTLogEvent' AND TargetInstance.LogFile='System' AND TargetInstance.SourceName='iScsiPrt' AND TargetInstance.EventCode=20"
$WMIEventFilterDown = Set-WmiInstance -Class __EventFilter -Namespace "root\Subscription" -Arguments @{Name=$FilterNameDown;EventNameSpace="root\cimv2";QueryLanguage="WQL";Query=$QueryDown}
Write-Host "Created new DOWN event Filter. " $WMIEventFilterDown.Path

$CommandLineTemplateDown = "powershell.exe -command `". 'c:\bin\iSCSI_OnIscsiChange.ps1' 'Down' '%TargetInstance.Message%'`""
$ExecutablePath = "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
$WMIEventConsumerDown = Set-WmiInstance -Class CommandLineEventConsumer -Namespace "root\subscription" `
    -Arguments @{Name=$ConsumerNameDown;CommandLineTemplate=$CommandLineTemplateDown;ExecutablePath=$ExecutablePath }
Write-Host "Created new DOWN event consumer."

$result = Set-WmiInstance -Class __FilterToConsumerBinding -Namespace "root\subscription" -Arguments @{Filter=$WMIEventFilterDown;Consumer=$WMIEventConsumerDown}
Write-Host "Created new DOWN event binding."


#
# Configure Up Events
#

$FilterNameUp = "iSCSI_TargetConnection_EventFilter_Up"
$ExistingFilterUp = Get-WmiObject -Namespace root\Subscription -Class __EventFilter -Filter "name='$FilterNameUp'"
if ($ExistingFilterUp -ne $null)
{
    $ExistingFilterUp | Remove-WmiObject -Verbose
    $ExistingFilterUp = $null
    Write-Host "Deleted existing UP event filter."
}

$ConsumeNameUp = "iSCSI_TargetConnection_EventConsumer_Up"
$ExistingCommandUp = Get-WmiObject -Namespace root\Subscription -Class CommandLineEventConsumer -Filter "name='$ConsumeNameUp'"
if ($ExistingCommandUp -ne $null)
{
    $ExistingCommandUp | Remove-WmiObject -Verbose
    $ExistingCommandUp = $null
    Write-Host "Deleted existing UP event command."
}

$ExistingBindingUp = Get-WMIObject -Namespace root\Subscription -Class __FilterToConsumerBinding -Filter "__Path LIKE '%$ConsumeNameUp%'"
if ($ExistingBindingUp -ne $null)
{
    $ExistingBindingUp | Remove-WmiObject -Verbose
    $ExistingBindingUp = $null
    Write-Host "Deleted existing UP event binding."
}

$QueryUp = "SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA 'Win32_NTLogEvent' AND TargetInstance.LogFile='System' AND TargetInstance.SourceName='iScsiPrt' AND TargetInstance.EventCode=34"
$WMIEventFilterUp = Set-WmiInstance -Class __EventFilter -Namespace "root\Subscription" -Arguments @{Name=$FilterNameUp;EventNameSpace="root\cimv2";QueryLanguage="WQL";Query=$QueryUp}
Write-Host "Created new UP event Filter. " $WMIEventFilterUp.Path

#$CommandLineTemplateUp = 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -File c:\bin\iSCSI_OnIscsiChange.ps1 -state Up -eventMessage %TargetInstance.Message%'
$CommandLineTemplateUp = "powershell.exe -command `". 'c:\bin\iSCSI_OnIscsiChange.ps1' 'Up' '%TargetInstance.Message%'`""
$ExecutablePath = "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
$WMIEventConsumerUp = Set-WmiInstance -Class CommandLineEventConsumer -Namespace "root\subscription" `
    -Arguments @{Name=$ConsumeNameUp;CommandLineTemplate=$CommandLineTemplateUp;ExecutablePath=$ExecutablePath }
Write-Host "Created new UP event consumer."

$result = Set-WmiInstance -Class __FilterToConsumerBinding -Namespace "root\subscription" -Arguments @{Filter=$WMIEventFilterUp;Consumer=$WMIEventConsumerUp}
Write-Host "Created new UP event binding."

Write-Host Done.

Pay attention to the namespaces used, you mostly want “root\Subscription” but the event filter’s EventNamespace must be “root\cimv2”.

Then have this file sitting on disk being referenced by the WMI EventConsumer:

c:\bin\iSCSI_OnIscsiChange.ps1

$state = $args[0]
$eventMessage = $args[1]

Function Write-Log {
    [CmdletBinding()]
    Param(
    [Parameter(Mandatory=$False)] [ValidateSet("INFO","WARN","ERROR","FATAL","DEBUG")] [String] $Level = "INFO",
    [Parameter(Mandatory=$True)] [string] $Message,
    [Parameter(Mandatory=$False)] [string] $logfile
    )

    $Stamp = (Get-Date).toString("yyyy/MM/dd HH:mm:ss")
    $Line = "$Stamp $Level $Message"
    If($Logfile) {
        Write-Output $Line
        Add-Content $logfile -Value $Line
    }
    Else {
        Write-Output $Line
    }
}

$Logfile = "C:\bin\iSCSI_EventLog.log"
Write-Log DEBUG "New iSCSI state: $state" $Logfile
Write-Log DEBUG "Event message: $eventMessage" $Logfile

if ($state -eq "Down") 
{
    Write-Log INFO "iSCSI reported 'connection to target was lost', turning virtual machines off!" $Logfile
    Get-VM | where {$_.State -eq 'Running'} | Tee-Object -Append -FilePath $Logfile | Stop-VM -TurnOff -Force | Tee-Object -Append -FilePath $Logfile
} 
else 
{
    Write-Log INFO "iSCSI reported Initiator reconnected to target, turning virtual machines back on" $Logfile
    Get-VM | Tee-Object -Append -FilePath $Logfile | Start-VM | Tee-Object -Append -FilePath $Logfile
}

I couldn’t get named parameters working with my CommandLineEventConsumer, just positional hence “$args” usage above.

Setup

  1. In PowerShell, run:
    1. PS C:\> .\bin\iSCSI_Monitor.ps1

Testing Options

Consider using “-WhatIf” parameters for Start-VM and Stop-VM in iSCSI_OnIscsiChange.ps1 unless you want to actually Stop/Start your VMs.

Confirm actions in log file “C:\bin\iSCSI_EventLog.log”.

Option A: Create fake events with “eventcreate”

  1. CMD> eventcreate /T INFORMATION /L SYSTEM /ID 20 /SO Test /D Test
  2. Note:
    1. You will need to remove the “AND TargetInstance.SourceName=’iScsiPrt’” filter conditions as the events will show up as Source “Test”

Option B: Disable/Enable the Targets from within Synology’s “iSCSI Manager” app.

This is faster than actually rebooting your Synology NAS if you’re okay with taking your iSCSI Targets offline which means also actually turning off your guest VMs.

  1. Launch “iSCSI Manager” in your Synology Dashboard
  2. 20180814 - Synology iSCSI Manager
  3. Select your Target you wish to test taking offline, click “Disable”, then click “Yes” to acknowledge the confirmation dialog.
  4. Once you’ve confirmed in the log file the event was received then click “Enable” to turn the Target back on.  State should eventually go from “online” to “connected” once your Hyper-V host connects again.
Advertisements
Categories: Uncategorized
  1. No comments yet.
  1. No trackbacks yet.

Comment?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: