Resolving BSOD with Cisco VPN on Windows 8/ Server 2012 Hyper-V

August 28, 2014 FoxDeploy

Recently I began to support a client remotely, and noticed a huge spike in blue screen events on my PC, after I installed their VPN client. I was using a Server 2012 R2 box configured with desktop experience, and as a Hyper-V host.

Symptom The errors would only come up during name resolution (if I pinged or performed an NSLookup on a server or workstation within the remote domain, or when trying to connect to a remote system within their network), and the Blue Screen Error Code was: UNEXPECTED_KERNEL_MODE_TRAP (7f).

I fired up WinDBG (Wind Bag, as it is affectionately referred to) and loaded up my minidmp file, found under C:\Windows\MiniDmp. When I performed the !analyze -v command, I received the following explanation:

This means a trap occurred in kernel mode, and it’s a trap of a kind that the kernel isn’t allowed to have/catch (bound trap) or that is always instant death (double fault). The first number in the bugcheck params is the number of the trap (8 = double fault, etc) Consult an Intel x86 family manual to learn more about what these traps are.

Looking at the Stack text for the memory at the time, I can see the following:

nt!KeBugCheckEx nt!KiBugCheckDispatch+0x69 nt!KiDoubleFaultAbort+0xb4 vmswitch!VmsMpNicSendNetBufferLists+0xa7 NDIS!ndisMSendNBLToMiniport+0xf4 NDIS!ndisInvokeNextSendHandler+0x40 NDIS!NdisFSendNetBufferLists+0x2ed dnelwf64+0x3243

NDIS Commands…smells like networking. Finally, the output of the analyze command pretty much dropped the faulting module into my lap. I’m bolding the most pertinent part:

vmswitch!VmsMpNicSendNetBufferLists+a7 fffff800`beef63d3 e8c81e0000 call vmswitch!VmsMpCommonSanitizeNblOffloadInfo (fffff800`beef82a0)


SYMBOL_NAME: vmswitch!VmsMpNicSendNetBufferLists+a7


MODULE_NAME: vmswitch

IMAGE_NAME: vmswitch.sys


IMAGE_VERSION: 6.3.9600.17114


FAILURE_BUCKET_ID: 0x7f_8_vmswitch!VmsMpNicSendNetBufferLists

BUCKET_ID: 0x7f_8_vmswitch!VmsMpNicSendNetBufferLists


FAILURE_ID_HASH_STRING: km:0x7f_8_vmswitch!vmsmpnicsendnetbufferlists

Cause: VMswitch.sys does not like me installing a VPN on my Hyper-V host. This makes sense, as in Hyper-V’s Type 1 Hypervisor, the host machine is treated as an elevated VM, which is exposed to both the virtual and the physical hardware on the machine. This is the reason you’ll see so many network and other devices listed in device manager:

VPNBbsod1](../assets/images/2014/08/images/vpnbbsod1.png) The Blue entries are actually Hypervisor related NICs. Don’t touch them!


And also why you’ll see so many network devices appear in Network Devices within the host. One for each Virtual Switch.


Again, some of these are only placeholders for VMswitch network devices, which you’re exposed to only from within the parent partition (that special sort of VM which can see physical and virtual devices).

If you note the names, these are the same as the entries from the above two pictures.

When a Hyper-V guest requests access to a network resource, its request goes to the VM queue which is passed along to the Virtual Switch agent within the Hypervisor (which is running on the hardware, not your parent partition / Host VM).

WorkAround: Some have reported success using the following KB ( but my work around was to use VPN Clients only within guest VMs, and not within the parent partition.

Unfortunately I never was able to determine the root cause of the problem, but I’m certain it has something to do with the Cisco VPN Client attempting some operation against the synthetic network adapter, which will not respond in the same manner as a physical device.

Continue Reading...

ScriptSharing : Quest Get-SwitchedStatus

August 26, 2014 FoxDeploy

This is another Quest Migration Manager for Exchange, Active Directory (QMM post).

Sorry for the big influx of these, as you can tell I’ve been doing a lot of Exchange and AD Migration work recently :)

So, during a Quest driven migration project, you generally have two goals.

  1. Migrate a User’s AD and E-mail accounts from the existing domain (Source) to the new domain (Target)
  2. Migrate a User’s Computer to the target

In order to do this, you have a few dependencies:

  1. Before a user’s PC can be moved, you need to sync user’s AD Objects and Mailboxes to ensure the user account is placed in the right OU for Group Policy, and to make sure that the user has all of their mail available when the switch e-mail servers
  2. When the above is synced, you then have to use the QMM Resource Update Manager to re-ACL all of the files, services and registry items on the user’s workstation, before the system can be migrated to the new domain.

So, a big prerequisite to this whole process that can keep you from moving a user’s PC is to know whether or not the user’s mailbox is Switched to the new Exchange Environment.

Here’s the general flowchart of when it is OK to migrate a user account.

Get-SwitchedStatus FlowChart

You’ll have Exchange server pairs setup for each of your core Exchange DAG’s and these will handle syncing mailboxes from Source to Target. Once a mailbox is synced, it can then be Switched, a process which handles mail routing and ensures that new mail is delivered to the mailbox the user is using. Essentially, Switching the mailbox will direct all new mail to the user’s new mailbox in the target, at which point you can safely unprovision and disable their mailbox from the source.

The easiest way to determine a user’s Switched status is to pull up their AD Account in the Source Domain and look for a ‘targetAddress’ attribute. Unfortunately, these attributes are not selectable as a column for Active Directory Users & Computers Search, nor can you view these Attributes from the user object when opened from source (if you attempt that, only a select few tabs of user object properties are shown).

To that end, I have created the Get-SwitchedStatus.ps1 PowerShell Script. You can use this, specifying -Server for Root AD Server to query for attributes. It accepts parameter input and also has some decent help to get you started.

My next incarnation of this will be a standalone C# version of the same functionality!

#     ----Name : Quest Switch Status
#     ---Author: Stephen Owen, 7.24.2014
#     Function : Use this tool to determine if a user account has switched or not, based on the account's Source targetAddress property
#     ---Usage : Get-SwitchedStatus <$username>
Function Get-SwitchedStatus {
       Use this tool to determine if a user account has switched or not, based on the account's Source targetAddress property
       This tool connects to the Source (legacy) AD Domain and looks up the specified user object, returning the .targetAddress property
       connects to the target domain to move the user account to the correct OU
    .PARAMETER UserName
       &lt;Mandatory&gt; Specify the username to inquire, will bind to aliases like UserName, LogonName or SAMAccountName.  Accepts Pipeline input.
    .PARAMETER Service
       &lt;Optional&gt;  Specify the Active Directory Service to Query
    .PARAMETER SourceCredential
       &lt;Optional&gt;  Specify the Credentials to use to query the Source AD Infrastructure.  If omitted, this tool will also look for a Global:$Credential object, and attempt to use that.
    .SWITCH WriteWarning
        Provide detailed Warning information if a user is encountered without needed properties.
       Get-SwitchedStatus -UserName Stephen.Owen
       &gt; This tool will connect to the source, and grab the targetAddress property of the object &lt;Stephen.Owen&gt;.  If this contains *@source.qmm, the account is most likely unSwitched.
         If this value contains *@target.qmm, then the account is most likely Switched and the workstation can be migrated.  If this account does not have a mailbox, the workstation can be migrated.
       "Stephen.Owen","Mark.Wuerslin" | Get-SwitchedStatus
       &gt; Perform Get-SwitchedStatus on both Stephen.Owen, then Mark.Wuerslin
      $Host.UI.RawUI.WindowTitle="---Get UserSwitchedStatus Tool"
      Write-Progress -Activity ("Checking for Quest Active Roles PSSnapIn...") -PercentComplete 25  -Status "Searching.."
            if (-not ((Get-PSSnapin).Name -contains "Quest.ActiveRoles.ADManagement")){
                try  {Add-PSSnapin Quest.ActiveRoles.ADManagement -ErrorAction Stop}
                      #Write-host "[ERROR]" -ForegroundColor Red
                      Write-Warning "This tool depends on the Quest Active Roles tools to operate"
                      $DL = Read-Host "Download? Y/N"
                      IF ($DL -eq "Y"){
                        Start ''
               finally{Write-Progress -Activity ("Checking for Quest Active Roles PSSnapIn...") -PercentComplete 100  -Status "Detected"
                       Start-Sleep -Milliseconds 150
        ELSE{Write-Progress -Activity ("Checking for Quest Active Roles PSSnapIn...") -PercentComplete 100   -Status "Detected"
             start-sleep -Milliseconds 150
            if (-not($SourceCredential)){
                if (-not($credential)){
                    $credential= Get-Credential -Message "Enter credentials which can browse AD in Source and Target"}
                ELSE{Write-Progress -Activity ("Connecting to source...") -PercentComplete 25  -Status "Cached credential detected, continuing..."
                $credential = $SourceCredential
        Write-host ("Detected: " + $UserNames.Count + " objects")
           #Connect to source
        start-sleep -Milliseconds 150
        Write-Progress -Activity ("Connecting to source...") -PercentComplete 50  -Status "Connecting..."
        try  {Connect-QADService $service -Credential $credential -ErrorAction Stop  | Out-Null}
              Write-warning "Error ocurred connecting to AMATLDC01 to pull source OU paths, check credentials..."
      finally{Write-Progress -Activity ("Connecting to source...") -PercentComplete 100  -Status "Connected!"
              Start-Sleep -Milliseconds 150
    forEach ($CurrentObject in $UserNames){
        Try  {start-sleep -Milliseconds 100
              Write-Progress -Activity ("Looking up $CurrentObject on $service") -PercentComplete 75 -Status "Gathering Info..."
              $user = get-qaduser $CurrentObject -DontUseDefaultIncludedProperties `
                -IncludedProperties targetAddress `
                -Service $service `
                -ConnectionAccount $credential.UserName `
                -ConnectionPassword $credential.Password `
                -ErrorAction Stop
              Write-Debug "Troubleshoot `$user"}
       catch {Write-Warning ("Unable to perform Get-QADUser for object $CurrentObject from service: $service `n check `$CurrentObject")
             Write-Debug "Troubleshoot `$user"
        if ($user){
            if (-not($user.targetAddress)){
                if ($WriteWarning){
                    Write-Warning ("Unable to find targetAddress: $CurrentObject has NOT Switched")
        if ($user.targetAddress -like "*@target.qmm"){
            $obj += [pscustomobject]@{UserName=$user;Status="Switched"}
            Write-Progress -Activity ("Looking up $CurrentObject on $service") -PercentComplete 100 -Status "Gathered!"
            Start-Sleep -Milliseconds 150
        if ($user.Email){
            $obj += [pscustomobject]@{UserName=$user;Status="Not Switched"}
            Write-Progress -Activity ("Looking up $CurrentObject on $service") -PercentComplete 100 -Status "Gathered!"
            Start-Sleep -Milliseconds 150
            $obj += [pscustomobject]@{UserName=$user;Status="No E-mail"}
            Write-Progress -Activity ("Looking up $CurrentObject on $service") -PercentComplete 100 -Status "Gathered!"
            Start-Sleep -Milliseconds 150
        $obj | ft

Continue Reading...

Quest Migration Manager : 'Resolving QMM No Matching was found'

August 22, 2014 FoxDeploy

If in Baretail (or your log file of choice) you see something similar to the following when attempting to Switch a mailbox.



Starting to process collection ‘SPEEDITUP’ (Source server ‘FOXXC04’, target server ‘FOXDAG01’, type ‘0’). Starting to process item ‘/o=Contoso/ou=Exchange Administrative Group ((01))/cn=Recipients/cn=Test.User’ (Collection ‘SPEEDITUP’, PST: ‘B5BDFEF890CD84439F04FC4B711E3E75186’, SyncMailboxInfo: ‘True’, SyncSwitched: ‘False’, SyncAllContent: ‘False’, SyncFolderContent: ‘False’). Retrieve info from RootDSE. Getting RootDSE Logging on to the mailbox ‘/o=Contoso/ou=Exchange Administrative Group ((01))/cn=Configuration/cn=Servers/cn=FOXXC04/cn=Microsoft System Attendant’ (Server: ‘FOXXC04’, user: ‘FOX_adAD_svc_exqmm’, has assocciated PFDB: ‘False’, connection flags = 32768). Trying to synchronize MAPI profile creation. Creating MAPI profile. Trying to logon. Trying to open private store. Trying to open address book. Setting search path. Setting cached Address lists. No matching was found for ‘/o=Contoso/ou=Exchange Administrative Group ((01))/cn=Recipients/cn=Test.User’. Please make sure that the user is migrated, the matching rules are correct, and the script components are functioning. Skipping collection ‘Resource Calendar Mailboxes’ because it is disabled. Skipping collection ‘All Mailboxes’ because it is disabled. No more items to process in this session. Logging off from the mailbox ‘/o=Contoso/ou=Exchange Administrative Group ((01))/cn=Configuration/cn=Servers/cn=FOXXC04/cn=Microsoft System Attendant’ (Server: ‘FOXXC04’, user: ‘FOX_adAD_svc_exqmm’). Session has been stopped.

If you run into this issue, first verify that a Quest AD Account Migration Completed Successfully. If not, then verify that the Exchange Quest service has permission to the mailbox in the source by using the following PowerShell Commandlette

Function Fix-MailboxPerms { $ServiceQMMAccount = "foxdeploy\\\_svcQMM"


#if $UserName came from MailboxStatistics, give us the size if ($username.TotalitemSize){Write-Host (($UserName.TotalItemSize).Value.ToMB() ) "MB Mbx" } $username| get-mailbox | % { $user = $\_; "Setting AD Rights for object...$ServiceQMMAccount" $user | Add-ADPermission -User $ServiceQMMAccount -AccessRights GenericAll -ExtendedRights Receive-As,Send-As

"Setting Mailbox Permissions...$ServiceQMMAccount" $user | Add-MailboxPermission -User $ServiceQMMAccount -AccessRights FullAccess



If this still doesn’t work, search in the Target Domain to verify that a mailbox exists for the user account. If not, this may signify that Calendar Sync agent isn’t configured to run regularly (possibly indicating a new user account). You can quickly resolve the matter by Mail enabling the user’s account in the target domain with the following syntax:

powershell Get-User Test.User1 | Enable-Mailbox -Database Region\_db01

Once completed, run another AD Object migration session, and then watch Calendar Sync (CSA.log) or Mail Source Agent (EMWMSA.log) to see if the changes are noted. You can also directly compare attributes within AD from the source and target accounts.

You should see the LegacyExchangeDN for the Source Account listed as an x500 proxyAddress in the target account. You should also see the LegacyExchangeDN for the Target account listed as an x500 proxyAddress in the source account.

In no time, you should see the mailbox switch/sync/or whatever you were trying to do.

Continue Reading...

PowerShell: Reapply ACLs Based on Folder Name

August 22, 2014 FoxDeploy

Recently in a project, we needed a way to reapply ACLs to user’s roaming profile / home folder directories on a file share.

Wait, why would you ever need to do that?

Somehow someone in the service desk noted that a user was missing permission to his share, and decided to resolve this by forcing permissions down from the top folder object all the way down.  They of course opted to choose the option to ‘replace all properties with inherited properties from this object.’  You know, the one which displayed a scary ‘Are you really sure you want to do this?’ alert.


This really caused a lot of issues, as we were in a project migrating users from Domain A to Domain B.

Fortunately for all involved, the user shares impacted had a naming convention that basically a user’ roaming profile was named after that person’s SAm or UPN name.  Using this olive branch, we were able to get very close to fixing 95% of users with one swift tool.

The overall goal of this tool is to determine what permissions the folder should have had based on the user name for Domain A. If the user account is valid, then the tool will apply an appropriate ACE for the user’s account in Domain A.

Once this was finished, we then used the functionality provided by Quest Migration Manager for Active Directory to apply a Domain B equivalent permission to all of the affected directories. You could also do the same by rerunning this script and manually specifying the name of Domain B within the Domain parameter.

#     ----Name : FoxDeploy ACL Fixing tool
#     ---Author: Stephen Owen, $company, 7.16.2014
#     Function : Post User-Migration, use this tool to add users back to a share with their own name
#     ---Notes : This tool can be used to restore appropriate Read-level of the $company account to a users share
#                 needed after a particularly overzealous application of permissions stripped Source Perms from certain shares
Function Fix-ACLs{
       Use this tool Post User-Migration, to give a user permission to a share, where the user's name matches the folder
       This tool receives a file share, then enumerates one level for directory objects. As the directories expected should be in the format of “userfirstname.userlastname”,
       it is trivial to determine the account which should have ownership of this share.
    .PARAMETER FileServers
       The path to the fileserver to process
       The AD Server to use to obtain the user object for permissions application
    .PARAMETER DomainA
        The Short Name of the original Domain
       Fix-ACLs -FileServers "\\aumjfp01\users" -ADServer "localserver"
       > This will return all of the directories under \\aumjfp01\users.  The rest of the example will assume the current user in question is OLDDOMAIN\Jimmy.John.
       > A Quest AD lookup is  performed to verify a valid user exists for the name of the folder (\\aumjfp01\“jimmy.john”).
       >  In this case an AD lookup is performed for OLDDomain\jimmy.john
       > If valid, give this user permission to the folder, if not, add user name to $badUsers, which is reported at the end of the function
    [Parameter(Mandatory=$true,Position=0,HelpMessage="Please specify the file shares to be processed, in the format \\server\share")]
            [string]$ADServer = "amatldc01",
            [string]$DomainA = "OldDomain"
    if (-not($credential)){
                $credential= Get-Credential -Message "Enter credentials which can be used to lookup a user account in AD on $ADServer"
                ELSE{"Cached credential detected, continuing..."
    Write-host "Checking for Quest Active Roles PSSnapIn..." -ForegroundColor White -NoNewline
        try  {Add-PSSnapin Quest.ActiveRoles.ADManagement -ErrorAction Stop}
                Write-host "[ERROR]" -ForegroundColor Red
                Write-Warning "This tool depends on the Quest Active Roles tools to operate"
                $DL = Read-Host "Download? Y/N"
                IF ($DL -eq "Y"){
                Start ''
    Write-host "[OKAY]" -ForegroundColor Green 
    #Connect to source
    Write-host "Connecting to Source..." -NoNewline
        try  {Connect-QADService $ADServer -Credential $credential -ErrorAction Stop | Out-Null}
                Write-warning "Error ocurred connecting to $ADServer to pull source OU paths, check credentials..."
        Write-host "[OKAY]" -ForegroundColor Green 
    $badUsers = @()
    ForEach ($FileServer in $FileServers){
        try  {$UserDirs = Get-ChildItem $FileServer -ErrorAction Stop | Where-Object PSIsContainer -eq $true}
        catch{Write-Warning "Error occurred connecting to $FileServer, check spelling and if valid path"
        $i = 0 
        ForEach ($userDir in $UserDirs){
            Write-Progress -Activity ("Setting permission for $userDir of $FileServer") -PercentComplete (($i/($UserDirs.count)*100 ))
            $acl = Get-Acl $userDir.FullName
            $DomainA_ADAccount = ($DomainA + "\" + $userDir.Name)
            "Verifying $DomainA_ADAccount is valid user"
            if (Get-QADUser $DomainA_ADAccount){
                Write-Host "User good, setting permissions..." -NoNewline
                    $inherit = []"ContainerInherit, ObjectInherit"
                    $propagation = []"None"
                    $permission = "$DomainA_ADAccount","FullControl",$inherit,$propagation,"Allow"
                    $accessRule = New-Object System.Security.AccessControl.FileSystemAccessRule $permission
                    $acl.SetAccessRule($accessRule) | Out-Null
                    Write-Verbose  "About to set $acl"
                    try   {#In a future version, this should create a job and then check for status while writing out progress
                           Set-Acl -path $userDir.FullName -AclObject $acl}
                    catch {
                Write-Host "[OKAY]" -ForegroundColor GREEN
                write-error "Not a valid user account"
                $badUsers += $DomainA_ADAccount
            "User Completed"
            #End of this userDir
    #End of this FileServer
#End of function
"The following user accounts were not found during an AD lookup, please investigate"
return $badUsers}
Continue Reading...

My most useful 8 liner yet

August 20, 2014 FoxDeploy

In my line of work, I’m constantly copying and pasting from e-mails, SharePoint, into PowerShell to do some meaningful work.  This means all day long I’m setting variables equal to paste, then removing blank entries, and then splitting them (because people never say ‘please migrate these computers’ and provide a list like $computers=”compA”,”compB”,”compC”.

I then end up Pasting, and running $variable= Paste, then $variable.Trim().Split(“`n”).  I finally decided to speed things up by making this short little few-liner.  I always forgot whether it was Split-N-Trim or not, so I made an alias for the other way around.

It now has a permanent place within my $PSProfile

```powershell Function Split-N-Trim { param( [Parameter(Mandatory=$True,ValueFromPipeline=$True)][string[]]$Objects) Write-Host (&quot;-Recieved &quot; + $Objects.Count +&quot; objects&quot;) $Objects = $Objects.Trim().Split(&quot;`n&quot;) Write-Host (&quot;Returning &quot; + $Objects.Count +&quot; objects&quot;) Return $Objects } new-alias Trim-N-Split Split-N-Trim


Continue Reading...

Using Try Catch to get Better Output in your Functions and Tables

August 19, 2014 FoxDeploy

Recently at a client, we have a situation arise in which we needed to verify which Domain a number of PCs were joined to.

One issue we encountered was that because of certain underlying settings conflicts between WINS, DHCP and DNS, occasionally the wrong computer name would reply to certain commands. To alleviate this, I wrote the following short code snippet to grab a computer from a CSV, run a Get-WMIObject command, and then return the domain name.

Write-Host "Ok output" import-csv .\\Desktop\\Ampcs.csv | select -expand Name | ForEach-Object { $name = $\_ Get-WmiObject -Class Win32\_ComputerSystem -ComputerName $name | select @{Name="RUMName";Expression={$name} },@{Name="ReplyName";Expression={$\_.Name} },@{Name="Domain";Expression={$\_.Domain} } }

The problem with this approach is that if the computer is offline or doesn’t allow remote procedure calls (a tell-tale for being in the wrong domain, as WinRM should be enabled if the PC is in the right place via Group Policy) emerges in the form of nasty error messages.

To ‘get the job done’ I told people just to ignore this message…but the dev inside of me couldn’t handle the ugliness of this whole process. With just a few small edits, I was able to get this much better output.

Write-Host "Better output" import-csv .\\Desktop\\Ampcs.csv | select -expand Name | ForEach-Object { $name = $\_ try {$a=Get-WmiObject -Class Win32\_ComputerSystem -ErrorAction Stop -ComputerName $name} catch{$a= \[pscustomobject\]@{Name=$name;Domain="Access Denied"}} \[pscustomobject\]@{RUM\_Name=$name;ReplyName=$a.Name;Domain=$a.Domain} } 

The core difference here Is that I’m setting an object $a equal to either the results of a Get-WMIObject Command, or as a PSCustomObject with a hashtable that contains a description of my problem and some other attributes. It never occurred to me before to try this route, using the -ErrorAction Stop to trigger an alternate output of my command.


I really like this method of communicating data, as it allows me to handle edge case scenarios gracefully. If I wanted to take it one step further, I could use multiple Catch Error blocks to provide logic for all different sorts of Error Messages, which would really elevate this code.

Hope you enjoy it!

Continue Reading...

Microsoft MVP

Five time Microsoft MVP, and now I work for the mothership

Need Help?

Get help much faster on our new dedicated Subreddit!

depicts a crowd of people in a night club with colored lights and says 'join the foxdeploy subrreddit today'

Blog Series
series_sml_PowerShellGUI series_sml_IntroToRaspberryPi Programming series_sml_IntroToWindows Remote Management Series The Logo for System Center Configuration Manager is displayed here Depicts a road sign saying 'Learning PowerShell Autocomplete'

Blog Stats