Forcing a time sync during Windows Autopilot OOBE to combat time related issues

[anyweb]

Introduction

I recently encountered an issue when after Windows Autopilot enrollment was completed, the end user was unable to browse on-premises network resources such as file shares when logged on with using the Windows Hello for Business login method. The workaround was to lock the screen, and then login again using username and password as credentials. We did not see the same problem when Windows Autopilot enrollment on home WI-Fi or even when using the Enterprise WI-Fi connections. To add to the confusion, the problem fixed itself by the following day with no user or other changes.

Troubleshooting the problem

Troubleshooting the issue was complex and took time as we didn't understand the root cause, but eventually it all fell into place, and I'll summarize it here. In a nutshell, the reason it was failing was because when on the wired on-premises LAN network, outbound connections to time.windows.com (Windows default NTP server) were blocked. This wasn't obvious immediately, but became clear later. The issue showed itself as soon as you tried to browse a UNC path in Windows File Explorer. You would see a message in the task tray stating that Windows needs your current credentials, as shown below.

Locking the screen, and unlocking using WHFB would not work, you had to lock the screen and enter the username+password to browse the on-premises resources without error.

To assist with troubleshooting the following were useful.

• Enable Kerberos event logging - Windows Server | Microsoft Docs
• Error code 0x3E Definitions (eventsentry.com)
• https://tech.nicolonsky.ch/mastering-windows-hello-for-business-with-your-hybrid-identity/

Further analysis of the problem revealed the following events in event viewer. The first was 0x3e KDC_ERR_CLIENT_NOT_TRUSTED

and another error in the Security-Kerberos event log made it even more clear as to what the problem was, event ID 8.

Quote

"The following error was returned from the certificate validation process: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file."

During the third phase of the ESP, (Account Setup) we prompt the end-user to enable Windows Hello for Business as a login method. That process of setting up Windows Hello for Business in turn creates a self-signed certificate which is used to authenticate against on-premise resources, and as we found out, when  Windows Autopilot enrollment was done on the wired on-premises LAN network, there was no ability to sync time against time.windows.com and therefore the Windows Hello for Business self-signed certificate was created in the Pacific Standard Timezone, and thus, in the future if you were doing enrollment in Sweden.

This explained why WHFB credentials didn't work for on-premises resources and also explained why the problem would self heal itself after one day.

To understand why this was happening we had to look at our Windows Autopilot PC's were delivered from the vendor. Windows Autopilot delivered computers come with a timezone of Pacific Standard Time, which is from Seattle, the home of Microsoft. If you press left-shift and F10 during the first screen in Windows Autopilot OOBE and run the Powershell command below, it would return this info.

get-timezone

If the computers system time (BIOS) was different to the configured timezone, then we'd have time sync issues which would show themselves later as the on-premises network disallowed access to time.windows.com.

 

The solution

The following PowerShell script detects the type of network we are on (Wired LAN, Zscaler VPN or Internet) and based on the type detected, attempt to sync time to the most appropriate NTP time server. This is a one time action during OOBE that takes place during the second phase of the ESP, namely Device Setup. This script was created to work with a Zscaler VPN solution so you might need to adapt it to work with your VPN. The logic is fairly simple.

When the script runs it attempts to determine the IP address of your on-premise NTP time server, in this example that could be time.windowsnoob.com, and that would resolve to  192.x.x.x if on the WIRED lan, or 100.x.x.x if on Zscaler VPN.

Note: Below is a mock up in my LAB of a wired LAN, everything works fine except the timesync to ads1.windowsnoob.lab.local. That is because an NTP server running on that fqdn doesn't exist. In a real production environment, this works just fine syncing against an on-premises NTP server whether on the Wired LAN or Zscaler VPN.

If the FQDN of the on-premise NTP server could not be determined then we assume that we are on the internet and thus use time.windows.com as the NTP server, finally, we trigger a time sync and log the results.

Below you can see the log on a device that syncs to time.windows.com

To use the script replace the on-premise NTP server address(s) with one you have configured yourself, or use any on-premise FQDN that can only be reached when on the internal network. Next, replace the datacenter IP addresses and FQDN's with your own on-premise FQDN server addresses hosting a local NTP service.

Finally, deploy the script like so

and wave bye bye to timesync issues DURING Windows Autopilot enrollments !
 

<#
.SYNOPSIS
    This script will sync time to on-premise or off-premise time servers based on what it detects
    
.DESCRIPTION
    This script should be deployed to Windows Autopilot devices as required in SYSTEM context
.PARAMETER [none]
    This script does not take any parameters.
.EXAMPLE
    
.NOTES
    Version: 0.1 2021/9/21 initial script creation   
    Version: 0.2 2021/09/22 re-encoding after cleaning up
    Version: 0.3 2021/09/23 datacenter check and set ntp server accordingly
    Version: 0.4 2021/09/23 byod confirmation in logging
    Version: 0.5 2021/10/11 synctime during OOBE
    
.LINK
    
.Author Niall Brady 2021/9/21
#>
Function LogWrite
{
   Param ([string]$logstring)
   $a = Get-Date
   $logstring = $a,$logstring
   Try
{
    Add-content $Logfile -value $logstring  -ErrorAction silentlycontinue
}
Catch
{
    $logstring="Invalid data encountered"
    Add-content $Logfile -value $logstring
}
   write-host $logstring
}

Function Cleanup
{# del the scheduled task so this won't run again...
#$TaskName = "SetTimeZone"
#$Delete = RemoveScheduledTask $TaskName
#LogWrite "Was the scheduled task removed: $Delete"
LogWrite "Exiting script."
break}

Function RemoveScheduledTask {
 
 try {
    LogWrite "About to remove scheduled task: $TaskName..."
    Unregister-ScheduledTask -TaskName $TaskName -TaskPath "\windowsnoob\" -Confirm:$false -ErrorAction Stop | Out-Null
    LogWrite "Successfully removed the scheduled task"
    return $true
    }

catch {
LogWrite "Couldn't remove scheduled task, please see the reason why in the debug line below this one."
$ErrorMessage = $_.Exception.Message  # Catch the error
LogWrite "DEBUG: $ErrorMessage"
    return $false
    }
    
   
}

function Set-SystemTime {

    LogWrite "Time resync forced"
    $ServiceName = 'W32time'
    $arrService = Get-Service -Name $ServiceName

while ($arrService.Status -ne 'Running')
{

    Start-Service $ServiceName
    LogWrite $arrService.status
    LogWrite 'Service starting'
    Start-Sleep -seconds 15
    $arrService.Refresh()
    if ($arrService.Status -eq 'Running')
    {
        LogWrite 'Service is now Running'
    }

}


    $whoami= & whoami
    LogWrite "DEBUG: whoami = $whoami"
    $timeOutput = & 'w32tm' '/resync', '/force'
    
    # get last sync time and other info
    $cmdOutput = & {w32tm /query /status}
    LogWrite "DEBUG: Here is the last sync time and other info from w32tm = $cmdOutput"

    # now let's try to sync time...
    
    LogWrite "DEBUG: w32tm /resync /force = $timeOutput"
    foreach ($line in $timeOutput) {
        LogWrite  "Time resync status: $line"
        $syncSuccess = ($line.contains('completed successfully'))
        LogWrite "TimeOutPut: $timeOutput"
    }
    
    # get last stync time and other info
    $cmdOutput = & {w32tm /query /status}
    LogWrite "DEBUG: Here is the last sync time and other info from w32tm = $cmdOutput"
    return $syncSuccess
}

function Set-NTPServer {
    param
    (
        [Parameter(Mandatory=$true)]$Server
    )

    LogWrite "Setting $Server as NTP server"
    $output = & 'w32tm' '/config', '/syncfromflags:manual', "/manualpeerlist:$Server"
    LogWrite "Time resync status: $output"
    $output = & 'w32tm' '/config', '/update'
    LogWrite "Time resync status: $output"
    $output = & 'w32tm' '/resync'
    LogWrite "Time resync status: $output"
}


######################################################################################################################

# Script starts here...
$Logfile = "$env:temp\win.ap.oobe.SyncTime.log"
LogWrite "Starting the oobe synctime script..."
LogWrite "Verifying if the time service is started..."

$serviceName = 'W32Time'
$service = Get-Service -Name $serviceName
    while ($service.Status -ne 'Running') {
        Start-Service -Name $serviceName
        LogWrite "$($service.DisplayName) service is: $($service.Status)"
        LogWrite "Starting $($service.DisplayName) service - Sleeping 15 seconds"
        Start-Sleep -Seconds 15
        $service.Refresh()
        if ($service.Status -eq 'Running') {
            LogWrite "$($service.DisplayName) service is: $($service.Status)"
        }
    }

 
    if ($service.Status -eq 'Running') {
       LogWrite "The time service is running!"
        # Get active network connection
        $defaultRouteNic = Get-NetRoute -DestinationPrefix 0.0.0.0/0 | Sort-Object -Property RouteMetric | Select-Object -ExpandProperty ifIndex
        $ipv4 = Get-NetIPAddress -AddressFamily IPv4 -InterfaceIndex $defaultRouteNic | Select-Object -ExpandProperty IPAddress
        LogWrite "Local IP: $ipv4"

        # set vars...
        $ZscalerDetected = $false
        $CompanyNetworkDetected = $false
        $InternetDetected = $false
        $ips = $null
        $fqdn = "time.windowsnoob.com"
        
        # let's see are we on Zscaler...
        LogWrite "Checking type of network access"    
        

        # are we on Zscaler private access ?
        try {$ips = [System.Net.Dns]::GetHostAddresses("$fqdn")
                #$ips.IPAddressToString
                    if ($ips.IPAddressToString.StartsWith("100.64.")){
                        $ZscalerDetected = $true}
            }
        catch {LogWrite "Error getting FQDN for $fqdn, not connected to Zscaler private access"
        }
        LogWrite "on Zscaler: $ZscalerDetected "

        # are we on the windowsnoob LAN ?
        LogWrite "Checking if on windowsnoob network access"
        try {$ips = [System.Net.Dns]::GetHostAddresses("$fqdn")
        
            if ($ips.IPAddressToString.StartsWith("192.")){
                $CompanyNetworkDetected = $true}
        }
        catch {LogWrite "Error getting FQDN for $fqdn, not on windowsnoob wired LAN network"
        }
        LogWrite "on windowsnoob wired LAN: $CompanyNetworkDetected "

        # if we are on windowsnoob LAN, let's determine the datacenter location...
        if ($CompanyNetworkDetected -or $ZscalerDetected){
        LogWrite "windowsnoob LAN was detected, checking which datacenter now..."
         try {$ips = [System.Net.Dns]::GetHostAddresses("$fqdn")
                LogWrite  "$fqdn ip address:  $ips.IPAddressToString"
                if  ($ips.IPAddressToString -eq "192.67.1.250" -or "100.64.1.1"){
                $DataCenter = "Sweden"
                $NTPServer = "ads1.windowsnoob.lab.local"}
                if  ($ips.IPAddressToString -eq "192.192.4.210"){
                $DataCenter = "Seattle"
                $NTPServer = "ads2.windowsnoob.lab.local"}
                if  ($ips.IPAddressToString -eq "192.138.1.12"){
                $DataCenter = "Australia"
                $NTPServer = "ads3.windowsnoob.lab.local"}
            }
        catch { LogWrite "Error getting FQDN for $fqdn, could not determine Datacenter location"
        }
         LogWrite "dataCenter: $DataCenter "}
        
        #if ($ipv4.StartsWith('192.') -or $ZscalerDetected) {Set-NTPServer -Server 'time.windowsnoob.com'
        if ($CompanyNetworkDetected -or $ZscalerDetected) {
        
        Set-NTPServer -Server $NTPServer
            LogWrite "Looks like we are on a windowsnoob network so setting '$NTPServer'"
            if (Set-SystemTime) { cleanup }
            }
            else {Set-NTPServer 'time.windows.com'
            LogWrite "we could NOT resolve 'time.windowsnoob.com' so we are setting NTP to 'time.windows.com'"
            if (Set-SystemTime) { cleanup }
            }
}
            
LogWrite "SyncTime script completed."

 

Lastly, keep in mind that you may still see access issues from Azure resources to on-premise resources as explained by Microsoft here, usually about 30 minutes. That is normal and nothing to do with the problem solved by this script.

Conclusion

Time sync issues during Windows Autopilot enrollment can be solved by using a PowerShell script deployed to all Windows Autopilot devices as long as the script can detect if on LAN, WAN or other network and flip to the corresponding NTP server(s).

Related - https://docs.microsoft.com/en-us/mem/autopilot/networking-requirements