I wanted to share the resolution to a crazy hair-pulling issue I’ve been having since late mid-July with Microsoft Support unable to determine the cause.
We had been using PXE booting from our locations for a while now. Got everything setup by our infrastructure teams with IP Helpers, no DHCP scope options. I tuned the registry settings for RamDiskTFTPBlockSize and RamDiskTFTPWindowSize for our VPN tunnels. Everything was working great for months.
Then all of a sudden PXE booting stopped working… But only for UEFI devices… Legacy clients were still booting correctly.
Error:
File: \Boot\BCD
Status: 0xc000000f
Info: The Boot Configuration Data for you PC is missing or contains errors.
Everything still worked great at our local office. So we figured something had changed on the networking side. But then we thought to test booting an unknown device and UEFI PXE works… Huh? Since Unknown Computers work, that basically rules out Networking. From any subnet that has to traverse a WAN connection to an ConfigMgr PXE server, UEFI PXE booting fails for computers that are in the ConfigMgr database. ‘Unknown’ computers boot fine. Legacy PXE works from all scenarios.
Computer Type | Client Location | Server Location | UEFI PXE | Legacy PXE |
Known Computer | Local Office | Local Data Center | Pass | Pass |
Known Computer | Local Office | Remote Data Center | Fail | Pass |
Known Computer | Remote Office | Local Data Center | Fail | Pass |
Known Computer | Remote Office | Remote Data Center | Fail | Pass |
Unknown Computer | Local Office | Local Data Center | Pass | Pass |
Unknown Computer | Local Office | Remote Data Center | Pass | Pass |
Unknown Computer | Remote Office | Local Data Center | Pass | Pass |
Unknown Computer | Remote Office | Remote Data Center | Pass | Pass |
I tried every bit of advice the internet had to offer:
- DHCP options 60, 66, 67 are NOT set on the subnet(s)
- IP Helpers are in place on the router config to forward DHCP broadcasts
- TFTP Options have been set to ensure that the block size is small enough for our VPN tunnels and prevent fragmentation
- [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP]
- “RamDiskTFTPBlockSize”=dword:00000550 (1360)
- “RamDiskTFTPWindowSize”=dword:00000004 (4)
- [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WDSServer\Providers\WDSTFTP]
- “MaximumBlockSize”=dword:00000550 (1360)
- [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP]
- Tried different task sequences with difference boot images.
- Tried updating boot images
- Disabled and re-enabled PXE on the DPs to rebuild the PXE role.
I asked for some help from the networking team and I was able to get a router setup like one of our locations with a mirrored port.
I did some Wireshark captures from the client. The only difference in the Wireshark captures from a working and non-working client is the DCHP ACK contains option 243 and option 252 with the BCD file path:
Non-working client:
Working client:
So what gives?
After looking at the SMSPXE.log though I noticed that there were two attempts made to update the Device Cached LastAdv Time. On clients that were failing to boot, the ‘Client boot action reply’ found no advertisements:
So this got me thinking, maybe there was some sort of timeout happening after the first time the Device Cached LastAdv Time was updated.
I starting to poke around the ConfigMgr DB at the stored procedures, filtered down to ones with just ‘PXE’ in the name.
In the SQL stored procedure dbo.NBS_GetPXEBootAction I noticed the following SELECT statement:
select * from LastPXEAdvertisement as lpa where lpa.MAC_Addresses = @MACAddress AND lpa.LastPXEAdvertisementID = sp.OfferID
AND LastPXEAdvertisementTime < DATEADD(SECOND, -40, @CurrentTimeInUTC
Based on this, it look like ConfigMgr is only allowing up to 40 seconds for a device to complete the download of smsboot\x64\wdsmgfw.efi. Since our locations take longer than 40 seconds to download the NBP, they would then fail to show up in this query when the second request for this device is made, therefore breaking the boot process.
So I tried changing -40 to -120, as the slowest location we have takes about 90 seconds to download this file:
select * from LastPXEAdvertisement as lpa where lpa.MAC_Addresses = @MACAddress AND lpa.LastPXEAdvertisementID = sp.OfferID
AND LastPXEAdvertisementTime < DATEADD(SECOND, -120, @CurrentTimeInUTC
Computers are now UEFI PXE booting as expected!
It’s still possible that something changed on the networking / server side that’s causing these initial downloads to be slower, but at this time, I’m going to say…
Hi
LastPXEAdvertisementTime < DATEADD(SECOND, -40, @CurrentTimeInUTC
does not seem to be present on the SCCM 1706 version?
LikeLike
Hi Thomas,
I upgraded to 1706 as a troubleshooting step before finding the stored procedure. You’re looking under CM_XXX > Programmability > Stored Procedures > dbo.NBS_GetPXEBootAction?
LikeLike
It looks like this in our 1706
USE [CM_UNI]
GO
/****** Object: StoredProcedure [dbo].[NBS_GetPXEBootAction] Script Date: 03-01-2018 11:29:10 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
—
— Name : NBS_GetPXEBootAction
— Version : 5.0.8540.1007
— Definition : SqlObjs
— Scope : PRIMARY_OR_SECONDARY
— Object : P
— Dependencies : ProgramOffers, vSMS_TaskSequencePackage
— Description :
—
ALTER PROCEDURE [dbo].[NBS_GetPXEBootAction]
@ItemKey int,
@UnknownItemKey int,
@SMBIOSGUID nvarchar(38),
@MACAddress nvarchar(38),
@DPName nvarchar(255),
@Architecture int = 1
AS
BEGIN
SET NOCOUNT ON
declare @SMBIOS nvarchar(38)
declare @MAC nvarchar(64)
DECLARE @CurrentTimeInUTC datetime
SET @CurrentTimeInUTC = getutcdate()
set @SMBIOS = @SMBIOSGUID
set @MAC = @MACAddress
IF (CHARINDEX (‘_’, @SMBIOSGUID) > 0 OR CHARINDEX (‘%’, @SMBIOSGUID) > 0)
BEGIN
IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID LIKE CommonSMBIOS_GUIDs.SMBIOS_GUID))
SET @SMBIOS = ‘Common’
END
ELSE
BEGIN
IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID = CommonSMBIOS_GUIDs.SMBIOS_GUID))
SET @SMBIOS = ‘Common’
END
IF (CHARINDEX (‘_’, @MACAddress) > 0 OR CHARINDEX (‘%’, @MACAddress) > 0)
BEGIN
IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress LIKE CommonMACAddresses.MACAddress))
SET @MAC = ‘Common’
END
ELSE
BEGIN
IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress = CommonMACAddresses.MACAddress))
SET @MAC = ‘Common’
END
SELECT MACAddress, SMBIOS_GUID, SMS_UniqueIdentifier0, LastPXEAdvertisementID, LastPXEAdvertisementTime,
OfferID, OfferIDTime, PkgID, PackageVersion, PackagePath, BootImageID, Mandatory, Known,
PresentTimeEnabled, ExpirationTimeEnabled, PresentTimeInUTC, ExpirationTimeInUTC
FROM (
SELECT @MACAddress AS ‘MACAddress’,
@SMBIOSGUID AS ‘SMBIOS_GUID’,
xref.GUID AS ‘SMS_UniqueIdentifier0’,
” AS ‘LastPXEAdvertisementID’,
” AS ‘LastPXEAdvertisementTime’,
sp.OfferID AS ‘OfferID’,
po.PresentTime AS ‘OfferIDTime’,
po.PkgID AS ‘PkgID’,
tspkg.Version AS ‘PackageVersion’,
cdp.URL AS ‘PackagePath’,
tspkg.BootImageID AS ‘BootImageID’,
CASE
WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
ELSE 0 END
AS ‘Mandatory’,
CASE
WHEN (EXISTS (select System_DISC.ItemKey from System_DISC where
System_DISC.ItemKey = xref.MachineID AND
ISNULL(System_DISC.Decommissioned0,0) != 0 AND
ISNULL(System_DISC.Obsolete0,0) != 0 AND
ISNULL(System_DISC.Unknown0,0) != 0))
THEN 0
WHEN (xref.MachineID = @ItemKey) THEN 1
ELSE 0 END
AS ‘Known’,
CASE WHEN ((po.TimeEnableFlag & 0x0001) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_PRESENT = 0x0001
AS ‘PresentTimeEnabled’,
CASE WHEN ((po.TimeEnableFlag & 0x0002) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_EXPIRATION = 0x0002
AS ‘ExpirationTimeEnabled’,
CASE WHEN ((po.TimeEnableFlag & 0x0100) != 0) THEN po.PresentTime ELSE dbo.fnConvertLocalToUTC(po.PresentTime) END — PROGOFFER_GMT_PRESENT = 0x0100
AS ‘PresentTimeInUTC’,
CASE WHEN ((po.TimeEnableFlag & 0x0200) != 0) THEN po.ExpirationTime ELSE dbo.fnConvertLocalToUTC(po.ExpirationTime) END — PROGOFFER_GMT_EXPIRATION = 0x0200
AS ‘ExpirationTimeInUTC’
FROM MachineIdGroupXRef xref
JOIN ResPolicyMap AS rpm
ON xref.MachineID = rpm.MachineID
JOIN SoftwarePolicy AS sp
ON (rpm.PADBID = sp.PADBID AND rpm.IsTombstoned != 1)
JOIN ProgramOffers AS po
ON sp.OfferID = po.OfferID AND po.[Action] 3
JOIN vSMS_TaskSequencePackage AS tspkg
ON (tspkg.PkgID = sp.PkgID AND (ISNULL(tspkg.BootImageID,”)!= ”))
JOIN SMSPackages
ON (tspkg.BootImageID = SMSPackages.PkgID)
JOIN PkgPrograms as pp
ON (tspkg.PkgID = pp.PkgID AND (pp.ProgramFlags & 4096) = 0 AND pp.[Action] 3) — 4096 = disabled
JOIN ContentDPMap AS cdp
ON (cdp.ContentID = tspkg.BootImageID AND cdp.ServerName = @DPName)
WHERE xref.MachineID IN (@ItemKey) AND
(@Architecture = 1 OR SMSPackages.Architecture = @Architecture) AND — Architecture of 1 means any, 0 means 32-bit, 9 means 64-bit
((CASE WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
ELSE 0 END) = 0 OR
(NOT EXISTS (select * from LastPXEAdvertisement as lpa where (lpa.MAC_Addresses = @MAC OR lpa.SMBIOS_GUID = @SMBIOS) AND lpa.LastPXEAdvertisementID = sp.OfferID))) AND
(po.OfferFlags & 0x00040000) != 0 — 0x00040000 = AP_ENABLE_TS_FROM_CD_AND_PXE
) AS X
WHERE ((PresentTimeEnabled = 0 ) OR (@CurrentTimeInUTC >= PresentTimeInUTC)) AND
((ExpirationTimeEnabled = 0) OR (@CurrentTimeInUTC <= ExpirationTimeInUTC))
ORDER BY Known DESC, Mandatory DESC, OfferID DESC
LikeLike
I am seeing the same thing as Thomas. Any thoughts?
LikeLike
Thomas, it looks like maybe a HitFix to 1706 changed this sproc. It was also changed in 1710. We just implemented a change today to resolve the issue again. We also diffed the sproc from 1706 and 1710 to see the change. And 1710 is the same as you report. Luckily jonk is a pretty smart guy and kept a backup of the 1706 sproc for us.
I’ll find the code for the updated sproc tomorrow and post it for you.
LikeLike
Thanks!
The issue we see is allbeit a bit different as we do not seem to see timeouts but just that Option 243 and 252 is missing.
Increase in TFTP package size seems to worsen it…
LikeLike
Hmmm, does sound a bit different. We’ll also pretend that today is the tomorrow I was speaking of…
Here is the full code that we have in our environment:
USE [CM_A01]
GO
/****** Object: StoredProcedure [dbo].[NBS_GetPXEBootAction] Script Date: 1/31/2018 12:00:50 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
—
— Name : NBS_GetPXEBootAction
— Version : 5.0.8577.1005
— Definition : SqlObjs
— Scope : PRIMARY_OR_SECONDARY
— Object : P
— Dependencies : ProgramOffers, vSMS_TaskSequencePackage
— Description :
—
ALTER PROCEDURE [dbo].[NBS_GetPXEBootAction]
@ItemKey int,
@UnknownItemKey int,
@SMBIOSGUID nvarchar(38),
@MACAddress nvarchar(38),
@DPName nvarchar(255),
@Architecture int = 1
AS
BEGIN
SET NOCOUNT ON
declare @SMBIOS nvarchar(38)
declare @MAC nvarchar(64)
DECLARE @CurrentTimeInUTC datetime
SET @CurrentTimeInUTC = getutcdate()
set @SMBIOS = @SMBIOSGUID
set @MAC = @MACAddress
IF (CHARINDEX (‘_’, @SMBIOSGUID) > 0 OR CHARINDEX (‘%’, @SMBIOSGUID) > 0)
BEGIN
IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID LIKE CommonSMBIOS_GUIDs.SMBIOS_GUID))
SET @SMBIOS = ‘Common’
END
ELSE
BEGIN
IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID = CommonSMBIOS_GUIDs.SMBIOS_GUID))
SET @SMBIOS = ‘Common’
END
IF (CHARINDEX (‘_’, @MACAddress) > 0 OR CHARINDEX (‘%’, @MACAddress) > 0)
BEGIN
IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress LIKE CommonMACAddresses.MACAddress))
SET @MAC = ‘Common’
END
ELSE
BEGIN
IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress = CommonMACAddresses.MACAddress))
SET @MAC = ‘Common’
END
SELECT MACAddress, SMBIOS_GUID, SMS_UniqueIdentifier0, LastPXEAdvertisementID, LastPXEAdvertisementTime,
OfferID, OfferIDTime, PkgID, PackageVersion, PackagePath, BootImageID, Mandatory, Known,
PresentTimeEnabled, ExpirationTimeEnabled, PresentTimeInUTC, ExpirationTimeInUTC
FROM (
SELECT @MACAddress AS ‘MACAddress’,
@SMBIOSGUID AS ‘SMBIOS_GUID’,
xref.GUID AS ‘SMS_UniqueIdentifier0’,
” AS ‘LastPXEAdvertisementID’,
” AS ‘LastPXEAdvertisementTime’,
sp.OfferID AS ‘OfferID’,
po.PresentTime AS ‘OfferIDTime’,
po.PkgID AS ‘PkgID’,
tspkg.Version AS ‘PackageVersion’,
cdp.URL AS ‘PackagePath’,
tspkg.BootImageID AS ‘BootImageID’,
CASE
WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
ELSE 0 END
AS ‘Mandatory’,
CASE
WHEN (EXISTS (select System_DISC.ItemKey from System_DISC where
System_DISC.ItemKey = xref.MachineID AND
ISNULL(System_DISC.Decommissioned0,0) != 0 AND
ISNULL(System_DISC.Obsolete0,0) != 0 AND
ISNULL(System_DISC.Unknown0,0) != 0))
THEN 0
WHEN (xref.MachineID = @ItemKey) THEN 1
ELSE 0 END
AS ‘Known’,
CASE WHEN ((po.TimeEnableFlag & 0x0001) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_PRESENT = 0x0001
AS ‘PresentTimeEnabled’,
CASE WHEN ((po.TimeEnableFlag & 0x0002) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_EXPIRATION = 0x0002
AS ‘ExpirationTimeEnabled’,
CASE WHEN ((po.TimeEnableFlag & 0x0100) != 0) THEN po.PresentTime ELSE dbo.fnConvertLocalToUTC(po.PresentTime) END — PROGOFFER_GMT_PRESENT = 0x0100
AS ‘PresentTimeInUTC’,
CASE WHEN ((po.TimeEnableFlag & 0x0200) != 0) THEN po.ExpirationTime ELSE dbo.fnConvertLocalToUTC(po.ExpirationTime) END — PROGOFFER_GMT_EXPIRATION = 0x0200
AS ‘ExpirationTimeInUTC’
FROM MachineIdGroupXRef xref
JOIN ResPolicyMap AS rpm
ON xref.MachineID = rpm.MachineID
JOIN SoftwarePolicy AS sp
ON (rpm.PADBID = sp.PADBID AND rpm.IsTombstoned != 1)
JOIN ProgramOffers AS po
ON sp.OfferID = po.OfferID AND po.[Action] 3
JOIN vSMS_TaskSequencePackage AS tspkg
ON (tspkg.PkgID = sp.PkgID AND (ISNULL(tspkg.BootImageID,”)!= ”))
JOIN SMSPackages
ON (tspkg.BootImageID = SMSPackages.PkgID)
JOIN PkgPrograms as pp
ON (tspkg.PkgID = pp.PkgID AND (pp.ProgramFlags & 4096) = 0 AND pp.[Action] 3) — 4096 = disabled
JOIN ContentDPMap AS cdp
ON (cdp.ContentID = tspkg.BootImageID AND cdp.ServerName = @DPName)
WHERE xref.MachineID IN (@ItemKey) AND
(@Architecture = 1 OR SMSPackages.Architecture = @Architecture) AND — Architecture of 1 means any, 0 means 32-bit, 9 means 64-bit
((CASE WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
ELSE 0 END) = 0 OR
(NOT EXISTS (select * from LastPXEAdvertisement as lpa where (lpa.MAC_Addresses = @MAC OR lpa.SMBIOS_GUID = @SMBIOS) AND lpa.LastPXEAdvertisementID = sp.OfferID AND LastPXEAdvertisementTime = PresentTimeInUTC)) AND
((ExpirationTimeEnabled = 0) OR (@CurrentTimeInUTC <= ExpirationTimeInUTC))
ORDER BY Known DESC, Mandatory DESC, OfferID DESC
END
LikeLike