Written by Jonk

ConfigMgr UEFI PXE Booting Error: \Boot\BCD 0xc000000f (SOLVED)

I wanted to share the resolution to a crazy hair-pulling issue I’ve been having since late mid-July with Microsoft Support unable to determine the cause.

We had been using PXE booting from our locations for a while now. Got everything setup by our infrastructure teams with IP Helpers, no DHCP scope options. I tuned the registry settings for RamDiskTFTPBlockSize and RamDiskTFTPWindowSize for our VPN tunnels. Everything was working great for months.

Then all of a sudden PXE booting stopped working… But only for UEFI devices… Legacy clients were still booting correctly.

Error:

File: \Boot\BCD
Status: 0xc000000f
Info: The Boot Configuration Data for you PC is missing or contains errors.

2017-08-25 18_54_32-Clipboard

Everything still worked great at our local office. So we figured something had changed on the networking side. But then we thought to test booting an unknown device and UEFI PXE works… Huh? Since Unknown Computers work, that basically rules out Networking. From any subnet that has to traverse a WAN connection to an ConfigMgr PXE server, UEFI PXE booting fails for computers that are in the ConfigMgr database. ‘Unknown’ computers boot fine. Legacy PXE works from all scenarios.

Computer Type Client Location Server Location UEFI PXE Legacy PXE
Known Computer Local Office Local Data Center Pass Pass
Known Computer Local Office Remote Data Center Fail Pass
Known Computer Remote Office Local Data Center Fail Pass
Known Computer Remote Office Remote Data Center Fail Pass
Unknown Computer Local Office Local Data Center Pass Pass
Unknown Computer Local Office Remote Data Center Pass Pass
Unknown Computer Remote Office Local Data Center Pass Pass
Unknown Computer Remote Office Remote Data Center Pass Pass

I tried every bit of advice the internet had to offer:

  • DHCP options 60, 66, 67 are NOT set on the subnet(s)
  • IP Helpers are in place on the router config to forward DHCP broadcasts
  • TFTP Options have been set to ensure that the block size is small enough for our VPN tunnels and prevent fragmentation
    • [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SMS\DP]
      • “RamDiskTFTPBlockSize”=dword:00000550 (1360)
      • “RamDiskTFTPWindowSize”=dword:00000004 (4)
    • [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WDSServer\Providers\WDSTFTP]
      • “MaximumBlockSize”=dword:00000550 (1360)
  • Tried different task sequences with difference boot images.
  • Tried updating boot images
  • Disabled and re-enabled PXE on the DPs to rebuild the PXE role.

I asked for some help from the networking team and I was able to get a router setup like one of our locations with a mirrored port.

I did some Wireshark captures from the client. The only difference in the Wireshark captures from a working and non-working client is the DCHP ACK contains option 243 and option 252 with the BCD file path:

Non-working client:
Wireshark1

Working client:
Wireshark2.png

So what gives?

After looking at the SMSPXE.log though I noticed that there were two attempts made to update the Device Cached LastAdv Time. On clients that were failing to boot, the ‘Client boot action reply’ found no advertisements:
smspxe1.png

So this got me thinking, maybe there was some sort of timeout happening after the first time the Device Cached LastAdv Time was updated.

I starting to poke around the ConfigMgr DB at the stored procedures, filtered down to ones with just ‘PXE’ in the name.

In the SQL stored procedure dbo.NBS_GetPXEBootAction I noticed the following SELECT statement:

select * from LastPXEAdvertisement as lpa where lpa.MAC_Addresses = @MACAddress AND lpa.LastPXEAdvertisementID = sp.OfferID
AND LastPXEAdvertisementTime < DATEADD(SECOND, -40, @CurrentTimeInUTC

Based on this, it look like ConfigMgr is only allowing up to 40 seconds for a device to complete the download of smsboot\x64\wdsmgfw.efi. Since our locations take longer than 40 seconds to download the NBP, they would then fail to show up in this query when the second request for this device is made, therefore breaking the boot process.

So I tried changing -40 to -120, as the slowest location we have takes about 90 seconds to download this file:

select * from LastPXEAdvertisement as lpa where lpa.MAC_Addresses = @MACAddress AND lpa.LastPXEAdvertisementID = sp.OfferID
AND LastPXEAdvertisementTime < DATEADD(SECOND, -120, @CurrentTimeInUTC

2018-01-02 09_25_55-wprdcm01v - Remote Desktop Connection

Computers are now UEFI PXE booting as expected!

It’s still possible that something changed on the networking / server side that’s causing these initial downloads to be slower, but at this time, I’m going to say…

success.jpg

7 thoughts on “ConfigMgr UEFI PXE Booting Error: \Boot\BCD 0xc000000f (SOLVED)”

    1. Hi Thomas,

      I upgraded to 1706 as a troubleshooting step before finding the stored procedure. You’re looking under CM_XXX > Programmability > Stored Procedures > dbo.NBS_GetPXEBootAction?

      Like

      Reply

      1. It looks like this in our 1706

        USE [CM_UNI]
        GO
        /****** Object: StoredProcedure [dbo].[NBS_GetPXEBootAction] Script Date: 03-01-2018 11:29:10 ******/
        SET ANSI_NULLS ON
        GO
        SET QUOTED_IDENTIFIER ON
        GO
        —
        — Name : NBS_GetPXEBootAction
        — Version : 5.0.8540.1007
        — Definition : SqlObjs
        — Scope : PRIMARY_OR_SECONDARY
        — Object : P
        — Dependencies : ProgramOffers, vSMS_TaskSequencePackage
        — Description :
        —
        ALTER PROCEDURE [dbo].[NBS_GetPXEBootAction]
        @ItemKey int,
        @UnknownItemKey int,
        @SMBIOSGUID nvarchar(38),
        @MACAddress nvarchar(38),
        @DPName nvarchar(255),
        @Architecture int = 1
        AS
        BEGIN
        SET NOCOUNT ON

        declare @SMBIOS nvarchar(38)
        declare @MAC nvarchar(64)

        DECLARE @CurrentTimeInUTC datetime
        SET @CurrentTimeInUTC = getutcdate()

        set @SMBIOS = @SMBIOSGUID
        set @MAC = @MACAddress

        IF (CHARINDEX (‘_’, @SMBIOSGUID) > 0 OR CHARINDEX (‘%’, @SMBIOSGUID) > 0)
        BEGIN
        IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID LIKE CommonSMBIOS_GUIDs.SMBIOS_GUID))
        SET @SMBIOS = ‘Common’
        END
        ELSE
        BEGIN
        IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID = CommonSMBIOS_GUIDs.SMBIOS_GUID))
        SET @SMBIOS = ‘Common’
        END

        IF (CHARINDEX (‘_’, @MACAddress) > 0 OR CHARINDEX (‘%’, @MACAddress) > 0)
        BEGIN
        IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress LIKE CommonMACAddresses.MACAddress))
        SET @MAC = ‘Common’
        END
        ELSE
        BEGIN
        IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress = CommonMACAddresses.MACAddress))
        SET @MAC = ‘Common’
        END

        SELECT MACAddress, SMBIOS_GUID, SMS_UniqueIdentifier0, LastPXEAdvertisementID, LastPXEAdvertisementTime,
        OfferID, OfferIDTime, PkgID, PackageVersion, PackagePath, BootImageID, Mandatory, Known,
        PresentTimeEnabled, ExpirationTimeEnabled, PresentTimeInUTC, ExpirationTimeInUTC
        FROM (
        SELECT @MACAddress AS ‘MACAddress’,
        @SMBIOSGUID AS ‘SMBIOS_GUID’,
        xref.GUID AS ‘SMS_UniqueIdentifier0’,
        ” AS ‘LastPXEAdvertisementID’,
        ” AS ‘LastPXEAdvertisementTime’,
        sp.OfferID AS ‘OfferID’,
        po.PresentTime AS ‘OfferIDTime’,
        po.PkgID AS ‘PkgID’,
        tspkg.Version AS ‘PackageVersion’,
        cdp.URL AS ‘PackagePath’,
        tspkg.BootImageID AS ‘BootImageID’,
        CASE
        WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
        WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
        ELSE 0 END
        AS ‘Mandatory’,
        CASE
        WHEN (EXISTS (select System_DISC.ItemKey from System_DISC where
        System_DISC.ItemKey = xref.MachineID AND
        ISNULL(System_DISC.Decommissioned0,0) != 0 AND
        ISNULL(System_DISC.Obsolete0,0) != 0 AND
        ISNULL(System_DISC.Unknown0,0) != 0))
        THEN 0
        WHEN (xref.MachineID = @ItemKey) THEN 1
        ELSE 0 END
        AS ‘Known’,
        CASE WHEN ((po.TimeEnableFlag & 0x0001) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_PRESENT = 0x0001
        AS ‘PresentTimeEnabled’,
        CASE WHEN ((po.TimeEnableFlag & 0x0002) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_EXPIRATION = 0x0002
        AS ‘ExpirationTimeEnabled’,
        CASE WHEN ((po.TimeEnableFlag & 0x0100) != 0) THEN po.PresentTime ELSE dbo.fnConvertLocalToUTC(po.PresentTime) END — PROGOFFER_GMT_PRESENT = 0x0100
        AS ‘PresentTimeInUTC’,
        CASE WHEN ((po.TimeEnableFlag & 0x0200) != 0) THEN po.ExpirationTime ELSE dbo.fnConvertLocalToUTC(po.ExpirationTime) END — PROGOFFER_GMT_EXPIRATION = 0x0200
        AS ‘ExpirationTimeInUTC’
        FROM MachineIdGroupXRef xref
        JOIN ResPolicyMap AS rpm
        ON xref.MachineID = rpm.MachineID
        JOIN SoftwarePolicy AS sp
        ON (rpm.PADBID = sp.PADBID AND rpm.IsTombstoned != 1)
        JOIN ProgramOffers AS po
        ON sp.OfferID = po.OfferID AND po.[Action] 3
        JOIN vSMS_TaskSequencePackage AS tspkg
        ON (tspkg.PkgID = sp.PkgID AND (ISNULL(tspkg.BootImageID,”)!= ”))
        JOIN SMSPackages
        ON (tspkg.BootImageID = SMSPackages.PkgID)
        JOIN PkgPrograms as pp
        ON (tspkg.PkgID = pp.PkgID AND (pp.ProgramFlags & 4096) = 0 AND pp.[Action] 3) — 4096 = disabled
        JOIN ContentDPMap AS cdp
        ON (cdp.ContentID = tspkg.BootImageID AND cdp.ServerName = @DPName)
        WHERE xref.MachineID IN (@ItemKey) AND
        (@Architecture = 1 OR SMSPackages.Architecture = @Architecture) AND — Architecture of 1 means any, 0 means 32-bit, 9 means 64-bit
        ((CASE WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
        WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
        ELSE 0 END) = 0 OR
        (NOT EXISTS (select * from LastPXEAdvertisement as lpa where (lpa.MAC_Addresses = @MAC OR lpa.SMBIOS_GUID = @SMBIOS) AND lpa.LastPXEAdvertisementID = sp.OfferID))) AND
        (po.OfferFlags & 0x00040000) != 0 — 0x00040000 = AP_ENABLE_TS_FROM_CD_AND_PXE
        ) AS X
        WHERE ((PresentTimeEnabled = 0 ) OR (@CurrentTimeInUTC >= PresentTimeInUTC)) AND
        ((ExpirationTimeEnabled = 0) OR (@CurrentTimeInUTC <= ExpirationTimeInUTC))
        ORDER BY Known DESC, Mandatory DESC, OfferID DESC

        Like

  3. Thomas, it looks like maybe a HitFix to 1706 changed this sproc. It was also changed in 1710. We just implemented a change today to resolve the issue again. We also diffed the sproc from 1706 and 1710 to see the change. And 1710 is the same as you report. Luckily jonk is a pretty smart guy and kept a backup of the 1706 sproc for us.

    I’ll find the code for the updated sproc tomorrow and post it for you.

    Like

    Reply

    1. Thanks!
      The issue we see is allbeit a bit different as we do not seem to see timeouts but just that Option 243 and 252 is missing.
      Increase in TFTP package size seems to worsen it…

      Like

      Reply

      1. Hmmm, does sound a bit different. We’ll also pretend that today is the tomorrow I was speaking of…

        Here is the full code that we have in our environment:

        USE [CM_A01]
        GO
        /****** Object: StoredProcedure [dbo].[NBS_GetPXEBootAction] Script Date: 1/31/2018 12:00:50 PM ******/
        SET ANSI_NULLS ON
        GO
        SET QUOTED_IDENTIFIER ON
        GO
        —
        — Name : NBS_GetPXEBootAction
        — Version : 5.0.8577.1005
        — Definition : SqlObjs
        — Scope : PRIMARY_OR_SECONDARY
        — Object : P
        — Dependencies : ProgramOffers, vSMS_TaskSequencePackage
        — Description :
        —
        ALTER PROCEDURE [dbo].[NBS_GetPXEBootAction]
        @ItemKey int,
        @UnknownItemKey int,
        @SMBIOSGUID nvarchar(38),
        @MACAddress nvarchar(38),
        @DPName nvarchar(255),
        @Architecture int = 1
        AS
        BEGIN
        SET NOCOUNT ON

        declare @SMBIOS nvarchar(38)
        declare @MAC nvarchar(64)

        DECLARE @CurrentTimeInUTC datetime
        SET @CurrentTimeInUTC = getutcdate()

        set @SMBIOS = @SMBIOSGUID
        set @MAC = @MACAddress

        IF (CHARINDEX (‘_’, @SMBIOSGUID) > 0 OR CHARINDEX (‘%’, @SMBIOSGUID) > 0)
        BEGIN
        IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID LIKE CommonSMBIOS_GUIDs.SMBIOS_GUID))
        SET @SMBIOS = ‘Common’
        END
        ELSE
        BEGIN
        IF (EXISTS (SELECT SMBIOS_GUID FROM CommonSMBIOS_GUIDs WHERE @SMBIOSGUID = CommonSMBIOS_GUIDs.SMBIOS_GUID))
        SET @SMBIOS = ‘Common’
        END

        IF (CHARINDEX (‘_’, @MACAddress) > 0 OR CHARINDEX (‘%’, @MACAddress) > 0)
        BEGIN
        IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress LIKE CommonMACAddresses.MACAddress))
        SET @MAC = ‘Common’
        END
        ELSE
        BEGIN
        IF (EXISTS (SELECT MACAddress FROM CommonMACAddresses WHERE @MACAddress = CommonMACAddresses.MACAddress))
        SET @MAC = ‘Common’
        END

        SELECT MACAddress, SMBIOS_GUID, SMS_UniqueIdentifier0, LastPXEAdvertisementID, LastPXEAdvertisementTime,
        OfferID, OfferIDTime, PkgID, PackageVersion, PackagePath, BootImageID, Mandatory, Known,
        PresentTimeEnabled, ExpirationTimeEnabled, PresentTimeInUTC, ExpirationTimeInUTC
        FROM (
        SELECT @MACAddress AS ‘MACAddress’,
        @SMBIOSGUID AS ‘SMBIOS_GUID’,
        xref.GUID AS ‘SMS_UniqueIdentifier0’,
        ” AS ‘LastPXEAdvertisementID’,
        ” AS ‘LastPXEAdvertisementTime’,
        sp.OfferID AS ‘OfferID’,
        po.PresentTime AS ‘OfferIDTime’,
        po.PkgID AS ‘PkgID’,
        tspkg.Version AS ‘PackageVersion’,
        cdp.URL AS ‘PackagePath’,
        tspkg.BootImageID AS ‘BootImageID’,
        CASE
        WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
        WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
        ELSE 0 END
        AS ‘Mandatory’,
        CASE
        WHEN (EXISTS (select System_DISC.ItemKey from System_DISC where
        System_DISC.ItemKey = xref.MachineID AND
        ISNULL(System_DISC.Decommissioned0,0) != 0 AND
        ISNULL(System_DISC.Obsolete0,0) != 0 AND
        ISNULL(System_DISC.Unknown0,0) != 0))
        THEN 0
        WHEN (xref.MachineID = @ItemKey) THEN 1
        ELSE 0 END
        AS ‘Known’,
        CASE WHEN ((po.TimeEnableFlag & 0x0001) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_PRESENT = 0x0001
        AS ‘PresentTimeEnabled’,
        CASE WHEN ((po.TimeEnableFlag & 0x0002) != 0) THEN 1 ELSE 0 END — PROGOFFER_ENABLE_EXPIRATION = 0x0002
        AS ‘ExpirationTimeEnabled’,
        CASE WHEN ((po.TimeEnableFlag & 0x0100) != 0) THEN po.PresentTime ELSE dbo.fnConvertLocalToUTC(po.PresentTime) END — PROGOFFER_GMT_PRESENT = 0x0100
        AS ‘PresentTimeInUTC’,
        CASE WHEN ((po.TimeEnableFlag & 0x0200) != 0) THEN po.ExpirationTime ELSE dbo.fnConvertLocalToUTC(po.ExpirationTime) END — PROGOFFER_GMT_EXPIRATION = 0x0200
        AS ‘ExpirationTimeInUTC’
        FROM MachineIdGroupXRef xref
        JOIN ResPolicyMap AS rpm
        ON xref.MachineID = rpm.MachineID
        JOIN SoftwarePolicy AS sp
        ON (rpm.PADBID = sp.PADBID AND rpm.IsTombstoned != 1)
        JOIN ProgramOffers AS po
        ON sp.OfferID = po.OfferID AND po.[Action] 3
        JOIN vSMS_TaskSequencePackage AS tspkg
        ON (tspkg.PkgID = sp.PkgID AND (ISNULL(tspkg.BootImageID,”)!= ”))
        JOIN SMSPackages
        ON (tspkg.BootImageID = SMSPackages.PkgID)
        JOIN PkgPrograms as pp
        ON (tspkg.PkgID = pp.PkgID AND (pp.ProgramFlags & 4096) = 0 AND pp.[Action] 3) — 4096 = disabled
        JOIN ContentDPMap AS cdp
        ON (cdp.ContentID = tspkg.BootImageID AND cdp.ServerName = @DPName)
        WHERE xref.MachineID IN (@ItemKey) AND
        (@Architecture = 1 OR SMSPackages.Architecture = @Architecture) AND — Architecture of 1 means any, 0 means 32-bit, 9 means 64-bit
        ((CASE WHEN ((po.OfferFlags & 0x00000620) != 0) THEN 1 — 0x620 = AP_ON_LOGON | AP_ON_LOGOFF | AP_ASAP
        WHEN (ISNULL(po.MandatorySched,”)!= ”) THEN 1
        ELSE 0 END) = 0 OR
        (NOT EXISTS (select * from LastPXEAdvertisement as lpa where (lpa.MAC_Addresses = @MAC OR lpa.SMBIOS_GUID = @SMBIOS) AND lpa.LastPXEAdvertisementID = sp.OfferID AND LastPXEAdvertisementTime = PresentTimeInUTC)) AND
        ((ExpirationTimeEnabled = 0) OR (@CurrentTimeInUTC <= ExpirationTimeInUTC))
        ORDER BY Known DESC, Mandatory DESC, OfferID DESC
        END

        Like

Leave a comment