Full Sync stuck in "Pending Downloads" for WinPE 3 Resources that do not exist

JasonWalker · November 17, 2017, 8:21pm

When using the “Upgrade Bare Metal Server” from the Bare Metal Server Manager dashboard, or performing a Full Sync on a bare metal server, the generated group actions get stuck on Pending Downloads for “Update WinPE 3 Resources on Bare Metal Servers” tasks. The downloads for both x86 and x64 SHA.zip.BFOSD files get “404 Not Found” responses.

The 404 doesn’t surprise me. As far as I know I don’t have any WinPE 3 Resources, I stopped deploying XP and 2003 quite a while back. But they may have existed in my dashboards at one time. Now when I look at the Bundle and Media Manager dashboard, I don’t see any PE 3 Resources; I have MDT bundles with PE 10, several Linux OS Resources, and Windows OS resources for Windows 7 x64 SP1, Windows 2008 R2 x64 SP1, WIndows 2012 R2 x64 SP0, and Windows 10 x64 SP0.

I retrieved the Dashboard Variables via
curl --insecure -u myoperator-X GET "https://myserver:52311/api/dashboardvariables/OSD_DataStore.ojo" > OSD_Variables.txt and searched for the sha1s of the two WinPE3 resources files that are getting stuck in ‘Pending Downloads’, and sure enough I have a lot of references to them in the dashboard variables.

Do I need to clear these out of the dashboard variable list somehow to prevent the Bare Metal Servers from trying to sync them? The files themselves are no longer present in my Uploads folder.

<DashboardData Resource="https://MYSERVER:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_FilePointer_DFD2E40C291E0B9BB98108B0D867C35E38B370F8">

<DashboardData Resource="https://MYSERVER:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Transient_Resource_PEX86"> <Value>{"resourceName":"PEX86","defaultBundle":false,"consoleUploadID":"OSD_Upload_DFD2E40C291E0B9BB98108B0D867C35E38B370F8_1","MDTBundleCreatorVersion":null,"type":"OS Resource","SyncTimeMDT":null,"MDTVersion":null,"WAIKVersion":null,"sizeOnDisk":"177134374","usmtVersions":null,"RBAgentVersion":null,"resourceInfo":null,"driverInfoCUID":"2"}</Value>

<DashboardData Resource="https://MYSERVER:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Upload_DFD2E40C291E0B9BB98108B0D867C35E38B370F8_1">

<DashboardData Resource="https://MYSERVER:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Upload_DFD2E40C291E0B9BB98108B0D867C35E38B370F8_1"> <Dashboard>OSD_DataStore.ojo</Dashboard> <Name>OSD_Upload_DFD2E40C291E0B9BB98108B0D867C35E38B370F8_1</Name> <IsPrivate>false</IsPrivate> <Value>{"uploadDateUTC":1363283711768,"url":"http://MYSERVER:52311/Uploads/DFD2E40C291E0B9BB98108B0D867C35E38B370F8/DFD2E40C291E0B9BB98108B0D867C35E38B370F8.zip.BFOSD","ticketID":1,"sha1":"DFD2E40C291E0B9BB98108B0D867C35E38B370F8","filename":"DFD2E40C291E0B9BB98108B0D867C35E38B370F8.zip","openActionIDs":[16601,16711,50335133,423268,577860,656498,658034,658093,658156,659061,659492,659499,659502,661019,661020,661021,661022,661023,661024,661026,661027,661028,661029,661030,661034,661068,661069,661071,661829,661830,664201,680230,694787,696221,696509,716058,716095,716649,717856,717881,717900,718473,718476,718696,725201,725234,725522,725536,725944,726220,736064,737379,738042,744158,744309,744350,744351,744865,744927,745939,748349,751186,751200,751212,753264,754233,754238,754244,16960691,759127,16960703,16960713,16960723,759621,759632,768894,16963156,16964939,16964949,770700,770710,770960,772325,779669,779738,779826,781950,833034,834802,834813,835763,847699,847966,848038,848300,850673,850678,850680,850683,850697,851297,855724,859365,859374,859397,859662,861953,870114,871246,905559,905579,905609,906816,907367,16993269,921414,923886,930577,931151,931179,931689,931701,931713,17000730,931979,932293,933119,933132,933140,934885,935490,935574,935577,935578,935620,936401,937717,937719,937723,937744,938036,938056,938057,938104,938427,938444,938461,938473,938474,938475,938533,938536,939124,941776,945627,945629,946729,946744,953983,954025,956652,993504,993710,993722,994068,17024124,17024128,17024243,17024245,17024252,17024254,1027326,1027329,1027332,1027498,1027501,1027508,1027511,1027518,1027520,1027998,1028001,1028006,1028011,1028013,1028035,1028037,1028039,17024577,17024830,17024887,17024889,17024893,17024896,1029187,1029188,1029190,1036590,1036607],"sha256":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF","size":175122376}</Value> </DashboardData>

Gianluca · November 20, 2017, 3:33pm

Hi Jason,

I’ve reviewed your post.
It’s a good analysis.
I’d like to review your “datastore” in order to verify the overall status of your variables.
I’d like to try to understand better what is happening in your environment and then eventually suggest if/which variables could be deleted to workaround this issue.
I can confirm that it is NOT normal to have a variable that links to a WinPE3 and then the corresponding MDT does not exists. Moreover it is a misalignment if the corresponding filename does not exist (Uploads/DFD2E40C291E0B9BB98108B0D867C35E38B370F8/DFD2E40C291E0B9BB98108B0D867C35E38B370F8.zip.BFOSD).
If possible I prefer you open a “support case” to my L3 team so we can work on this more specifically. You can mention this message to L2 to speed it up it reaches my team.

This will also allow you to upload the ‘datastore’ file on ecurep in a secure way.
Then I’d like to have also the output of the ‘tree /F’ command of the folder …BES Server\wwwrootbes\Uploads. It has to be run on the BigFix Server. The purpose is to double check the existing folders and files.
Please could you add also a screenshot of action that failed where the error (Download Error) is expanded?

Regards,
Gianluca G.

JasonWalker · November 20, 2017, 3:54pm

Thanks Gianluca, I’ve just opened case TS000048915 on this. We also mentioned this in relation to an existing case but I think this is worth a separate ticket. I’m uploading the files now.

How do I upload the “datastore”? Do you just mean the output of the TREE /F, or a file related to the dashboard variables, or something else?

Gianluca · November 21, 2017, 4:28pm

Thanks Jason, on ecurep everything is fine. The datastore and the TREE /F output are correctly saved and uploaded.

Gianluca · November 21, 2017, 5:35pm

I’ve just updated the “support case”.

*****____
…
we have analyzed the provided doc. It was complete, thanks for that.
For workaround the issue it is needed to delete the two following variables from the datastore:
‘OSD_Transient_Resource_PEX86’ and ‘OSD_Transient_Resource_PEX64’.

<DashboardData Resource="https://ndjsmspwdsa01.ndc.nasa.gov:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Transient_Resource_PEX86">
        <Dashboard>OSD_DataStore.ojo</Dashboard>
        <Name>OSD_Transient_Resource_PEX86</Name>
        <IsPrivate>false</IsPrivate>
        <Value>{"resourceName":"PEX86","defaultBundle":false,"consoleUploadID":"OSD_Upload_DFD2E40C291E0B9BB98108B0D867C35E38B370F8_1","MDTBundleCreatorVersion":null,"type":"OS Resource","SyncTimeMDT":null,"MDTVersion":null,"WAIKVersion":null,"sizeOnDisk":"177134374","usmtVersions":null,"RBAgentVersion":null,"resourceInfo":null,"driverInfoCUID":"2"}</Value>
</DashboardData>

and

<DashboardData Resource="https://ndjsmspwdsa01.ndc.nasa.gov:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Transient_Resource_PEX64">
        <Dashboard>OSD_DataStore.ojo</Dashboard>
        <Name>OSD_Transient_Resource_PEX64</Name>
        <IsPrivate>false</IsPrivate>
        <Value>{"consoleUploadID":"OSD_Upload_6F2C5884A5850F7A4B0A1C04AF74C5D2A9B0CC2F_1","type":"OS Resource","defaultBundle":false,"SyncTimeMDT":null,"MDTBundleCreatorVersion":null,"MDTVersion":null,"resourceName":"PEX64","sizeOnDisk":"214376501","usmtVersions":null,"WAIKVersion":null,"driverInfoCUID":"2","RBAgentVersion":null,"resourceInfo":null}</Value>
</DashboardData>

We expect that the deletion of these two vars should allow you to run the Bare Metal server upgrade successfully because the WinPE3 Update action should be avoided.

For deleting the variable one method is by the Presentation Debugger of the BigFix console.
-1- From the console press the menu DEBUG if it is present. If it is not present press CTRL+SHIFT+ALT+D to have this menu on the upper taskbar.
-2- Then run the ‘Presentation Debugger’
-3- On the ‘Presentation Debugger Panel’ press the radio button ‘Presentation’.
-4- On the upper part write the following text to delete the reference to the WinPE3 x86:

<script>
DeleteSharedVariable("OSD_DataStore.ojo","OSD_Transient_Resource_PEX86")
</script>

for deleting WinPE3 x64 use

<script>
DeleteSharedVariable("OSD_DataStore.ojo","OSD_Transient_Resource_PEX64")
</script>

We are continuing the investigation to understand if it is needed to clean also the references to the corresponding OSD_FilePointer and to the OSD_Upload.
…

Regards

Gianluca G.

Gianluca · November 21, 2017, 5:41pm

I’ve just noticed that the editor has removed some parts of my update, so it seems to be incomprehensible.
I’m uploading my text again…

…
we have analyzed the provided doc.
For workaround the issue it is needed to delete the following variables from the datastore:
‘OSD_Transient_Resource_PEX86’ and ‘OSD_Transient_Resource_PEX64’.

<DashboardData Resource="https://ndjsmspwdsa01.ndc.nasa.gov:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Transient_Resource_PEX86">
		<Dashboard>OSD_DataStore.ojo</Dashboard>
		<Name>OSD_Transient_Resource_PEX86</Name>
		<IsPrivate>false</IsPrivate>
		<Value>{"resourceName":"PEX86","defaultBundle":false,"consoleUploadID":"OSD_Upload_DFD2E40C291E0B9BB98108B0D867C35E38B370F8_1","MDTBundleCreatorVersion":null,"type":"OS Resource","SyncTimeMDT":null,"MDTVersion":null,"WAIKVersion":null,"sizeOnDisk":"177134374","usmtVersions":null,"RBAgentVersion":null,"resourceInfo":null,"driverInfoCUID":"2"}</Value>
</DashboardData>

AND

<DashboardData Resource="https://ndjsmspwdsa01.ndc.nasa.gov:52311/api/dashboardvariables/OSD_DataStore.ojo/OSD_Transient_Resource_PEX64">
		<Dashboard>OSD_DataStore.ojo</Dashboard>
		<Name>OSD_Transient_Resource_PEX64</Name>
		<IsPrivate>false</IsPrivate>
		<Value>{"consoleUploadID":"OSD_Upload_6F2C5884A5850F7A4B0A1C04AF74C5D2A9B0CC2F_1","type":"OS Resource","defaultBundle":false,"SyncTimeMDT":null,"MDTBundleCreatorVersion":null,"MDTVersion":null,"resourceName":"PEX64","sizeOnDisk":"214376501","usmtVersions":null,"WAIKVersion":null,"driverInfoCUID":"2","RBAgentVersion":null,"resourceInfo":null}</Value>
</DashboardData>

For deleting the variable one method is by the Presentation Debugger of the BigFix console.
-1- From the console press the menu DEBUG if it is present. If it is not present press CTRL+SHIFT+ALT+D to -have this menu on the upper taskbar.
-2- Then run the ‘Presentation Debugger’
-3- On the ‘Presentation Debugger Panel’ press the radio button ‘Presentation’.
-4- On the upper part write the following text to delete the reference to the WinPE3 x86:

<script>
DeleteSharedVariable("OSD_DataStore.ojo","OSD_Transient_Resource_PEX86")
</script>

for deleting WinPE3 x64 use

<script>
DeleteSharedVariable("OSD_DataStore.ojo","OSD_Transient_Resource_PEX64")
</script>

We are continuing the investigation to understand if it is needed to clean also the references to the corresponding OSD_FilePointer and to the OSD_Upload.
…

Gianluca G

JasonWalker · November 21, 2017, 6:53pm

Thanks so much for the quick analysis, @gianluca!
I’m in the middle of my BES upgrade to 9.5.7 now, but should be able to try this a bit later this afternoon.

JasonWalker · November 21, 2017, 10:39pm

Thanks, @gianluca, the workaround worked for me.

I’d point out also that I have had a similar issue where drivers won’t sync to the OSD server when I delete all driver mappings for XP/2003. The generated “Sync Drivers” fixlet still try to download the last version of the XP/2003 driver binding mappings, instead of skipping the XP/2003 driver downloads entirely. End up in the same state of ‘Pending Downloads’ for a file that’s no longer present.

I’ve worked around those by creating at least one dummy driver mapping for XP/2003. Now I’ll investigate whether there’s a similar variable I can remove for the XP/2003 drivers.