I’ve been using the BES Download Cachers for years to precache fixlet downloads to handle an airgap environment. I’m encountering an issue now that I think may be caused by the size of my sha1 folder.
Right now on the system executing the BES Download Cacher, the sha1 folder has 44,000 files totaling a little over 1.1 TB. The Download Cacher never seems to complete.
For a test, I’m passing it the BES Support.efxm (where I expect the download count to be relatively small). Watching in ProcMon, I can see BESDownloadCacher.exe execute a QueryDirectory for BES Server\wwwrootbes\bfmirror\downloads\sha1*, which returns a list of every file in sha1; then, for every file in sha1, it seems to execute a CreateFile, QueryInformationVolume, QueryAllInformationFile, CloseFile operation.
More disturbingly, I see the this loop repeated many times.
Apparently the BES Download Cacher is spending a lot of time in checking what’s already present in sha1. I suspect that it is enumerating and statusing every file in sha1, for each file that needs to be downloaded. In my environment, it takes 52 seconds each time through that loop, so if it’s going through a lot of repetitions (once for each download link maybe?) that’s going to take a long time to run.
Am I reading this wrong? Is there a better way to precache download files? Is this PMR-able?
I opened a PMR on this but it doesn’t look like there’s a real solution yet. The BESDownloadCacher just can’t handle a large sha1 folder - which is a little irritating because I’d expect the people using BESDownloadCacher are exactly the same folks who would have large sha1 caches to begin with.
IBM Support did clue me in to a parameter that builds a list of download URLs, without comparing them to sha1 or executing the downloads. I’ve wrapped that in a set of actions in a task I just posted to https://bigfix.me/fixlet/details/22467
Basically, Action 1 will download BESDownloadCacher, execute it against a set of content sites, and build a list of download URLs. Action 2 will compare the download URLs to the sha1 folder, and build a list of download URLs that are not cached already (for diagnostic purposes). Action 3 will check the download URLs generated from Action 1, compare them to the sha1 folder, and download anything that’s missing.
Last night I ran this on my server against BES Support, Patches for Windows, Updates for Windows Applications, and OS Deployment and Bare Metal Imaging. I finished in about 6 hours.
I’ve opened a couple of RFE’s on BESDownloadCacher. Give them a vote if you think this functionality would be worthwhile to you!
BESDownloadCacher: Improve existing file check
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=96906
The logic that BESDownloadCacher uses to determine whether a download already exists in the sha1 folder is inefficient. With a large number of files in the sha1 folder, the comparison can take more than 24 hours (even if no new downloads are required). The reason is that for every download URL, the entire sha1 folder is scanned, rather than checking for the specific download file sha1 value.
BESDownloadCacher: Option to ignore “superseded” and “MANUAL_BES_CACHING_REQUIRED” downloads
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=96907
The BESDownloadCacher has no option to ignore Superseded fixlets, which I’d like added as a command-line parameter. Additionally, the BESDownloadCacher does not ignore MANUAL_BES_CACHING_REQUIRED downloads. It tries to download these files and logs errors when they are not available.
1 Like