Delayed Patch downloads - 1 per 10 minutes

Hi all,

New user of Bigfix here, and so far so good, except for the delay in downloading patches.
When I select multiple fixlets to install to a single Windows server, the main bigfix server only downloads one patch every 10 minutes. How do I configure it so that it can download multiple files at a time, and/or download them more frequently than every 10 minutes?

Downloading 6 files an hour is terrible! Imagine if I have servers that need 50+ patches (8+ hours to download) the same day.

A bit more background info:
UDP is not enabled as this is a NAT’d environment, so I am only using the Command Polling.

_BESClient_Comm_CommandPollEnable=1
_BESClient_Comm_CommandPollIntervalSeconds=3600

Bigfix server <> Main Relay (direct/same subnet at BF Svr) < Domain Relay (NAT) <> Domain Agent (same subnet as domain relay).

I saw other forum posts that sound similar, but no resolution in them.
I have added the following settings but it hasn’t made any difference

_BESClient_Download_RetryMinutes=1
_BESClient_Download_CheckAvailabilitySeconds=120

Any advice or help would be great.
Thanks.

Something doesn’t sound right there… I don’t see these kinds of delays (but I usually do configure my actions ahead of time, with an ‘Action Start Time’ set in the future, so I might not notice if there were download delays).

Even if the BES Server is only downloading one file at a time (which I don’t expect is the case), it shouldn’t wait ten minutes between downloads.

The _BESClient_Download_RetryMinutes affects how long it takes the server to retry, after a download fails.
_BESClient_Download_CheckAvailabilitySeconds affects how frequently a client/relay checks to see whether the file is available from its upstream relay (in cases like yours where the UDP notifications won’t make it down to the client).

1 Like

Thanks Jason, I guess those settings aren’t relevant in this scenario. Surely there must be something else though.

It definitely doesn’t download more than one fixlet at a time in a multiple action group.
If there are 2 files (ie: x86 & x64) in a fixlet, it will download the 2nd file immediately after the first file is finished downloading rather than waiting 10 mins.

The ‘\BigFix Enterprise\BES Server\wwwrootbes\bfmirror\downloads\sha1’ folder on the Bigfix server shows the timestamps of each file. They are all 10 mins apart.

bigfix-dl-10mins

Oh, ok.
I don’t have a particular insight on that, but I can posit a theory to fit the results. What it looks like to me, is that it’s waiting for your client to request the downloads before the server goes to fetch them. 10 minutes would fit well with the client report interval. The client, having received a baseline, may only be requesting the next download after completing the previous component.

Try checking the box for “Start client downloads before constraints are satisfied” checkbox on the Take Action / Execution tab, and see whether that helps.

1 Like

No luck unfortunately…
One of those batches in the image above was using that option.
I tried again just now and still 10 mins between each download.

These latest files are less than 1mb in size and still have to wait 30 mins to download 3 files that total 1.8MB :frowning:

Just to let you know, that is not expected behavior, looks like there may be something wrong in your environment. I think you should open a PMR and let the support team go through the logs with you to determine what’s going on.

Here’s a sample shot from my server’s downloads…

12/28/2018 01:16 PM 6,096 159609fb464994ef53910bffaf1c8d7205355017
12/28/2018 01:16 PM 15,888 95d3623aeb0adef1cd29cb757098e247f8436eb5
12/28/2018 01:16 PM 30,732 45e2f0472091cbbc3a0a381a7c0ed85810aaaf9d
12/28/2018 01:16 PM 6,888 f0f258f1a6c0d100859667d76931ae8a0116b45b
12/28/2018 01:16 PM 1,397,996 de29ebdb0965ecb43e1968e972fd4e0cce492f03
12/28/2018 01:16 PM 9,996,816 61c5e28c43254feada3543603cecb49b0962a930
12/28/2018 01:16 PM 17,369,996 609e62289045facf946903ea89279634c6548ed6
12/28/2018 01:16 PM 25,422,352 808f28d9172a8a3f8c8a33bf952c47ab24f32c64
12/28/2018 01:16 PM 30,348,100 f6174282ca18db9f7bc82747c20c46481565e5e1
12/28/2018 01:16 PM 32,471,800 5ba84c6eb87283a00ea20e6c06cbd569818aa925
12/28/2018 01:16 PM 38,195,500 2801de9d2ae1269fffb86c00acb948fbab53a5e9
12/28/2018 01:16 PM 40,995,364 038486d7b225391ea74572cf57ab5de259df9e53
12/28/2018 01:16 PM 73,127,700 457dc1c0c245f6f29152f9a071bada1951314dea
12/28/2018 01:16 PM 72,999,568 1c695156a2921cf94d7ea61b31506d6a35864cd0

I know that the default retry time is 10 minutes for downloads to the root server.

Are you able to see the status of the download in the console while is waiting for the root to download the patch and then cache it on the relay?

After 20+ mins the status hasn’t changed in the console.
It says “Pending Downloads (member action 00-728)” which is the first of the 3 fixlets. It has downloaded all 3 though.

I get that it’s because it’s not updating the status immediately since we are in NAT environment, but does that affect the downloading of file from Microsoft from the Bigfix server as well?

I created the action from the bigfix server because those patches were relevant to the agent. Does the server then download what has been configured in the action, or does the agent need to request them after it’s told that it can have them?

Just to be sure, the screenshot you posted earlier with the timestamps…that was a directory on your BES Root server? By default C:\Program Files (x86)\BigFix Enterprise\BES Server\wwwrootbes\bfmirror\downloads\sha1 ?

The “Pending Downloads (member action 00-728)” message is actually an action status from the client. This can be delayed significantly because this status only gets updated when the client posts a report - so it’ll say ‘pending downloads’ while the server downloads the file, while the relay downloads it from the server, and while the client downloads it from the relay. Once the client downloads the file, its report back up to the server can be delayed while the client is busy running the downloaded patch, or waiting for its next scheduled report interval.

To see the file download status at the server, you’d need to look at the Summary tab of the Action History. For a Baseline, you’d need to select the component individually and look at its Summary tab. Example

This could have statuses of “Cached on server”; “Downloading” (and showing the bytes downloaded / bytes total counts); or several different failure codes (file not present on website, website unreachable, downloaded file doesn’t match expected sha1, etc.)

I have now raised a support case with IBM for this as well.

Yes, that is the BES Root server directory I showed earlier.

I see this message for the files that have been downloaded.

Action status
Pending Downloads (member action 00-733) - 100% - Completed

Download status
Windows8.1-KB3100473-x64.msu - Complete - Cached on Server

And this for the fixlets that haven’t been downloaded
Status = Waiting
Doesn’t display Download status yet

733

734

735

baseline-files

Please let us know the results of the support case.

There are some settings with which you might experiment on the server/relays, but I honestly haven’t needed to tune these. The full list is at IBM Documentation

Servers/Relays:

_BESGather_Download_ChannelThreshold
BigFix Gather can simultaneously download two files at a time by using one “main channel” and one “thin channel”. The main channel is used for all downloads, but if the main channel is currently downloading a large file, the thin channel can be used to download smaller files if the download size is less than the specified threshold. If this setting is set high, BigFix Gather will use the thin channel to download larger files, which could slow down actions because two large files may be downloading at the same time (each using half the bandwidth) instead of one file after the other. If this setting is set low, the thin channel will be used for only very small file downloads.

Clients:

_BESClient_Download_MinimumDiskFreeMB
This setting stops both stages of downloading (normal stage and pre-caching stage) if the free space of the disk on which the client stores downloads is less than the value of this setting.

_BESClient_Download_NormalStageDiskLimitMB
This setting stops normal stage downloading if the client is already using this much normal stage download disk space. Actions marked for normal downloads will report constrained if the total space used for downloads exceeds this limit.
Note
Normal stage downloads may exceed this limit by borrowing some space from the pre-cache stage space if it is not full.

_BESClient_Download_PreCacheStageDiskLimitMB
This setting stops pre-cache stage downloading if the client is already using this much pre-cache stage download disk space. Actions marked for pre-caching will report constrained if the total space used for downloads exceeds this limit.

_BESClient_Download_ChannelThreshold
The BigFix Client can simultaneously download two files at a time by using one “main channel” and one “thin channel”. The main channel is used for all downloads, but if the main channel is currently downloading a large file, the thin channel can be used to download smaller files if the download size is less than the specified threshold. If this setting is set high, the BigFix Client will use the thin channel to download larger files, which could slow down actions because two large files may be downloading at the same time (each using half the bandwidth) instead of one file after the other. If this setting is set low, the thin channel will be used for only very small file downloads.

_BESClient_Download_DownloadsCacheLimitMB
This configuration setting sets the BigFix Client download cache limit (TEM 6.0+). The Endpoint Manager client keeps all files that were cached in the same day, regardless of the download cache limit. If there is only one file which is larger than the configured download cache size, the Endpoint Manager client keeps this file, regardless of the age or the download cache limit.

Thanks guys. I will update if there is any solution from IBM support.

I’ve also added all of those settings yesterday while troubleshooting.

Both top level & low level relays are the similar settings except for relay URL and file path.

_BESClient_ActionManager_SkipVoluntaryOnForceShutdown 1 
_BESClient_ArchiveManager_FileSet-citsearch C:\Program Files (x86)\BigFix Enterprise\BES Client\LMT\CIT\10802484_cit.xml.bz2 
_BESClient_ArchiveManager_FileSet-isotagsearch C:\Program Files (x86)\BigFix Enterprise\BES Client\LMT\CIT\10802484_isotag.zip 
_BESClient_ArchiveManager_FileSet-itsitsearch C:\Program Files (x86)\BigFix Enterprise\BES Client\LMT\CIT\10802484.xml.bz2 
_BESClient_ArchiveManager_FileSet-slmtagsearch C:\Program Files (x86)\BigFix Enterprise\BES Client\LMT\CIT\slmUploadDir\* 
_BESClient_ArchiveManager_LastArchive 1ff2b67f9f5995b9144f6cbd540d4be656c3f236 
_BESClient_ArchiveManager_MaxArchiveSize 52428800 
_BESClient_ArchiveManager_OperatingMode 0 
_BESClient_Comm_CommandPollEnable 1 
_BESClient_Comm_CommandPollIntervalSeconds 300 
_BESClient_Download_CheckAvailabilitySeconds 120 
_BESClient_Download_DirectOnFail 1 
_BESClient_Download_DownloadsCacheLimitMB 10240 
_BESClient_Download_MinimumDiskFreeMB 5000 
_BESClient_Download_NormalStageDiskLimitMB 500 
_BESClient_Download_PreCacheStageDiskLimitMB 1000 
_BESClient_Download_RetryMinutes 1 
_BESClient_Download_UtilitiesCacheLimitMB 500 
_BESClient_LastShutdown_Reason Service manager stop request 
_BESClient_Log_Days 30 
_BESClient_Log_MaxSize 1536000 
_BESClient_Query_NMOMaxQueryTime 30 
_BESClient_Query_SleepTime 500 
_BESClient_Query_WorkTime 250 
_BESClient_Resource_AccelerateForPendingMessage 1 
_BESClient_Resource_StartupNormalSpeed 0 
_BESClient_Upgrade_UTF8Settings 1 
_BESClient_UploadManager_BufferDirectory C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\__Global\Upload 
_BESClient_UsageManager_EnableAppUsageSummary 1 
_BESClient_UsageManager_EnableAppUsageSummaryApps -:noapp: 
_BESClient_UsageManager_EnableAppUsageSummaryPath 1 
_BESGather_Comm_ParentRelayURL http://bigfixr01:52311/cgi-bin/bfenterprise/BESGatherMirror.exe 
_BESGather_Comm_UseUrlMoniker 0 
_BESGather_Download_CacheLimitMB 10240 
_BESGather_Download_CheckInternetFlag 0 
_BESGather_Download_CheckParentFlag 1 
_BESGather_Mirror_SiteVersionPollingPeriod 5 
_BESRelay_GatherMirror_ClientActionSiteFolder C:\Program Files (x86)\BigFix Enterprise\BES Client\\__BESData\actionsite 
_BESRelay_HTTPServer_HttpLogDirectoryPath   
_BESRelay_HTTPServer_LogFilePath C:\Program Files (x86)\BigFix Enterprise\BES Relay\logfile.txt 
_BESRelay_HTTPServer_PortNumber 52311 
_BESRelay_HTTPServer_ServerRootPath F:\wwwrootbes\ 
_BESRelay_PostResults_ParentRelayURL http://bigfixr01:52311/cgi-bin/bfenterprise/PostResults.exe 
_BESRelay_PostResults_ResultTimeLimit 3 
_BESRelay_UploadManager_BufferDirectory C:\Program Files (x86)\BigFix Enterprise\BES Relay\UploadManagerData\BufferDir 
_BESRelay_UploadManager_CleanupHours 120 
_BESRelay_UploadManager_ParentRelayHost http://bigfixr01:52311 
_Enterprise Server_ClientRegister_BatchCount 100 
_Enterprise Server_ClientRegister_BatchDelay 1000 
_Enterprise Server_ClientRegister_ParentRelayURL http://bigfixr01:52311/cgi-bin/bfenterprise/clientregister.exe 
_Enterprise Server_ClientRegister_UDPMessagePort 52311

I think I got it. It’s working much better now.
Thanks Jason for sharing that link. I went through all config options listed on that webpage and added this one to retry every 1 minute.
Now the files are downloading quicker.

_BESGather_Download_RetryMinutes
When BigFix Gather fails to download a file from the Internet or its parent during an action, it will wait for the specified amount of time then try again.
Default value 10
Setting type Numeric (minutes)
Value range 0 - 1440
Server, Relay

bigfix-dl-1min

1 Like

I’m glad it’s helping, but this also implies that the first attempt at each download is failing. I think you should still investigate why each first attempt fails.

1 Like

I think what is happening is the client requests the download from the relay, the relay requests it from it’s parent, but it’s parent doesn’t already have it so that request fails. The parent relay gets the download and then notifies the child relay over UDP but the child relay doesn’t get it, so then it takes 10 minutes for it to check again.

I would recommend making sure that your BigFix Root Server and Relay caches are set to large enough values so they don’t roll over too quickly, but also if UDP isn’t working between relays, then setting _BESGather_Download_RetryMinutes to 1 is the right fix, though there are other settings that should ALSO be adjusted to help in these situations.

4 Likes