Issue with "Waiting for downloads to be mirrored." related to Office patches

aasheim 2016-01-18 10:18:23 UTC #1

After deploying MS Office 2013 / 2016 to 10 endpoints, it seems that the TEM-server got a hang up where it’s imposible to empty to the cache and or get the download plugin to work properly. The result is that is not possible to neither to distribute software or patches to endpoints…

The software and patches is distributed from the TEM-server to a main relay in DMZ and from there to local relay. The cache limit is set to 4096 on all relays and on the server…

Any suggestions?

"Waiting for downloads to be mirrored." Everytime file is deployed

strawgate 2016-01-18 17:30:50 UTC #2

4096mb? You should consider increasing the server cache to a couple hundred gigabytes and the relay caches to 20+gb

JasonWalker 2016-01-18 19:36:50 UTC #3

Yes, a few -

In general, start a new topic. People may skip over additions to a thread this old and not see your problem.
Use Task 148 “BES Relay / BES Server Setting: Download Cache Size” (from the BES Support site) to increase the cache sizes on your relays and servers. When the Server or Relay get a download request from a client, they will store a cached copy of that download so that if it is requested again (by that client or by another) the file does not have to be re-downloaded from the Internet. If the cache is small, the file would have to be re-downloaded every time a client requests it, increasing your bandwidth use and delaying the installs. If the cache is too small to hold the file at all, the download will never complete.

Sizing the cache is environment-specific, depends a lot on how frequently you deploy updates, whether you have a common configuration baseline so most clients are downloading the same things, tradeoffs between bandwidth and storage, etc; but generally you’ll want the caches to be as large as you can afford. On my BigFix Server I use 500 GB, and on my Relays I generally set at least 100 GB if their disks are large enough.

Likewise you might also need to increase the _BESClient_Download_NormalStageDiskLimitMB and _BESClient_Download_PreCacheStageDiskLimitMB settings on clients, but there’s no built-in task for that; you’d need to write your own or grab one from bigfix.me. I set mine to 8 GB (8192 MB) on the clients.

jgstew 2016-01-18 23:38:03 UTC #4

I’m aware of a BigFix environment where there could be as much as 700gb of software being deployed at any given time with BigFix. In this case the Root server has a 8TB volume dedicated to the Uploads & WebCache with the Cache set to 3TB. Most environments won’t need this much storage, but it doesn’t need to be super fast. Speed matters most for FillDB and the Databases.

It is also a good idea to have the top level relays have large caches to take some load off of the root server. If you had a single top level relay that everything used, you could put a lot of storage there and not need as much storage on the root server.

You probably don’t need quite as much storage on all of the relays that connect to the top level relays and root server that are on very fast links, like 10 Gigabit or greater, though even then I’d recommend at least 50gb.

You should consider having a decent cache size on every relay behind a WAN to save bandwidth at those WAN links.

aasheim 2016-01-19 21:09:00 UTC #5

I got this strange messages in the logs:

ItemizedDownloadsAvailable: false (action id ....)

Any opinions what that tells?

jgstew 2016-01-20 00:08:51 UTC #6

I assume you mean the client logs.

This means that the client is checking its parent relay to see if the download is available and it is not. That client’s parent relay then checks with the relay above it. This process continues until it reaches either the root server or a relay configured to do internet downloads directly, at which point the file is downloaded, cached on the relay, and then sent down the chain until it gets to the client. If the relay and/or root server’s caches are set to be too small, then the download will never come available because the cache will have contention by other downloads that are inflight in other active actions. This error could also be caused by other things as well, including that the file is no longer available at the source URL in the prefetch / download statement.

In general, all relays should have caches 3 times the largest single download or 20gb, whichever is larger. I would recommend 50gb or more in all cases. I would recommend the root and top level relays have significantly larger cache sizes. Ideally the top level relays would have at least 3 times the cache size of all downloads in active actions combined.

aasheim 2016-01-20 08:13:56 UTC #7

Thanks!

Maybe it can has something to do with BFGather to do?

http://127.0.0.1:52311/relaydiagnostics

In addition I do recognized that the proxy agent log tells that the proxy agent does not work properly, even I do not know there was a proxy agent running at the server… How to we uninstall that proxy agent? Or do we need it to be configured this way?

Problem with BFGather

jgstew 2016-01-20 08:16:43 UTC #8

That could be the cause, but I’m not certain what you mean when you say proxy agent, do you mean for VMWare? A proxy agent is a completely different thing than an HTTP proxy, so it shouldn’t be related to the issue, but again, I’m still a little unclear on what you are referring to.

There is also proxy settings but that is not the same as a proxy agent, but if you configured a HTTP proxy and it isn’t working, that would be an issue.

aasheim 2016-01-20 08:32:57 UTC #9

I recognized that there is a lot av errors in the ProxyAgent

Mon, 18 Jan 2016 22:41:09 +0100 -- Main Thread (1176) -- Error connecting to local relay: "HTTP Error 28: Timeout was reached: Connection timed out after 10000 milliseconds". Retrying in 1 minute.

I do not know if it has something to do with this issues, but in the article I refer to they explain how to get the proxy agent to work…

aasheim 2016-01-20 08:40:51 UTC #10

Maybe this one can explain the issue:

Among the download links that fails is fixlets with an ID that I do not find in the console - And it seems that MS Office 2013/16 distribution package is among them. So, how to I deleted ghosts in the database for downloads?

http://127.0.0.1:52311/cgi-bin/bfenterprise/BESMirrorRequest.exe

Failed:
Action: 2282 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows8.1-KB3093503-v6-x64.msuAction: 2749 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows8.1-KB3093503-v6-x64.msuAction: 2752 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows6.1-KB3007196-x64.msuAction: 2784 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows6.1-KB3007196-x64.msuAction: 2803 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows8.1-KB3093503-v6-x64.msuAction: 3026 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows8.1-KB3093503-v6-x64.msuAction: 3047 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows8.1-KB3093503-v6-x64.msuAction: 3103 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/Windows8.1-KB3093503-v6-x64.msuAction: 3174 Retry Mirror Request
url 1: http://MANUAL_BES_CACHING_REQUIRED/download/Windows6.1-KB2577795-x86.msuAction: 3185 Retry Mirror Request

jgstew 2016-01-20 17:01:56 UTC #11

This could be because there is no connection to be made, or it could mean that the timeout needs to be increased.

This is probably the actual issue. This means that you have to download the files manually by going out to the vendor website and then you have to place it into the root server’s cache with the file renamed to it’s SHA1. If you haven’t done so, then this is why it is failing. It will never succeed until you have done this.

You will find MANUAL_BES_CACHING_REQUIRED whenever the vendor makes it hard to download the file directly. In some cases it could be that IBM needs redistribution rights to legally include it and doesn’t have it, but I’m not sure that is the case since they are not technically redistributing the file itself.

aasheim 2016-01-21 12:48:42 UTC #12

Thanks! That explains very clearly what to do!

aasheim 2016-01-21 12:54:10 UTC #13

The IBM BigFix support team advice me to download and BESAuditTrailCleaner after deleting old fixlets and run it with the following parameters: BESAuditTrailCleaner-3.0.10.95.exe -V -O -b 1000000

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Endpoint%20Manager/page/BES%20Audit%20Trail%20Cleaner

That seems to work very well!

jgstew 2016-01-21 19:14:50 UTC #14

That will remove deleted items from the BigFix database and help with performance, but using the BESAuditTrailCleaner doesn’t typically solve problems.

aasheim 2016-01-22 10:58:00 UTC #15

How should I understand this one:

Waiting for downloads to be mirrored.
Status: Pending Downloads
Start Time: Not Executed
End Time: Not Executed

Exit Code
None

What’s happening? Downloads that usually that a few minutes is hanging…

jgstew 2016-01-22 20:47:16 UTC #16

Not sure. Depends on the client, the download, the fixlet, the parent relay, caches, and many other things.

You should probably file a PMR with IBM.

What are the sizes of the download caches of your clients, relays, and root server?

aasheim 2016-01-24 17:56:01 UTC #17

I have adjust the cahce size in accoradance with your advice, some of them to 50 GB.

May I have a suggestion what can be the issue: IEM Software Distribution - Could not connect to DSN:
bes_bfenterpriseusing username:

It can be a possible reason that the Software distribution upload and maintenance service does not work properly.

So, how do I reinstall the service by downloading the application or is use of a fixlet the only way to do it?

http://www-01.ibm.com/support/docview.wss?uid=swg1IV74448

jgstew 2016-01-24 21:20:24 UTC #18

Where are you seeing this message? What log?

I’m not really sure what this is about.

Seems like you should probably file a PMR. It seems like you are having multiple issues: How to ask for IBM product help: PMRs, RFEs, and more

aasheim 2016-01-25 11:35:33 UTC #19

The message was from the OSD_UMS.log file…

aasheim 2017-05-17 09:54:16 UTC #20

Once again: ItemizedDownloadsAvailable: false (action id 12037)

First it recognize as following:

Relevant - Update to Pattern 20170515_081236 - Core Protection Module (fixlet:12037)

But afterwards it reports it’s impossible to download the files for the pattern…