Manual pre-cache to BES Relays

Usage and Config Platform
1

(imported topic written by SystemAdmin)

http://support.bigfix.com/cgi-bin/kbdirect.pl?id=232

I am attemping to follow the article above to manually cache a large package (Outlook 2007) to one of our relays that I have a test PC pointed to in order to rapidly test parts of my action script. I have folllowed the instructions exactly, copying our Oulook2007.7z package to C:\Program Files\BigFix Enterprise\BES Relay\wwwroot\bfmirror\downloads\sha1\ on the Relay I am testing with and then renaming the file to its sha1 value with no file extension (like the other files that are already in the folder. Then I submit the task to my test PC to take the action, it checks in with the relay where I have already manually dropped the file, but the relay doesn’t think it has the file and then contacts it’s parent relay to get the file, which in turn contacts our main BES server to get it.

Is this kb article outdated? Is there another step or file to modify to make the relay realize it already has the file in its cache?

2

(imported comment written by BenKus)

Hi cyberbrad,

Can you verify that your download command matches the size and sha1 of the file?

Also, what is the size of the file?

Ben

3

(imported comment written by SystemAdmin)

Hi Ben,

I am actually using a prefetch command in my action to download the file. Could that be the problem? I read in the BigFix action scripting guide that the prefetch command was preferred over the download command, so I started using it in my action scripts, but I never really understood the benefit other than it eliminated the need for a “continue if” line.

The sha1 and file size in the prefecth command match that of the file I manually copied exactly. It is 412,886,741 bytes (393MB). Here is a copy of the prefectch command I am running, with our internal server name changed to a generic one.

//Download Outlook 2007 install files

prefetch Outlook.7z sha1:2e7d7db99d1dde2656da9c864b262fbf63fb36bb size:412885741 http://bigfix.company.com:52311/Uploads/2e7d7db99d1dde2656da9c864b262fbf63fb36bb/Outlook.7z

4

(imported comment written by BenKus)

Hey cyberbrad,

prefetch is fine – you are correct and it works the same as “download / continue if” and it is the preferred method for downloading files (simply because it eliminates the ugly and errorprone “continue if” statement).

So it looks like you did everything correctly and I am not sure why it didn’t work.

The way it is supposed to work is that the client will ask the relay to download the file (and it gives the sha1 during the download request). The relay first checks its sha1 cache (by name, size, and sha1) and if it can’t find the file, it will check its parent relay…

Try executing this on your relay… I think it should return “true” if the file is cached properly:

q: exists file whose (name of it = “2e7d7db99d1dde2656da9c864b262fbf63fb36bb” AND sha1 of it = “e7d7db99d1dde2656da9c864b262fbf63fb36bb” AND size of it = 412885741)of folder “C:\Program Files\BigFix Enterprise\BES Relay\wwwroot\bfmirror\downloads\sha1”

Ben

5

(imported comment written by SystemAdmin)

It does return true, so I believe it is pre-cached properly. I did some more testing and here is what I found out.

Let’s say you have your parent BigFix Server(1), main Relays servers at Corp HQ (2), Regional BigFix Relay servers (3), your local Relay server (4). I was trying to manually pre-cache the file to my local Relay server (4), but when I sent my PC the action, even though 4 had the file, it still checked in with 3, which didn’t have it, so it checked in with 2, which didn’t have it, so it downloaded it from 1. So each parent relay in the hierarchy downloaded the file. Once 3 had the file, it notified 4, and THEN 4 checked its local pre-cache and figured out it already had the file and doesn’t need it, so it notifes my client that the file is ready.

I then tried pre-caching the file to both 4 and 3 (so each relay in the hierarchy had the file except 2). Same behavior as before. 4 had the file, but still checked in with 3, which had the file, but still checked in with 2, which didn’t have the file so it had to download it from 1. Once 2 had finished the download, it notified 3 that the file was ready and 3 just instantly continued because it already had the file in its local cache and notified 4 that the file was now ready, and 4 looked at its local cache and determined it didn’t need it so it notified the client the file was ready.

So basically, if I want to manually pre-cache a file, I have to pre-cache it to each Relay server in the particular hierarchy of my test PC. It is as if the relay checks its parent relay for availability of a file BEFORE it looks at its own cache and only will check its own cache once its parent has notified it that the file is ready for download.

Test it out for yourself (you will need to have at least two nested child relays below your BF server and pre-cache the file to the relay that the client is pointed to, but not the parent relay of that relay), but I have tried it several different ways and still get the same results. It would be nice if I could JUST copy the pre-cache to my local relay for testing purposes and not have to wait for each parent relay of my local relay to have the file. This really is a bit of a bug as BigFix might be needlessly downloading files to parent relays when the local relay already has it, wasting time and network resources. Let me know your test results.

6

(imported comment written by BenKus)

Sounds like you have done a lot good and conclusive work… Let me check in with the developers on whether this is expected (and why) or not…

Ben

7

(imported comment written by SystemAdmin)

Update

Today I pre-cached a file on each leg of my relay hierarchy. (1-4). I sent the action to my client and watched the logs, at first it said “Downloads Available False”, but it checked back after only 10 seconds, but it still said “Downloads Available False”, after another 10 seconds it checked back a 3rd time and said “Downloads Available True” and began the action. The package I was sending it was rather large so the local relay couldn’t have re-downloaded it from its parent relay in 30 seconds. I believe that the 30 second lag time was due to the fact that my local relay was checking in with its parent, which was then checking in with its parent, and so on. So again, even with each relay already having the file, I believe they are still at least communicating to one another to check that the parent has the file before checking its own cache.

8

(imported comment written by BenKus)

Hey Brad,

Yes… that is how we would expect it to work… The relay needs to check with its parent to understand the files that are available for this action, but if the files are cached locally, it will use it rather than re-download.

Sounds like we don’t have a bug then… :slight_smile:

Ben

9

(imported comment written by SystemAdmin)

Could someone provide some insight on pre-caching custom tasks? Here’s the scenario. Office 2010 task created using the Software Distribution Dashboard. How do I schedule a pre-push to my remote BES relays for after hours?

I’ve looked a the document listed at the top of this page and it talks about manually copying files to the “default download cache location” Is that referring to the cache location on the BES Relay? That document was last updated in 2007. Is that still the correct method? How would you schedule this?

Previouisly I would take my install task, strip out the install line and push it to my relays. For tasks created with SWD Do I simply take the Begin–>End prefetch block, copy that into a new task and push it that way?

It’s REALLY too bad that I can’t use the “File Pre-Cache Wizard”, point it at my task or files listed in the Software Distribution Dashboard and be done!! At mininum an option to create a relay pre-cache task in the SWD or at least an option in the task to pre-cache on relays.

Is the the TEM Download Cacher the only option left? Having to go outside of the console, manually run a program on all of my replays and point it to a specifc sha1 value???

Am I missing something here??

10

(imported comment written by BenKus)

Hey j2johnson,

I will check with the application team to see about the download pre-cache file… but one thing you can do is make any child client of a relay download the file and it will be cached automatically be the relay…

Couple ways to do this:

  1. Manually modify the action to just download the file (don’t run it) and target the Relay (this is basically what the precache wizard does).

  2. Click the option in the take-action-dialog for at least one computer reporting to the relay to “start client downloads before the constraints are satisfied”. This will cause the relay (and the agent) to pre-cache the file.

Ben

11

(imported comment written by SystemAdmin)

I tried just taking the prefetch block from the task,

begin prefetch block

add prefetch item name=8C16CBAD2E3D70412C9D97B6849687A81854D78A sha1=8c16cbad2e3d70412c9d97b6849687a81854d78a size=1056859374 url=SWDProtocol://127.0.0.1:52311/uploads/8C16CBAD2E3D70412C9D97B6849687A81854D78A/compressedPackageData.bftemp.bfswd

end prefetch block

removing everything else and running it against the couple of test relays. That seemed to work with a bit of a caveat. The action status reports back as “Not Relevant” even though the file was copied out.

I had used this as my relevance so that the task would only be relvant to relays that don’t already have the file:

(exists relay service OR exists main gather service) AND ((not exists file “8C16CBAD2E3D70412C9D97B6849687A81854D78A” of it) of folder ((value “wwwRootFolder” of key “HKEY_LOCAL_MACHINE\SOFTWARE\BigFix\Enterprise Server” of registry as string) & “bfmirror\downloads\sha1”))

Taking a queue from a task generated by the pre-cache wizard I had set a custom success criteria of :

((not exists file “8C16CBAD2E3D70412C9D97B6849687A81854D78A” of it) of folder ((value “wwwRootFolder” of key “HKEY_LOCAL_MACHINE\SOFTWARE\BigFix\Enterprise Server” of registry as string) & “bfmirror\downloads\sha1”))

Is it simply that the relevance re-evaluated before the task had completed? If I remove the custom success criteria will the task simply report successful on the prefetch has completed?

12

(imported comment written by SystemAdmin)

Is it possible that with this new SWDProtocol it doesn’t follow the typical parent relay structure? I recently pushed out large packages (Office 2010) using the prefetch block in my previous post. Is it possible that a relay machine could pull a package from another relay that is not listed as it’s parent relay??? Each of my relays has the parent server listed as it’s relay however from my network monitoring tools it would appear the BigFix relays are downloading the package from somewhere other than the parent server as I don’t see the traffic outgoing to the relay. What is bad is that this means it is pulling from somewhere else where I can’t manage the traffic!!!

13

(imported comment written by BenKus)

Hey j2johnson,

It certainly is supposed to download from its parent relay…

Maybe start a support case to investigate further?

Ben