Large file size software distributions

(imported topic written by rdamours91)

Hey you crazy kids…first time caller long time listener.

I’m pushing out a Quickbook 2007 install out to 200 sites next week sometime. The sites are distributed with each having their own relay.

My install done through the software distribution wizard is about 488 MB and looks something like this. My setup.iss file is in the same folder that the setup.exe is in.

download http://bigfix.epsb.ca:52311/Uploads/b4f503b10bd43171c884ab288cfaf71f48eb2519/big10AB.tmp

continue if { (size of it = 520767410 and sha1 of it = “b4f503b10bd43171c884ab288cfaf71f48eb2519”) of file “big10AB.tmp” of folder “__Download”}

extract big10AB.tmp

wait __Download\Qbooks"setup.exe -s"

This biatch is going to take a while to come down to each relay. Is there a way to run this from a share on the bigfix server as an alternative or will this eventually wake it down and run?

(imported comment written by jessewk)

rdamours,

You will have a much more reliable experience with a download than a share.

Check that action script though. It doesn’t look right to me.

It should probably be: wait __Download\Qbooks\setup.exe -s

And you probably need to add a switch to use the .iss file too. I’d try it first by manually caching the download somewhere on the client and just running the install command. That way you don’t have to do the download every time and you can work on just getting the install command right.

(imported comment written by rdamours91)

I threw the command in quotes as I wasn’t sure how it would handle the -s after the setup.exe.

I can force feed a path with a -f1 parameter but I tested it at a command line and as long as the setup.iss was in the same folder as the setup.exe I should be ok.

Will the whole 488MB thing be a problem or should I be looking at a repackaging application to only push down the changed files?

(imported comment written by BenKus)

Hey rdamours,

If things are set up correctly, the 488MB shouldn’t be a problem at all… And you definitely don’t want things running from a network share because you will still have to transfer the file on the network and you wouldn’t be able to take advantage of the nice BES features like automatic download restarts, download throttling, download caching, etc.

Here are a couple of things to note:

  • Make sure your relays have enough disk space.
  • Consider raising the cache size of the BES Relays (I believe there is a Task for this).
  • Consider using the “Pre-cache wizard” that is available in BES 6.0 to get the file to the relays during off-hours.
  • Make sure your clients are choosing good close-by relays (so that they are not downloading this big file over the network).
  • Make sure that you have bandwidth throttling set if you have bandwidth concerns between the clients and relays.

Let us know how it goes… For some reason, I have heard many people who routinely push gigabytes (or even terabytes) of patches, updates, service packs, etc. through BES on a regular basis without problems, but suddenly get scared of bandwidth conerns when it comes time to install an application… Remember that BES originally was built as a general purpose systems management / security configuration management tool, and although we are the best patch management tool in existence (arguably – but a very solid argument) , we also do many other things very well too.

Ben

(Sorry for the mini rant…)

(imported comment written by rdamours91)

That’s the vote of confidence I was looking for…

I’ll let 'er rip for some testing tomorrow and let you know how it goes…

(imported comment written by BenKus)

Hey rdamours,

How did this go? Was your software distribution job successful?

Ben

(imported comment written by rdamours91)

We’ve done about 100 sites of 200 or so with incredible success. It seems that the 600MB weren’t that big of a deal.

Found the odd missing client and had to restart the odd relay but things are going swimmingly.

(imported comment written by pcaprio91)

We have IT Admins that want to download and cache all relevant patches locally to their servers to stage them prior to installing.

I am testing the use of the File Pre-Cache Wizard to cache Enterprise Security (patches for Windows).

I set the _BESGather_Download_CacheLimitMB size on my test system to 300Mb.

I select Fixlet ID(s) and enter in the Fixlets I are needed.

902903 902949 903711 903717 903719 903721 903723 903803 904101 904203 904407 904505 904603 904705 905109 905111 905203 905413 905503 905603 905703 905803 905901 906201

Click Search, then select all of the desired Fixlets.

Click Finish and deploy.

The status ends up at “Not Relevant”, and the fixlets are not cached nor installed.

But yet I am still showing them listed as Relevant Fixlets.

I have tried some of the suggestions that I found in related posts, but with no success.

BES 8 should have this feature built-in, but until then, is there another way I can work around this to download all fixlets to a system first, then schedule a time to run them?

(imported comment written by JasonO91)

pcaprio,

The old school way to do caching was to:

Open the task that was created during the software deployment wizard. The sha1 value is located between the quotes after sha1 of it = “Sha1 value here” in the action script. This information will be used to find the correct file on the BES Server.

The file will be located in the BES Server directory under wwwrootbes\bfmirror\downloads\sha1. Copy the desired file to the relay server’s sha1 directory.

This is the correct location for the file to reside on the remote server. When a client requests the file, the relay server will look the file up in this directory, and if it is there, will send the file to the client.

We used to use robocopy or some other tool to copy the file out to the relays before BigFix started using the nifty bandwidth throttling features.

Hopefully this will help you get the files out there and we’re not threadjacking rdamours.

Jason

(imported comment written by BenKus)

Hey pcaprio,

The pre-cache wizard was designed for relays so it will only run on relays (it will report “not relevant” if you target other computers). You can change this by removing the relevance that checks for the relay before you send the action…

Also, note that the setting you mentioned “_BESGather_Download_CacheLimitMB” is a relay setting… To increase the size of the agent cache, you want to use the similar (but BES Client specific) setting of “_BESClient_Download_DownloadsCacheLimitMB”.

Default “_BESClient_Download_DownloadsCacheLimitMB” is 20MB and default “_BESGather_Download_CacheLimitMB” is 1000MB.

Ben

(imported comment written by pcaprio91)

Wow, big difference.

BES Support said for me to set the reg key _BESGather_Download_CacheLimitMB.

I deleted this key and set _BESClient_Download_DownloadsCacheLimitMB.

Selected the 23 patches, modified the relevance to (true) and pushed.

The patches cached on my local client.

Thx Ben for the correct info!!!

(imported comment written by Shembop91)

Ben,

Am I missing something, because I don’t see where the pre-cache wizard lets you cache software. It only allows you to cache items under subscribed sites. Software Distribution isn’t in there.

(imported comment written by SystemAdmin)

I have the same question, I need to deploy large packages and I want to pre cache those packages and be sure that the endpoint will take the packages from the replays not from the server, How can we do that?

(imported comment written by BenKus)

Can you just use the option “pre-cache on agents before deployment” option in the take action dialog?

Ben

(imported comment written by SystemAdmin)

Are there any size limitations for Software Distribution? We are attempting to package up Adobe CS 5.5 which is 4.85 GB in size. and we have successfully had this file uploaded to the TEM server, however we are continually having issues deploying the task.

At first we received the Download Failed status message “The download size exceeds the maximum value set in the client setting _BESClient_Download_PreCacheStageDiskLimitMB, which can be modified through the Edit Computer Settings dialog.” We set this setting to 10240 and it appears to get further in the deployment but in the end the task seems to fail at this size check

Failed continue if {(size of it = 5204393423 AND sha1 of it = “8a735014e65617e94b519e0593377b5a5a3c3201”) of file “Adobe_CS55_32.tmp” of folder “__Download”}

When I check the BES Client__BESData__Global__Cache\Downloads directory I see the sha1 file and when I look at the properties it has the Size as 5204393423 but the size on disk is 5204393984. Looking at the deployment logs it looks like it creates the .tmp file properly.

logs from time of the deployment:

Command succeeded (Using download manager collected file) download http://XXX.YYY.com:52311/Uploads/8a735014e65617e94b519e0593377b5a5a3c3201/Adobe_CS55_32.tmp (fixlet 75100)

At 18:02:58 -0600 -

GatherHashMV command received.

No matching site.

At 18:03:10 -0600 -

GatherHashMV command received.

No matching site.

At 18:03:41 -0600 -

GatherHashMV command received.

No matching site.

At 18:29:50 -0600 - actionsite (http://XXX.YYY.com:52311/cgi-bin/bfgather.exe/actionsite)

Command failed (Relevance substitution failed) continue if {(size of it = 5204393423 AND sha1 of it = “8a735014e65617e94b519e0593377b5a5a3c3201”) of file “Adobe_CS55_32.tmp” of folder “__Download”} (fixlet 75100)

At 18:33:38 -0600 -

ActionLogMessage: (action 75100 ) ending action

At 18:33:38 -0600 - actionsite (http://XXX.YYY.com:52311/cgi-bin/bfgather.exe/actionsite)

The old school way to do caching was to:

Open the task that was created during the software deployment wizard. The sha1 value is located between the quotes after sha1 of it = “Sha1 value here” in the action script. This information will be used to find the correct file on the BES Server.

The file will be located in the BES Server directory under wwwrootbes\bfmirror\downloads\sha1. Copy the desired file to the relay server’s sha1 directory.

This is the correct location for the file to reside on the remote server. When a client requests the file, the relay server will look the file up in this directory, and if it is there, will send the file to the client.

We used to use robocopy or some other tool to copy the file out to the relays before BigFix started using the nifty bandwidth throttling features.