Using prefetch to download files but without sha1 hash

[lordtux]

Hi all,

I was reading some post and wiki but I don’t found the exactly answer.

I have some scripts to run in some servers, this part I understood but I need some external files, that are update every day by a external team. From the clientes, I can’t download those files directly, so I from what I see I need use the prefetch command that will cache those files on the IEM Servers to send to the clients.

So, for example:

prefetch unzip.exe sha1:e1652b058195db3f5f754b7ab430652ae04a50b8 size:167936 http://software.bigfix.com/download/redist/unzip-5.52.exe

I found the example above, I tested and ok, this would work to download some files from external sources to the clients but my problem is:
As the file is changed everyday and I don’t have control of that, how I can get the sha1 hash, seems that the prefetch command need the sha1 parameter right? As well the size parameter, how I can do to prefetch ignore this and only download the indicated package and put on the clients?

I wanted all of this because I need create a task that run a shell script on a set of servers, but the scripts use some files that are update every day from one source link that I have.

[TimRice]

I don’t believe you can use the PREFETCH command do to this.

You need to use the DOWNLOAD NOW AS command.

You can read about DOWNLOAD NOW AS in the BigFix Action Guide

ftp://public.dhe.ibm.com/software/tivoli/IEM/9.2/Platform/Action_Guide.pdf

[AlanM]

You can use the plain “download” command ( https://developer.bigfix.com/action-script/reference/download/download.html ) which does not specify a hash but uses the caching mechanism though the download now might be needed as the server may think it already has the download for the action once it downloads it.

You could also use a dynamic download which has a no-hash version ( https://developer.bigfix.com/action-script/reference/download/add-nohash-prefetch-item.html )

[BigFixNinja]

And note, if you use dynamic downloading be prepared to whitelist the download source on the BigFix server to allow the download through:

http://www-01.ibm.com/support/knowledgecenter/SS63NW_9.2.0/com.ibm.tivoli.tem.doc_9.2/Platform/Config/c_dynamic_download_white_lists.html

http://www-01.ibm.com/support/knowledgecenter/SS63NW_9.2.0/com.ibm.tem.doc_9.2/Platform/Action/c_dynamic_downloading.html

[lordtux]

Thanks all for the answers, I’m making some tests with the information here.

[finsel]

Hi, many BES Server does not allow you to download files from different sites (by Security reason).
You can think in a alternative solution, for example, create some secure script to make your downloads and then store the downloaded files in your Relay server as sha name convention. Then your endpoints should prefetch it normally.
tks

[lordtux]

Hi finsel,
Yes, I see that, as I don’t manage every aspect from the IEM server I asked our IEM team to check for grant access the url but I’m thinkin In another solution for this as I explained here:

Transfers files between servers

Hi All, My scenario, I have a couple of servers, for example, Server A is where I have some files that I need sent to server B, C, D. I want to use IEM to gather the files from server A and put on server B, C, D on specific folders. But I want to automate this, on case would be logs generated on the servers B,C,D that will be transfered to one location on server A. I read about the upload manager to accomplish this, do you have more suggestions and examples? And more, if I create a baseline …

From one of our servers clients, that have internet access through proxy, I thinkin in setup one script to download the required files in regular basis and use IEM to collect those files and redistribute to the other clients.

[finsel]

Hi Lordtux,
Yeah, this works too !
Just keep safe your script, repository and also get all approvals needed for this implementation.
tks
Frank

[jgstew]

As the file is changed everyday and I don’t have control of that, how I can get the sha1 hash

You could use the REST API to automatically generate a fixlet to do what you need whenever the files change. It would dynamically update the prefetch statement to be correct for the new files.

This would allow the files to flow through the relay infrastructure using the hash values for proper verification.

[matthias.wulkow]

I’ve just tried to download a dynamic created xml file:
https://www-01.ibm.com/support/docview.wss?uid=nas4PSPbyNum&aid=1
using
download as ptf.xml https://www-01.ibm.com/support/docview.wss?uid=nas4PSPbyNum&aid=1

Unfortunately, BigFix safes a very cryptic docview.wss as ptf.xml

Any Ideas ?

[JonL]

download now as ptf.xml https://www-01.ibm.com/support/docview.wss?uid=nas4PSPbyNum&aid=1

Tested it and it works fine for me. Just need to add the ‘now’.

[JonL]

As a fun aside, with the “download now as” command, you can then do relevance substitution within the URL if desired. That opens up all kinds of possibilities with internal corporate web apps and ERP systems.

[matthias.wulkow]

Hi, thanks for that. Bad luck for me: using “download now as” leads to:
(Can’t download file ‘General transport failure. BAD SERVERNAME (winsock error 4294967290’) download now https://www-01.ibm.com/support/docview.wss?uid=nas4PSPbyNum&aid=1

Using curl with same proxy settings on besserver works like a charm. Strange.

[JasonWalker]

download now as triggers the client to perform the download directly - these download requests do not use the root server or relay hierachy and do not get cached.

[matthias.wulkow]

As using prefetch shows a similar result with downloading “docview.css” I played around with curl (our BESServer runs on Linux):
wait curl -x --proxy-user <USER+PW> -L https://www-01.ibm.com/support/docview.wss?uid=nas4SF99728&aid=1.xml -o ptf.xml

This runs like a charm, unfortunately Proxy und and Password appears in in cleartext in logfile.
Is there any way to use the defined credentials, maybe as parameter or something like that ?

[itsmpro92]

Try using the action log command to turn off parameter logging:

action log command
wait curl -x --proxy-user ...
action log all

[matthias.wulkow]

Works perfekt, thanks very much.

[brolly33]

User and password will still appear in the action and fixlet .FXF files on the client.

If you want to remove those as well, you should check out the Secure Parameters techniques mentioned on https://bigfix.me/fixlet/details/3678 It’s trickier to implement, since you have to work with javascript inside of the Fixlet Description, but the security is worth it in the long run.