Dynamic Downloads with NO Hash for 3rd party integration

Content Authoring
1

Hi all,

Just wanted to reach out and see if something might have some kind a trick that I haven’t thought of cause I am running into a brick wall every which way I try it. Just short background story - I am building a quick and dirty CMDB integration (using BigFix to distribute CMDB data files to each machine and then read the different ones back via properties in analysis) for CMDB system that is currently not supported (!= ServiceNow). To minimize traffic I broke the CMDB data files to one file per computer named with the SerialNo of the machine and I built my own logic to only make the distribution task relevant to machines whose data has recently changed (I have CMDBDataChange client setting that I change via RestAPI to highlight that).

Anyway, the problem I am stuck with is the distribution of those files require dynamic updates with no HASH and without having any data for the files (size, shas) but require relevance substitution to make the same code work for each endpoint. This is the code that I put in my distribution fixlet:

parameter “Repository” = "http://<root_server>:52311/CMDB"
parameter “PrefetchFile” = "{(if (it contains " ") then (concatenation “%2520” of substrings separated by " " of it) else (it)) of (it as trimmed string & “.json”) of (… Code to Evaluate Serial Number…)}"
download now as cmdb.json {(parameter “Repository”) & “/” & parameter “PrefetchFile”}

And it works fine on most machines but since “download now” forces direct download without using relay architecture I have about 10% of the machine that are on networks where they cannot reach the root server where all the download files directly and here comes my problem:

  • I tried to change it to “download as” but that command does not support Relevance substitutions (don’t understand why…)
  • Tried “add prefetch item” but that forces you to supply Hashes/Sizes and since those data files are changed periodically OUTSIDE of the system (I have a script that checks the CMDB data source, if the data for any machine has changed, it recreates the data file automatically and flags the machine with the client setting described above that a change is required), so that doesn’t work
  • Tried with “add nohash prefetch item” but again, that doesn’t support relevance substitutions (again, don’t understand why)

I’ve also explored generating a manifest but for 20k endpoints the manifest itself is 6mb in size and it’s ridiculous to have to download 6mb manifest every hour to accommodate maybe 50 machines having CMDB data changes to then have to download 50 2kb data files… Besides, since the manifest file itself would be changing frequently outside, I think I will have the same problem with the dynamic downloads with no size/hash.

Also, I did try on one machine to actually hardcode the URL with no hash, but what I found out is the “download” command doesn’t seem to be handling changes to the underlying files very well - you download the file now it’s fine, I make a change to the file but since the old file was already cached on relays it just uses that and doesn’t give you any control to re-cache it… Obviously with “download now” that problem with caching doesn’t exist!

Anyway, does anyone have any suggestions of something I may have missed? It seems that such a simple thing and last hurdle type on a long road, and still can’t seem to find a way around it…

2

One approach could be to have your REST API interface POST the small, changed per-machine file to the mailbox site of each endpoint. That approach is used in some of our other integrations.

Another could be to POST a manifest file - containing the url, size, and hashes of all of the files - to a shared site, and parse that file to dynamically build a prefetch command that is unique to each endpoint.

A third could be to develop a “download plug-in” which is certainly a greater development effort but is within the realm of possibility. That method is used in our Linux patch content to dynamically query RPM repositories on our root server, where the repo metadata is not known to the client ahead of time. The main challenge there is building a download plug-in that works across all of the operating systems that you need.

I would probably favor the first method, posting to the client mailbox sites, since you already have a working REST API framework.

3 Likes
3

Thanks for the suggestions, really appreciate it! Will explore those and see what comes out of it.

4

@JasonWalker I really like the first suggestion here. I had not thought of add files to site into the client mailbox for this use case before.

1 Like
5

Thanks, @JasonWalker, the mailboxing suggestion is in fact great! Powershell (language of choice) was a bit of a PAIN to write the upload with but got it working at the end!

1 Like
6

Glad to hear it’s helpful! There are a couple of edge-cases to be aware of…

  1. Clients may exist in the CMDB but do not yet exist in BigFix, and get added to BigFix later…so you cannot necessarily assume that only “new” things in CMDB are “new” to BigFix. You’ll need a way to identify systems that get added to BigFix that may need an older CMDB entry sent to them. One simple way may be to create a Fixlet or Analysis to check whether the mailboxsite file exists, so you can query 'relevant computers of ’ the fixlet to check the CMDB for content. An Analysis has the bonus that you could check the date of the file, or a date or version value inside the file, to see whether the CMDB has something newer to post to that client.

  2. Clients that get reset on BigFix get a new mailboxsite. Sometimes upgrades, system restores, etc. can reset the BigFix client, at which time they will delete their mailboxsite and download a new one. You can probably overcome this by having the client copy their CMDB file out of the mailboxsite and store it somewhere more permanent. When the client is reset and gets a new mailboxsite, the custom copy of the file can remain valid until the client gets a new CMDB file from its new mailboxsite.

3 Likes
7

Yea, those two were not a problem on my side as I load up the data sets from CMDB & BigFix, and only call the function to post of the record exists already In BigFix & if they have no CMDB data (have a property just checks if the CMDB file already exists or not). It’s all conditional with error-checking - don’t call anything blindly, but good tips. Thanks!

2 Likes