How to get all fixlet results that have changed since last time query was run

Content Authoring
1

I’m new to the BigFix relevance language but I am trying to write a relevance query to get all of fixlets results that have changed since the last time I ran the query. My goal is to have a script run this relevance query on an interval to get new fixlet result events and pass those event to another tool to do analysis on.

Below is the relevance I have come up with so far. I am testing it using BigFix’s SOAP API. However, it seems to have poor performance against large data sets (200k computers, 100k fixlets). Any ideas how to optimize this (if at all) or if there is another way to pull this data from BigFix?

(	
	id of fixlet of it as string,
	id of computer of it as string,
	relevant flags of it,
	first became relevants of it as string,
	last became relevants of it as string,
	last became nonrelevants of it as string
) of results 	
whose (	
	first became relevant of it >= "26 Sep 2018 14:05:00 -0700" as time 
	OR last became relevant of it >= "26 Sep 2018 14:05:00 -0700" as time  
	OR last became nonrelevant of it >= "26 Sep 2018 14:05:00 -0700" as time 
) of bes fixlets
2

Anybody have any ideas on this one? trying it now myself to extract deltas instead of full dumps.

3

With or without the filter, this query would effectivly touch 20 billion rows of data, and then filter them down, assuming 200k endpoints and 100k fixlets, as the original poster stipulated.

The filter uses OR, so only one of the 3 elements of the whose will cause the filtered results out of the query.
A small tweak would be to reorder the filter. last became relevant, then last became nonrelevant and finally first became relevant, because of the way those fields would change over time would get “hits” in the filter faster. But I would only expect marginal improvements.

One could also consider filtering the Fixlet set or the Computer set to be smaller before using the date filters.

4

I did a little ad hoc testing this morning. 3 computers, 28k Fixlets, 1104 Fixlet results (many of my computers are not subscribed to all sites, and so will not have results for every Fixlet) My test box is a wimpy little virtual box running on my laptop, so I don’t expect great performance.

q: number of bes fixlets
A: 28025
q: number of bes computers
A: 3
q: number of results of bes fixlets
A: 1104

The query as it stands (42 seconds runtime - 264 results) - Note I added Number Of in front of the query to remove the “creating the giant strings” portion of the query. It lets me better quantify the runtime of the underlying query vs the time spent constructing strings for the browser to display.

q: number of (id of fixlet of it as string, id of computer of it as string, relevant flags of it, first became relevants of it as string, last became relevants of it as string, last became nonrelevants of it as string) of results whose (first became relevant of it >= "26 Sep 2018 14:05:00 -0700" as time OR last became relevant of it >= "26 Sep 2018 14:05:00 -0700" as time OR last became nonrelevant of it >= "26 Sep 2018 14:05:00 -0700" as time) of bes fixlets
A: 264

Adjusted the date to last week to reduce total results (42 seconds - 24 results)

q: number of (id of fixlet of it as string, id of computer of it as string, relevant flags of it, first became relevants of it as string, last became relevants of it as string, last became nonrelevants of it as string) of results whose (first became relevant of it >= "Fri, 10 Apr 2020 10:44:35 -0400" as time OR last became relevant of it >= "Fri, 10 Apr 2020 10:44:35 -0400" as time OR last became nonrelevant of it >= "Fri, 10 Apr 2020 10:44:35 -0400" as time) of bes fixlets
A: 24

The query without First Became Relevant (42 seconds run time - 24 results)

Q: number of (id of fixlet of it as string, id of computer of it as string, relevant flags of it, last became relevants of it as string, last became nonrelevants of it as string) of results whose (last became relevant of it >= "Fri, 10 Apr 2020 10:44:35 -0400" as time OR last became nonrelevant of it >= "Fri, 10 Apr 2020 10:44:35 -0400" as time) of bes fixlets
A: 24

Without the filter at all (42 seconds - 264 results)

Q: number of (id of fixlet of it as string, id of computer of it as string, relevant flags of it, last became relevants of it as string, last became nonrelevants of it as string) of results of bes fixlets
A: 264

Without accessing any the 3 timestamps (<1 sec - 1104 results)

Q: number of (id of fixlet of it as string, id of computer of it as string, relevant flags of it) of results of bes fixlets
A: 1104

If you do not need the last became relevant, last became nonrelevant, first became relevant timestamps, which are very expensive for Web Reports to work with, then drop those timestamps out of your query and simply query the entire dataset.

1 Like