Hoping someone has come across the same issue or could shed some light on the problem I’m having. We are doing a simple api call to get gather Compliance data and feeding it into splunk. We are having issues pulling back the measured values data…when we add that field we get memory heap errors, so we had our splunk team try pulling a smaller data set, but they then started to see “NULL” “ERROR” or “ActiveX error encountered”…We don’t currently have all analyses enabled for all checklists. Even in the Compliance Webui, if we try generating a report with Measured Values, it takes 10-15 minutes to get the report generated. Does this particular value have a different way of generating that data that is causing these issues?
Any information someone could shed on this subject would be very helpful, thanks!
The measured values should also be available using the BigFix Platform REST API to get results from the BigFix analyses that actually get the measured values raw data.
And in both Compliance and the BigFix interface, they are optional, and ideally the analyses should not be activated unless you are trying to determine why a system is non-compliant. The Measured Values analyses can return huge volumes of data in some cases, and are not needed for calculating whether a system is compliant or for generating scores.
Your Splunk ingest may need to have very long timeouts and/or very large buffers to handle the volume of data you can get from a Measured Values result, as well as handling all sorts of data formats.
Since he’s the best expert on Splunk I know of that lurks here, I’ll page in @jmaple
Thanks for the response Jason. My suggestion was to not enable these analyses in production anyway…we are dealing with 75k to 100k machines, and we have a Dev and Test environment to work out issues with Checks failing before they get implemented into Prod. I’ve tired explaining that this data shouldn’t be needed on this scale, and, if we are having issues with a subset of machines, simply turn on analysis mode, remediate the machines, and turn if back off. However, someone higher up the food chain determined that we need this data, so all the info I can bring back the better.
They don’t pull in the description field because of this very issue, so they may just have to leave measured values out. Appreciate the help!
For reference, at a previous job I was in an environment with large NFS shares. The NFS Server itself did not resolve the UID and GID of users to user accounts, it was up to the NFS clients to do that - and not every user was known to every NFS client.
There’s a check that “every file has a known Owner and Group”, that fails on these clients and on the server because while the GID and UID are valid, they don’t resolve.
The “Measured Values” analysis for this check retrieves a list of the pathname for every file that didn’t have a resolvable owner/group - and this analysis could return properties of several hundred megabytes for each client in my case.
Funny you should ask. I’ve been working recently with a customer of mine to ingest Compliance data using the Compliance API.
In our testing, we found it was easier to take in the checks with their ID’s and other metadata that describes what they are and a second input that gives us the ID of the host, the ID of the check, and the state of it. We then turned the list of checks into a lookup for easy correlation of data. We did it this way because it scaled better in larger environments.
I am working on bringing that work into the existing add-on I’ve been developing over the past year to query the BigFix REST API for data collection. Give it a look if you haven’t already.
Thanks jmaple! I’ll definitely give this a look through, really appreciate the info, and I may reach out with some a few more questions as we dive into this.
I’m going to post it in the other thread where I introduced the add-on but I’m going to note that due to scaling issues with the current app, I’ve been working with some of my fellow Splunkers internally to make it much more scalable in larger envrionments and I will hopefully be able to post the first iteration of what that looks like in the near future. It will likely require you reinstall the add-on from scratch to ensure there aren’t any conflicts with older data ingestion methods.