Improving BigFix integration with Splunk

[jmaple]

Greetings Programs!

It’s been a while since I’ve been around and that’s because I’ve had a job change in these past few months. With this change comes some time on my hands and with that time, I believe I want to improve integration with BigFix and Splunk.

Currently Splunk has a Splunk for BigFix app on Splunkbase that uses Python scripts to query the Web Reports SOAP API for deployment information. While this app does bring in information, I feel there is a much better way to do it using the REST API as well as adding the potential for automatic actions to be kicked off from analytic results from Splunk directly.

Is this something that the BigFix community would be interested in seeing? I know that my previous experience with the Splunk app was that while the information was basically just a host dump with some properties attached, I always felt there was more that was unexplored because the versions of BigFix and Splunk were many years ago when the apps and scripts were developed.

[Sean]

Welcome back! Due to my lack of experience with it, I always love seeing more REST API info / posts. I have seen a few posts about Splunk also, so I’m sure people would find it useful.

[Bighize65]

I would also like more information on this. I have a client looking to utilize this and I don’t have any experience with putting splunk and bigfix together.

[jgstew]

Many BigFix customers have said they are interested in this. There are different possible integration points between BigFix & Splunk.

One is using BigFix to get Asset Info for Splunk.

Another would be to use BigFix as a possible source for events in splunk.

Another is analyses in BigFix to audit the status of splunk on systems where it is installed. (what versions, is it running properly and communicating?, is it configured for X/Y/Z?)

Another is using BigFix to install/update/manage/configure splunk clients / forwarders / servers / etc…

Examples:

• https://bigfix.me/fixlet/details/21108
• https://www.bigfix.me/analysis/details/2998317
• https://www.bigfix.me/fixlet/details/4005
• https://www.bigfix.me/fixlet/details/25265
• https://bigfix.me/relevance/details/3017250
• https://www.bigfix.me/cdb/analysis/2994522
• https://bigfix.me/relevance/details/3021513
• https://bigfix.me/relevance/details/3017248
• https://bigfix.me/fixlet/details/25280
• https://www.bigfix.me/fixlet/details/4008
• https://www.bigfix.me/fixlet/details/4003

Related:

• How to configure splunk with Bigfix & syslog connector

[jmaple]

Bringing this thread back up, I wonder if I could ask the BigFix community for some assistance.

The development of this integration would work best if I had as many sources of information with a plethora of events as possible. The different components of BigFix contain a lot of information and will help us paint a great picture of the health of BigFix.

To the community, if you are willing to do it, provide me some log samples of BigFix clients, server logs, audit logs, relay logs, download plug-in logs, web reports logs, and any other logs you may stumble across relating to BigFix.

I strongly recommend the longs are sanitized to remove any trace of where it came from.

Please send any sanitized samples to jimmy at splunk dot com.

[jmaple]

Hitting this thread again to feel the room.

I believe I have a pretty boilerplate app ready that is used just for bringing in most of the flat logs as well as a REST API call that collects the hosts of a deployment with their DNS name, IP addresses, and, if available, the MAC addresses of the network adapaters of hosts.

I need to do some finishing touches with eventtypes and tagging for events in the flat logs but I’m hoping to have it ready by Thanksgiving.

[jmaple]

BigFix Technical Add-on for Splunk (1.0.0) is now available.

jimmyatSplunk/TA-bigfix

Coordination between BigFix and Splunk administrators is a really good idea. Check out the README for installation and configuration requirements.

Currently the add-on does NOT contain any eventtypes or tagging for CIM compliance but that is hopefully coming by the end of the year or early next year.

Feel free to contact me or reply here with issues.

[jmaple]

Greetings once again, BigFixers!

Over the past year I’ve been doing a lot of personal development on this add-on and have found the current implementation works well in smaller environments but tends to struggle at larger scales. Further development is going on now to try to deal with the scale issues for the larger customers. Depending on the drive for customers in larger environments wanting to ingest data from more than 50k endpoints, another add-on may be introduced or this one will be updated with a new version that is compatible with larger environments.

The current version of the add-on should work comfortably in environments with ~10,000 endpoints but the higher the number of endpoints is, the higher chance the queries will start to fail or cause other issues. So to get a gauge on scale, if you want to use the add-on, I’d like to get a rough estimate on how many endpoints are in environments this could go into so please answer the poll.

How many endpoints are in your environment?

• 1-99
• 100-4999
• 5000-19999
• 20000-49999
• 50000+

Show results

[DerrickD]

We’re new to BigFix, but are very interested to add this integration as we get started.

[Akshay19]

Hi Jmaple,

I required the splunk app for bigfix earlier it was availabe in the splunk app store, the same link is shared by you but now it is not available can you please me to find the splunk app for bigfix, is their any new link or new app?

[jmaple]

I believe this was due to the version of Splunk the app was configured to be “compatible” with. I believe it’s automatic based on versions of Splunk coming to EOL. Let me see what I can do.

EDIT: I forgot that I took 6.0.1 (the last version on Splunkbase before it was removed) and created a tree of it in my Git that I’m working to update.

jimmyatSplunk/splunk-app-for-bigfix

I would recommend NOT using the updated version (7.0.0) as the data that backs it uses the TA that’s being reworked at the moment. But if you’re just looking for 6.0.1, the link above should be what you need.

[_bsmith]

Hi jmaple,

I am trying to use your TA-bigfix app and am running in to some issues. I am unable to open the app after installing. I receive the dreaded horse-man error page: “Oops.
An error occurred while reading the page template. See web_service.log for more details Click here to return to Splunk homepage”

The only related entries in the web_service.log related to the request IDs are:
view:1054 - bypass module system fast path

Right now we are on Splunk 7.2.3. I have removed the app and re-downloaded/reinstalled.

Any ideas or suggestions?

[jmaple]

The name of the TA has to have the “master” part removed. If you remove that, you should see what your looking for. The app in the apps directory should just be “TA-bigfix”

[_bsmith]

Thanks, so much. That worked! I appreciate the quick response.

[rfjohns1]

TA-bigfix

Is there any available solution for Splunk 7.0.0+

[rfjohns1]

thanks! works.

I see in the Splunk logs
"SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)"

Does this need to be resolved for data collection into Splunk work?

[jmaple]

You may see that message but you should still get data indexed appropriately. Do you not see any data?

[rfjohns1]

no. Index is still at zero events.

Setting logging to Debug

[jmaple]

The workaround we use for this would be to add the certificate being presented at the time of connection to a file with a list of trusted certificates.

/opt/splunk/etc/apps/TA-bigfix/bin/ta_bigfix/httplib2

1. Export your certificate via a web browser or openssl. Splunk comes with openssl embedded in the software so you could use that in tandem with this post to export it to your host
2. Add the raw certificate in the same format as the others to the bottom of the file above and see if that solves your issue.

Let me know if you need further assitance.

[rfjohns1]

httplib2

I found that the BES server does not have a cert installed. Working to correct that. The log does seem to say that the connection should still be made.

“…is not trusted, this add-on will proceed to connect with this certificate.”

Otherwise I see from the log that port 52311 is being used, but I don’t see comm between splunk and BigFix using that port. I do see a variety of other ports beings used.

Does 52311 need to be opened on the firewall?

+++++++++++++++++++++++++++++++++++++++++++

**The response status=400 for request which url=https://xxxxxxxxxxxxx/:52311/api/query?

+++++++++++++++++++++++++++++++++++++++++++++
Creation Time Sender Facility Situation Action Src Addr Dst Addr Service IP Protocol Src Port Dst Port Network Application User Anomalies Severity Rule Tag Nat Rule Tag Nat Src Nat Dst Nat Src Port Nat Dst Port Logical Interface Physical Interface Src IF Src VLAN Protocol Agent Alert Syslog ICMP Type ICMP Code ICMP Id Inbound SPI Round Trip Elapsed Time Bytes Sent Bytes Rcvd Event Information Message
2019-11-13 12:09:24 xxxxxxxxxxxxxxxxx Inspection TLS_Certificate-Verify-Failed Permit xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx gTCP_Bigfix_63422 TCP 54886 63422 Untrusted-Generic-TLS Low 1228.0 Interface #6 801
2019-11-13 12:09:41 xxxxxxxxxxxxxxxxx Packet Filtering Connection_Closed xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx gTCP_Bigfix_63422 TCP 54886 63422 Untrusted-Generic-TLS 1974.5 Interface #6 801 17s 3556 2931 Connection closed Connection was reset by client
2019-11-13 12:09:57 xxxxxxxxxxxxxxxxx Packet Filtering Connection_Allowed Allow xxxxxxxxxxxxxxxxx 1xxxxxxxxxxxxxxxxx gTCP_Bigfix_63422 TCP 54967 63422 1974.5 Interface #6 801 New connection
2019-11-13 12:09:57 xxxxxxxxxxxxxxxxx Packet Filtering Connection_Closed xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx gTCP_Bigfix_63422 TCP 54967 63422 Generic-Web-HTTP 1974.5 Interface #6 801 0s 731 1188 Connection closed