Relevance check on IOCs

Content Authoring BigFix.me
1

The relevance check below is returning ‘true’ for all endpoints but doing manual check using PowerShell proves the check return false.

Basically we have to do a lot of this kind of activity, where the Authority sends us a list of malicious MD5, SHA1 and SHA256 hashes, we have to check through all the files of all the computers to see if there is a match. I had created BigFix analyses, simple one, most of the time it returns false, but one particular list of MD5 hashes has returned true for all the endpoints, which is alarming.

Is there anything wrong with the relevance below?

exists descendant whose (md5 of it as lowercase = “baa93d47220682c04d92f7797d9224ce” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “23041caef38d4991296ffbe42743c691” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “da701d0e0ab6bfbddd747feebed96546” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “d41d8cd98f00b204e9800998ecf8427e” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “efcb51d4d8a55d441d194e80899bb2b0” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “231617ad2dc2a0c3f2d8e3241c57626f” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “92a0680fea369ae11f900c1a92e5499c” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “cf68e5165e3b89c0ece9b4905abf861a” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “5c0a4f9e67ced69eaea17092444b2c1a” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “22f49b12cb818728d293ae43082d8949” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “01c0e5316c7bba2ebdc00754a1d83f2a” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “5e501430acba545b719c0887357226dd” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “37fabfab797e631603a696b7ac2296d7” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “c10780e19363abda168c5861ce481635” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “671f4fb0c657d89c924064db6be0442e” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “a425d258e0ddf17fe412040b81d41aac” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “9cfb80616de943facef57fabbece780a” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “55e1897e20dbef5db7b4a718fd539ef7” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “83734ab1f8e17720271dc4b429ea0f6c” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “18f194fd3ae2455d8e26aad2e0dd6685” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “5fa71bdf383d16a6b25955bff53efb90” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “5af578a4785cc0683866fa19e262eb4d” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “e31fd661c75ca688e967a8cb3acaf667” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “ee501cdb0da38b6674f2156044a7c4fa” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “01772205e022a2ffd1809a471bd44333” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “6292ff91b59460d11cb00c8553b79b2d” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “c8d0ecf5c22d5806a5af87953844408c” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “7db95ed8565bbdbfc5ed4c5e80c68a4f” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “387bb23a8901baa300e42ce92310530e” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “f0411cd79ef1b71082f0817fe17fe1e6” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “25afe34ab1b36cc1ee118c9165f8619c” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “1bb7ba760f7f7cba0addd4a273b464f6” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “922af695fe14a7f70f8e068dcadc0584” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “729c12997f9639810666bb171ea9241d” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”) OR exists descendant whose (md5 of it as lowercase = “729c12997f9639810666bb171ea9241d” as lowercase) of root folders of drives whose (type of it =“DRIVE_FIXED”)

2

I would recommend against using ANY query that crawls the entire filesystem – it will virtually lock up your endpoints for minutes and likely in many cases return the Inspector Interrupted error. Instead, write a script (you mentioned PowerShell) and have the script executed at a set interval via a Task. This way, the burden of execution lies outside the context of the BESClient service. You could then use an analysis/fixlet to parse the results of the script.

5 Likes
3

While I wholeheartedly agree with Josh, the reason you are getting a true result for all of your endpoints is because your list of checksums contains the md5 result for a zero byte file (d41d8cd98f00b204e9800998ecf8427e).

So if you are scanning the whole drive you’re pretty likely to hit at least one empty file. Here is an example showing the checksums you should avoid for some popular checksum algorithms:

q: (size of it, md5 of it, sha1 of it, sha256 of it, sha512 of it) of file "C:\Temp\empty1.txt"
A: 0, d41d8cd98f00b204e9800998ecf8427e, da39a3ee5e6b4b0d3255bfef95601890afd80709, e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
T: 0.722 ms
I: singular ( integer, string, string, string, string )

q: (size of it, md5 of it, sha1 of it, sha256 of it, sha512 of it) of file "C:\Temp\empty2.txt"
A: 0, d41d8cd98f00b204e9800998ecf8427e, da39a3ee5e6b4b0d3255bfef95601890afd80709, e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e
T: 0.733 ms
I: singular ( integer, string, string, string, string )

Oh, and your query has 729c12997f9639810666bb171ea9241d listed twice.

5 Likes
4

I also wholeheartedly agree with Josh, and would also point out that if you’re determined to scan the entire hard drive in relevance, you’re also doing it in the least efficient way possible. Each IOC in your list is triggering its own independent file scan.

This snippet scans the entire content of every fixed disk, twice. With 50 IOCs in your list, you scan the drives 50 times. You could refactor that (but it will still be cripplingly slow) with something like

exists descendant whose (md5 of it as lowercase is contained by set of (( “729c12997f9639810666bb171ea9241d”;“c8d0ecf5c22d5806a5af87953844408c”) as lowercase)) of root folders of drives whose (type of it =“DRIVE_FIXED”)
At least this way all of the files on the drive only get checksummed once.

edit: I grabbed the one duplicate md5 to use in the example, it figures. Which brings up the next point on efficiency - by using a "set of " for the list of md5 hashes, any duplicates get discarded too because a set contains unique elements.

5 Likes
5

I know about the inefficiency thing - not the right way. For now I only want to know if the relevance statements are correct or not.

PowerShell - we had done that but not efficient and not effective. There is a policy wanting to disable PowerShell (though not enforced yet).

The best thing I can think of is to ask antivirus to incorporate the malicious hashes inside their definition files. However, it is still not good enough, I can’t automate it (have to manually create service request and upload files to them), then there are many malicious hashes submitted by Authority which antivirus company has no record of.

6

Thanks.

Very useful post, though I have difficulty trying to understand.
So the true result is because of a hash of zero byte file?

7

Thanks, very useful.

8

So the true result is because of a hash of zero byte file?

Yes, exactly. In my example you can see that I have two different zero byte files, and they return the same exact checksums. So whoever gave you that list inadvertently hashed an empty file. And as you saw from the results from your endpoints, pretty much any computer is going to have at least one empty file on it. So I’m not surprised they all returned True.

A zero byte file run through a message digest algorithm will always generate the same checksum for that algorithm. So if you are using md5 you should avoid checking for d41d8cd98f00b204e9800998ecf8427e, if you are using sha1 you should avoid checking for da39a3ee5e6b4b0d3255bfef95601890afd80709, etc.

1 Like
9

Alternatively, if you have BigFix Inventory enabled in your environment you can use it to collect checksums for files as well. I think by default it only looks at executables, but you can configure it to scan for more file extensions (or even entire filesystems – but you’d want to throttle the CPU for the scan in this case, which is configurable). You can see more info about this feature here.

1 Like
10

Totally agree. If you’re going to do this scan activity within the BigFix toolset (not using PowerShell or external tool to collect the data for the agent to parse and collect), BFI is the place to do it. This way it is using the independent scan engine, which doesn’t directly affect the agent’s activities. Like Josh said, you would want to make sure you throttle the CPU to keep from taxing machines unnecessarily. Good luck!

11

Too bad we don’t have BigFix Inventory license. I need to know more, if it is cheap and can do many wonders, it is justifiable to request to purchase.

12

I was interested in using BigFix for this purpose, so I whipped up a small Powershell script that trawls through the hard drive, hashes all the “interesting” files, and dumps the data to a SQLite database. You can then use BigFix to query the SQLite very efficiently. It doesn’t make any sense at all to scan through the drive repeatedly. Either scan using Powershell or VBScript or whatever, or deploy an EDR tool that monitors file writes and hashes on write.

The code for this is available at https://code.stanford.edu/secops-public/secops-posh/blob/master/Gather-InterestingFileHashes/Gather-InterestingFileHashes.ps1 - still beta quality, but it works, and querying it from BigFix with the SQLite inspectors is easy.

3 Likes