Hello, I’m attempting to track a specific item in the server_audit.log file and have built the relevant link below for that. However, it looks rather expensive to run each time, and I can’t think of any other way out. Could you kindly examine and recommend any effective ways to shorten the timing?
Q: ((preceding text of last " " of (preceding text of first ", " of it & ", " & preceding text of first ":" of (following text of first ", " of it))) of (it as string) of (preceding texts of firsts "|" of following texts of firsts "|" of lines whose (it contains "my_string") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log" as string))
A: Wed, 29 Nov 2023
A: Sun, 03 Dec 2023
A: Sun, 03 Dec 2023
A: Sun, 03 Dec 2023
A: Mon, 04 Dec 2023
A: Tue, 05 Dec 2023
A: Wed, 06 Dec 2023
A: Wed, 06 Dec 2023
A: Wed, 06 Dec 2023
T: 21790.934 ms
Q: number of (preceding text of last " " of (preceding text of first ", " of it & ", " & preceding text of first ":" of (following text of first ", " of it))) whose (it as string = (current date as string)) of (preceding texts of firsts "|" of following texts of firsts "|" of lines whose (it contains "my_string") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log" as string)
A: 3
T: 11311.387 ms
This is probably not the best use-case for BigFix, I’d really suggest using a SIEM tool like Splunk for log parsing/aggregating/reporting, but there’s nearly always a way forward.
At first check of this relevance, it looks fairly efficient - no cross-product tuples are generated, the ‘lines of file’ retrieval only runs once…but I tried a couple of variations, and I was surprised to find that actually casting to time & retrieving date from it appears to be faster than some of the string manipulations you’re doing. I searched for an entry that appears a lot in my log - “ServerPluginService” has about 27,000 entries in mine. Your query ran in 6168 ms on mine, while the following gives the same result and ran in 3754 ms:
q: dates (local time zone) of (it as time) of preceding texts of firsts "|" of following texts of firsts "|" of lines whose (it contains "ServerPluginService") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log"
Likewise, this one runs in about 3700 ms, where your original ran in about 6300:
q: number of dates (local time zone) whose (it = current date) of (it as time) of preceding texts of firsts "|" of following texts of firsts "|" of lines whose (it contains "ServerPluginService") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log"
We know how long the returned string should be, so how about using substring instead of all the parsing
((substrings(0,16) of following texts of firsts "|" of lines whose (it contains "my_string") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log" as string))
I’m not even sure if the start of the line is always a single digit follwed by |
I tried both of your queries, but the results are almost the same on my computer—not much different.
Q: ((substrings(0,16) of following texts of firsts "|" of lines whose (it contains "my_string") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log" as string))
A: Wed, 29 Nov 2023
A: Sun, 03 Dec 2023
A: Sun, 03 Dec 2023
A: Sun, 03 Dec 2023
A: Mon, 04 Dec 2023
A: Tue, 05 Dec 2023
A: Wed, 06 Dec 2023
A: Wed, 06 Dec 2023
A: Wed, 06 Dec 2023
T: 12392.588 ms
Q: number of dates (local time zone) whose (it = current date) of (it as time) of preceding texts of firsts "|" of following texts of firsts "|" of lines whose (it contains "my_string") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log"
A: 3
T: 11072.724 ms
Q: dates (local time zone) of (it as time) of preceding texts of firsts "|" of following texts of firsts "|" of lines whose (it contains "my_string") of file "C:\Program Files (x86)\BigFix Enterprise\BES Server\server_audit.log"
A: Wed, 29 Nov 2023
A: Sun, 03 Dec 2023
A: Sun, 03 Dec 2023
A: Sun, 03 Dec 2023
A: Mon, 04 Dec 2023
A: Tue, 05 Dec 2023
A: Wed, 06 Dec 2023
A: Wed, 06 Dec 2023
A: Wed, 06 Dec 2023
T: 11219.860 ms
I think Jason is correct - the requirement is much better suited to a log ingestion tool
The other idea I had was to find the total number of lines, then use that to read the last n lines and parse just those (this does assume you are mostly interested in more recent results than digital archeology)
Is there still a built in 10 second limit for a property to complete its evalation when processed by the service?
I recall many versions ago (I think it was in the v7 and v8 era) it being mentioned that a property inspection would abort if it had failed to complete within 10 seconds. If so then with a 11219.860 ms time it may not even produce any results outside of the FixletDebugger
