Architecting a Log Search Solution in Restricted Environments (No SMB/RDP)
Hi everyone,
I’m currently designing a workflow to centralize client log searching and viewing in an environment where direct access via SMB or RDP is strictly prohibited. Since I cannot "reach out" to the endpoints, I need to leverage BigFix to "push" the data inward.
I’ve done some initial testing and wanted to share my logic and see if anyone has implemented a more elegant approach.
The Challenge
We need a way to search and read full client logs centrally.
Session Relevance in Web Reports doesn't natively expose BES Site Files.
REST API allows us to list and download site files, but we cannot query the content of those files without downloading them first.
My Proposed Workflow (The "ETL" Approach)
Ingestion: Use the BigFix Upload Manager to periodically move specific client logs to the Root Server.
Staging: Use a script to organize these logs into a dedicated Content Site. To keep things searchable, I’m using a naming convention: [ComputerName]-[ID]-[FileName].log.
Transformation (Middleware): A background process polls the REST API for new/updated files, downloads them, and transforms the raw text into a structured format (like a SQLite DB or a consolidated flat file).
Presentation: Creating a Custom Web Report that reads from this processed data source to provide a searchable UI for helpdesk/admins.
Questions for the Community
Database Bloat: Has anyone seen performance degradation on the Root Server when using the Upload Manager for high-frequency log collection?
Optimization: Is there a way to leverage the Action Site or Custom Sites more effectively to bypass the need for an external DB?
Alternative: Does anyone have a "lighter" way to display the contents of an uploaded file directly in Web Reports without a middle-ware transformation step?
I’m curious to hear how others are handling log aggregation when the standard ports are closed.
Current Strategy: For clients with active communication, we leverage the Query app to retrieve the current day's BES Client log. For clients experiencing connection issues, we utilize an external tool capable of querying the machine through alternative channels.
I believe this is generally achieved through log monitoring tools that already have a lot of capabilities built-in for this.
If you only available tool is BigFix, I feel like you are going to use a lot of DB resources to storage the data and read it. I am not even sure how slow a report in WebReports could get if you have a lot of data coming from the Clients’ Logs.
As much as I ever hate to say that BigFix is not the right tool for a job, in this case I have to agree with @fermt 's assessment that there are better purpose-built log monitoring & searching tools.
To be clear we're talking specifically about the BigFix Client Logs, right? I do think it's a fairly normal use-case to upload the log files themselves through the Archive Manager on a periodic basis, and then handle them on the server-side.
But once the files are on the server, I don't know of any customers that convert them to a SQLite format, and I think it will be problematic to try to build a Web Report to read them.
I do think there might be an approach using a SIEM tool to pull the log files from the root servers' UploadManagerData directory and import them from there into the SIEM tool of your choice. While I think it would be better to use a client-side agent that can filter the logs locally before uploading, if BigFix is the most reliable communication channel you have that could be valid.
When it comes to SIEM tools, I think many of our customers use QRadar or Splunk, if you'd like to compare features on commercial products; lately I've been working a lot with Loki and Grafana, which are available in free versions for self-hosting and I think would be worth exploring in this use-case.
Not quite the same use case as yours but I’ve used Bigfix for capturing certain event log data via a property which held possibly several hundred KB per endpoint. Though no problems with the inspection and the data getting up to the server other than an increase in DB size, retrieving the data via API calls to Web Reports was somewhat sluggish, and Web Reports itself struggled if you tried to filter for text within the property, presumable as that type of data and usage isn’t really what the DB was designed or optimized for.