Recently, within the past month or so, and at an accelerating rate, the “BES Web Reports Server” service has been crashing, recently on a daily basis (often more than one time a day). I can always start the service back up with "net start “BES Web Reports Server”’ or within services.exe and it always comes back up, without any issues. Also, there are no other issues on this 9.5.6 server - everything else is working just fine.
Its gotten to the point where I’ve written a small fixlet that generates a batch script that runs the net start command to start “BES Web Reports Server” up and scheduled it to run once an hour, forever, on the Bigfix server (it also logs a few things for me)… This is total hackery and not at all how I want to solve this, and so far IBM has not been of much help.
One thing I’ve noticed is that the service crashes often at the same period of time during the day - 3:10pm, or 12:10pm, or 7:10pm … I see 3:10pm the most often. At first we thought that this had to do with our nessus / tenable scanners somehow causing the service to crash when probing it for vulnerabilities, but we took the bigfix server out of the scan rotation, and then I went a step further and null routed the bigfix server on the nessus scanner boxes themselves (which had the side effect of making them stop checking into the Bigfix server… heh!). This did not help. I have removed the null route, but we are still, for the moment, excluding the server from scans.
Has anyone ever seen anything like this?? Any advice or help or nudge in the right direction would be greatly appreciated… Thanks!
That’s interesting because we’ve been experiencing regular crashes of the BESWebReportsServer service on our stand-alone instance (Linux) for the past two releases. We also implemented a policy to restart the service if the process disappeared. We worked a PMR regarding the matter and were unable to determine root cause or correlate a specific report, etc. We’re not at 9.5.6 yet in production, but will be interested to see if the issues persist.
Just curious, have you engaged IBM via PMR on your issues?
I run WR on that level of OS, so wondering what might be interacting with the system. I presume the system is fully patched? And if you are running WebUI that you have patched your OS (as indicated by the fixlets) as there are some OS leaks on that version of the OS?
I’d definitely open a PMR as there is a lot of info which might glean some ideas.
@cmcannady I presume you opened a PMR, I might have to look into that info
Yes, Win2k8R2, fully patched. Nothing else is running on the system except MSSQL (as a prereq for Bigfix), and a couple of agents that are required by policy (symantec EP mainly).
I have engaged IBM with a PMR but so far we have not come to a determination as to what is causing the crashes. I thought it might be our nessus/tenable scans, but we removed the BIgfix server from the scan asset list altogether, and it made no difference.
So, so far no solution… I will update thread if I get something more concrete from IBM. At this point they basically want me to run a ProcMon and catch a crash of the BESWebReportsServer.exe - something we have not been able to setup yet.
This may not be helpful, but we have found that running certain custom reports regularly causes our web reports service to crash. We are running BigFix 9.2.8 and our web reporting server is remote from our main BES server. The quarterly reports that cause the service to crash are very large, and they do complete, but we always expect to have to restart the service after running them.