Our number issue with BigFix/IEM in our organization is Console performance. One of the main reasons for our issues is due to having over 300 console operators, with as many as 100 operators simultaneously. The number of operators also causes there to be many actions.
We have opened a PMR: 68487,7TD,000
This is an issue that we have struggled with for over a year now. It seemed to really start with the upgrade to 9.0.x and has not gotten better since.
Does anyone else have similar issues? Possible solutions?
We have thrown hardware at the problem with little effect. We have tweaked many different settings to try to help with the issue, and are planning more tweaks. We have not changed much in the way of Windows Server OS settings themselves in an effort to enhance performance.
Details:
Our current root server version: 9.2.3.68
OS: Windows Server 2012 R2
Our database is on a separate system. The DB Server, App Server, and Terminal Server (console) are all using 10 gigabit NICs on the same switch. Almost all storage is split between multiple RAID1 SSD volumes.
The 9.2.3 upgrade did not help, I just added some details above.
We are not exactly sure where the issues are or why they are occurring, but certain things are worse than others. Content Creation, Wizards/Dashboard, Taking Action, Permissions, etc… but issues do sometimes occur when you are doing nothing at all. The console often locks up for 1 to 10 minutes. Sometimes it locks up, will seem to come back for about 30 seconds, and then lock up again… and this could go on for 30 minutes in some cases.
If you haven’t added the details yet to the PMR can you indicate what the configurations of the Console Terminal Server machines are like, and include the cache settings that you have used and what kind of differences you have noticed in behaviour. Like have you tried a no cache scenario as you have such a high connection rate to the server?
Also indicated in the PMR, but I just witnessed the following behavior again:
The instant each of the freezes occur we can watch memory used by that user’s BESConsole.exe process climb up by about 800-1000MB and consume about 3% CPU for the duration of the WSoD. In this scenario, my console was using ~1900MB pre-freeze, then sat at ~2700MB for a minute or so during the entire period. The instant the freeze thaws out the process’s memory utilization dropped back down to 1700MB with 0% CPU and was again functional.
During this last freeze I was about to take an action on one of our custom tasks, specifically it occurred immediately after hitting “OK” to actually take the action where I would then be prompted to enter my CO password. As expected, when the freeze stopped the top-most window was where I would re-authenticate to then hit OK to initiate the action. I don’t recall the issue always occurring during the re-authentication process, but will specifically keep on the lookout for that.
As James indicated the problem seems to occur more frequently when using a dashboard or wizard (doesn’t seem to matter which ones).
My organization is having issues with the IEM console freezing as well. We are currently trying to get some Terminal Servers, but currently using IEM console directly from our laptops.
It is amazing how much “stuff” can be forgotten about what you’re working on while you’re waiting and seeing the busy mouse pointer spin away…
Hardware for the root server is different, and we’re still using 9.0 at the moment, but should be at 9.2.2 by the end of June.
The reason for stopping at 9.2.2 is that 9.2.3 requires a 64 bit OS to run on. The Terminal Servers, when in place, will help with that transition.
We are currently working at the Console freeze issue through PMR 68487,7TD,000, in order to determine the root cause. We’ll provide a further update here as we move forward in getting to the root cause.