Computers stuck in <not reported>, have to manually restart BES service

Usage and Config
41

lol…I thought it was weird that you were recommending to change all the Properties to Every Report considering the machines taking long to eval, but yeah…I TOTALLY misread that. I thought it was to get data faster from the machines…and after 24 hours I would have updated data, which I am obviously not.

Either way I just changed it to their default values and refreshing on 3 different systems that I am currently testing with. I’ll see what comes up for those.

Thanks!

Edit: I also found this article regarding performance counters. By the looks of it, it reports the same data about cycle times. https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0023415 Shouldn’t make a difference to use the one your recommended vs this one?

42

ok it’s been about 12± hours since I reverted back to the original reporting times of the analysis and it doesn’t seem that much different.

During my fall down the rabbit hole yesterday, I noticed that there’s a column called Client Evaluation - Last UDP Ping. I have a good number of machines that are not receiving that ping. They reside in vlans where other machines do get the ping so it’s not a corporate fw issue. We do use the Windows Firewall, and I got curious on and was able to get on some machines in which the BESClient fw inbound rules were missing! How does that even happen? How can the client be installed but somehow missing the fw rules? The end users are not admins to the machines so it’s not the end user just deleting rules.
image

I exported the fw rules from my machine and took out the 3 registry lines and came up with this relevance and action to add them to the registry and to restart the client afterwards.

not exists key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules” of native registry

regset64 “[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules]” “{{E9671E3F-116D-4260-ACB3-362EEE9D3BB5}}”=“v2.30|Action=Allow|Active=TRUE|Dir=In|Protocol=58|App=C:\Program Files (x86)\BigFix Enterprise\BES Client\BESClient.exe|Name=BES Client (ICMPv6)|”

regset64 “[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules]” “{{68913DCF-3D98-4DFC-87BD-C0F56DD9FF2B}}”=“v2.30|Action=Allow|Active=TRUE|Dir=In|Protocol=1|App=C:\Program Files (x86)\BigFix Enterprise\BES Client\BESClient.exe|Name=BES Client (ICMPv4)|”

regset64 “[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules]” “{{F52B19FB-7CA6-4354-8EFD-E2DB8B23CDB5}}”=“v2.30|Action=Allow|Active=TRUE|Dir=In|App=C:\Program Files (x86)\BigFix Enterprise\BES Client\BESClient.exe|Name=BES Client|”

// Delay for 10 seconds
waithidden timeout /t 10

// Restart BESClient service
wait cmd /C net stop “BESClient” & net start “BESClient”

funny thing is that I searched the forum and I found someone who mentions fixlet 518 Windows Firewall is Blocking BES Traffic but it doesn’t come up as relevant. I looked over the relevance and with my limited knowledge I don’t know if it picks up missing BES Client fw inbound rules.

so back to my registry fixlet - I tested this on a few test machines and it worked. I then ran it on the machines in question but still they’re showing up not receiving the UDP ping in the analysis.

I can only assume this could be related to what we’re experiencing with the clients not reporting on a timely fashion and taking long to eval?

Here’s the analysis sorted by Client Evaluation - Receiving UPD Ping:

image
image1566×606 43 KB

If I sort it by Last UDP Ping then I have all recent days, many for today and yesterday to about a week back.

image
image1574×1022 81.6 KB

43

If there are no resource problems with these devices, try turning on the following settings on a couple of them and see if the reporting improves or not.

Set the _BESClient_Resource_SleepNormal client setting to value 50
Set the _BESClient_Resource_WorkNormal client setting to value 50

44

I’m afraid I disagree on this point, I think those settings are too aggressive. It would allow the BESClient to constantly use 50% of a processor core for background evaluation and I think that’s unnecessary.

@ceez I’m afraid it may be some time before I can give a more full answer on this, but I think the UDP notifications are a separate issue to work on later. The first thing to address is the client evaluation cycle; and I think you may be right about the clients continuing to report old values in the Analysis. It’s been a while since I looked into this area but I’m refreshing my recollection with the article at Data Collection: BigFix Client - Customer Support . Find the section about the ‘Client Usage Profiler’, you may need to either use the listed fixlets from the BES Support site, or custom settings, to configure how long to keep the tracked fixlets.

I believe the default is “for as long as the client is running” - which means that the ten slowest evaluations are tracked, until you restart the BESClient service. You may have disabled or removed the property that was evaluating slowly, but the client profiler retains the record from when it was the slowest. You might try restarting the BESClient on some of those machines and then using the ‘Force Refresh’ option to have a few of them report and ensure the analysis result is updated; then use those Client Usage Profiling fixlets to change the defaults to reset the list of slowest fixlets each day, so you’ll see improvements over time as you change the properties.

45

@JasonWalker alright I got homework to do here. It seems extensive but a good challenge, I’ll see when I have time to go through that and will return my findings.

I know the udp is another thing, but how about the missing fw rules? Why would they be missing? Are they truly not needed?

@vk.khurava I have a machine that no one is using and out of curiosity I’ll set those at 50 to see what happens.

Thanks gents.

46

Ok so I got the BESClientDiagnostics logs and files…but now what do I look for?

On the diagnostic log both workstations share the same Summary of ALL Warnings below

======= Summary of All Warnings ===========
- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\engine.dll does not exist

- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\ADInspectHelper.dll does not exist

- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\Inspectors\ADInspect.dll does not exist

- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\Inspectors\Core.dll does not exist

- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\Inspectors\Client.dll does not exist

- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\Inspectors\Inspect.dll does not exist

- WARNING: C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\Inspectors\RegExp.dll does not exist

- WARNING: Can't contact http://dmz.externalurl.com:52311/cgi-bin/bfenterprise/clientregister.exe.

- WARNING: Can't contact http://dmz.externalurl:52311/cgi-bin/bfenterprise/besgathermirror.exe.

- WARNING: Analysis reference failed to parse: C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\BES Inventory and License\Analyses.fxf

- WARNING: Analysis reference failed to parse: C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\OS Deployment and Bare Metal Imaging\Analyses.fxf

- WARNING: Analysis reference failed to parse: C:\Program Files (x86)\BigFix Enterprise\BES Client\__BESData\Power Management\Analyses.fxf

- WARNING: Could not find a site folder for siteurl: http://rootserver.domain.local:52311/cgi-bin/bfgather.exe/actionsite

- WARNING: Can't find C:\Program Files (x86)\BigFix Enterprise\BES Client\BESLib\Inspectors

In attempt to get better visibility to machines that are online, I bumped up the Mark as offline after from 45 min to 120 min

I get a better number of machines showing as online, but even then I get some that I can ping showing up as gray.

The one I just checked shows up in the evaluation analysis as

average cycle 55.0 (I assume these are in mins)
max cycle 182
image

YET it doesn’t show any slow client evaluations

So it’s either not reporting correctly or there’s something up with the workstation or maybe bes client? is a repair/reinstall an option? Is there a way to reinstall\repair via fixlet?

Starting to feel like groundhog day here…losing track of what to troubleshoot at this point.

Edit:
I restarted the BESclient on the workstation and now it’s average 0.18 and 0 for max cycle…
image