Computers stuck in <not reported>, have to manually restart BES service

Usage and Config
1

Hey everyone. Need some help in trying to figure out this issue.

We have machines for will stay as<not reported> for days and the only way for the action to complete is to manually restart the BES service (or if the user just happens to reboot their machines which we know users never do).

If I look at the logs there are no “GatherHashMV command received”, but once I restart the service it starts to work showing the GatherHashMW and showing the relevant actions in which it then runs to completion and it reflects as Pending Restart on the console.

We are running bfix and client ver 10.0.9.21.

Thanks!

1 Like
2

Since your client reports after a restart, I am ruling out any permissions or site subscription issues. It appears that you are experiencing problems with UDP communication.:

  • Verify and enable UDP communication for every endpoint.

  • Turn on command polling for every 20 minutes on all devices. This will force the clients to check for content every 20 minutes or at any other interval you choose. It is not advised to set the polling frequency too low, though, as this can lead to many performance problems.
    https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Config/r_client_set.html#r_client_set__cpol
    _BESClient_Comm_CommandPollEnable
    _BESClient_Comm_CommandPollIntervalSeconds

  • Enable persistent connection. BESClient is intelligent enough to determine when data should go over TCP and when it shouldn’t. I have done so on all of my devices and have not noticed any negative effects.

https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Config/c_persistenconn.html
_BESRelay_PersistentConnection_Enabled

Note: While I have not observed any issues when command polling and persistent connection are both enabled, however it is recommended that you disable command polling if you choose to use persistent connection.

3

Agree with all of the post, with exception of this part. Enabling both Command Polling and Persistent Connections is pretty widely-used and I’d recommend enabling both.

4 Likes
4

thanks for this information and input, unfortunately way above my knowledge of bfix but willing to give it a try.

In regards to
_BESClient_Comm_CommandPollEnable
_BESClient_Comm_CommandPollIntervalSeconds

I can set these parameters manually on the computer(s) by right click and editing computer settings. I do see there’s a task called BES Client Setting: Enable Command Polling so I can mass deploy.

The default action is

_setting "BESClient_Comm_CommandPollEnable"=“1” on “{parameter “action issue date” of action}” for client

_setting "BESClient_Comm_CommandPollIntervalSeconds"=“3600” on “{parameter “action issue date” of action}” for client

What @vk.khurava is saying I should set the BESClient_Comm_CommandPollEnable"=“1” to =“20” but what is the commended value for BESClient_Comm_CommandPollIntervalSeconds which defaults to 3600?

Now for the _BESRelay_PersistentConnection_Enabled, I see there’s also a task for that as well called Persistent Connection: Enable Relay. This task has 3 different settings so not sure if I make the change using the task

_setting "BESRelay_PersistentConnection_Enabled"=“1” on “{parameter “action issue date” of action}” for client
_setting "BESRelay_PersistentConnection_MaxNumber"=“{parameter “MaxNumberOfPersistentConnections”}” on “{parameter “action issue date” of action}” for client
_setting "BESRelay_PersistentConnection_NumberPerSubnet"=“{parameter “MaxNumberOfPersistentConnectionsPerSubnet”}” on “{parameter “action issue date” of action}” for client

Or should I just manually set the _BESRelay_PersistentConnection_Enabled"=“1” as mentioned in the link you provided.

We do have 2 relays, one in the dmz to reach the machines outside the network and the other relay inside the network.

On both of these machines I went to Edit Computer settings and I dont see that Name listed, so this obviously means that this setting is not applied to our relays, correct? Is the main bfix server considered a relay as well? Should I set the persistent connection on it as well?

Thank you again for the help with this.

Edit:
And what sort of issues could be expected if we made these changes to all workstations (Win & MAC environment) as well to our two relay servers? I understand that @JasonWalker says this is widely used, but if anything, what should we be on the lookout for? Thanks again!

Edit2:
I was checking the log on one of the machines I made the change in and I see the change in the log file showing “PollForCommands” every 15 mins which is the setting I set (900 seconds). So to further understand what this will do is force the machines to poll every 15 mins and this will reflect in the console as the computer showing as available (black text) instead of grey when in fact I know that the machine is online and responding to ping?

pollforcommands
pollforcommands1248×594 179 KB

But even with this setting, when an action is pushed the client will pick up the action without having to wait to the next 15 min polling interval?

5

It’s up to you how you want to set it, but 20 minutes was just an example and based on personal recommendation; it’s not too aggressive on the client and does fit for the majority of the team.

As I mentioned in my earlier comments, both of them were enabled in my environment, and I didn’t find any problems. After implementing persistent connection, I continued to monitor a few of my devices and didn’t find anything unusual, but I read somewhere that if persistent connection is implemented, command polling is not necessary because persistent connection keeps the TCP channel open and is faster than command polling to grab the content.

Exactly !

Based on this log snapshot and the fact that the client can complete the BigFix query, I don’t think there is a problem with UDP communication; however, if you have enabled persistent connection, it can also be received via TCP.

You can easily verify whether BESClient can receive UDP packets by looking for the entries “GatherHashMV command received” OR “ForceRefresh command received” in the client log.

Another relevant/fixed item is that it is being updated in the client logs based on the evaluation of the gathered content and indicates that it was found to be relevant and has since been fixed. It is not an action, more of a fixlet/task.

6

Thanks again for the wealth of information.

I have left the polling time at 15 mins (900 secs) and I have seen a few machines that I know are persistently powered on and do not sleep that eventually show up gray. Not sure what is going on. We have both 52311 UDP and TCP to and from all relays to endpoints. When reviewing the corporate firewall there is no traffic block on the port\protocol. Also the bes logs on these machines do show the PollForCommands line every 15 mins so it’s odd that it should be gray. I can successfully send a refresh, send a fixlet action or just a blank action successfully to these machines. The GatherHash shows within seconds or a minute or two max. When sending a refresh it’s reflected in the bes logs within a min or two max.

Do you have such issues with machines that you know are powered on? For what it’s worth I am focusing on our Windows 10/11 workstations.

The only other thing that could be is that we have the Windows firewall enabled, so not sure if those might somehow be causing issues. We had lots of back and forth with our bfix msp in getting the firewall properly enabled (Which registry to use for example), so I wonder if with so many changes the win fw on some machines stayed in some strange state. Is there a best practice fixlet or tips on getting the Win fw configured properly? Just basic enabled for all profiles? domain, private & public?

Thanks once more!

7

When you look at the BESClient logs, what is the difference in time between “Report Posted Successfully” on that device? Clients will be marked as black in text or Online, otherwise greyed out or Offline, if this is less time than the time specified under “Mark as Offline after” in your console preferences.

Default reporting time is 15 min, in below screen shot I have my custom timings.

image

If “Report Posted Successfully” is not occurring at the appropriate intervals, the client may be busy with other matters. To find out why, you should examine the BESClient logs to see what is taking longer. Enabling BESClient Debug logs & usages profiler are the two option you can try.

FIxlet ID # 157 - BES Client Setting: Enable Debug Logging
FIxlet ID # 361 - TROUBLESHOOTING: Enable BES Client Usage Profiler

Another possibility is that, although your client is sending reports on time, they are not reaching the root server from your local site relay. In this case, you can verify whether or not the relay is not reporting and whether the other children from the same relays are exhibiting the same behavior.

1 Like
8

So the logs show a PollForCommands every 15 mins:

polling-every-15-mins
polling-every-15-mins580×834 185 KB

System preferences are as follows:
bfix-preferences

And in case it makes a difference, the computer settings for this machine:

bfix-computer-settings
bfix-computer-settings754×530 55.9 KB

We have many machines pointing to this relay, and this one in particular is not the only one that randomly shows greyed out, I am just using it as one example. And actually this happens to random machines that either contact the primary server or either the inside relay or dmz relay. Again, fw is open tcp\udp 52311 both ways.

Now back to this machine, I am trying to deploy the 2 fixlets you mentioned but it’s just not getting to this particular client. No Gather log entries and send refresh seems to do nothing. I have restarted the BES service and even restarted the machine itself but no luck; action status is .

I would really like to get to the bottom of this since even when we had an msp handling our bfix instance they really couldn’t figure out as to why we had this issue.

Thanks

9

I see the issue; according to this screen shot, it takes your client an average of 1.5 hours to post a report, which suggests that they are undoubtedly busy with something.

You can look into below link for enabling debug & profiler logs manually.

https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0022437

10

I am curious, but how did you get to 1.5 hours :slight_smile: I’ll check out that link.

and I was poking around the relay server log (logfile.txt) and there are many lines like this one but with different ip’s, this is the one for the machine in question

Tue, 23 Jan 2024 05:48:08 -0500 - /cgi-bin/bfenterprise/clientregister.exe (5704) - Uncaught exception in plugin ClientRegister with client 192.168.184.42: HTTP Error 55: Failed sending data to the peer: Connection died, tried 5 times before giving up

Does this help?

11

done editing registry for debug and profiler. I’ll let this cook overnight and report again tomorrow

:+1:

12

From here:

image image

It appears that your internet-facing client attempted to connect but was unsuccessful. There may be a number of reasons for this, but I don’t believe this error message has anything to do with your problematic computer. You can safely disregard that if the majority of your clients and your relay are able to report correctly in the console.

13

Also check the "long-running evaluations topic at Tip: Troubleshooting Client Reponsiveness for some Analyses to help identify what is taking so long.

Common culprits would be some custom property that recurses through the whole disk using 9ne of the ‘descendant’ inspectors, or something that collects hashes of many files or of large files.

2 Likes
14

I was looking at that last night, good article you put together. I’ll look at it more today.

Could it be a resource issue on the relay server? There are no resources alerts and we have 830 machines registered.

I was sorting the computers by last report time and by this particular relay and I see that there 466 machines on that relay and I have 55 with today’s date that are showing up gray. I did a spot check and some are not replying to ping so maybe not available, but there are many that do reply and show gray.

@vk.khurava I have 10 usageprofiler.txt.000#.log files on BES client folder. Any advice on how to read them or do I post them here?

15

If you examine these log files, you’ll see something like this below. The elapsed timing should be your primary focus; if it’s increasing in seconds or minutes, you need to verify the content & fix it.

0000) 28.407: actionsite.1368:Evaluate Property 1
0001) 21.206: Enterprise Security.500222905:Background Evaluation

Breakdown:

0000)                                : Serial numbering
28.407                               : Timing in seconds, here its 28 seconds
actionsite.1368                      : Site Name & content ID
Evaluate Property 1                  : Seems Retrieved Property evaluation
Background Evaluation                : Fixlet/task/Baseline background evaluation
16

ok, so there’s definitely an issue with evaluations. First of all I found this query to use with q&a on this machine: Analyzing profiler output

lines whose (((following text of first ") " of preceding text of first “.” of it) as integer) >= 1) of files whose (name of it contains “usageprofiler.txt” and modification time of it = (maximum of modification times of files whose (name of it contains “usageprofiler.txt”) of parent folder of regapp “BESClient.exe”)) of parent folder of regapp “BESClient.exe”

Looking at the last file with the above query (is that called a query?) it’s taking up to 280 seconds!

Start:Wed, 24 Jan 2024 09:36:31 -0500
Elapsed Time:01:00:04
Tracking: Top 100
Samples:8837
Elapsed Evaluation Time:00:36:11

  1. 280.483: actionsite.6170:Background Evaluation
  2. 187.204: Enterprise Security.281736901:Background Evaluation
  3. 170.502: Enterprise Security.281736903:Background Evaluation
  4. 116.516: CustomSite_-2eShared_Content.123603:Background Evaluation
  5. 28.692: Patching Support.344:Evaluate Property 1
  6. 25.184: Enterprise Security.281743003:Background Evaluation
  7. 24.519: Enterprise Security.281744103:Background Evaluation
  8. 23.570: Enterprise Security.281743301:Background Evaluation
  9. 22.787: Enterprise Security.281743303:Background Evaluation
  10. 22.730: Enterprise Security.276791601:Background Evaluation
  11. 22.064: Enterprise Security.281743001:Background Evaluation
  12. 21.200: Updates for Windows Applications.8200035:Background Evaluation
  13. 21.115: Enterprise Security.281744101:Background Evaluation
  14. 20.595: Updates for Windows Applications.8200044:Background Evaluation
  15. 20.531: Updates for Windows Applications.8200043:Background Evaluation
  16. 20.395: Enterprise Security.276021203:Background Evaluation
  17. 19.718: Updates for Windows Applications.8200037:Background Evaluation
  18. 19.671: Enterprise Security.281743503:Background Evaluation
  19. 19.646: Enterprise Security.1308401:Background Evaluation
  20. 19.575: Enterprise Security.281743501:Background Evaluation
  21. 18.955: Updates for Windows Applications.8200039:Background Evaluation
  22. 16.508: Enterprise Security.282564001:Background Evaluation
  23. 16.458: Enterprise Security.1308403:Background Evaluation
  24. 14.938: Enterprise Security.1302309:Background Evaluation
  25. 14.088: Enterprise Security.276021201:Background Evaluation
  26. 12.567: Enterprise Security.276784901:Background Evaluation
  27. 11.950: actionsite.26413:Evaluate Property 2
  28. 11.774: actionsite.26413:Evaluate Property 2
  29. 11.176: Enterprise Security.1402415:Background Evaluation
  30. 10.976: Enterprise Security.288301903:Background Evaluation
  31. 10.166: Enterprise Security.281730101:Background Evaluation
  32. 9.588: Enterprise Security.281730103:Background Evaluation
  33. 7.851: Enterprise Security.1205706:Background Evaluation
  34. 7.035: BES Inventory and License.1:Background Evaluation
  35. 6.306: Enterprise Security.281742601:Background Evaluation
  36. 6.297: CustomSite_-2eShared_Content.59450:Background Evaluation
  37. 5.511: Enterprise Security.1402413:Background Evaluation
  38. 5.497: Enterprise Security.288055301:Background Evaluation
  39. 4.737: Enterprise Security.1408207:Background Evaluation
  40. 4.723: Enterprise Security.12034107:Background Evaluation
  41. 4.717: Enterprise Security.246004301:Background Evaluation
  42. 4.700: Enterprise Security.1402417:Background Evaluation
  43. 4.700: actionsite.129466:Background Evaluation
  44. 4.678: Software Distribution.8:Evaluate Property 1
  45. 3.983: Enterprise Security.1402423:Background Evaluation
  46. 3.973: Enterprise Security.1402211:Background Evaluation
  47. 3.965: Enterprise Security.259708701:Background Evaluation
  48. 3.955: Enterprise Security.1402217:Background Evaluation
  49. 3.949: Enterprise Security.258937501:Background Evaluation
  50. 3.947: Software Distribution.8:Evaluate Property 2
  51. 3.943: Enterprise Security.288301901:Background Evaluation
  52. 3.936: actionsite.13:Evaluate Property 1
  53. 3.929: Enterprise Security.1107421:Background Evaluation
  54. 3.921: Enterprise Security.1307501:Background Evaluation
  55. 3.912: Enterprise Security.1302311:Background Evaluation
  56. 3.902: Enterprise Security.1206029:Background Evaluation
  57. 3.885: Enterprise Security.1310601:Background Evaluation
  58. 3.875: Enterprise Security.1307405:Background Evaluation
  59. 3.199: Enterprise Security.1107423:Background Evaluation
  60. 3.191: actionsite.13:Evaluate Property 1
  61. 3.188: BES Support.1987:Background Evaluation
  62. 3.186: BES Support.518:Background Evaluation
  63. 3.183: Enterprise Security.302503603:Background Evaluation
  64. 3.173: Enterprise Security.1310605:Background Evaluation
  65. 3.171: BES Support.3664:Background Evaluation
  66. 3.160: Enterprise Security.258937503:Background Evaluation
  67. 3.152: Enterprise Security.1206603:Background Evaluation
  68. 3.149: Enterprise Security.1307403:Background Evaluation
  69. 3.145: Enterprise Security.1300301:Background Evaluation
  70. 3.142: BES Support.3101:Background Evaluation
  71. 3.126: Enterprise Security.1307503:Background Evaluation
  72. 3.124: BES Support.5177:Background Evaluation
  73. 3.111: Enterprise Security.1300235:Background Evaluation
  74. 3.108: Enterprise Security.903304:Background Evaluation
  75. 3.105: Updates for Windows Applications.7057081:Background Evaluation
  76. 3.092: Enterprise Security.1310603:Background Evaluation
  77. 3.086: Enterprise Security.259708703:Background Evaluation
  78. 3.086: Updates for Windows Applications Extended.17100101:Background Evaluation
  79. 3.067: Enterprise Security.1300307:Background Evaluation
  80. 2.831: CustomSite_-2eShared_Content.35432:Background Evaluation
  81. 2.396: BES Support.4967:Background Evaluation
  82. 2.396: Updates for Windows Applications.7056232:Background Evaluation
  83. 2.373: Enterprise Security.258929803:Background Evaluation
  84. 2.371: Enterprise Security.258929801:Background Evaluation
  85. 2.370: Enterprise Security.1300304:Background Evaluation
  86. 2.364: Enterprise Security.282564003:Background Evaluation
  87. 2.358: BES Inventory and License.20:Background Evaluation
  88. 2.354: Enterprise Security.276034403:Background Evaluation
  89. 2.349: Enterprise Security.1300305:Background Evaluation
  90. 2.348: actionsite.172049:Background Evaluation
  91. 2.348: BES Support.3662:Background Evaluation
  92. 2.348: CustomSite_-2eShared_Content.4332:Evaluate Property 4
  93. 2.348: Enterprise Security.1300303:Background Evaluation
  94. 2.345: Enterprise Security.903306:Background Evaluation
  95. 2.345: CustomSite_-2eShared_Content.33593:Evaluate Property 2
  96. 2.343: Enterprise Security.276034401:Background Evaluation
  97. 2.343: CustomSite_-2eShared_Content.4332:Evaluate Property 4
  98. 2.342: Updates for Windows Applications.7057081:Background Evaluation
  99. 2.337: Enterprise Security.1300311:Background Evaluation
  100. 2.335: Enterprise Security.1402409:Background Evaluation

So there’s obviously a big delay on this particular client, can we also presume that other clients will be experiencing the same delays? And this delay is what is causing the bes client to not properly report on time and make the machines look offline?

And to fully understand:

  1. 280.483: actionsite.6170:Background Evaluation
  2. 187.204: Enterprise Security.281736901:Background Evaluation

it’s taking 280 seconds to evaluate actionsite 6170, then it moves on to the next one which takes 187 seconds, etc…

Thanks again so much for your help, this is all intriguing and fun at the same time! :+1:

EDIT:

So I was browing my computer and I see that I usageprofiler logs as well, probably enabled by the msp at some point.

On my machine I dont have those long times for evaluation:

Start:Thu, 25 Jan 2024 09:14:39 -0500
Elapsed Time:00:53:09
Tracking: Top 100
Samples:44612
Elapsed Evaluation Time:00:20:12

  1. 16.206: Patching Support.344:Evaluate Property 1
  2. 13.415: actionsite.26413:Evaluate Property 2
  3. 8.466: Enterprise Security.317864401:Background Evaluation
  4. 8.442: Enterprise Security.281736903:Background Evaluation
  5. 7.717: Enterprise Security.317864401:Background Evaluation
  6. 7.631: Enterprise Security.401118301:Background Evaluation
  7. 7.568: Enterprise Security.281736903:Background Evaluation
  8. 6.999: Enterprise Security.281736901:Background Evaluation
  9. 6.990: Enterprise Security.401108301:Background Evaluation
  10. 6.911: Enterprise Security.401160101:Background Evaluation
  11. 6.218: Enterprise Security.281736901:Background Evaluation
  12. 6.177: Enterprise Security.317864401:Background Evaluation
  13. 6.175: Enterprise Security.401113301:Background Evaluation
  14. 6.166: BES Inventory and License.1:Background Evaluation
  15. 6.162: Enterprise Security.401108301:Background Evaluation
  16. 6.161: actionsite.6170:Background Evaluation
  17. 6.146: Enterprise Security.281736901:Background Evaluation
  18. 6.144: Enterprise Security.311518003:Background Evaluation
  19. 6.141: Enterprise Security.311518003:Background Evaluation
  20. 6.138: Enterprise Security.401160101:Background Evaluation
  21. 6.134: actionsite.6170:Background Evaluation
  22. 5.451: Enterprise Security.401118301:Background Evaluation
  23. 5.425: Enterprise Security.281736901:Background Evaluation
  24. 5.402: Enterprise Security.401160101:Background Evaluation
  25. 5.395: Enterprise Security.401108301:Background Evaluation
  26. 5.328: Enterprise Security.401113301:Background Evaluation
  27. 4.765: actionsite.6170:Background Evaluation
  28. 4.665: Enterprise Security.311518003:Background Evaluation
  29. 4.627: Enterprise Security.317864401:Background Evaluation
  30. 4.623: Enterprise Security.401108301:Background Evaluation
  31. 4.606: Enterprise Security.311518003:Background Evaluation
  32. 4.603: Enterprise Security.401118301:Background Evaluation
  33. 4.592: actionsite.6170:Background Evaluation
  34. 4.549: BES Inventory and License.1:Background Evaluation
  35. 3.897: Enterprise Security.281736903:Background Evaluation
  36. 3.874: CustomSite_-2eShared_Content.123603:Background Evaluation
  37. 3.863: BES Support.4422:Background Evaluation
  38. 3.852: Enterprise Security.401113301:Background Evaluation
  39. 3.842: actionsite.6170:Background Evaluation
  40. 3.812: Enterprise Security.401113301:Background Evaluation
  41. 3.787: Enterprise Security.1308401:Background Evaluation
  42. 3.133: Enterprise Security.401160101:Background Evaluation
  43. 3.096: Client Manager for TPMfOSD.6:Background Evaluation
  44. 3.090: OS Deployment and Bare Metal Imaging.Analyses.fxf@00000000:Background Evaluation
  45. 3.088: Enterprise Security.281743003:Background Evaluation
  46. 3.083: Enterprise Security.281743301:Background Evaluation
  47. 3.080: Enterprise Security.500109201:Background Evaluation
  48. 3.078: Enterprise Security.407311911:Background Evaluation
  49. 3.078: CustomSite_-2eShared_Content.123603:Background Evaluation
  50. 3.077: Enterprise Security.281743501:Background Evaluation
  51. 3.077: Enterprise Security.1206029:Background Evaluation
  52. 3.073: Enterprise Security.281743003:Background Evaluation
  53. 3.070: Enterprise Security.281743001:Background Evaluation
  54. 3.069: Enterprise Security.281744103:Background Evaluation
  55. 3.068: BES Inventory and License.54:Background Evaluation
  56. 3.066: Enterprise Security.1308403:Background Evaluation
  57. 3.061: CustomSite_-2eShared_Content.123603:Background Evaluation
  58. 3.059: Enterprise Security.281744103:Background Evaluation
  59. 3.059: Enterprise Security.903306:Background Evaluation
  60. 3.054: Enterprise Security.276021203:Background Evaluation
  61. 3.049: Enterprise Security.1308403:Background Evaluation
  62. 3.035: Enterprise Security.281744103:Background Evaluation
  63. 2.883: CustomSite_-2eShared_Content.123603:Background Evaluation
  64. 2.444: Enterprise Security.281743001:Background Evaluation
  65. 2.374: BES Support.3342:Background Evaluation
  66. 2.365: CustomSite_-2eShared_Content.61037:Background Evaluation
  67. 2.355: Enterprise Security.401118301:Background Evaluation
  68. 2.349: CustomSite_Management.4502:Background Evaluation
  69. 2.347: BES Support.1987:Background Evaluation
  70. 2.342: actionsite.133572:Background Evaluation
  71. 2.335: CustomSite_-2eShared_Content.59450:Background Evaluation
  72. 2.330: actionsite.13:Evaluate Property 1
  73. 2.330: CustomSite_-2eShared_Content.59450:Background Evaluation
  74. 2.322: Enterprise Security.276791601:Background Evaluation
  75. 2.320: Enterprise Security.281743303:Background Evaluation
  76. 2.319: Enterprise Security.903304:Background Evaluation
  77. 2.318: Enterprise Security.281743001:Background Evaluation
  78. 2.317: actionsite.2113937577:Background Evaluation
  79. 2.316: Enterprise Security.903304:Background Evaluation
  80. 2.316: Client Manager for TPMfOSD.6:Background Evaluation
  81. 2.315: Enterprise Security.281743503:Background Evaluation
  82. 2.314: Enterprise Security.281743503:Background Evaluation
  83. 2.314: Enterprise Security.276021203:Background Evaluation
  84. 2.313: Enterprise Security.281744101:Background Evaluation
  85. 2.312: Enterprise Security.281743003:Background Evaluation
  86. 2.310: Enterprise Security.276021201:Background Evaluation
  87. 2.308: OS Deployment and Bare Metal Imaging.136:Background Evaluation
  88. 2.307: Enterprise Security.1302309:Background Evaluation
  89. 2.303: Enterprise Security.281744101:Background Evaluation
  90. 2.303: Enterprise Security.1308403:Background Evaluation
  91. 2.303: Enterprise Security.281743003:Background Evaluation
  92. 2.302: Enterprise Security.276021203:Background Evaluation
  93. 2.298: Enterprise Security.903306:Background Evaluation
  94. 2.297: Enterprise Security.281743301:Background Evaluation
  95. 2.292: Enterprise Security.1308403:Background Evaluation
  96. 2.292: BES Asset Discovery.18:Background Evaluation
  97. 2.287: Power Management.69:Background Evaluation
  98. 2.286: actionsite.133572:Background Evaluation
  99. 2.285: Enterprise Security.282564001:Background Evaluation
  100. 2.283: Enterprise Security.1302309:Background Evaluation

Not sure if 16 seconds is acceptable but it’s definitely not 280 seconds! And what is the ‘recommended’ time range for evaluations to complete?

17

That longest-evaluating entry is from ‘actionsite’, i.e. the Master Action Site. That could be an Action, or a custom Fixlet/Task/Baseline/Analysis.

Because the evaluation type is ‘Background Evaluation’, we know that this is the Relevance of a fixlet/task/action, and not an Analysis Property. If this were a Property, it would have a message like ‘Evaluate Property 2’ on lines 26 & 27.

In the Console’s Fixlets/Baselines views, be sure to add a column for ‘ID’ and then you can see whether this is a Fixlet, Task, or whatever, and can see whether the Relevance for it can be optimized.

The next two long-running entries are from the ‘Enterprise Security’ site; this is the internal name of ‘Patches for Windows’. You won’t be able to do much to tune that, unless you’ve configured the clients to ‘EnableSupersededEvaluation’ and they’re continuing to spend time evaluating superseded fixlets; you could return that to the default of Disabled.

But there are some optimizations that HCL can, and does, do on external sites like Patches for Windows that apply to External Content but not so much to your Custom Content. For example, we can configure fixlets to only evaluate once a day, or every six hours, or other intervals for fixlets we know will be expensive; a fixlet in the external site that takes 180 seconds but only evaluates once a day, is much better than something that takes 120 seconds but re-evaluates for every client loop.

Your Row 1 entry 187.204: Enterprise Security.281736901:Background Evaluation corresponds to a Fixlet in Patches for Windows, that is an update for Office 2010 released in year 2014. It might be worth checking those problematic machines to see if they still have Office 2010 installed, or registry traces that it used to be; this relevance evaluation runs fast on my machine, but I can see it might take much longer if the machine actually had Office 2010 installed (because there are a lot more files/registry values to check, that are short-cutted if the Office 2010 registry paths don’t exist).

18

It’s really difficult to say what a ‘good’ evaluation time is, it’s all about your expectations. I’d suggest using the link I had earlier to retrieve the client performance analysis, import it, and activate it, so you can start getting a baseline of how your machines are performing.

What you’d want to watch for, are outliers (some machine evaluating much more slowly than others); as well as changes over time (good evaluation times in December and much longer times now might indicate a new bad property/relevance added).

Once you know your expected eval times, you can tune the Console’s graying-out to match your times. I am aware of some large customers with hundreds of thousands of computers, with a ton of properties, that take two or three hours to complete an evaluation cycle. Those properties are important to them though, so they live with those times and accept that the actions they issue can take longer to respond (generally they issue patching actions several days ahead of their maintenance windows anyway).

1 Like
19

I also want to add the importance of https://help.hcltechsw.com/bigfix/10.0/platform/Platform/Config/c_real_time_av.html - I’m having a customer that complained about a Huge Average Evaluation Cycle (More than 10 hours) - We Started with a Clean Image with just BigFix Client, the average evaluation cycle, was up to 15 minutes… so we knew that there is something that caused the Client to work a lot harder. after he Installed Carbon Black… The average evaluation cycle sky rocketed to … 15 Hours
Of course at first , he said that he already excluded as we asked for, but we confirmed that he did not do that… :slight_smile:

Always make sure to make the baseline with a Clean Image and BigFix Client

3 Likes
20

@JasonWalker thank you for all that information, I am trying to digest this all to make sense of it, improve our environment and learn along the way.

Now for some Q’s…

^I found ID 6170 in Master Action Site and it was Baseline created back in 2019 when we first were getting started with bigfix. No longer needed it was removed, so that shouldn’t be an issue for any other besclient.

^I found the Analyses, I changed it from evaluating every hour to every day.

^I dont see EnableSupersededEvaluation on the settings of this particular client nor on other clients while spot checking. I was trying to find a fixlet\task that would tell me which computers have it but I am unable to. Is there such fixlet\task or how can I found out overall in the environment if this is set? Searched the forum for that and it seems like it’s not a good idea to this set, am I correct in this?

^So I was not finding this fixlet until I decided to click on Show Hidden Content and then Show Non-Relevant Content…that’s when I saw both ID’s 281736901 and 281736903.
There should be no machines with Office 2010 and both of these fixlets show that they’re applicable to 0 machines. Why would these come up in the logs as being evaluated? Then I searched for some more IDs from the log and there’s old adobe reader DC fixlet from 2020 but yet the machine in question has reader XI. Why would these old fixlets for Patches for Windows or Updates for Windows Applications being scanned by the besclient? These applications are not even on the machine I am currently troubleshooting with?

I know that’s a lot of Q’s but thanks for your time.