Clients not checking in regularly

Usage and Config Platform
1

Hello,

I am managing part of an installation of BigFix where we have many teams in our organization. One of our scientific teams is in the process of adopting BigFix for automation to replace an existing custom solution.

I am reaching out to the community after having tried to resolve the issue on my own.

The issue we’re seeing right now is that their clients are checking in about once per day. Something to note is that they are all on a NAT’ed network. I am intermittently allowed to use two of their computers for testing which I believe I’ve correctly switched to using command polling but I am still seeing the same behavior. I am in the process of deploying a test machine on their network to see if I can reproduce the behavior.

We’ve had discussions with their team regarding setting up a relay on their network vs using command polling and they prefer the command polling option.

Can you make any recommendations to get these clients to check in regularly?

Looking forward to hearing from anyone,
– Tareq

2

Is it a problem with reporting up, or with gathering new content/actions?

You can check by whether the “Last Report Time” for the computers are updating in the Console. It should, by default, update roughly every 5 minutes.

Command Polling should help with gathering content/actions, but not with report failures, and we’d troubleshoot with in two different paths.

If Reporting is not working, we’d check the BES Client logs for “Report posted” messages, and check the client evaluation loop times to see whether there is some long-running property that is blocking the client evaluation.

If Gathering is not working, we’d look to relay selection and command polling.

1 Like
3

I should add that enabling Command Polling on the client requires two client settings, described at IBM Documentation

_BESClient_Comm_CommandPollEnable

_BESClient_Comm_CommandPollIntervalSeconds

I’d recommend going no shorter than 900 seconds (15 minutes) for command polling, and less frequently than that for larger deployments.

If the relay is servicing a large number of clients, it may be refusing the connections as well based on

_BESRelay_HTTPServer_MaxConnections

You may need to increase that value to allow the relay to consume more connections from clients.

If you can get your deployments up to version 9.5.11, I’d suggest looking at Persistent Connections and Peernest to avoid command polling and reduce bandwidth use across their firewall/NAT device.

1 Like
4

Hi Jason,

The problem sounds like the first item you described. The “Last Report Time” property is not updating for many hours.

How would I check the client evaluation loop times for long-running properties?

Thank you for your time,
– Tareq

5

Based on the information you provided, I’m looking into the following resources:
https://www-01.ibm.com/support/docview.wss?uid=swg27049171&aid=1
https://www-01.ibm.com/support/docview.wss?uid=swg21505873#clientdebuglogging
https://www-01.ibm.com/support/docview.wss?uid=swg21505873#clientusageprofiler

Thank you

6

Hello,

After running the client profiler I have some logs. I’m not sure how to read the logs.

usageprofiler.txt.0018 from a machine that hasn’t checked in since 4/18/2019, 3:14:34 AM

Is the log saying that it took over two hours to evaluate relevance and it may still be going?

Is the top consumer “actionsite.6891:Evaluate Property 7” taking over 5000 seconds to evaluate?

How do I identify the top consumer in the BigFix console?

Is this all covered by the “IBM Security Master Skills University”?

Happy for any information anyone can provide,
– Tareq

7

I forgot to attach the logs: https://drive.google.com/drive/folders/1y1tx46p81DahP8YSsZEoYJ-BJjs7pJoe?usp=sharing

1 Like
8

Hello,

I’ve generated a client log “besclientdebug.log” and wrote a script to parse the timestamps of the results. Below are the results where the timestamp difference are greater than 1 hour. Am I correct in understanding that the issue is that the clients are spending all of their time evaluating properties?

Format: (time elapsed in minutes between records) (previous record) (next record)

Loading file: 'besclientdebug.log'
162.76666666666668: Fri, 19 Apr 2019 09:01:17 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Fri, 19 Apr 2019 11:44:03 -0700 ReportTimer EvaluateProperties 9814496105 microseconds elapsed
165.0: Fri, 19 Apr 2019 12:57:19 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Fri, 19 Apr 2019 15:42:19 -0700 ReportTimer EvaluateProperties 9983226737 microseconds elapsed
164.63333333333333: Fri, 19 Apr 2019 17:30:39 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Fri, 19 Apr 2019 20:15:17 -0700 ReportTimer EvaluateProperties 9927589691 microseconds elapsed
164.7: Fri, 19 Apr 2019 21:29:00 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sat, 20 Apr 2019 00:13:42 -0700 ReportTimer EvaluateProperties 9963337488 microseconds elapsed
163.55: Sat, 20 Apr 2019 01:48:11 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sat, 20 Apr 2019 04:31:44 -0700 ReportTimer EvaluateProperties 9867358879 microseconds elapsed
165.21666666666667: Sat, 20 Apr 2019 05:44:19 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sat, 20 Apr 2019 08:29:32 -0700 ReportTimer EvaluateProperties 9959545570 microseconds elapsed
164.16666666666666: Sat, 20 Apr 2019 10:02:44 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sat, 20 Apr 2019 12:46:54 -0700 ReportTimer EvaluateProperties 9897854129 microseconds elapsed
164.3: Sat, 20 Apr 2019 14:00:16 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sat, 20 Apr 2019 16:44:34 -0700 ReportTimer EvaluateProperties 9941993691 microseconds elapsed
165.35: Sat, 20 Apr 2019 18:32:00 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sat, 20 Apr 2019 21:17:21 -0700 ReportTimer EvaluateProperties 10007246890 microseconds elapsed
159.78333333333333: Sat, 20 Apr 2019 22:29:38 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 01:09:25 -0700 ReportTimer EvaluateProperties 9640408716 microseconds elapsed
163.73333333333332: Sun, 21 Apr 2019 02:43:26 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 05:27:10 -0700 ReportTimer EvaluateProperties 9880933209 microseconds elapsed
165.0: Sun, 21 Apr 2019 06:40:41 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 09:25:41 -0700 ReportTimer EvaluateProperties 9991174747 microseconds elapsed
165.3: Sun, 21 Apr 2019 10:59:22 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 13:44:40 -0700 ReportTimer EvaluateProperties 9979017525 microseconds elapsed
166.7: Sun, 21 Apr 2019 13:47:20 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 16:34:02 -0700 ReportTimer EvaluateProperties 10092645804 microseconds elapsed
165.23333333333332: Sun, 21 Apr 2019 16:36:02 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 19:21:16 -0700 ReportTimer EvaluateProperties 9968763257 microseconds elapsed
165.78333333333333: Sun, 21 Apr 2019 20:34:54 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Sun, 21 Apr 2019 23:20:41 -0700 ReportTimer EvaluateProperties 10036247764 microseconds elapsed
165.1: Sun, 21 Apr 2019 23:22:32 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Mon, 22 Apr 2019 02:07:38 -0700 ReportTimer EvaluateProperties 9954338987 microseconds elapsed
164.26666666666668: Mon, 22 Apr 2019 02:11:11 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Mon, 22 Apr 2019 04:55:27 -0700 ReportTimer EvaluateProperties 9932830451 microseconds elapsed
166.41666666666666: Mon, 22 Apr 2019 06:44:11 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Mon, 22 Apr 2019 09:30:36 -0700 ReportTimer EvaluateProperties 10074117009 microseconds elapsed
165.13333333333333: Mon, 22 Apr 2019 10:43:29 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Mon, 22 Apr 2019 13:28:37 -0700 ReportTimer EvaluateProperties 9967770065 microseconds elapsed
166.05: Mon, 22 Apr 2019 15:01:55 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Mon, 22 Apr 2019 17:47:58 -0700 ReportTimer EvaluateProperties 10012103134 microseconds elapsed
166.68333333333334: Mon, 22 Apr 2019 19:00:58 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Mon, 22 Apr 2019 21:47:39 -0700 ReportTimer EvaluateProperties 10092547758 microseconds elapsed
166.06666666666666: Mon, 22 Apr 2019 23:33:55 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Tue, 23 Apr 2019 02:19:59 -0700 ReportTimer EvaluateProperties 10015526227 microseconds elapsed
164.68333333333334: Tue, 23 Apr 2019 03:53:33 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Tue, 23 Apr 2019 06:38:14 -0700 ReportTimer EvaluateProperties 9932758340 microseconds elapsed
166.18333333333334: Tue, 23 Apr 2019 07:52:00 -0700 DebugMessage Inside IsAnyRestartPending() checking HKLM\System\CurrentControlSet\Control\Session ManagerPendingFileRenameOperations Tue, 23 Apr 2019 10:38:11 -0700 ReportTimer EvaluateProperties 10063845342 microseconds elapsed
9

I’ll try to get back to this when I’m at a computer…but the easiest thing to do is to find & download one of @strawgate’s C3 analyses from bigfix.me, wheich reports the average client eval loop and the top 10 worst-offending properties on the client.

10

Hi Jason,

Thanks for your help.

I tried to find the analyses but this is the closest one I found: https://bigfix.me/analysis/details/2994765

11

That’s close, but I was referring to this one
https://bigfix.me/analysis/details/2998424

Check the notes for the related Task to enable fixlet evaluation tracking on the client.

1 Like
12

Have you confirmed port 52311 udp is allowed through all the firewalls between the Bigfix server, relays and endpoints?

13

Thanks all. I was able to narrow it down using the following:

https://www-01.ibm.com/support/docview.wss?uid=swg21669200
Session relevance via /webreports?page=QNA
names of bes properties whose ((item 1 of it is AnalysisID and item 2 of it is PropertyID) of id of it)

1 Like