Decrease BigFix client registration time on Linux

Usage and Config Platform
1

I’ve seen this thread https://forum.bigfix.com/t/forcing-new-clients-to-check-in-faster-client-refresh-offer-cache and we are setting the BES Client CPU usage to 20% (and even 90% during testing) but it still takes a RHEL 7.4 Client running v9.5.4.38 15minutes to complete evaluating all content. How can I speed that up? I know it’s tough when the amount of content to evaluate is a wildcard, but does anyone have any other suggestions?

The subscribed external sites are
Patches for RHEL7
Linux RPM Patching
Patching Support
BES Support

The Master Operator Site and 4 custom sites, along with ~200 operators to evaluate.

1 Like
2

Are you talking about on initial install of an agent? There is a setting that when the agent starts, it will run at normal speed until it goes through the content once

_BESClient_Resource_StartupNormalSpeed

    Type: Boolean 
    Version: 9.0 
    Platform: All 
    Default: 0 
    Requires Client Restart: YES 
    Description: Set to 1 to process the first evaluation loop at the WorkNormal/SleepNormal settings. Sets itself to 0 after that loop completes.
3

Yes, on the initial install of the agent. So even with WorkIdle and SleepIdle set to 20% via the besclientsettings file on install, the first evaluation loop is still running at the default 2% CPU usage? (I’m seeing CPU usage of 50% during first evaluation).

4

If you set Work/SleepIdle to 20% correctly during install you will see some of the operations go to 20%, but as a subscription action executes (which is required for subscribing to the other sites) the “Normal” speed which defaults to 50% will be triggered. This will drop after the action though and the content evaluation would drop to the “Idle” speeds

5

So if i’m correctly setting the WorkIdle/SleepIdel to 20%, there is no need to set _BESClient_Resource_StartupNormalSpeed to “1”?

6

From initial install of the agent to it showing up in the console for a master operator should be between 30 seconds and 120 seconds if using client settings to speed up the initial registration / provisioning, at least in my experience. It will always take longer for a non-master operator to see the system in the console than a master operator, but even that time can be greatly reduced, but that depends on the complexity of the logic that causes an operator to get the computer assignment.

For Linux hosts, they do not use the clientsettings.cfg file like Windows/Mac, so that doesn’t work.

I have a script that solves this problem: tools/bash/install_bigfix.sh at master · jgstew/tools · GitHub

It creates a clientsettings.cfg if one does not exist, but then it also populates /var/opt/BESClient/besclient.config with the settings from it on Linux using awk.

The goal of the script is to greatly simplify the process of installing bigfix on ANY linux host that has an internet connection.

I have an example clientsettings.cfg file here: An example clientsettings.cfg to set the default BigFix client settings at install time. Only works with the Windows EXE installer or the OS X installer. For linux, see here: https://github.com/jgstew/tools/blob/master/bash/install_bigfix.sh · GitHub

I try to populate it with settings that I think are a good idea as defaults in general, as well as those that would speed up initial provisioning. The workidle setting should probably be lowered after initial provisioning to lower idle CPU usage, but there is a power save mode setting that actually makes it possible to have a higher work idle without impacting the CPU usage as negatively. If an aggressive command polling setting is used, it should also be set to something less often after initial provisioning.

The settings that will affect initial provisioning / registration the most are:

_BESClient_Resource_StartupNormalSpeed=1
_BESClient_Resource_WorkIdle=20
_BESClient_Download_RetryMinutes=1
_BESClient_Download_CheckAvailabilitySeconds=120
_BESClient_Resource_AccelerateForPendingMessage=1
_BESClient_Comm_CommandPollEnable=1
_BESClient_Comm_CommandPollIntervalSeconds=600

This also helps: Delay BigFix site subscription to speed up client provisioning

In my opinion, delaying a client subscribing to patching sites makes sense because the highest priority when BigFix is first installed is processing all of the operator stuff so that the system shows up in the console for a particular operator as fast as possible. The next most important thing is for the system to start processing policy actions for configuration and software installation. It is only some time later that patches really matter. It is usually a given that a newly installed client is going to find tons of relevant patches, but by delaying that processing by an hour or more you can greatly speed up the other initial work that I feel is more important to accomplish as quickly as possible in the first few hours after BigFix client installation.

I implemented this approach on my own testing root servers, as well as those at a previous employer and definitely saw improvements.

At a previous employer, we had ~250 custom sites and ~350 operators. If using the clientsettings.cfg our initial registration / provisioning time was much shorter than you are experiencing.

2 Likes
7

Thanks for all that info. But does _BESClient_Resource_StartupNormalSpeed=1 need to be set if WorkIdle/SleepIdle is bumped up as part of the besclient.config?

8

It’s effect will be less or nil if you bump up WorkIdle a lot. I bump it up some, but the normal speed is still faster so it still has a benefit. It won’t hurt anything to add it as it gets set back to 0 after a few evaluation loops.

9

hmm, if the effects will be less or nill if I bump up the WorkIdle/SleepIdle (which I am), then i guess i’m back to square one to figure out why initial evaluation is taking 15min.

10

if you want to send me an initial client log privately I can review it to see if I spot anything odd. You should string replace the root server/relay/similar details.

11

One thing I’d suggest is to upgrade the clients to 9.5.7. There were performance issues with the “lines of file” family of inspectors in all of the 9.5.x versions up to 9.5.7, which are heavily used in the Linux content.

1 Like
12

If that is the problem, then delaying the client’s subscription to the patching sites would help because it would delay the client evaluation of a lot of content that uses the lines of file as well, though upgrading is the better long term solution.

13

Test results: Set WorkIdle/SleepIdle and WorkNormal/SleepNormal to 90%
Tested with client at v9.5.4 and v9.5.8 (both resulted with about the same results)
10min to install and complete evaluation. Better then the 15min from before but still seems slow.

We only have about 200 Console users but there are over 1000 opsites to evaluate. Can those old opsites be purged and maybe that would reduce evaluation time?

[root@xxxx Logs]# grep -i “Assign and Revoke Management Rights For __op” 20180112.log | wc
1314 13797 106990

14

That is likely part of the problem, but even with all of those I think it shouldn’t take this long, but I’m not certain.

There might be a way to clear the old ones up, but I think it is not possible or difficult. Good question for @AlanM or @Aram .

15

I’ve been working with @cstoneba a bit on this. After looking over 2 client logs captured just after client install and initial provisioning, there is definitely 10 minutes or more that is taken doing all of the management rights actions for all of the old and current opsites and similar. This is even with the WorkIdle set to the max, it would be much worse at the default WorkIdle setting.

I don’t think 10 minutes is terrible, but it really should be a lot faster if only the “real” operators had to be evaluated. It is hard to get around this problem without a way to reissue all of the management rights actions for just those that are current, but I think I might have figured out a possible way to get around this:

The idea is to pre-populate all of the possible opsite assignments with a false value set far in the past ( Thu, 01 Jan 1970 05:23:11 -0800 ) so that any rights assignments that would take away an operators rights would be skipped and only those that provide management rights would be run and change the pre-populated value.

This should work because the relevance for a client_role is:

(not exists setting "__Client_Role_###" of client) OR ((not exists effective date of it) OR ((effective date of it <= universal time "Fri, 30 Oct 2015 11:55:47 +0000") AND ((not exists value of it) OR (value of it does not equal (( if ( false ) then "1" else "0" ) as string))))) of setting "__Client_Role_###" of client

And the relevance for an operator is:

(not exists setting "__Group___AdminBy___op_###" of client) OR ((not exists effective date of it) OR ((effective date of it <= universal time "Fri, 30 Oct 2015 16:55:47 +0000") AND ((not exists value of it) OR (value of it does not equal (( exists (setting "__Client_Role_####" of client) whose (value of it = "1") ) as string))))) of setting "__Group___AdminBy___op_###" of client

An example /var/opt/BESClient/besclient.config on linux looks like this:

[Software\BigFix\EnterpriseClient\Settings\Client\__Client_Role_####]
value                          = 0
effective date                 = Mon,%2030%20Oct%202017%2016:57:45%20-0700

[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy___op_###]
value                          = False
effective date                 = Mon,%2030%20Oct%202017%2017:08:46%20-0700

[Software\BigFix\EnterpriseClient\Settings\Administrators\__op_###]
effective date                 = Mon,%2030%20Oct%202017%2017:08:46%20-0700

The bummer is that this can’t be done with a clientsettings.cfg at install time because using one does not set an effective date and an effective date seems to be required, otherwise these would not be skipped. ( In this case, I’m not referring to linux since a clientsettings.cfg won’t work there anyway )

This should be the set of operators to set:

masthead operator names of bes users whose(not master flag of it)

This would be the format for in the besclient.config 

("[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy_" & it & "]%0d%0avalue                          = False%0d%0aeffective date                 = Thu,%252001%2520Jan%25201970%252005:23:11%2520-0800") of masthead operator names of bes users whose(not master flag of it)
2 Likes
16

For UNIX, the besclient.config cannot be populated pre-install to include these opsite settings?

17

It can be, just not using a clientsettings.cfg.

I have an awk command to turn a clientsettings.cfg into a valid besclient.config to get around this limitation, but my current command does not set the effective date, so it would need adapted to address that. The other option would be to take a generated besclient.config and put it into place before installing BigFix.

18

I tested pre-populating the besclient.config file with the output of opsite query above. After the client installed, evaluation times were still about 10 minutes but I noticed in the log file that opsites were still being set to false yet they were not in the output of my opsite query:

Example from log file:

   Command succeeded setting "__Group___AdminBy___op_376"="False" on "Wed, 30 Aug 2017 15:10:33 +0000" for client (action:946923)
   Command succeeded administrator delete "__op_376" on "Wed, 30 Aug 2017 15:10:33 +0000" 
   Command succeeded (evaluated false) continue if { value of setting "__Group___AdminBy___op_376" of client = "True" } (action:946923)
   Not Relevant - Assign and Revoke Management Rights For __op_376 (fixlet:946923)

Output from the opsite query (with no op_376)

[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy___op_374] value = False effective date = Thu,%2001%20Jan%201970%2005:23:11%20-0800
[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy___op_385] value = False effective date = Thu,%2001%20Jan%201970%2005:23:11%20-0800
[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy___op_403] value = False effective date = Thu,%2001%20Jan%201970%2005:23:11%20-0800

Why wouldn’t the query contain all the opsites the client sets during initial evaluation?
log file attached: ops comparison.xlsx.pdf (21.4 KB)

1 Like
19

I’d bet the query doesn’t contain the output from operators that have been deleted, which I didn’t really even consider until just now, but seems rather obvious.

I wonder if your pre-population caused it to skip those that were in the query but were not being set to True.

You could add custom operators that are not returned by the query like this:

("[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy_" & it & "]%0d%0avalue                          = False%0d%0aeffective date                 = Thu,%252001%2520Jan%25201970%252005:23:11%2520-0800") of ( "__op_375"; "__op_376"; masthead operator names of bes users whose(not master flag of it) )

I also wonder if they are sequential, if it would be sufficient to return from the query the maximum operator number then pre-populate everything from __op_100 to the max… though it seems like you have an __op_3

The max operator should be:

maxima of (it as integer) of following texts of firsts "__op_" of masthead operator names of bes users whose(not master flag of it)

This is all the operators from the current minimum to the current maximum:

("__op_" & it as string) of ( integers in ( ( minima of (it as integer) of following texts of firsts "__op_" of masthead operator names of bes users whose(not master flag of it) ) ,it) ) of maxima of (it as integer) of following texts of firsts "__op_" of masthead operator names of bes users whose(not master flag of it)

This may miss a few before the current minimum, but seems like it is one of the better options to get most of those required without definitely getting extras.

This would be it all together:

("[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy_" & it & "]%0d%0avalue                          = False%0d%0aeffective date                 = Thu,%252001%2520Jan%25201970%252005:23:11%2520-0800") of ("__op_" & it as string) of ( integers in ( ( minima of (it as integer) of following texts of firsts "__op_" of masthead operator names of bes users whose(not master flag of it) ) ,it) ) of maxima of (it as integer) of following texts of firsts "__op_" of masthead operator names of bes users whose(not master flag of it)

If you know the actual minimum value, then that could be substituted:

("[Software\BigFix\EnterpriseClient\Settings\Client\__Group___AdminBy_" & it & "]%0d%0avalue                          = False%0d%0aeffective date                 = Thu,%252001%2520Jan%25201970%252005:23:11%2520-0800") of ("__op_" & it as string) of ( integers in (3,it) ) of maxima of (it as integer) of following texts of firsts "__op_" of masthead operator names of bes users whose(not master flag of it)

Also, thanks for giving this a try. I can’t say for certain that this will work, but the relevance seems like it should.

@cstoneba I made many updates above ^

20

one thing i noticed was the output of the WR query had 3 underscores infront of the “op” but the BESClient log file only had 2:

client log: __op_182
WR query; ___op_1002

What that just a typo in the query?

Looking at the opsites, the client log has about 640 entries that are almost consecutive. The WR query results has about 200, similar to the number of active user accounts in the Console, so your thinking sounds rights.

I suppose I could use the opsites listed in the log file as the data to pre-populate in the config file?