BESClient consuming most I/O on WInXP

(imported topic written by crichmon91)

Of all the processes on my WinXP machine, BESClient is causing the most I/O of any process.

This data is from ProcessManager and sorting I/O Reads, I/O Writes, and I/O Other columns.

The next highest task is the video driver. It seems that BESClient is hogging some resource

such that even character echo gets delayed up to 2-3 seconds. Once the BESClient process

gets killed, normal response returns. What is BESClient doing that causes so much I/O

traffic? Can this be tuned? Its

really

annoying, to say the least.

(imported comment written by BenKus)

Hi crichmon,

The BigFix Agent very slowly investigates various parts of the system to do continuous checks to make sure everything is up-to-date and configured properly. It tends to use 0-1% of the overall system resources and absolutely is not supposed to cause any system problems. The reason you see the agent have many I/O Read/Writes is because the agent is always on and so the numbers accumulate over time, but it doesn’t mean that the agent is affecting system performance.

Have you been able to isolate that the BigFix Agent is indeed the cause of the computer being slow and the typing being slow? The simple way to test this is to try to run the system for a while with the agent disabled. If the agent running appears to be correlated with the slowness, then it is certainly an issue and I recommend that you start up a support case so we can look at it in more detail.

Here is what to expect from the BigFix agent in terms of resource usage:

http://support.bigfix.com/cgi-bin/kbdirect.pl?id=43

Ben

(imported comment written by ErnieF91)

Bummer, I seem to be having the same problem with a new Vista machine running 7.0.1 Client. Opening email and typing is delayed by 2-3 seconds and it goes back to normal once I stop the BESClient.

CPU usage is less that 2%, but Disk I/O is huge. Using Task Manager, I added columns for I/O Read, I/O Writes and I/O Other. BESClient is the largest consumer of resources in the list, with I/O Other the largest of the three. The numbers are incrementing by several hundred on every refresh more that anything else running.

Is there some way to see what it’s doing and maybe find a fix for it.

(imported comment written by SystemAdmin)

Hey guys,

We probably want to do some more advanced troubleshooting with you through tech support. Specifically, we would like to gather the BES Client Diagnostics and review the EMsg logging at a level of 10,000. The EMsg log will tell us exactly what the client is doing and how long its taking to do it.

http://support.bigfix.com/cgi-bin/kbdirect.pl?id=186

I don’t think that the Disk I/O characteristics that you described are the root of the problem. The BES Client has the same characteristics you mentioned on every computer but this doesn’t lead to any problems. Further, the agent is not event driven so it seems weird that you have issues when doing specific activities (ie, the client doesn’t change what its doing when you open a document and type into it). We certainly want to get to the root of the problem and fix it for you but the disk I/O is probably a red herring.

(imported comment written by crichmon91)

Pardon my ignorance, but where would I find this console thing? I’m betting we got a partial install from IT that didn’t include this. Assuming I can get the registry hacks in place, where do the logs go, and how would I get them to you? BESClient is version 6.0.21.5. The only other executable I have is WinBESAgentUI.exe, which won’t start interactively.

Chris

(imported comment written by crichmon91)

As another follow-up, one of the other big consumers of I/O is McAfee virus scan. I’m pretty sure its set to scan all file reads and writes. In the performance tuning tips on the BigFix site, it makes mention that there are lots of small files that are read or written when BESClient runs. I can’t kill McAfee, so I can’t tell if the combination is causing the impact. The other thing I found was in the BESClient logs: C:\Program Files\BigFix Enterprise\BES Client__BESData\BES Support\1Performance.fxf warnings about Windows file indexing being enabled, BES Server/Relays are Set to use Compression, The listed BES Servers/Relays have the BES folder compressed, BES Server Does Not Meet Recommended Requirements, etc. This file is dated 4/4/2006, so it may or may not reflect current status. Are there any other logs I can get some clues from?

(imported comment written by BenKus)

Hi Chris,

The BES Server/Relays can suffer performance issues from some of the AV scanners if the components are under high load. This does not tend to be a problem for the agents.

How many agents are having this problem? does this issue occur often or is it sporadic?

It is possible you can have an issue like this that is related to some custom properties that are too aggressive, but as Tyler mentioned, we can do some investigations to try to find this…

Also, if you temporarily disable the McAfee service, does this issue seem to go away?

Ben

(imported comment written by Shlomi91)

Hi,

i have another issue on the Bigfix client:

Disk I/O - the numbers are indeed high, but do not affect system performance.

McAfee Antivirus - i excluded the C:\Program Files\BigFix Enterprise\BES Client__BESData folder from scans in ePO, and that solved this problem.

McAfee HIPS, however, is a whole new story:

i am currently testing HIPS deployment in my organization, and i found that when both BESClient and McAfee HIPS are working, the computer has terrible response time.

disabling any of these (wither HIPS or BESClient service) will release the computer’s resources.

looking in McAfee KB i found nothing.

Chris / Ernie, do you also have McAfee HIPS installed?

has anyone found a solution for this yet?

thanks,

Shlomi

(imported comment written by BenKus)

Hi Shlomi,

One of our other customers reported that McAfee HIPS was causing an issue similar to the one that you described. The status update I heard from the customer was that McAfee was looking into it now and we have made ourselves available to McAfee to help answer any questions that they might have.

I will update you when I hear more info.

Ben

(imported comment written by crichmon91)

RE: McAfee HIPS

As a matter of fact, we had this pushed to our machines in the last few months and have

no way to disable it for testing. I’m very interested to hear what comes of this.

Thx, Chris

(imported comment written by BenKus)

Hey folks,

We are still working with McAfee on the issue and they have reproduced it on one of their versions.

Which version are you guys using 6.1 or 7.0?

More info to come soon hopefully…

Ben

(imported comment written by crichmon91)

It appears we are on versions:

3.6.0.453 for ePolicy Orchestrator Agent, AutoUpdate, Prod Coverage Rpt

6.0.6.116 for HIP

8.5.0.781 for VS

8.5.0.163 for AntiSpyware

5200.2160 for ScanEngine, 5200.0000 for definitions

Thx, Chris

(imported comment written by Shlomi91)

Hi,

Our versions:

ePO - 3.6.1.202

VirusScan enterprise + AntiSpyware enterprise - 8.5.0i

i think these are irrelevant, because we used them for about a year before starting PoC of HIPS.

our HIPS version is 7.0.0, build 688, security content version 1756

BTW - i opened a ticket in McAfee, see what they have to say…

Shlomi

(imported comment written by Shlomi91)

small update -

HIPS is made of 3 parts: IPS, firewall and app blocking.

i can disable each of these elements seperately, and i found that the element causing the problem is ONLY IPS…

after disabling the IPS part, resources were released…

(imported comment written by BenKus)

Quick update on this:

McAfee has an open case with one of our customers and is able to reproduce the issue. They are looking into it and hopefully will have a status soon.

Ben

(imported comment written by BenKus)

We heard back from McAfee and they said that their 6.1 agent did not work well with applications that did a lot of registry access. McAfee 7.0 HIPS agent apparently resolves this issue.

Ben

(imported comment written by Shlomi91)

hi ben,

sorry to dissapoint you (or mcafee), but i have HIPS 7 fresh install, no upgrade) and still experience these issues.

i still have have an open case with them, and will update if any progress is made.

Shlomi

(imported comment written by Efairrell91)

Hey everyone,

I am with the customer Ben is referring to. This problem was supposed to have been fixed with HIPS version 7, however, this is not the case. If you have HIPS 6.1 or 7.0 and BESClient 7.0, then this will cause you a problem. There is no problem with HIPS and BESClient 6 I have tested a work around for the problem and it seems to be working. Here’s what we did:

First, in the Policy Catalog, we added BigFix to our Trusted Applications list. We then went into the IPS Rules and put in an exclusion in our Application Protection Rules for Bigfix. Finally, we created an exception rule in our IPS Rules for the BESClient for rule 1002.

Essentially, the problem is that McAfee does not allow BigFix to enumerate the registry for a number of HIPS related keys. Rule 1002 is the signature that protects the registry from viewing. Granting BigFix an exception to this signature seems to do well as a workaround.

Eric

(imported comment written by Shlomi91)

Yup, thats what they told me. working fine so far…