I have a deployment that is about 2500 clients in size. I’m having the following issue as of now.
I can see a lot of files in my FillDB buffer directory and it is clearing things only every 15-20 mins or so. Which means I can see temp files that were created about 15 minutes ago in the buffer directory. They do clear, but very slowly and then again, new files are coming in by that time. The DB and app is both running on the same machine, same partition which is configured via RAID5.
Any idea why this could be happening? My application is 7.2.5.22 and SQL Server is 2008 64-bit.
I checked the activity monitor in SQL and saw that every now and then the update task for BFEnterprise by FillDB is in status SUSPENDED.
I don’t have any other application installed on the server that might use SQL.
I’m not a FillDB expert, but we’ve seen Database/FillDB performance issues similar to this. In some cases, it can be disk performance. If it’s causing tasks/actions/fixlets to process slowly, I would open a case with support immediately. A few things you can check:
What is the disk queue length on the server (disk IO is a common bottleneck)?
Do you see any errors in the BES relay logs on the root server?
How much RAM do you have, and how much is in use?
Are there any errors/warnings in the BES Health Check dashboard?
Are clients still checking in and processing actions?
Another thing to look at is your virus scanning settings or any indexing that might be affecting the folder and files.
We have set the FillDB.exe as a low risk process in AV to avoid conflicts and it really helps in performance. Also excluding the folder could be considered, but we have been happy with just tuning AV at the process level.
As a general rule we don’t let the index anything unless we specifically, and it would be a real resource waste to index anything in that folder.
For a deployment of only 2500 computers, you should never see a large backup in the FillDB folder. Very large deployments (like 100,000+ agents) will have some expected backup in certain situations, but most small/mid deployments typically shouldn’t see this issue. It sounds like boyd/mrfixit are right and there might either be a harddisk issue or maybe some sort of software conflict (like AV).
I would suspect your harddrive performance (for the database) is very low right now for some reason and this is likely where you probably should start your investigation.
In the relay log at the root server, there are lots of repetitions for the same error which is unable to post results, error overload buffer directory.
The server has 8GB of RAM out of which currently about 4GB is in use. SQL alone is occupying about 2GB of RAM. Not sure what is causing it.
The error in health check dashboard is the one related to overload buffer directory which says the server has a lot of buffer waiting for input into FillDB.
Clients are still getting actions and processing them, but their status is not updated for a long time. Even after the execution finishes, the status remains as not reported.
The problem just started recently. The license in this place had expired and the new license was put in 1 day late. It is from then that the problem started.
Depending on the size of your database, it’s not surprising that SQL is using 2GB of RAM. That’s actually a good thing, from an IO perspective. I don’t recall ever seeing that error, but does seem to indicate that your FillDB directory is backing up. You should definitely open a support case with BigFix. They may want you to turn on the FillDB performance log: http://forum.bigfix.com/viewtopic.php?id=431 .
If you’re not running AV software, as Gary suggested, it sounds like a disk performance issue to me, especially since FillDB is still processing results. I’d look at those disk response times and disk queue lengths. What type of disks are you running (SAS or SATA)? Did you recently add any analysis properties that could return large amounts of data?