(imported topic written by Shlomi91)
Hi,
recently i noticed a great decrease in BES performance.
i ran the BESDiagnostics tool, i got these results:
Test Failed: BESMirrorRequest Plugin Reason: The GetHttp call failed with error err_SOCKET_RECEIVE Test Failed: Checking that TCP/IP is enabled on SQL server Reason: Failed to determine
if TCP/IP is enabled on SQL server
what do they mean?
again, BES is working, but very slow.
thanks,
Shlomi
(imported comment written by BenKus)
Hey Shlomi,
It looks like your BigFix Root Server is not properly accepting connections from relays/agents. That would explain that error + the fact that things are slow…
What version of the server are you running? If you look in Task Manager, how many threads is the BESRootServer.exe process using?
Ben
(imported comment written by Shlomi91)
Hi Ben,
we’re using version 7.1.1.315 and have 6-8 threads under the BESRootServer.exe process. while running the bes diagnostics, it comes up to 10 threads.
forgot to mention earlier, but also rebooting the server (twice) didnt help.
thanks,
Shlomi
(imported comment written by BenKus)
Is it only the BESMirrorRequest plugin that is failing? Or is it other plugins too?
Ben
(imported comment written by Shlomi91)
Hi Ben,
Other than the SQL test, its the only other test failing- all other plugins pass.
(imported comment written by BenKus)
If you go to the “wwwrootbes\bfmirror\downloads” folder of your BES Server folder, how many folders exist?
Ben
(imported comment written by Shlomi91)
Hi Ben,
you may be on to something… i have over 14,000 folder under “downloads”
if it is too much (and i suspect it is), how do i remove them?
thanks,
Shlomi
(imported comment written by BenKus)
Hi Shlomi,
If you look at the Deployment Health Checks Dashboard, this is the first check “Download Folders”. The check is keyed off a Fixlet in the BES Support site that contains a description of this issue and lets you run a tool to clean things up.
Let me know if that seems to address your issue.
Ben
(imported comment written by Shlomi91)
Hi Ben
i ran the fixlet, but it failed:
Completed prefetch BESDownloadsCleaner.exe sha1:0e1294af27a665406d21c9ed57ab745151f7dff6 size:1855555 http:
//software.bigfix.com/download/bes/util/BESDownloadsCleaner.exe Completed wait __Download\BESDownloadsCleaner.exe -N 10 -w -W __Download\downloadcleanerdata.txt Failed
continue
if
{exists file
"downloadcleanerdata.txt" of folder
"__Download"
} delete
"{value "wwwRootFolder
" of key "HKEY_LOCAL_MACHINE\SOFTWARE\BigFix\Enterprise Server
" of registry as string}\downloadcleanerdata.txt" move
"__Download\downloadcleanerdata.txt"
"{value "wwwRootFolder
" of key "HKEY_LOCAL_MACHINE\SOFTWARE\BigFix\Enterprise Server
" of registry as string}\downloadcleanerdata.txt"
can i run it manually?
Shlomi
(imported comment written by BenKus)
That is interesting…
I wonder if you have some sort of file system corruption that might be affecting things… You should run it manually and see what it has to say…
Ben
(imported comment written by Shlomi91)
Hi Ben,
i ran the tool manually.
at first, i ran it just as it appears in the fixlet - it gave me an error it could not find the path to write “downloadcleanerdata.txt” to.
then i manually entered the full path for downloadcleanerdata.txt, and it found and deleted 488 folders, which left me with about 13,500 folders - still too much.
then i ran the file with no arguments (all defaults), which ended up deleting additional ~700 folders, so i am now with about 12,900 folders - still too much.
any ideas?
thanks,
Shlomi
(imported comment written by Shlomi91)
Hi Ben,
looking at the “actions” tab, i see only 57 open actions.
(imported comment written by sthull)
This does sound like the behavior you can see with too many downloads folders, but that issue does not affect 7.1, so it must be something else. Do you have a lot of clients reporting directly to the BES Server?
You should probably open a support ticket because it sounds like some more detailed investigation will be needed.
Regards,
Steve
(imported comment written by sthull)
Let me clarify my last message a bit. Having an excessive number of download folders is known to have a performance impact on propagations, site gathers, and registrations; which is why it is included in the health check dashboard. But 7.1 introduces some changes that prevent this impact regardless of the number of download folders. So can you double check your RootServer.exe and make sure it is version 7.1.1.315? If it is 7.1, then the overall performance decrease you mentioned is not the issue we are aware of related to excessive download folders.
Still the number of download folders you have seems high, especially if you only have 57 open actions. The BESMirrorRequest plugin is the one that gives you the status of the downloads for each action, so this is surely causing the failure in BES Diagnostics. Are all/most of your 57 actions baselines or multi-action groups with 200+ subactions? Do you need to have all of these actions open?
Regards,
Steve
(imported comment written by Shlomi91)
Hi Steve,
the besRootServer.exe is indeed version 7.1.1.315. out of the 57 open actions (actually, its 68 now), i have 2 (baselines) with 144 subactions. most of the other actions have 20 actions or less.
i stopped most of the open actions, deleted the stopped actions and the expired actions, and ran the cleanup tool again - to no availe… it showed “deleted 0 folders”.
BUT - this morning i took another look, and imagine my surprise - i found only ~1200 folders under C:\Program Files\BigFix Enterprise\BES Server\wwwrootbes\bfmirror\downloads !
the health checks all passed (except for the last one - “Test Failed: Checking that TCP/IP is enabled on SQL server Reason: Failed to determine if TCP/IP is enabled on SQL server”), and performance has improved…
regarding the second failed health test (TCP/IP on SQL) - i am using remote SQL DB - should i be worried about it?
thanks,
Shlomi
(imported comment written by BenKus)
Hey Shlomi,
It sounds like you have cleared up your issue (although it seems a bit of a mystery what actually cleared up the folders).
Did your performance issues go away?
Ben
(imported comment written by Shlomi91)
Hi Ben,
seems pretty clear to me - stopped, expired (or both) actions still keep their folders in the downloads folder. once i stopped unneeded actions and cleared stopped / expired actions, (and probably waited for a while) it also cleared the folders.
regarding performance - let me put it this way: if previously it took me 10-12 minutes to issue a single fixlet to a single computer, it now takes me ~3.5 minutes. does that seem ok?
one more thing i mentioned before - the TCP/IP on SQL test which failed - should i be worried about it if i am using a remote SQL DB?
thanks,
Shlomi
(imported comment written by sthull)
Hi Shlomi,
That does make sense. You do need to delete old stopped and expired actions periodically, and that process will also remove the associated download folders.
I have seen the TCP/IP enabled in SQL test fail erroneously in certain environments. If you have a remote database, TCP/IP must be enabled in order for your BES consoles and services to communicate with it. Since those are working correctly, you can ignore this failure.
Regards,
Steve
(imported comment written by BenKus)
I will double check with the developers… but I believe that stopping an action will clean up the download folders and expired actions will periodically be cleared up… It seems that this wasn’t working quite right on your system though…
Ben