All Endpoints stop updating for a specific time of the day

[washnware]

Hi All,

All Endpoints stop updating for a specific time of the day. 2 hours everyday at same time. No Backup was running or security scan. DB connection was working fine it is just no updating the DB. Is there some logs from Bigfix to check where is the choke point during that time? Endpoint - relay -Main server connection/reporting was fine when checking logs.

[Aram]

Might this happen to be around midnight per the Root Server’s timezone?

I’d recommend turning on the FillDB Performance log perhaps 3-4 hours before and after the specific time of day in question. Do be careful with the log however as it can grow in size relatively quickly.

[Meydey]

We had an issue like that where clients would stop checking in around 6 am each day. Turns out we share qos with over-the-wire daily backups. One of our PCI teams went from incremental to full backups at 6 am which basically pushed out all other traffic, with UDP being the lowest priority. Everything picks up again like normal once the daily backups complete.

[washnware]

Hi Aram,

Below is the log but only starts 1:03 AM our endpoints stop reporting at midnight.

Mon, 03 Feb 2020 01:03:02 -0500 – 7456 – Unexpected exception: Database Error: [Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot continue the execution because the session is in the kill state. (S1000: 596)
[Microsoft][SQL Server Native Client 11.0]Unspecified error occurred on SQL Server. Connection may have been terminated by the server. (S1000: 0)
Mon, 03 Feb 2020 01:03:12 -0500 – 7456 – Signature Algorithms: sha256, sha1
Mon, 03 Feb 2020 01:03:12 -0500 – 7456 – Download Algorithms: sha256, sha1
Mon, 03 Feb 2020 01:03:27 -0500 – 660 – UploadDirectoryMonitor::MaybeNewFileVersion: ODBCException processing file “D:\Program Files (x86)\BigFix Enterprise\BES Server\UploadManagerData\BufferDir\sha1\9\8775209\Index.txt”: Database Error: [Microsoft][SQL Server Native Client 11.0]TCP Provider: An existing connection was forcibly closed by the remote host.
(08S01: 10,054)
[Microsoft][SQL Server Native Client 11.0]Communication link failure (08S01: 10,054)
Mon, 03 Feb 2020 01:03:27 -0500 – 660 – UploadDirectoryMonitor::MaybeNewFileVersion: ODBCException processing file “D:\Program Files (x86)\BigFix Enterprise\BES Server\UploadManagerData\BufferDir\sha1\9\8775209\citsearch_0_8779_cit.xml.bz2”: Database Error: [Microsoft][SQL Server Native Client 11.0]Communication link failure (08S01: 0)

[washnware]

Hi Meydey,

I think this is a good area to check. Thanks for sharing.

[Aram]

This doesn’t appear to be the Performance log, but in any case, if this is occurring consistently at midnight, I would suggest it is more than likely related to FillDB’s Historical Counts process. Please see https://bigfix-wiki.hcltechsw.com/wikis/home?lang=en-us#!/wiki/BigFix%20Wiki/page/Troubleshooting%20FillDB%20Operations for more information, but essentially, once a day, FillDB performs some work to count the total number of relevant Fixlets by severity (and total computer count) for use in some specific Web Reports. You can disable this process with the following setting:

HKLM\Software\BigFix\Enterprise Server\FillDB
"UpdateHistoricalCounts" [DWORD]=0

[washnware]

HI Aram,

Thanks for the info.We will try that option. I just want to add that if we restart SQL service all endpoints reports fine. And if we do not touch anything it will be auto-correct after 3 hours.

[Aram]

If there’s concern with trying the option, FillDB’s performance log would confirm whether or not the historical counts process is a factor here

[washnware]

Hi Aram,

We will try that option and I will post update here. Thank You Sir!

[Aram]

This is the area where you can configure both the FillDB Performance logging (if interested), as well as disabling the historical counts process, yes.

[washnware]

Hi Aram,

It works like magic. Any issue with the web reports if we disable below? Thanks

HKLM\Software\BigFix\Enterprise Server\FillDB
"UpdateHistoricalCounts" [DWORD]=0

[EmilianoN]

The UpdateHistoricalCounts setting controls the execution of update_historical_counts database procedure: configuring it to 0, you are going to disable the procedure.

This procedure inserts data into BFEnterprise tables HISTORICAL_FIXLET_COUNTS, for the historical information about the number of Fixlets at different severity levels, and HISTORICAL_COMPUTER_COUNTS, for the information about historical computer count objects.

The data from these tables are taken by a couple of session relevance inspectors, runnable from WebReports QNA: “all fixlet count” and “all computer count”.

For this reason, if you have any custom report based one of the two inspectors, you won’t see the report data to change and to get updated but you will keep seeing the data collected until you set the UpdateHistoricalCounts to 0.

[washnware]

Thanks for the Info.

[washnware]

Is it possible to set schedule of the “UpdateHistoricalCounts”? Thanks

[Aram]

It’s not a configurable option to specify the schedule, no. Is there a different time of day you would have preferred it to run? Or perhaps on a weekly rather than daily basis?

[washnware]

I think weekly will do to update it on a weekend. Thanks Aram.

[washnware]

Hi Aram,

Is it possible to schedule it weekly? Thanks

[Aram]

There is not an officially supported way to schedule it weekly, no, even if there might be some unofficial/unsupported methods of doing so.

That said, have you identified whether or not you actually use this data? I would consider it ‘legacy’ at this point.