November rollup caused massive BigFix platform issues

I just wanted to alert everyone to the fact that KB4467697 (November OS quality rollup; now superseded by KB4471320) caused huge I/O issues for our platform. After investigating for two weeks why our I/O and FillDB were through the roof, we realised that it had started shortly after this update was applied. We rolled the update back and rebooted the server on Wednesday morning of this week, and the issues all but disappeared.

I contacted IBM to let them know, and they saw one other customer had an issue with different symptoms, but caused by the same patch:
http://www-01.ibm.com/support/docview.wss?uid=ibm10743209

Just a head’s up, in case anyone else is struggling with similar symptoms. We have a large environment (125k endpoints), so may be affected more than most… But @jgstew encouraged me to share the info, so here we are :slight_smile:

3 Likes

We installed that KB4467697 on our Win2K12-R2 BES 9.5.8 which has a local SQL 2016 on 11/26. The article link you provided mentioned the Compliance database. We have both Compliance and BFI using remote SQL 2016; they also have the patch. I don’t believe we are having the issue and the article doesn’t make clear the log (I’m not familiar with tema.log) for us to check.

Were does a solution stand at this time? With Microsoft?

Thanks for the info.

1 Like

Yeah, we’re pushing this to Microsoft and holding off any further Microsoft updates on the master server until our freeze period is over (in January) and we have found a root cause/resolution.

And we have BigFix Patch 9.5.9 and Compliance, and have SQL 2014 also local.

1 Like

It is very likely that Compliance and Inventory would not have the same performance issue as FillDB since those products typically only do an import once a day or similar, not around the clock like FillDB. You might want to look at the time of ETL from before the patch and after the patch for Compliance & Inventory and see if there is a difference there. If the patch causes slower performance when using Compliance / Inventory / etc… then that affect could be much more subtle and harder to detect.

It does seem like the linked article is due to the same patch, but a different problem.

1 Like