I have noticed now for a while that patching windows systems using BigFix is not at all working as intended. I am facing a number of issue’s and listing down a few below.
There is a mismatch in the patches applicable to server/workstations with BigFix either showing more or less patches(Source Microsoft -> Category -> Security Updates) applicable on machines when compared to windows update being run manually.
Some patches show as completed successfully on the BigFix console but do not appear installed on the server when checked manually (both in Windows Update history as well as Installed updates in Programs and Features.)
Failed patches appear as installed successfully in BigFix console while they appear as failed when checked on the server. The same patch then does not appear as applicable on the server when checked in BigFix console.
Servers/Workstations show patches as applicable even after being installed ( either by BigFix or Manually by running Windows Update.)
The applicability of Monthly Quality Rollup / Monthly Security Only patches is not consistent with half the machines showing both of them applicable while others show only one of the two applicable.
With all the Security Vulnerabilities and Zero Day patches being released, using BigFix with the existing inconsistencies has made it very difficult to ensure that the environment is completely patched and not exposed to any threats.
Any guidance/ help on how to smoothen out these issue’s or perform any tasks to ensure a consistent patching environment would be really appreciated.
Some of these topics are pretty broad, and will likely require more indepth analysis on your part, but here are some general comments.
Mismatches do occur, and in my experience there are two reasons for this
a. If you have a WSUS server in your environment, you have to ensure you release updates for endpoints and/or ensure that you check for updates online to get the most up to date deployments available.
b. Sometimes, although this is rare and is usually fixed when brought to IBMs attention (PMRs are your friend), fixlet relevance provided by IBM does not necessarily match what Microsoft is using for applicability. We see this most commonly with Office updates that share common files, and even then only on machines with mismatched Office products. So for example, an endpoint has Office 2016 but then installs a standalone 2013 product.
c. In either of these cases, thorough examination of the fixlet relevance is necessary to determine if you are dealing with a false positive/negative as in example b. This is time consuming and cumbersome, but it really is the only way to quantitatively determine what is going on.
I have never really encountered this scenario. Can you elaborate on what you mean by “success”? Are you seeing exit code 0 from the fixlet? If yes, I would encourage you to inspect the CBS logs, as some updates are hidden from view and will not show up in Add/Remove Programs or even Installed Updates.
Same as 2, I cannot really say I’ve encountered this issue. Care to elaborate?`
How long are you waiting for the reporting to reach the main BigFix server? If you look at an endpoint log after deploying a patch, and you see the fixlet become non-relevant, this report should filter up to the main server as the report makes it’s way from client to relay(s) to main server. This can take a varying amount of time depending on your infrastructure.
I suspect that this may be related to 2. and 3. though, as fixlet relevance determines if any given fixlet remains relevant after running an action. If you are seeing a patch install in the logs but the fixlet remains relevant, then the conditions in the fixlet relevance remain true. This goes back to 1, where the only real option is to begin digging into the fixlet relevance to determine what is causing a given patch fixlet to report relevant (i,e, which conditions are true prior to and after installing the patch).
I also have to point out that some patches, Windows in particular, need a reboot to complete installing. Make sure you reboot your systems…
This is a feature, if memory serves. The Security Only updates are only ever relevant if a system is fully patched to prior month security updates. As a quick example, a system that was fully patched with the December updates will have both the Quality Rollup and Security Only patches for January relevant, while a system that was only patched to November will only see have the Quality Rollup relevant.
Now, there is a caveat in the above statement in that Microsoft has added a requirement for January that endpoints must have a registry key set either by compatible Anti-Virus software or manually, but the same above statement is true for October to November vs September to November, for instance.
mwolf is spot on. There is no blanket answer, though 2 & 5 are correct behavior. We don’t install via Windows Update, so I would not expect that history to update; and not all patches appear in Programs and Features, in my experience, even when manually installed.
The other items require investigation of specific instances of that behavior to confirm what the cause is, though mwolf provided likely explanations and good starting points.
In my experience, there can be differences in the update applicability you get depending on if you check for updates from WSUS or Windows Update or Microsoft Update, and that is entirely outside of BigFix, so there is already some variability in play. That is not even taking into account “hidden” updates.
The best bet is to compare what Microsoft Update says is applicable to what BigFix says is applicable.
Updates do not always appear in those locations. You must check certain places in WMI and/or in the Windows Registry to be able to tell what updates are currently “installed”.
Also, it is possible for there to be a difference between what updates have been “installed” vs what updates are “effective”. If somehow you manually roll back the files themselves to a pre-patch state without uninstalling the patch, then you can have files that are effectively unpatched while having the patch appear to be installed. This is generally pretty rare, but it is possible and can explain some discrepancies.
I have a fixlet and analysis combo to check windows update results and report on them:
