Patching Windows Servers to Completion

Hello,

I feel the answer should be pretty straightforward here, but wanted to reach out. Currently we patch a large variety of servers all at different patch levels. We implemented Bigfix to manage Microsoft deployment of patches from a central location, however we are looking to simply patch servers to completion in a given window just as you would if you logged in to the server and ran Windows Updates manually. We are currently using baselines to deploy applicable patches, but need a confident way to deploy all patches to an end system the same way you would if you ran Windows Updates manually.

Here is a breakdown of the above to make more sense of our current stance;

As we push patches from our created baselines we cannot confidently say a server is 100% patched even after all relevant baselines are completed and not relevant. We can confirm this by pushing all relevant baselines to a said server, logging into that server after the fact, running windows updates, and can see updates available when we are expecting this to be patched to 100% completion. With us not being able to be 100% confident we lose faith in the BF tool and with us having to go in there manually to check defeats the purpose of the BF tool. I honestly don’t think this is a BigFix problem, but more of our approach on using the BigFix tool to deploy patches. Keep in mind we onboard servers all with different patch levels and we are not consistently patching the same endpoints on a recurring basis. We would ideally like to get something in-place (if possible) to be able to pull all relevant windows updates from a specific endpoint and be able to create a baseline off that. I believe that would allow us to get to the 100% confident level of saying patching is truly completed. Keep in mind we ONLY want to be applying patches that Windows Update would pull down (No hotfixes, Cumulative updates, etc)

How are you creating the baselines?

We may have some different answers depending on whether you are finding that a specific patch appears non-relevant in BigFix but relevant via Windows Update; or whether installing one patch makes the system relevamt for another, newer patch.

A couple of options you might explore is Patching Policies (via the optional WebUI), or Server Automation (available in the Bigfix Lifecycle license).

If you are inboarding servers with varying/unknown levels of patches installed, it may take a few rounds of patching to get them caught up.

We initially created baselines by the Windows Patch Wizard and essentially cherry picking patches that were applicable and not hotfixes, CU, security advisories,etc. We only want patches that you would get from Windows Update.

The problem we have today is even setting baselines and picking patches that are applicable to our endpoints we cannot 100% say they are fully patched. We were using the Web Reports to gain a visual on missing patches post deploying all relevants MS baselines, but that wasn’t very accurate even after doing some in-depth filtering.

We did look at the patch policies, but that seems like its more of of cherry picking patches as well. We did just purchase the SA via BigFix lifecycle so hopefully that can provide some additional assistance in patching.

I do understand it can take a few rounds for outdated systems to be caught up. We are just trying to find a seamless approach to using BigFix Patch in a mirrored fashion that you would get if you ran Windows Updates manually. Bigfix provides us the central console we need, but if we have to go run Windows Updates manually on the server after the fact to see if there are still applicable patches it defeats the purpose of the tool entirely. Is there anyway today to utilize the client Rest API to generate a baseline of applicable patches from a specific endpoint and or a group of endpoints?

Ok, just trying to figure out whether you were having an issue with Fixlet relevance (false positive / false negative) or an issue with the workflow. At this point it sounds like a workflow thing.

As far as I’ve seen, Bigfix doesn’t give yhe kind of automation you’re looking for out of the box. It expects an operator to vet which patches to deploy and choose those, unlike Windows Update. For those of us in tightly-managed environments this is a good thing, but is contrary to what a lot if people who like Windows Update’s hands-off approach would expect.

Patch Policies might be helpful to you, I haven’t tried it myself. Server Automation probably not so much, as you still need to choose which fixlets to add to your baselines; Server Automation can dynamically create Actions from those baselines, but you still need to pick the patches.

Custom code using REST can do what you’re looking for, but you’ll want to be careful in how you automate selecting the fixlets to apply. There are a lot of fixlets you should not apply; for example there are conflicting fixlets for Spectre mitigations - a fixlet to turb the mitigations on, and another to turn them off.

Using REST requires a Console Operator credential, so you wouldn’t want all of the endpoints running it, but you could have a script on any single machine run and create baselines and actions targetting any computers or computer groups in the environment.

Have a look at what’s available for REST API at https://developer.bigfix.com - I think you’ll find examples there for creating baselines, sending actions, etc. that should be helpful.

Jason,

You are correct. It is more of a workflow problem than anything. We do have the occasional reports that flag a pre-patch report as relevant and then show it as non-relevant after the fact. We understand that installing a specific patch can cause other patches to become non-relevant, which is confusing, but just the way Windows Updates work.

Ill look further into Patch policies.

I agree that developing a custom code using REST would be the way to go and we are very mindful of the meltdown/spectre patches since there dependencies that can cause issues (Anti-virus). That is one positive item on us leveraging BF to patch so we can control what patches are deployed vs Windows Updates deploying everything.

Appreciate the link on REST API insight.

Looking at this in a different angle would it be possible to utilize the web reporting, generate a missing patch on a server or group of servers then take that missing patches report and create a baseline strictly from that report? I assume there is and would be done either manually or via API. We are really just trying to identify the best approach to take here in terms of deploying patches in a controlled manner, but also validate servers are completely up to date on patches.

I’ve gotta say I’m somewhat ignorant on what the default solutions are for this, since I went a custom code route several years ago. I’d expect there’s something in the console to handle some of this by default by now.

I recall one other similar posting on this recently, so I just want to check that you are aware of one workflow option. You know that in the console, you can select a computer group, go to the “Applicable Fixlets and Tasks” tab, and select fixlets from that list to create a new baseline? Then you get everything that’s applicable to any member of the group.

In my customization, I have a Console Dashboard that I wrote. I create new Baselines with several customizable filters (category, severity, OS Applicable, etc.) and also match against “not already in a baseline in my custom site” so I don’t get a lot of duplicate baselines. Something similar can be done via REST if you don’t want to use the Console for that.

I can’t post the whole dashboard because I have external dependencies that are too embedded, but I can post some of the queries that I’m using if you are willing to go down the custom dashboard / REST route.

Yeah, one would think, but it doesn’t seem like there is anything embedded by default and we have been in close contact with our reps on this. We are currently riding on 9.5.5 193 and don’t see anything compelling along those lines in the newer released versions, which is why we haven’t moved forward with upgrades.

We are aware of the applicable fixlets tab when selecting a said group. I suppose we could go down the route of creating specific baselines per each group, but still is a very manual effort each time in patching, but that may be the best route to go at this time to set accurate expectations.

If you wouldn’t mind, I’d be interested in seeing some of the queries that you are utilizing.

I appreciate all of your feedback here!

We decided to take a step back on this and adjust our procedure. At this time I think it makes the most sense to create our existing baselines from applicable fixlets, deploy them in a given window, and then run a report to show MS missing patches. That should allow us to get a visual on outstanding patches and also give us a relevant list that excludes hotfixes, CU, SP, etc. We could then take that defined list from the report and use the applicable fixlets tab on a specific server/group to create a multiple actions task to deploy the remaining missing patches. If someone happens to have a better approach please do share.

In terms of reporting, we have modified some existing reports out there and added additional filters to exclude patches that wouldn’t come with traditional MS updates. Digging around I see a lot of the reporting posts are quite old. Is anyone aware of any new reports available for MS missing patches?

This new path makes sense to me.

Because I already started down a path to explaining some of this, I’ll be creating a new forum post on creating & importing baselines via script. Watch for it shortly.

edit Nevermind, looks like this has already been covered pretty thoroughly by @jgstew, as usual :slight_smile:Automatic Baseline Creation and eventually AutoPatching

1 Like

FYI - I do have autopatching working with the REST API entirely that can do pretty much anything with some customizations. It would be great if I turned it into a console dashboard to make it easier to use. It isn’t super easy to maintain and customize, but it is very powerful.

There is now autopatch functionality in the WebUI which is easy to use, but limited in what it can do.

@jgstew can you provide some further insight into this and or specific direction on how to implement? I took a look at Automatic Baseline Creation and eventually AutoPatching and https://bigfix.me/fixlet/details/6132, but the fixlet fails, which I think is because of http://skanthak.homepage.t-online.de/download/curl-7.43.0.cab resulting in a 404 error.

It is really more of an example, not exactly meant to be implemented as is since I think the currently published form will just start patching everything automatically, which you probably don’t want.

The CURL binary is probably no longer available since that is a bit old and it would need updated to a newer currently available version.