Any way to automate patch management?

Hi,

I am new to IEMs patch management so this may be a silly question but essentially I would like to know if it is possible to do the following, and if so how?

At the moment, we do patching every week and it is a manual process to generate the patching baseline and then to Take Action against our computer groups.

Without using the API (if possible), and using IBM Endpoint Manager Console or any other program, is it possible to do all or part of;

  1. Every month, automatically generate new baselines with patches (with category filters) relevant to certain Computer Groups
  2. Once the baseline is generated, take action against the computer group at a specific time

We currently achieve this by manually doing this. Since it is done every week, it is extremely time consuming i.e. generate baseline, ensure the baseline matches up with Microsofts Recommended and Important updates, check that updates have been applied, troubleshoot any patches that failed to apply.

Happy to take any suggestions that any of you may have.

Cheers,
Blake

3 Likes

Without using the API, I donā€™t think what you are asking for is possible using IEM. And I donā€™t think it should.

Personally, it only takes me a few minutes to create the Monthly Baselines (one for Critical/Important and a second with the rest of the MS patches for the month). I include All Patches rather than just the Relevant ones.

Our server group then deploys the baselines against our Test machines, then if there are no problems, they update the Production servers in waves.

Workstations, can be patched by the same Baselines because I include all the patch Fixlets in them.

Iā€™m not responsible for patching anything other than the IEM environment itself, but Iā€™d be unwilling to simply have IEM automatically install patches before I had a chance to test them first. Microsoft has released several ā€˜badā€™ patches in the last few months. It highlights the need to TEST patches before simply rolling them out.

2 Likes

You may be able to use a dashboard to make the creation of a patching baseline just a few clicks, but to automate it completely would require the use of the API.

@TimRice You could fully automate deploying patches to a set of test machines, then automate deployment of the patches to a wider set of machines at a future date and then so on until all machines you wish to automatically patch have been patched.

I am also interested in coming up with something to automate patches like this.

Here is an example piece of relevance that is just one step on the way to fully automate patches:

http://bigfix.me/relevance/details/2999610 Fixlets to apply to computers in group ā€œautomatic updatesā€

This relevance will give you the names of all fixlets which contain version numbers in the ā€œUpdates for Windows Applicationsā€ site that are relevant to the group of computers named ā€œAutomatic Updatesā€. (this same relevance could be tweaked to include other patching sites)

My idea is this: You would have an automatic group called ā€œTest Machinesā€ which would get all of the fixlets deployed to them right away. You would have another automatic group called ā€œAutomatic Updatesā€ and this group would get all relevant fixlets older than 2 weeks installed to them. Membership of these groups would be controlled by a task that would be deployed to set a client setting, which would cause the machines to join the group that they are desired to belong to. The automation would stop and restart a new baseline every 2 weeks that would contain the correct set of fixlets targeted one to each of the Automatic Groups.

This is an interesting topic, and Iā€™d be curious to have a few questions answered for our design and development team. Trying to move away from the specific language of BigFix/IEM, it sounds like there are two issues here.

First is the ease of creating a patching policy that can be leveraged from within BigFix. This ā€œpolicyā€ might be ā€œall critical or important Microsoft patches should be remediatedā€ (and it sounds like Tim can quickly create this) or ā€œall Microsoft Recommended updatesā€ (which Blake finds more difficult- Iā€™d be interested in getting more detail here). I know some customers only include patches they know are currently applicable to some machines. Is this just to reduce the size of baselines, or is there some functional reason?

The second issue seems to be how to deploy the patches. Some products have a set it and forget it approach: define the policy, and tell all your computers to follow it. As Tim and jgstew point out, this can make you a bit exposed to bad patches. A couple related questions:

  1. How many of you would leverage the ā€œset it and forget itā€ approach, and what kind of progress tracking would you be looking for there (e.g. just call out failures, or maintain the current level of detail on all patch deployments)?

  2. Would a simplified version of jgstewā€™s approach work? Create a policy, apply it to all machines on a regular schedule (e.g. 3rd week of the month) but schedule test groups to run it first, and enable it to be canceled if error reports come in.

2 Likes

It would be interesting if the historical success/failure rate of all fixlets/tasks were tracked and ones that had any failures, or failure over a certain threshold would have to be manually remediated instead of being installed automatically, but otherwise set it and forget it would apply to all patching. (anything that would be installed by Windows Update, including recommended updates, plus most things in the Updates for Windows Applications site)

My general feeling is that all systems should have sufficient and reliable backups, and that most issues that occur arenā€™t catastrophic and thanks to using BigFix/IEM, most non-catastrophic problems can be remediated easily after they occur. This leads me to a push out all updates until something breaks, and then react to address the breakage. Iā€™d just rather not break all machines at once if something like this were to happen, so thats why I prefer to have the updates applied progressively to larger and larger sets of computers.

Yes, just to reduce the size and complexity of the baselines, partly for performance reasons.

Thanks everyone for your responses. Its good to know that there are active members in this forum.

@AaronBauer, creating the patching baseline is simple and doesnā€™t take that long. May take 15 minutes at most. But we do patching weekly, so 15 minutes times 52 weeks is 13 hours.

Option 2 is what I am after but knowledge of option 1 would also be good. My plan is to automate every Microsoft KB patch (same as Windows Update) to a group (mainly IT staff and some Admin staff), cutting out some of the time it takes me to patch.

If any fail to install, Iā€™d like those emailed to me with a reason. This can be done through Web Reports so need for anyone to explain that.

If there are no failures, Iā€™d like for those patches to be deployed across to every machines unless I cancel that due to some user complaints.

Iā€™m guessing going down the API route is an easier solution to this.

1 Like

Definitely going down the REST API route is what I would recommendā€¦

You can use /api/query?relevance to get at the latest patch fixlets that you care aboutā€¦ probably something like

(names of it, ids of it) of bes fixlets whose (source severity = ā€œCriticalā€ and now - source release date of it < 30 * days) of bes sites whose (name of it =ā€œEnterprise Securityā€)

Then youā€™d POST to /api/import/{site type}/{site name} to actually create your baseline (this is tricky, I think eventually weā€™ll have /api/baseline

So this will create a new baseline for you every time, which is part of the way there. Ideally though, I think youā€™d want to introduce some way of checking the fixlets in your previously created baselines against the latest version of your relevance query, and only adding the new stuff. That part is a bit trickier.

Iā€™m actually not seeing a way to modify a baseline in all the REST API stuff right now, although adding that functionality through /api/baselines is definitely a pending bug.

@jgstew, could you please advise that where I have to put your given relevance to do the patching automation. if you could just provide me the steps to achieve this. I would be really thankful to you.

There actually isnā€™t anywhere to put that relevance to automate patching.

You should have to use the REST API to create and deploy a baseline that would contain all of the fixlets output by that relevance and then deploy it to the automatic group that should get the updates.

I donā€™t have anything written to do this as of yet, but it is a way that this could be done.

Got itā€¦Thanks

Have you looked at the automation plans in Server Automation Application.
It could give you this capability ?

First step of the plan would deploy the baseline to your first group, and if that has no failures it will proceed to apply it to the second group and so on.

If thereā€™s a failure you can set the plan to stop, and thereā€™s other conditional options as well.

There are some gotchas to making baselines via the API but itā€™s not terribly difficult.

I canā€™t share any code on how we do it but we do this exact same thing in our test environment:

  1. Grab all applicable critical and important patches from various sites
  2. Bundle them into Component Groups in a baseline
  3. Action the baseline against a set of test endpoints starting now
  4. Action the baseline against a larger set of test endpoints starting in 2 days
  5. Action the baseline against production starting in 7 days

Is there any chance that you can please elaborate on the process, Iā€™ve exported the baseline definition for a windows baseline for Critical and security patches and the action to install the baseline for reference.

The XML looks pretty convoluted for both objects, any help direction would be greatly appreciated.

1 Like

I already have something that uses session relevance to turn a set of fixlet/task objects into baseline component XML.

Hi Jgstew,
Now that we are on BigFix 9.5.5 has this processes changed? We are looking at using a RestAPI call from HPOO to patch imaged servers before releasing to the business unit. I was asked from the HPOO team if they can just call the same action every time.
Thanks,
Frank

I am trying to run this file using :
. .\BESImportFile-1.3.exe servername username@pqr.com password ā€œWindows_update.besā€

but itā€™s erroring out:

BESAPI.XMLImporter.ImportFile error: HTTP Error 28: Timeout was reached: Connection timed out after 10000 milliseconds

If you have the WebUI installed you can use Patch Polices which recently now allow auto refresh on new Patches. Below is a Screenshot on how the Policy looks you can target Devices or Computer Groups with policies.

One thing to add though it creates a bunch of empty actions every so often so do not be alarmed if this happens.

3 Likes