OpenMic: Deploying Windows Updates via Baselines (March 16th, 2016)

Announcing the following upcoming OpenMic event:

Deploying Windows Updates via Baselines

Date/Time: March 16th, 2016 at 11:00 AM EDT (15:00 UTC/GMT, UTC-4 hours) for 60 minutes.
Presented by: Adam McDonald from our IBM BigFix L2 Support organization along with a panel of other product experts arrayed from our product management, development, and services teams.

Please click here to view/accept the invitation.

Ahead of and after the OpenMic; please post any questions you would like answered either during the OpenMic or through this forum.

To receive email notifications for upcoming Security Support OpenMic web casts, send an e-mail to isssprt@us.ibm.com with the following in the subject line: ste subscribe Endpoint And Mobility Management.


2 Likes

Is the link already available?

Will there be a recording available? I’m off site this day unfortunately, but this is a topic I’d find valuable. I have a few questions related to the topic.

-when I have several targets that are several updates behind, what is the best method of deploying the patches? what I often find is that the logic in applying the patches will push updates until one of them requires a restart, and it will restart, instead of installing other patches that require a restart and doing a single restart. I find the success rate of the machine getting through all of the required restarts when it follows this sort of logic pretty unreliable.

-often times, I’ll deploy updates to the company’s workstations and for whatever reason, a number of machines will ignore the design of the deployment and not actually restart when it’s finished. They’ll get some or all of the updates and get into a weird state. Some of the users manually restart after running into weird OS behaviors and that’s the end of it, some restart their machine and bigfix will continue off with the update it was deploying even though it is far outside of the scheduled patching window now.

-lastly, are there any known issues patching vmware based servers? for whatever reason, I have a number of vmware servers that will patch, and upon restarting, they’ll get into a hung state and have to be forcefully power cycled. not fun to deal with. I don’t run into this on our other (non vmware) virtual servers, and I don’t run into this when I manually restart the vmware servers.

1 Like

This was one of my questions too I wanted to ask: If you send the patches as an offer and you have 3-4 different offers, you need to restart 3-4 times instead of just once. Is there any workaround for this? It seems useless to reboot a device 3-4 times when it’s finished installing after the first reboot.

Another question I have is the following, but that’s more a functionality:
We use the Patch Wizard to create our baselines. We search always on OS and critical/important. So I search for Windows 7, Critical then I select the patches and then I search for Windows 7, Important and select them too to create the baseline. When I want to create the baseline for W8.1, I click reset but I noticed that it says that there are still patches selected (those from W7). And the strange thing is that sometimes they are indeed still selected, but sometimes they are not selected… A little bug in the wizard there or?

1 Like

Hello and thanks for the invitation, allow me to introduce my self, I am Sameer, and i work for a Managed Service Provider so we have multi customer environment, most of them are IBM’s Customer, basically working on behalf of IBM and provide several services and Patching via BIG FIX is one of them.

using BIG FIX in a multi tenant environment, is fun but at the same time also have to deal with the pain points that we have to deal with. and via this “OpenMic” i would like to know if theres a solution or a workaround to the pain points as mentioned below.

Question-1
so on every 2nd Tuesday when Microsoft releases the Bulk of their patches, we compile a list of the same on the 2nd wednesday and send it out to the customer for their review and approval. when the list comes back to us we group all of the “Approved Patches” and in a baseline. and deploy them as per the approved change window to multiple environments that the customer may have.

the Biggest Pain in this is that we have to have an Excel Spreadsheet open on one side and the Big Fix Console on the other and then Cherry Pick the KB numbers or the ID numbers on the big fix console from a very long list straining the eyes and the risk of miss selecting some thing else that may not be approved.

so the question is the way we have the ability of “enter device names” in the “take action window”, why cant we do the same with the KB numbers ??? like Create a Baseline by a selected list of KB or ID numbers by copy/pasting the same from an excel or a text file in to the “Create Baseline window”. just feed the list of KB numbers that a baseline needs to be created with.

Question-2
Once the Baseline is created, why dose’nt the system alert if a patch has been added twice or has a duplicate entry.

Question-3
Once a Patch is added to a baseline , why cant we see the ID or the KB number ? it just shows the MS number, wouldn’t it be good to have the ability to find a patch with a KB number or an ID number within a baseline ??? like if one needs to ensure that an Important KB is a part of the baseline ???

Question-4

how can we export the contents of a Baseline into an excel spreadsheet ??, this is a frequent task with us here where the customer requests to provide a list of all patches in the baseline that was applied to their environment ??

Question-5
How can we generate a report on what was installed by BIG FIX for a specified computer or specified set of computers on a given date or Dates.

Hope to see some answers here

Thanks
Sameer

3 Likes

I added the link for the invite to the description of the event.

There will be a recording taken which will eventually make its way up to our IBM Security Support channel here:

I’ll do me best to answer the questions you have posted soon.

@steini44 @Sameer @Entaille @Wouter I’ll do my best to answer these questions ahead of the OpenMic

1 Like

This can be accomplished through session relevance

(id of source fixlet of it, display source severity of source fixlet of it, name of source fixlet of it ) of (components of component groups of bes fixlets whose (baseline flag of it AND name of it is "Name of your baseline"))
2 Likes

Is it possible to improve the sound quality of the next recording?
It is very difficult to understand the previous recordings. Which is too bad since the topics are very interesting.

1 Like

Reboots should be suppressed so that a restart doesn’t happen in the middle of a baseline.

It can happen that a baseline completes but not all patches were applied because either they aren’t relevant until the patches that have just been applied are actually fully installed after a reboot, then after the reboot, the baseline becomes relevant again, applies more patches, and then another reboot.

This could also happen because of the order that the patches are placed in the baseline. If one patch requires a previous patch to be applicable, then if it is first in the order in the baseline, it will get skipped, while the later patch will be applied. Baseline components always execute in order, so this will mean that the baseline will have to apply again in order for all the patches to be applied.

I don’t recommend restarting using baselines, and instead tag the machines as needing a restart and use a separate process to actually do the reboots.

You have to set the baseline to restart the system when completed, or you need a separate process to cause it to reboot. All patching fixlets and tasks and the default behavior of baselines is not to restart the system at all so that can be handled specifically if needed or separately.

You should definitely file a PMR with IBM about this.

Don’t reboot as a part of the baselines, do that differently. Also have different baselines for patches that don’t require reboots (applications) than baselines for patches that do require reboots (OS).

Only require reboots once after everything has applied.

I don’t use that wizard, particularly because many patches are shared between OSes. We typically create 1 baseline for windows desktops containing all relevant Critical & Important patches.

I have some proof of concept code to automatically generate baselines for patching using the REST API on BigFix.Me

Probably. You should file a PMR with IBM on this.

You can do this with the REST API.

If you do it with the REST API, you can prevent this with the logic that generates the baseline.

Just because it isn’t available in the title of the baseline component doesn’t mean it isn’t available. It should be available through session relevance / REST API / WebReports.

Use the REST API to dump out the baseline contents to a CSV file. WebReports might also work for this.

Session Relevance through WebReports or the REST API or similar.

Probably not. The audio conference bridge they use is pretty bad and they are calling into it with Cell or Land Line phones, which are pretty much the worst case in quality.

I’d love to see a VoIP only audio option that would be much higher quality with some decent quality headsets. This should be possible, but they always use that damn audio conference bridge and it is awful.

2 Likes

I definitely look into the REST API on BigFix.

the reason we are creating different baselines ( for W7, W8.1, Windows Servers etc) is that IBM BigFix advised us to limit a baseline to 100 components. If you put all the OS in one baseline, you’ll exceed that number… Don’t know if this is true/has big impact etc?

1 Like

Yes, it will be recorded and eventually be made available on the IBM Security Support YouTube channel:
https://www.youtube.com/user/IBMSecuritySupport

The slide deck will be made available even earlier

I recommend putting a restart task as the first and last component of the baseline. I recommend making a custom copy of Task # 177 - “Restart Needed” in the BES Support site and changing the actionscript from:

action requires restart

to:

restart 60

Restarting an endpoint before patching, if a restart is needed, ensures the baseline will execute immediately after the system comes back online. And having the restart at the end of the baseline as an action, if a restart is needed, helps to get the machine restarted as soon as the baseline executes on a pass. However, the restart task should not be set to allow baseline to be relevant, I will show this in the OpenMic presentation.

I am not too sure what the problem would be here. I would wonder if the issue would happen independently apart from BigFix, as BigFix is only responsible for the part of bringing the patches to the endpoint, initiating the installers, and then sending the restart command to the OS. Everything else (the actual application of the patches is handled by the OS)

1 Like

Generally you don’t want too many components in a baseline, but if you end up putting many of the same components in multiple baselines, this could actually be worse, not better.

The 100 component limit isn’t strict. It depends on many things, but I wouldn’t be afraid to have 300 components in some cases.

The more components, then the more open actions the root has to handle, but if you arbitrarily create more baselines with many of the same components, then you are actually increasing the number of open actions, not decreasing them.

The other issue is with too many components in a baseline is that can negatively affect the evaluation loop of the client.

It would be nice for IBM/BigFix to better document why you want to minimize components in a baseline and what the effects are, rather than just having an arbitrary recommendation which could lead to things being worse instead of better.

2 Likes

@jgstew Everything you stated are the reasons why there is an arbitrary recommendation of 100. 100 is what the health check on the Deployment Health Checks dashboard looks for in order to flag baselines of “questionable” sizes; however as you said, there is no hard and fast limit.

See the following article:

http://www-01.ibm.com/support/docview.wss?uid=swg21636385

I did put an explanation for the size limit in the article:

“SIZE: Limit the number of components per baseline to a maximum of anywhere between 75 and 150 components. There is no hard maximum number of components that a baseline can contain. However, a baseline which is too large can take a very long time for the client to evaluate and process based on the number and types of components being added to the baseline and the size of the component relevance clauses. If this happens, this could tie up the client for several hours to a couple of days and in the meantime block the endpoint from being able to process further actions or send reports. There is no exact way of determining how many components your baselines can contain before client performance suffers, so start within the recommended range and move up from there; testing the to see if the clients can take larger baselines and still perform well.”

Your point about the multiplicity of the same components across multiple baselines resulting in a multiplicity of actions is also well taken, I will update the article to include some of this reasoning or recommendations for avoidance as well. :slight_smile:

It would be a very complex algorithm to come up with to determine the right maximum size of baselines (if even feasible). Many variables unique to individual deployments would need to be taken into consideration.

1 Like

If one of the primary concerns is endpoints evaluating baseline applicability, which I do believe is the primary issue that leads IBM to suggest a limited number of components, as @BigFixNinja seems to suggest, then there is actually a solution for that.

You could have all components of a patching baseline not make the baseline relevant, and then make the baseline relevant on all systems of a particular OS, but only if they have never run the baseline.

This would cause all systems to run the baseline once, but only the components that are actually relevant, which could be none of them. This means the evaluation loop of the client would not be affected.

You could use this method to have a giant catch all baseline, and then use other baselines to supplement this baseline that use relevance the normal way, but only be relevant on machines that have run the catch all baseline at least once.


###The actual mechanics of doing this would be like this:

  1. Add ton of patches to the baseline you want to have applied.
  2. Set the baseline relevance to windows of operating system or mac of operating system or similar.
  3. uncheck all of the boxes for “this component makes the baseline relevant” or whatever the phrasing is in the console.
  4. put a component at the very end of the baseline that writes a file or client setting or something to the machine to mark the machine as having run this baseline. This component’s relevance would be checking for the non-existence of the file or client setting or whatever. The box would be checked for this to make the baseline relevant.

One of the concerns though with large baselines might also be evaluation and execution at run time. We want the client to be able to get through one thing relatively quickly in order to get on to its next and other things. Since the client runs on a single evaluation / execution cycle; we don’t want to have one thing overly evaluating and blocking other things from happening within a given slice of time. So if we break things up to be evaluated / executed upon into more manageable pieces then the hope is every job or item to be evaluated / executed plays nice and shares the client with all the other jobs or items to be evaluated / executed. Also, generally speaking, increasing the number of items in anything increases the likelihood of probability for single point blocking failure.

You can prevent the client from evaluating any of the components except for at runtime using the method I described above, making the evaluation concern not an issue.

any word on when this will make it’s way to youtube, or the slide deck? it’s just about patch time again, was hoping to use this as a resource to fine tune the process.

:slight_smile:

The Slide Deck is here:
http://www-01.ibm.com/support/docview.wss?uid=swg27047538&aid=1

Working on getting the video together and up on YouTube. Hopefully soon.

2 Likes