OpenMic: Deploying Windows Updates via Baselines (March 16th, 2016)

Is it possible to improve the sound quality of the next recording?
It is very difficult to understand the previous recordings. Which is too bad since the topics are very interesting.

1 Like

Reboots should be suppressed so that a restart doesn’t happen in the middle of a baseline.

It can happen that a baseline completes but not all patches were applied because either they aren’t relevant until the patches that have just been applied are actually fully installed after a reboot, then after the reboot, the baseline becomes relevant again, applies more patches, and then another reboot.

This could also happen because of the order that the patches are placed in the baseline. If one patch requires a previous patch to be applicable, then if it is first in the order in the baseline, it will get skipped, while the later patch will be applied. Baseline components always execute in order, so this will mean that the baseline will have to apply again in order for all the patches to be applied.

I don’t recommend restarting using baselines, and instead tag the machines as needing a restart and use a separate process to actually do the reboots.

You have to set the baseline to restart the system when completed, or you need a separate process to cause it to reboot. All patching fixlets and tasks and the default behavior of baselines is not to restart the system at all so that can be handled specifically if needed or separately.

You should definitely file a PMR with IBM about this.

Don’t reboot as a part of the baselines, do that differently. Also have different baselines for patches that don’t require reboots (applications) than baselines for patches that do require reboots (OS).

Only require reboots once after everything has applied.

I don’t use that wizard, particularly because many patches are shared between OSes. We typically create 1 baseline for windows desktops containing all relevant Critical & Important patches.

I have some proof of concept code to automatically generate baselines for patching using the REST API on BigFix.Me

Probably. You should file a PMR with IBM on this.

You can do this with the REST API.

If you do it with the REST API, you can prevent this with the logic that generates the baseline.

Just because it isn’t available in the title of the baseline component doesn’t mean it isn’t available. It should be available through session relevance / REST API / WebReports.

Use the REST API to dump out the baseline contents to a CSV file. WebReports might also work for this.

Session Relevance through WebReports or the REST API or similar.

Probably not. The audio conference bridge they use is pretty bad and they are calling into it with Cell or Land Line phones, which are pretty much the worst case in quality.

I’d love to see a VoIP only audio option that would be much higher quality with some decent quality headsets. This should be possible, but they always use that damn audio conference bridge and it is awful.

2 Likes

I definitely look into the REST API on BigFix.

the reason we are creating different baselines ( for W7, W8.1, Windows Servers etc) is that IBM BigFix advised us to limit a baseline to 100 components. If you put all the OS in one baseline, you’ll exceed that number… Don’t know if this is true/has big impact etc?

1 Like

Yes, it will be recorded and eventually be made available on the IBM Security Support YouTube channel:
https://www.youtube.com/user/IBMSecuritySupport

The slide deck will be made available even earlier

I recommend putting a restart task as the first and last component of the baseline. I recommend making a custom copy of Task # 177 - “Restart Needed” in the BES Support site and changing the actionscript from:

action requires restart

to:

restart 60

Restarting an endpoint before patching, if a restart is needed, ensures the baseline will execute immediately after the system comes back online. And having the restart at the end of the baseline as an action, if a restart is needed, helps to get the machine restarted as soon as the baseline executes on a pass. However, the restart task should not be set to allow baseline to be relevant, I will show this in the OpenMic presentation.

I am not too sure what the problem would be here. I would wonder if the issue would happen independently apart from BigFix, as BigFix is only responsible for the part of bringing the patches to the endpoint, initiating the installers, and then sending the restart command to the OS. Everything else (the actual application of the patches is handled by the OS)

1 Like

Generally you don’t want too many components in a baseline, but if you end up putting many of the same components in multiple baselines, this could actually be worse, not better.

The 100 component limit isn’t strict. It depends on many things, but I wouldn’t be afraid to have 300 components in some cases.

The more components, then the more open actions the root has to handle, but if you arbitrarily create more baselines with many of the same components, then you are actually increasing the number of open actions, not decreasing them.

The other issue is with too many components in a baseline is that can negatively affect the evaluation loop of the client.

It would be nice for IBM/BigFix to better document why you want to minimize components in a baseline and what the effects are, rather than just having an arbitrary recommendation which could lead to things being worse instead of better.

2 Likes

@jgstew Everything you stated are the reasons why there is an arbitrary recommendation of 100. 100 is what the health check on the Deployment Health Checks dashboard looks for in order to flag baselines of “questionable” sizes; however as you said, there is no hard and fast limit.

See the following article:

http://www-01.ibm.com/support/docview.wss?uid=swg21636385

I did put an explanation for the size limit in the article:

SIZE: Limit the number of components per baseline to a maximum of anywhere between 75 and 150 components. There is no hard maximum number of components that a baseline can contain. However, a baseline which is too large can take a very long time for the client to evaluate and process based on the number and types of components being added to the baseline and the size of the component relevance clauses. If this happens, this could tie up the client for several hours to a couple of days and in the meantime block the endpoint from being able to process further actions or send reports. There is no exact way of determining how many components your baselines can contain before client performance suffers, so start within the recommended range and move up from there; testing the to see if the clients can take larger baselines and still perform well.”

Your point about the multiplicity of the same components across multiple baselines resulting in a multiplicity of actions is also well taken, I will update the article to include some of this reasoning or recommendations for avoidance as well. :slight_smile:

It would be a very complex algorithm to come up with to determine the right maximum size of baselines (if even feasible). Many variables unique to individual deployments would need to be taken into consideration.

1 Like

If one of the primary concerns is endpoints evaluating baseline applicability, which I do believe is the primary issue that leads IBM to suggest a limited number of components, as @BigFixNinja seems to suggest, then there is actually a solution for that.

You could have all components of a patching baseline not make the baseline relevant, and then make the baseline relevant on all systems of a particular OS, but only if they have never run the baseline.

This would cause all systems to run the baseline once, but only the components that are actually relevant, which could be none of them. This means the evaluation loop of the client would not be affected.

You could use this method to have a giant catch all baseline, and then use other baselines to supplement this baseline that use relevance the normal way, but only be relevant on machines that have run the catch all baseline at least once.


###The actual mechanics of doing this would be like this:

  1. Add ton of patches to the baseline you want to have applied.
  2. Set the baseline relevance to windows of operating system or mac of operating system or similar.
  3. uncheck all of the boxes for “this component makes the baseline relevant” or whatever the phrasing is in the console.
  4. put a component at the very end of the baseline that writes a file or client setting or something to the machine to mark the machine as having run this baseline. This component’s relevance would be checking for the non-existence of the file or client setting or whatever. The box would be checked for this to make the baseline relevant.

One of the concerns though with large baselines might also be evaluation and execution at run time. We want the client to be able to get through one thing relatively quickly in order to get on to its next and other things. Since the client runs on a single evaluation / execution cycle; we don’t want to have one thing overly evaluating and blocking other things from happening within a given slice of time. So if we break things up to be evaluated / executed upon into more manageable pieces then the hope is every job or item to be evaluated / executed plays nice and shares the client with all the other jobs or items to be evaluated / executed. Also, generally speaking, increasing the number of items in anything increases the likelihood of probability for single point blocking failure.

You can prevent the client from evaluating any of the components except for at runtime using the method I described above, making the evaluation concern not an issue.

any word on when this will make it’s way to youtube, or the slide deck? it’s just about patch time again, was hoping to use this as a resource to fine tune the process.

:slight_smile:

The Slide Deck is here:
http://www-01.ibm.com/support/docview.wss?uid=swg27047538&aid=1

Working on getting the video together and up on YouTube. Hopefully soon.

2 Likes

Are there still plans to post this video on youtube? I am interested in reviewing the session again, and I have coworkers who may be interested as well.

The audio was horrible. I need to re-record it. It is on my to do list :smile:

The replay is posted at https://www.youtube.com/watch?v=LWLVRiAGFoI
I’m sorry about the quality of the audio.

I’m sorry for the delay. The replay now is at https://www.youtube.com/watch?v=LWLVRiAGFoI

Adam, I posted it anyway, since we wanted it as prerequisite viewing for the 2017-02-23 Baselines Q and A Open Mic.

2 Likes

Forgive me if I’m missing something here, but after watching the presentation via the youtube channel am I correct in saying that every month that the BigFix console receives new updates from Microsoft an entirely new baseline will have to be created? The synchronizing feature is only for the fixlets/components that existed in previous baselines correct?

1 Like

I believe that’s the recommended practice/routine. This keeps the baseline size manageable.

When you create a baseline, it creates it based on the current content of those fixlets / tasks. If there are any updates, changes or things become superseded, the baseline still references the original versions. Syncing helps update those fixlets/tasks to the latest version, or lets you know when you can remove the superseded ones etc…

1 Like

I have seen this only with Server 2012R2 VMs - it is annoying, I just got into the habit of forcing it to apply patches and shutdown, instead of apply patches and reboot. Have to do it manually, but that is better than it going into the fugue state where it ignores all rdp/ctrl-alt-del/etc but ping works. wierd.

Yes, that is correct, but there are some existing dashboards / wizards / WebUI Autopatch / REST API automation to help with this.

1 Like