What is the best practice for achieving the following with BigFix. We have a real struggle with this:
How do we get it so a new XP or Windows7 workstation that is joined to the domain, downloads and installs any relevant updates. We could create one massive baseline with all fixlets, and submit that as an action, regardless of relevance to be sure they get patched (since we can’t always predict what will become relevant at any given time), but that seems like it would kill the BES agent. We are used to WSUS allowing us to approve an update for installation for ANYTIME it ever becomes relevant. Is there a way to replicate this with BigFix?
Sure we could submit a new action each month, but that doesn’t always ensure new PC’s get the needed updates, since the baselines created are based on what’s relevant at the time.
I think a chapter in a book could be written on baseline management and strategies. My organization has discussed internally quite a bit different benefits and ideas about how to achieve what you are looking for.
There are a couple things to think of, and then find a happy balance between them all.
Don’t make baselines too large - forum rumors about max number of fixlets per baseline
Don’t make baselines too large - we’ve observed processing time (client profiler) and big ones can take a WHILE
Baselines aren’t interruptable on the client. Once they start it appears that’s all the client does (so to have the client responsive to other requests out there, make baselines run in less than 5 minutes? 10 minutes?)
Everytime you update a baseline, from the client’s perspective its two updates…one to delete the old baseline, one to bring in the new one. (So don’t make baselines too large)
They are not easy to manage if you have lots of people involved with patching.
So what we (try to) do is set a date for when our base desktop image is created, say the date WinXP SP3 came out. Then we have yearly baselines for Windows Desktop fixlets. In the current year we have one baseline per month (so we don’t update teh same one every month) until the END of the year, where we roll all the monthlys into a yearly baseline. We maintain separate baselines for Desktops and servers, as well as for applications. Then, we keep an open action (policy) for each baseline.
This approach has kept the management to a minimum, kept the baselines small(ish) and keeps the number of actions to a reasonable amount.
We’ve struggled with most of the issues you mention. The baselines are still, in our opinion, a work in progress. Right now they aren’t much more than just groups of fixlets.
As for the issue of targeting new systems as the come on line:
rmnetops
since the baselines created are based on what’s relevant at the time.
When you create your Policy Action, target a group - on the action dialog box under the Target tab, this is termed "All Computers with the property values selected in the tree below - and not computers themselves. Then as new systems come on line and join the group, the action will apply to them.
The downside to all this is that you will need to update that baseline or create a copy and update the copy when new fixlets are published. Then you will need to create a new Policy Action and target the group again.
We have similar challenges. Baselines with either a large number of components or with components having complex relevance can quickly bog down machines and take a while to process. This is especially true with patching older machines.
We’ve been semi-successful at heading off these baseline issues. Here’s what we try:
We try to capture new images at least annually that contain the latest service packs and security patches. Then as the year progresses, we add each month’s patches to our security baseline. This works well for the first few months, but by the back half of the year, the baseline turns into a monster.
By default, baselines “double-dip” relevance evaluation on the client which can really slow things down. When editing a baseline, if you expand a fixlet in that baseline, there is a check box for “Baseline will be relevant on applicable computers where this component is relevant”. Essentially it is taking the relevance from each fixlet or task in the baseline, mashes it together creating a HUGE relevance that the clients can struggle to evaluate. When the baseline runs, the client also re-evaluates the relevance of each item again at runtime as it goes through the baseline. Hence the “double-dip”. While this “double-dip” is useful in some scenarios, it seems to add significant overhead to the client in relevance evaluation.
The first inclination may be to uncheck all the boxes. However, the baseline may not run at all (it used to run in older BES versions, but they changed baseline behavior in more recent versions).
A strategy that we use in some situations is to create a task with an action that creates a flag of some sort (file, reg value, etc.) and that has as its relevance the non-existance of that flag and perhaps a few other things such as “not pending restart” or a regex of the naming convention of the machines being targeted. We then make this task the last item in the baseline. The key is that this item is the ONLY item with the box checked. The relevance for the task is copied verbatim into the relevance of the baseline.
Now the clients quickly evaluate the simple relevance of the baseline. When the baseline is run, only the applicable items within the baseline will run on a given client. The “double-dip” has been avoided, but the client still only runs the appropriate items.
This strategy also works will when constructing a “work-flow” where a subsequent baseline becomes relevant upon the completion of a previous. In this case, “work-flow” logic is added to the relevance of the task and baseline in addition to the previously mentioned items.
Baselines are ideal for software distribution, but less so for checking hundreds of patches in a single baseline. Doing so takes away some of the BigFix advantage by making the agent act like a legacy vulnerability scanner. You can get the best of both worlds by using multiple baselines; there are natural groupings of fixlets by time or affected feature that are good places to set baseline boundaries.
What is the best method to ensure that a Machine that’s been hiding in a storage closet for a year gets patched when it finally talks to the network? Big Fix Support tells me I’m all wrong for having Baselines per month and leaving them active.
I’d love to have a way to state approved/disapproved, if approved patch at any time, if disapproved report on it but do nothing.
The other side of the coin that the other folks aren’t mentioning is when the patch gets updated for whatever reason and you get the ohhhhhhh so common source fixlet differs message so with the wonderful baselines you need to stop your policy action update the baseline then re-issue the policy action after you’ve figured out what Big Fix has actually changed.
I think what BigFix needs to take away from the numerous and often repeated discussions on patch compliance / baselines is this:
BigFix customers need a way to create a true baseline and have any machine which falls into the scope of that baseline report and patch against that baseline. We need a simple way to do this. We do not want to spend non-value added time maintaining (babysitting) baselines and worrying about compliance and patch status of multiple baselines for a common area - Ex: Windows Security Patches.
We understand this may not have been the intention of baselines when BigFix designed them, but it is the need and desire of your user community to have this functionality - whether implemented as the current Baseline feature or some other yet to be named feature.
There’s a number of issues here that I’ll try to take a stab at. The most important thing I can say is that we do see the issue with applying multiple patches and we don’t think it’s going away; if anything, it will get worse as virtualization becomes more common (more hardware resource sharing, more patch churn from long absences or backup restoration).
I understand the annoyance of a fixlet change deactivating a baseline, but I’ve also received the unhappy call (at another company) when a vendor content change rolled out to thousands of workstations without any admin knowing about it. Content is never really set in stone; sometimes we have to make changes, and sometimes our upstream vendors such as MS and Adobe make changes that we have to reflect. We have to balance safety against convenience, but we do revisit these balances to make sure we continue to support the needs of our customers.
I can’t speak for support, but I suspect they’re focusing on performance when they say to reduce the number of active baselines. Perhaps a common ground can be found by reducing frequency of scan for those baselines or focusing on service packs; I’ll ask. However, I want to stress that this is an industry problem as much as it’s a BigFix problem. Relevance vs. VBScript, real-time scan vs. scheduled scan, different types of payload delivery and cache… all of these are irrelevant to the problem. Finding whether dozens or hundreds of changes need to be made is performance intensive, and then making them is even more so. I’ve seen a lot of proposals to solve the problem which generally boil down to “make your own service packs”, which seems even worse than what the industry provides now.
Thanks for taking the time to discuss, I’m interested in learning more about “true baselines” as opposed to what we do now.
Jack, speaking only for myself and my current company, we fully understand and appreciate how BigFix handles updates to fixlets. This is perhaps one of the best features and one of the many reasons we chose the product. But I think improvements can be made in the way the end user (BigFix Admins / Users) has to go about updating baselines and policy actions. Automated reports of changes would be a helpful start. The ability to update a baseline and reissue the policy action in one seamless step might be another. Some type of ability to auto create baselines based on certain fixlets aspects and client properties would be another. We can do a lot of this with other tools we have right now, but those tools targets are much more limited in scope. We can give specific examples if needed.
As for my “true baseline” terminology - for us it is very clear. A baseline for us is a target baseline, not a starting baseline. We need to take a machine from an unknown state to a known state - aka the target compliance baseline. So the “true baseline” is that target compliance baseline. Every machine that meets the criteria for that baseline relevance should then be able to attain compliance to that target compliance baseline. In the BigFix environment, that would normally be accomplished by applying every fixlet and task within that baseline via a policy action. We also need to be able to check the status and report against that target compliance baseline.
So if I have to create more than one baseline for say, Windows security patches, then the way BigFix wants us to use baselines, we need to maintain a baseline per month per OS or perhaps every six months per OS or perhaps some other scheme depending on who you talk to at BigFix. We then need to update all those baselines when the underlying fixlets are updated. Then we need to report on all those baselines to make sure each client received the baseline and applied it and is in compliance. On the other hand if I have a single baseline I have a single update to make and a single action to check and a single report to run.
To us, a (BigFix) baseline is not a baseline if it cannot bring the target system up to the target complaince baseline. If a baseline to BigFix is just a loose grouping of fixlets, then perhaps the terminology should be changed to “Collections”. I hope you truly understand that we’re not being adversarial; we are just trying to explain our needs as the customer. I did not mean to hijack this thread, as rmnetops has a valid question which has been asked time and time again in the forums. I hope he finds an acceptable means of performing the function, as I think his question is still unanswered.
Your not hijacking. It’s good to see other customers are experiencing/observing the same thing we are. We were just wondering if customers had figured out the “best practice” yet, to save us the time experimenting trying to find it on our own.
Customers are using baselines for many purposes, perhaps beyond what Bigfix designed or intended them to do. Hopefully Bigfix will incorporate some of this feedback into the product.
Perhaps offering different types of baselines or options for tuning baseline behavior to be optimized for various uses would be appropriate. Possible varieties might include:
Classic - existing baseline functionality
Compliance - apply security patches or security settings or configuration changes in such as was as to always use the most current version of either the security fixlet (when superceded) or custom fixlet/task that has been modified. This option could be set on the entire baseline and/or by check box on each item (much the way items are included in relevance checking or not). This would be jspanitz’s “true baseline”.
Performance - Similar to classic except that only the baseline relevance is evaluated by the agents, not individual components, until runtime. This would be very useful in applying large amounts of patches to a set of cloned machines, for example, that all need the same 10 months of security patches. It would also speed patch application during short outage windows. (Some of our older hardware struggles to evaluate all the security patch baseline relevance within the alloted maintenance window.)
Workflow / Business Process - Optimize a way to create and maintain a “workflow” of baselines for business processes. Examples would include ERP information and configuration flows. Another example would be a “staging” process for automatically performing standardized configuration flows automatically.
So, I just went through this discussion with the support folks, and the consensus was that multiple baseline policies of a dozen or two fixlet messages each is the way to go.
all content is not equal; the most important issues can go into a critical baseline that’s watched more closely. This also lets you separate “reboot worthy” and “user notification worthy” issues from merely important patches.
performance will remain better than in the “everything in one baseline” model.
a fixlet change that un-synchronizes the baseline won’t have a huge effect.
That last one brings up issues like JonL’s option 2 and jspanitz’s though; the desire to have an auto-update when a fixlet is modified. That continues to make me nervous, but there are enhancements being tracked in the system for that and related behavior.
The introduction of this topic on the forums got me to thinking about how we can better enhance our patching at our organization. I have done today what some of you have suggested, setting up multiple baselines and then establishing policy actions for all of my computers based on what is relevant on my systems. Thanks for bringing to light one of the toughest things IT people face with regards to patching computers. It has been a great discussion and a great thought provoker!
JonL - I really like the process behind option your #3.
This brings me to another question. If BigFix had default collections for things like OS and the fixlets used those collections instead of having the relevance in the fixlet, would that speed up the client? In other words, having a baseline with 100 fixlets and each of those fixlets having the exact same relevance for OS (ex: if(name of operating system starts with “Win” ) then)) has got to be somewhat time consuming.
Jack - I hear what you are saying. And I know I’m being a pain, but it seems like that is still the same way we do things now. It’s Admin resource heavy and time consuming to maintain. If you can tell me that something along the lines of what many of us are suggesting (in the way we want to use baselines) is at least somewhere on the roadmap, I’ll go away
operating system lookups are very fast (measured in microseconds) and the agent is highly optimized for inspectors like this… Generally file version lookups are the slower part (by slower, I mean milliseconds, but often times in these big baselines there are lots and lots of version lookups and the agent only gets a very small percentage of the resources of the system).
I breezed through this thread as I don’t have time now but it interests me greatly as we just moved from WSUS to BigFix… I’m also used to just “approving” updates and then having old PC’s coming online getting the appropriate approved updates automagically…
Question though that’s somewhat off topic but still related… I’ve read in this thread about the performance hit baselines can have on the systems… Can someone elaborate on this? I don’t want to provide unnecessary overhead to any of my systems by something I configured incorrectly.
If there was a way you could select all (or multiple) fixlets, right click and have it create multiple baselines for you in a single action. Then you select all those baselines, right-click and create a single action to run those multiple baselines as one action.
Have a way to sync all fixlets in a baseline in a single action, instead of having to edit the baseline and sync each one manually.
