Does BigFix pre-test patches?

(imported topic written by mauricev91)

Patchlink claims to pre-test patches before releasing them, and I’m under the assumption that if a patch misbehaves, the patch isn’t distributed to the repository and the vendor is notified. But it doesn’t seem as if BigFix does this. It would seem that pre-testing the patches is brilliant way to avoid hosing 500 systems overnight , so why doesn’t BigFix offer this functionality?

(imported comment written by StacyLee)

I’d be interested in what is Patchlink’s testing criteria?

My understanding BigFix does pre-test patches, but will let a BigFix person clarify this point.

In our environment we always pre-test patches in this order:

  1. Crash and Burn Test systems (usually virtual machines)

  2. Development Systems

  3. Productions Systems

We have over 20K endpoints and it would be impossible to test every different setup and scenario. Our virtual and test machines are a representation of the base systems (XP, Vista, 2003) with essential apps, we test patching them and make sure basic apps work. If for some reason 1 or more user installs a piece of software not part of the standard applications that conflicts with a patch and blue screens, we are not responsible. BigFix needs to make sure the fixlets written do the appropiate version, error and verification checking, however I still consider it only a delivery mechanism of patches to the end point.

If a MS patch breaks an application or crashes a computer it wouldn’t matter if you used patchlink, bigfix, WSUS or any other automated patch deployment tool. It’s the patch thats the problem not the messager (so to speak).

In short you need to you have your own testing plan for you base set of systems and applications. I wouldn’t think you would push out patches the moment they come out to 500 machines w/o prior testing.

(imported comment written by mauricev91)

I’d be interested in what is Patchlink’s testing criteria?

I’m interested too, but haven’t been able to get a clear answer nor an answer to how they handle misbehaving patches.

My understanding BigFix does pre-test patches

This would be news to me but certainly a very welcome feature.

We have over 20K endpoints

So have you run into misbehaving patches? Are there reports of unfortunate experiences post-patch and of patches that have been re-released?

I wouldn’t think you would push out patches the moment they come out to 500 machines w/o prior testing.

Stanford University

This almost seems ironic, but my belief-and it could be off the wall wrong–is that your university is a unusual case, that most universities treat users like they are at home and just instruct them to use automatic updates (along with the same policy toward virus definition updates). And I would just assume that virtually no home user test patches (then again, may be they’re not even updating!). That is, I think the automatic mechanisms like BigFix are employed almost exclusively by large corporate environments. This gets back to question of where are the problematic patches.

(imported comment written by jessewk)

This almost seems ironic

What’s really ironic is that Stanford was one of BigFix’s first major customers, but most of the technical folks at BigFix are Cal grads. See this post for a little about how we feel: http://forum.bigfix.com/viewtopic.php?id=82

I think the automatic mechanisms like BigFix are employed almost exclusively by large corporate environments.

Actually, there a lot of academic environments that use BigFix. One of our larger deployments (100,000 seats) is a school district and there are quite a few universities and K-12 groups that use BigFix to manage their environments. Schools need to keep their computers secure and up-to-date too! I wonder if

this

would have ever happened if BigFix AntiPest was installed at Norwich Middle School.

This gets back to question of where are the problematic patches.

I know there’s been a few… MS06-040 springs to mind. MS05-019 and MS06-015 too. There have certainly been others… anybody else care to chime in?

My understanding BigFix does pre-test patches

BigFix carefully tests all Fixlets to ensure successful delivery, application, and verification. In the case of patch Fixlets, we do not test the patches themselves for functionality or stability, although in the past we have certainly noticed things and reported our findings back to Microsoft or other vendors.

(imported comment written by SystemAdmin)

Just to add my 2 cents, you are right that many universities do treat users like ‘home’ users and expect them to patch their computers themselves through auto-update. This model simply doesn’t work in practice though, not enough patches get applied in the home user model and you end up with a large pool of computers with vulnerabilities that can quickly spread viruses to each other. The problem gets worse because these computers will then propagate the virus to internal computers as laptops move around. Standford is leading their industry in my opinion by pro-actively resolving these vulnerabilities instead of relying on home users to patch computers themselves. Your average user won’t keep their computer up to patch standards on their own, this is true even at a University (though maybe not the engineering computers!).

(imported comment written by StacyLee)

So have you run into misbehaving patches? Are there reports of unfortunate experiences post-patch and of patches that have been re-released?

–As Jesse mentioned there have been some in the past. we’ve never ran into any problems here. I don’t remember details but when I read the past details of a patch problem it was for a specfic circumstance that never applied to us.

This almost seems ironic, but my belief-and it could be off the wall wrong–is that your university is a unusual case, that most universities treat users like they are at home and just instruct them to use automatic updates (along with the same policy toward virus definition updates). And I would just assume that virtually no home user test patches (then again, may be they’re not even updating!). That is, I think the automatic mechanisms like BigFix are employed almost exclusively by large corporate environments. This gets back to question of where are the problematic patches.

–That was the problem before, trusting the end users to do the right thing, we as IT professional would think patching is not a big deal and just go ahead and click OK to windows update. However many people have no idea what that little shield icon let alone read what the security bulletins mean and may choose not to do anything or WUA may even be disabled. The Blaster worm caused a major outage on several thousand machines on campus in the order of 1.5 million dollar in clean up costs. A university has many essential services just like a corporation and more: finance, HR, student, medical school,facilities. etc… Stanford’s peak population is around 30,000 people made up of staff, students and faculty. Can you imagine if suddenly 10K people could not get their pay checks because the payroll departments machines were all comprimised and had to be rebuilt or a reaseach computer with XP hooked up to sensitive medical or reasearch equipment comprimised and needed to be rebuilt. Any computers particpating on the Stanford Network use BigFix or have some kind of automatic updating requirement. Times are changing we need to have a way to push updates to the campus. BigFix helped saved us from Crisis (see http://news-service.stanford.edu/news/2005/september14/bigfix-091405.html) where as big companies such as CNN, the New York Times, Caterpillar Inc., SBC, DaimlerChrysler AG and General Electric were suffering from the zotob/esbot worm.

(imported comment written by BenKus)

To be clear, the answer to the original question is that we absolutely test Fixlets/patches before releasing them. Please see this KB article for more information:

http://support.bigfix.com/cgi-bin/kbdirect.pl?id=170

Note that we will always release the Fixlets as soon as they are tested, but we note any issues we find in the descriptions of the Fixlets (and there are lots of examples in our standard Fixlet content). We do this because we feel that it is up to the customer to decide whether they want to patch their computers and it is not our decision to make for them. We simply give them all the information and the capability to patch.

Ben

(imported comment written by mauricev91)

Thanks for all your comments.

Anyway, I’m bit confused.

The tech note says

we test to ensure the patch …works in different environments

That says to me functionality and stability

are

being tested just like PatchLink is claiming to do

but

jessewk said

we do not test the patches themselves for functionality or stability

Aren’t these statements contradictory?

So assuming BigFix does do this automated testing and just happened to discover that a patch that did something potentially devastating such as preventing a particular version of Symantec antivirus from running, would BigFix hold back the patch until the vendor could check it out, fix it and reissue?

(imported comment written by paulc91)

If Bigfix, or any other vendor, does or does not test the patches should not replace, in my opinion, testing in your own environment working through test, development, qa and production environments. There is no way that any vendor could confirm 100% that the patch will not cause issues in your environment.

If Patchlink say they ‘pre-test’ patches have a look for their disclaimer, I am sure that somewhere it says something like ‘we cannot be held responsible for any issues caused by installing patches’.

Don’t forget that MS test the patches, but you still see bugs and issues!

(imported comment written by BenKus)

Hi mauricev,

We do plenty of testing on the patches to make sure they work in all sorts of environments (we have so many pieces of software from all sorts of vendors and we have hundreds of pre-setup ghost images and VMWare systems with common configurations). My guess is that we do more testing than Patchlink does, but I know they hype this fact big time as a sales pitch (a fairly weak sales pitch in my personal opinion).

When we test the patches, we that during our testing we are looking for specific issues:

  • Make sure the patch applies successfully.
  • Make sure all the files/registry keys are updated.
  • Make sure that our Fixlets properly detect applied, not applied, and partially applied patches.
  • Look for any issues on the system that might have resulted from the patch.

I think Jesse is trying to say that we don’t do full regression testing of patches on all combinations of environments (we select common environments similar to our customers) and we can’t guarantee that just because we ran some tests on the patches, we would be able to detect bugs. Microsoft does full regression testing and they have lots of testing resources in place and they test for a long time before releasing the patches. It would be silly to say that we could begin to replicate that process.

The nice thing that we do have going for us is that BigFix agents are on millions of computers around the world and if we hear reports that patches are causing issues, we can update the Fixlets so that all our customers can benefit from other customers testing. For instance, look through your Fixlet list and you should be able to find:

“MS04-011: Security Update for Microsoft Windows - Windows 2000 May Crash After Patch Installation”

This is a Fixlet that actually identifies the condition that might cause the patch not to work. Note that we continue to offer this Fixlet and we don’t refuse to offer Fixlets just because they might cause a problem in certain configurations. We believe that the we need to provide all the relevant information and each customer will assess the risk themselves.

Ben