How do I know that a machine has been truly patched and not just overlooked because of a false negitive?
That is, if a fixlet reports back that a machine doesnt need a patch because of a false relevance(s), but the machine really does need the patch according to MS Updates(assuming that MS Updates is the ultimate authority which is questionable to begin with), how do you know without running MS updates on each and every machine ?
or
How do I know what I dont know ?
Is there something built into Bigfix to avoid this problem or if not then could there be?
It is a tough question in general for all applications detecting vulnerabilities… Some customers will point out that Windows Update or MBSA sometimes differs from our Fixlets, but often when we look into it, we see that we are correct (if not, we make changes to the Fixlet site).
I guess the only way I can think of to deal with false negatives (or alternately false positives) in a general sense is to get two products, compare the results and examine discrepancies (hopefully the multiple tools don’t have the same false negative).
We expect very high accuracy from BigFix and we are constantly refining our results so hopefully this is not a problem you or anyone else have to deal with often.
Thanks for the reply. Sorry for the delay. I did an inventory of 144 machines using MBSA/MSUpdates and discovered approximately 27 false negative security updates that Bigfix shows as unneeded but MBSA shows as required. Most have to do with MS Office security updates. I called support on 5 of those 27 and they explained that Bigfix uses the KB’s as supreme authority on secruity updates and of those 5 Bigfix was spot on - so apparently Microsoft is not reading their own KB’s.
I still find this very disconcerting if for no other reason than Microsoft is dealing with millions of machines on a daily basis. Not that they are perfect by any means. But it seems that it would be a very nice feature for Bigfix to offer a second or third means of verification for it users. For example would it hurt anything to continue to use the Kb’s as primary and then set up somethng for Bigfix to report false neg/pos relative to MSBA/MSUpdates and be able to apply fixlets to bring machines into compliance with MSBA/MSU? And then ditto for something relative to Nessus or other vulnerbility scanners. It seems to me that this would add greater confidence and trust to the Bigfix product and allow for better/faster false neg/pos detection and reporting.
I am also wondering now what I should do with those 27 possible out of compliance security updates. I feel the need to apply them just to be on the safe side ??? I dont want to have to worry about checking/fixing those all of the time. I cant get my boss to spring for Nessus at this point to do more real vulnerbility testing. Maybe I have some other deeper or masked errors here too, I am not sure?
Well… Remember that we also deal with millions of computers… We have several million BigFix Enterprise Agents and also several million consumer agents (Gateway and eMachines distribute a BigFix “Consumer Edition”).
Here is how we look at this:
We have been creating patch Fixlets for 10+ years for multiple vendors: Microsoft, Sun, RedHat, IBM, Adobe, WinZip, etc… We are not the vendor, of course, but we are experts on patching and we have a whole team dedicated keeping this body of info up-to-date and accurate. Sometimes we have a mistake (either due to problematic public information from the vendor or because we made a mistake ourselves) and we will fix the issue very soon and we publish the info about the change… And remember that our customers will help us if they find a mistake…
The experience that you mentioned with the 5 discrepencies is similar to what we generally experience: If there is a difference, we will compare the info and let you know what we think. Often times, we are correct over MBSA/WU/other apps, as you mentioned in your 5 issues OR if we have an issue we can fix it.
We consider it our job to be as accurate as possible and we don’t consider it our job to compare to MBSA (or Nessus or others) and so we don’t focus any energy on making sure we are the same as them (we often find they aren’t accurate).
If you have specific questions about discrepancies, you can send them in to support and we can look at them together to get answers.
