It’s what community and Open-Source is all about!
2 Likes
Your post, couldnt have said it better myself.
I also wont be surprised if some update eventually comes to Logpresso to replace the jar with 2.17 or 2.12.2 etc
1 Like
There was already a GitHub Issue filed for this here: Question: What is Logpresso with vulnerable 2.16 jar files ? · Issue #173 · logpresso/CVE-2021-44228-Scanner · GitHub
Yeah interesting, 2 days ago, I guess annual leave ignorance has been bliss.
hahahaha surprised you found it, thats a terrible issue name!
1 Like
It was very hard to find. I think I ended up searching for 2.17 to find it.
1 Like
The BigFix Compliance team has released a new version of Compliance, that removes Log4j entirely in the product. Note that we believe there was not a vulnerability due to the way we were using Log4j previously, but updating to this version should remove any concerns for Compliance.
1 Like
I was checking for a corrected version of the hashes for scanning Linux and AIX systems containing log4j files.
if you are only looking for hashes of the log4j-core files or those file by path, then you are missing a tons of log4j vulnerabilities. The Logpresso scan utility will find tons more and help mitigate them. See the summary info about it here: Log4j CVE-2021-44228, CVE-2021-45046 Summary Page
1 Like
This should now download a temporary JAVA JRE/JDK for every BigFix platform except for HP-UX:
We can’t add HP-UX support due to HP not allowing direct downloads of their JRE/JDK. It might make sense to handle the HP-UX case in this same fixlet using a fallback to a system installed JRE/JDK, but there is another fixlet for system JRE that already that I would test on HP-UX instead and give us feedback on if it works there as far as the relevance detecting the JRE or not.
We haven’t finish testing the part that expands the JRE and runs the commands on all platforms. It is some AIX platforms and Solaris that still need work in some cases.
Right now the JRE in the download for 32bit is Win2012r2+ only it seems. Other alternatives that are Win7+ require VC++ redist to be installed. It is hard to find a good option for older end of life operating systems, but that is kind of expected. The right answer in these cases might be use the “already installed JRE” on the OS to do the scan, which is a separate fixlet/task.
Had a pretty rough day with a bit of stuff developed in relation to Log4j but not time to do any reading on the new developments.
Anyways good news is logic wins, and we are not doing the individual server certain path exclusions, or inclusions if you will. The entire server will be mitigated (yay!).
Not sure if it was mentioned but RHEL 6 seems to have glibc 2.12 on 64 bit which happily runs OpenJDK JRE 1.8, but… not sure if its been noticed here as seems everyone is running on latest releases. This is from 2.6.1 updated version of the pre compiled linux scanner on github, I was wondering why my version with jar scanner worked…
So glad I put those glibc checks in anyways. Ill hopefully get a chance to check your updates @jgstew tomorrow, need bed, stat!
EDIT: For under Win2012R2 releases and equivalent desktop versions, I am hoping to find some version of Java 1.7 JRE that is usable but I havent had time to look. Found linux versions of OpenJDK 7/1.7 which I ahvent checked if they are in your task yet? Ideally I want to go down to AIX 6 (5 if possible!), RHEL 5 and its equivalents (3/4 if possible…!), Windows 2008 before R2… (2003 if possible), SunOS 5.9 (5.8 if possible). AIX and SunOS I have researched the least ouf of the lot, but JRE 1.7 should allow at least RHEL 5.x and its equivalents, and thats reasonable enough. I should note I would only be using these unsupported EoL 1.7 versions on the old old stuff, anything else latest 1.8’s
1 Like
This is a major challenge. We can never really say we can definitely support EoL products with EoL versions of BigFix, but if it is trivial to extend support and others can demonstrate how, then great, please help us enhance our templates to do so.
Amazon Correto JRE/JDK8 does seem to go further back in support than Azul or AdoptOpenJDK but it seems to have more requirements for VC++ redist? Correto doesn’t go back further than v8.
We have also been trying to look for solutions here.
Until you know you have something that breaks, this is the right approach if you want to get things mitigated in a timely fashion.
Unless the system is critical to the point of life or death or the system impacts many other systems (domain controller, LDAP, SAN, etc) then it might be better to break it than to have a remote code exploit remain open.
And if something does break, do you roll back everything? or maybe only 1 or 2 things broke and you go fix them by trying to update the JAR to 2.17 or look for vendor patches or something else. Or just roll back that one app.
And if you do find something that breaks, please please please tell us all here so everyone can know about it!
1 Like
Absolutely, and I agree this is the right approach overall, but in real life enterprises still stick with some odd choices and something of this criticality I see myself catering for, its not like they will suddenly upgrade, if they havent already 
sad story… it should be relatively trivial to get old old old OS’s supported for this task in particular just due to the fact that the major dependency is the JRE, and a little bit of extra relevance thats basically copy/paste with some editing and testing of course.
I would be a bit more upset if it meant completely redesigning something from ground up, at least for this task its essentially more of the same.
1 Like
We know this is the case and we feel for it, but we also hope that the number of those systems is smaller and more manageable.
The issue is the people that automatically build JREs can’t responsibly do it on platforms that are EoL for long because they cannot patch them. Then we can’t generally test our stuff on platforms that are EoL for the same reasons. Our best best is to boot up an old VM we keep around for this purpose, test it, then shut it back down.
This is exactly the agreed approach, after some convincing and much light seeing that this is least work and uniform approach. Then deal with breakages if they do actually happen, which we think is minimal.
We are going to leave backups in place until some agreed to time, few weeks?? till reboots happen as that will be ultimate test.
BTW, I have taken the liberty to create some content and update the remediation tasks with some assistance from a colleague due to time restraints, as well as a roll back task. And an analysis to read the separate log from the force-fix option. Any interest?
1 Like
yes, though you should put them on GitHub in addition to or instead of BigFix.Me just because it makes it easier to accept updates and fixes from anyone. Easier collaboration.
Also, all of the stuff we are posting around Logpresso and JAR replacement is actually being built using templates, so we need to integrate any changes into the templates themselves.
We are working very hard to refine the detection and mitigation stuff. Rollback will be something we consider in the future.
1 Like
I may do this tomorrow then as Its 3am again heh… and yeah I really need to use github for this much more. So much more convenient but I have just been putting it off. Will make a crack at it tomorrow, feel its quite useful additions
Yeah this is the challenge, I had to do something today out of necessity but it actually came out really well.
I will detail it tomorrow as it is mainly enhancements to existing tasks and of course new tasks for rollback and separate analysis for mitigations, including analysing the mitigated files, locked files and numbers of archived zip files (now stored inside BPS-scan folder).
1 Like
FYI, the Logpresso Scan Utility based detection of Log4j fixlet is now live in the BES Inventory and License site.
5 Likes
Thanks. I also see there is a corresponding analysis too. I deleted the analysis in favor of the one in the site to avoid seeing duplicates in webreports.
2 Likes
I think congratulations are in order - nice work by all involved to get to an official published solution in 2 weeks. 
5 Likes
@JasonWalker and myself took this extremely seriously. Jason was on this sooner than me, and I’m thankful for that. We deferred our vacations and put in over 200 hours worth of work combined in less than 2 weeks. We don’t want all of you to suffer over the holidays without stepping up. We also had help from our internal teams doing QA and refinement against as many of our supported and technically unsupported platforms as possible. The one that is in production supports all but 3 platforms and the one on GitHub that uses an preinstalled JRE/JDK should cover those if it detects JAVA. We are not done, though the lack of downloadable JRE on HP-UX does not seem to be solvable anytime soon.
We could not have done this without help and feedback from all of you. Through this very busy thread we found holes in all other approaches that we started down based upon other recommendations and solutions. Thanks to @nicksberger for some specific recommendations and investigations. Thanks to @Mario for doing initial work on a universal JRE solution we ended up shipping.
I also want to mention that @strawgate ‘s and others approach of looking at running processes / file handles is supplemental to this approach, in that it is more narrow and quicker and covers highest priority problems. The main difference is that it can be run more frequently with lower impact.
Also, a ton of thanks to the developers of the Logpresso scan utility. They have been producing multiple updates per day even on weekends, integrating feedback from myself, @JasonWalker, and many others, including passing on feedback from all of you. I’m not a Java developer, but I did look over all of the code at one point and was impressed how compact and straightforward it was. The Logpresso scan utility developer has specifically named HCL BigFix as the first official integration: https://github.com/logpresso/CVE-2021-44228-Scanner#tool-integrations
We don’t know if we will ship it in production but we do have mitigation options that use the same methods on my GitHub right now and we plan to investigate automatic roll back as well all based upon the exact same Logpresso scan utility capabilities. The difference between a full scan, full remediation, and full rollback seems to just be very slight modifications.
Please please please let us know if you come across any breaking applications from mitigation. Also let us know if you have deployed mitigation widely without issues. We think the community can help each other with the confidence to deploy mitigations in ways that we can only do collectively.
Happy Holidays to all. We really mean it.
9 Likes