Log4j CVE-2021-44228 Detection and Mitigation

Hehe that was my thought initially but I have checked over the results myself and its not that clear cut.
It would actually increase their times drastically to remediate using this method, and I actually agreed with them.

Sadly it is a case of one shoe size does not fit all

1 Like

Also, the Logpresso scan utility now has a rollback / restore option. So you can mitigate all, restore what breaks.

From what we can tell, an app breaking is very rare. If they know for certain an app would break from the mitigation, please report it here and/or to the logpresso scan utility.

I would not avoid mitigating an app unless I knew it broke personally.

That said, I would still roll out the mitigation very carefully and slowly, testing and validating as I go to build up confidence that it doesn’t break anything.

I feel like over cautiousness without knowing it will actually break something is not warranted. What is warranted is careful rollout and validation until you know something breaks, rather than assume it will. It breaking something somewhere is probably inevitable. But I don’t like the approach of designing around the assumption that everything will break when that isn’t the case.

2 Likes

I feel this may also be the case, however this is coming back from the vendor, I am interested in checking out more specifics, will update if there is anything interesting found out tho. I am very glad to see the restore option, I wasnt aware of it till 5 and a bit hours ago, but I had a plan to add the option through BigFix, so bit of relief!

Sadly we had had some of this with plain old deletion/renaming already, very early on. I have expressed my concerns before that was attempted but didnt win…

With the mitigation/removal of the JNDI class on 2.x, Id say its a safe bet, much safer then living with the vulnerability potentially.

Testing is a must at least to just make sure its working initially as well as core applications

I’m curious what those issues were, but yes, that is more likely to cause problems so it doesn’t actually reflect poorly on this mitigation method for certain, though also hard to say for certain.

agreed it is a safer way to go.

Yep. You likely have to reboot for the mitigation to be effective, so not only do you have to roll out the mitigation carefully, but then you have to reboot and see.

I thankfully didnt get involved directly in the incidents, they went elsewhere but they were linked to the crude mitigation methods. So I lack the full details but believe applications crashed out when the file was missing causing the service outage.

I am hearing conflicting information on this, so I am not sure what to think yet, but everyone is of course hopeful that the impact is minimal or none. Time will tell!

There has been plenty of work here on mitigation/replacement to not affected versions and its been successful as far as I am aware, so it is promising. Thankfully everyone focused attention to publically available services/servers first. Which is the best approach, then hit everything else/internal asap.

1 Like

A lot of great progress and discussion this morning. Thank you very much @Mario for posting you progress. I’m still reading through it, but it looks very useful and may be a great reference we can use as we move forward.

Late last night we posted some major updates to the content in @jgstew’s GitHub. I’ve detailed the instructions on using, reporting, remediating, and some beginning notes on rolling back in the Summary thread at Log4j CVE-2021-44228, CVE-2021-45046 Summary Page

Please have a look, see which bits are useful and which may need more details. We’re continuing work on adding more JRE downloads and our Content teams are working to produce official content on this. We need to be guided by your feedback on what’s working well and what’s not so great in the community content.

2 Likes

Yeah, this is not a surprising result from just deleting the Log4j JAR file entirely. Either doing the logpresso style mitigation or swapping in the 2.17 version for the older file would be a much safer approach.

It makes me sad that they are now shy to try mitigation methods that are very unlikely to cause breakage because they tried a mitigation method that is almost certain to provide breakage.

My whole point and approach is to try the mitigation methods that are the least likely to cause breakage and easiest to roll back, even if they don’t resolve all CVEs as other approaches. Then move forward with more mitigation after.

This is why I still like the idea of using the Environmental Variable approach even though it is known to not fully resolve the CVEs because it is likely the quickest way to identify breaking applications with the least impact and the easiest rollback.

The next best mitigation seems to be the Logpress approach, both based upon speed, ease of use, reach, and the ability to rollback.

Swapping out the JAR files for Log4j-core for those that are 2.17+ version is the only way to close all of the CVEs completely, but that is actually only possible easily in cases where you actually find the JAR file exposed that you can swap out easily. Otherwise you will miss TONS of applications that have JARs bundled in JARs or similar with this method.

2 Likes

Thanks, hopefully something useful but see how you go, its a big task. Guess its not too bad to have a number of developments even if one wins out over the other or merge parts for the best of worlds.

Saying that, I did a small update to just de-duplicate some stuff thats been bugging me but didnt have time till this second just before bed time and a test on windows and linux… 3:30am :thinking:
https://bigfix.me/fixlet/details/26903

I will be taking a good look in the morning, sounds like I got a bit of catching up to do. Thanks!
Will make feedback/testing as soon as I can too.

1 Like

It’s what community and Open-Source is all about!

2 Likes

Your post, couldnt have said it better myself.

I also wont be surprised if some update eventually comes to Logpresso to replace the jar with 2.17 or 2.12.2 etc

1 Like

There was already a GitHub Issue filed for this here: Question: What is Logpresso with vulnerable 2.16 jar files ? · Issue #173 · logpresso/CVE-2021-44228-Scanner · GitHub

Yeah interesting, 2 days ago, I guess annual leave ignorance has been bliss.
hahahaha surprised you found it, thats a terrible issue name!

1 Like

It was very hard to find. I think I ended up searching for 2.17 to find it.

1 Like

The BigFix Compliance team has released a new version of Compliance, that removes Log4j entirely in the product. Note that we believe there was not a vulnerability due to the way we were using Log4j previously, but updating to this version should remove any concerns for Compliance.

1 Like

I was checking for a corrected version of the hashes for scanning Linux and AIX systems containing log4j files.

if you are only looking for hashes of the log4j-core files or those file by path, then you are missing a tons of log4j vulnerabilities. The Logpresso scan utility will find tons more and help mitigate them. See the summary info about it here: Log4j CVE-2021-44228, CVE-2021-45046 Summary Page

1 Like

This should now download a temporary JAVA JRE/JDK for every BigFix platform except for HP-UX:

We can’t add HP-UX support due to HP not allowing direct downloads of their JRE/JDK. It might make sense to handle the HP-UX case in this same fixlet using a fallback to a system installed JRE/JDK, but there is another fixlet for system JRE that already that I would test on HP-UX instead and give us feedback on if it works there as far as the relevance detecting the JRE or not.

We haven’t finish testing the part that expands the JRE and runs the commands on all platforms. It is some AIX platforms and Solaris that still need work in some cases.

Right now the JRE in the download for 32bit is Win2012r2+ only it seems. Other alternatives that are Win7+ require VC++ redist to be installed. It is hard to find a good option for older end of life operating systems, but that is kind of expected. The right answer in these cases might be use the “already installed JRE” on the OS to do the scan, which is a separate fixlet/task.

Had a pretty rough day with a bit of stuff developed in relation to Log4j but not time to do any reading on the new developments.
Anyways good news is logic wins, and we are not doing the individual server certain path exclusions, or inclusions if you will. The entire server will be mitigated (yay!).

Not sure if it was mentioned but RHEL 6 seems to have glibc 2.12 on 64 bit which happily runs OpenJDK JRE 1.8, but… not sure if its been noticed here as seems everyone is running on latest releases. This is from 2.6.1 updated version of the pre compiled linux scanner on github, I was wondering why my version with jar scanner worked…

So glad I put those glibc checks in anyways. Ill hopefully get a chance to check your updates @jgstew tomorrow, need bed, stat!

EDIT: For under Win2012R2 releases and equivalent desktop versions, I am hoping to find some version of Java 1.7 JRE that is usable but I havent had time to look. Found linux versions of OpenJDK 7/1.7 which I ahvent checked if they are in your task yet? Ideally I want to go down to AIX 6 (5 if possible!), RHEL 5 and its equivalents (3/4 if possible…!), Windows 2008 before R2… (2003 if possible), SunOS 5.9 (5.8 if possible). AIX and SunOS I have researched the least ouf of the lot, but JRE 1.7 should allow at least RHEL 5.x and its equivalents, and thats reasonable enough. I should note I would only be using these unsupported EoL 1.7 versions on the old old stuff, anything else latest 1.8’s

1 Like

This is a major challenge. We can never really say we can definitely support EoL products with EoL versions of BigFix, but if it is trivial to extend support and others can demonstrate how, then great, please help us enhance our templates to do so.

Amazon Correto JRE/JDK8 does seem to go further back in support than Azul or AdoptOpenJDK but it seems to have more requirements for VC++ redist? Correto doesn’t go back further than v8.

We have also been trying to look for solutions here.

Until you know you have something that breaks, this is the right approach if you want to get things mitigated in a timely fashion.

Unless the system is critical to the point of life or death or the system impacts many other systems (domain controller, LDAP, SAN, etc) then it might be better to break it than to have a remote code exploit remain open.

And if something does break, do you roll back everything? or maybe only 1 or 2 things broke and you go fix them by trying to update the JAR to 2.17 or look for vendor patches or something else. Or just roll back that one app.

And if you do find something that breaks, please please please tell us all here so everyone can know about it!

1 Like

Absolutely, and I agree this is the right approach overall, but in real life enterprises still stick with some odd choices and something of this criticality I see myself catering for, its not like they will suddenly upgrade, if they havent already :frowning: sad story… it should be relatively trivial to get old old old OS’s supported for this task in particular just due to the fact that the major dependency is the JRE, and a little bit of extra relevance thats basically copy/paste with some editing and testing of course.

I would be a bit more upset if it meant completely redesigning something from ground up, at least for this task its essentially more of the same.

1 Like