Log4j CVE-2021-44228 Detection and Mitigation

nicksberger · December 17, 2021, 9:58am

//CVE-2021-44228 - Log4j2 - Vulnerable files

unique values of (it as trimmed string) of preceding texts of firsts "," of following texts of firsts "vulnerability in " of (if exists property "locked lines" then locked lines of it else lines of it) whose (it as lowercase does not contain " (mitigated)" as lowercase) of files "results-log4j2-scan.txt" of folders "BPS-Scans" of parent folders of parent folders of client folders of sites "actionsite"

//CVE-2021-44228 - Log4j2 - Mitigated files

unique values of (it as trimmed string) of preceding texts of firsts "," of following texts of firsts "vulnerability in " of (if exists property "locked lines" then locked lines of it else lines of it)  whose (it as lowercase contains " (mitigated)" as lowercase) of files "results-log4j2-scan.txt" of folders "BPS-Scans" of parent folders of parent folders of client folders of sites "actionsite"

Mario · December 17, 2021, 11:06am

I’m working my way to get the Java version for all OS. There is a lot of OS not covered by the precompiled options.

The jar should work on any os and architecture

jgstew · December 17, 2021, 12:03pm

Whoops, I forgot to use the newer locked lines inspector! My bad!

I have seen some very fast scan times on my NVMe systems, but 1.26 seconds does seem a bit fast for that many files.

Example:

Logpresso CVE-2021-44228 Vulnerability Scanner 1.6.3 (2021-12-16)
Scanning directory: /

Scanned 3462 directories and 35526 files
Found 0 vulnerable files
Found 0 potentially vulnerable files
Found 0 mitigated files
Completed in 3.15 seconds

This seems less surprising to me given the number of files being scanned on NVMe, but your result seems somewhat plausible.

I’m still working on a solution for non x64 based systems to run the JAR directly instead of using the Linux or Windows binary which will only work on Intel/AMD x64 systems currently.

yep, that is what I should be doing, good call. You want to file a pull request against my analysis in GitHub?

I’m planning on doing this today. What do you have so far?

FYI, there have been 4 releases of the log4jscan utility in the last 19 hours, including this important addition:

Support Log4j 1.x CVE-2021-4104 vulnerability scanning using --scan-log4j1 option.

So now it supports detection of the CVE within the EOL Log4j 1.x versions. I’ll look to add this to the default command shortly.

Also this:

Redefined exit code
-1 failed to run
0 for clean (No vulnerability)
1 for found
2 for some errors
Use --old-exit-code for legacy automation.

I’ll be updating the scan tasks to use this newer version soon, as well as incorporating a method for custom path exclusions.

Mario · December 17, 2021, 12:41pm

Soooooooo, I am actually not that far into non Windows as I would have liked today…

However, I started the all Java approach on Windows and I am very happy with it thus far.
Its not tested on anything apart from Windows 2012R2 and Windows 2016, however the reason why I mention this is because it will be largely re-usable with the NIX stuff, with just some commands slightly different, unzip vs tar etc and how its run of course, but the base structure is set up.

If you would like to have a look, here it is attached to this post. I am not opposed to using the compiled versions for x64 linux/windows but for now it was easier for testing for me to start on windows with java, can change that to use precompiled versions easily later.
Log4j logpresso scanner WIP 006.1.bes (15.3 KB)

Its almost midnight here and I am wasted from this week. If you think this is usable to port to other OS, which I believe should be doable, please let me know. I plan to continue it when I can but again, if you feel it is suitable as a foundation to cross over to linux flavours, and feel like it, go for it heh.

Please note:

I just did a manual update to latest logpresso jar
Please note the description screen, option for keeping JRE on the system or not
Additionally a quick 2 fixlets yet to be made to disable or enable scan of particular systems, this check is already included in the relevance of this task
The non windows section is basically rubbish placeholder for now as its not developed yet, its in fact a copy of some select stuff from BFI scanner install/upgrade task just to have the relevance for various linux flavours and architectures without having to think about it too much, but has to be verified of course…
The relevance is mostly stuff that wont be there in the future but its just to keep it relatively happy and ill change it later, the only good part is the check to not run on systems where scan is disabled…
Possibly more stuff I forget, but im really quite fried now…
The OpenJDK I settled on using for JRE due to licensing, requires glibc 2.12 for x64 linux and 2.17 for anything else, I would like to incorporate a check for this in the relevance but not got around to it yet.
The linux section include/exclude parameters are from before the updates came to logpresso to exclude certain fs types, and will not be used
There is some more then neceessary relevance for the JRE folder, but its there in case multiple versions of JRE end up on the system, to use the latest available JRE if multiple exist, as I have option to keep JRE for future scans, to prevent re-downloading
x86 Windows JRE is included, hoping to test some out, hoping to also go down to Windows 2003… still got plenty of these for some reason in the environments
I am hoping that I can get linux down to 8.2 client level, but it may depend on the JRE I guess, specifically the glibc requirement, AIX 5.1… SunOS from year 1879, z390 from yesteryear, etc… we seem to collect old hardware… so I am trying to maintain down to 8.2 as best as possible
Need to add log management, like 10 keep logs, or 10 days worth or even just last 2… etc

If you think its worth a laugh, then please do so

Also, my plan is to use JRE from https://adoptium.net/releases.html?variant=openjdk11&jvmVariant=hotspot for anything that can
Temurin 11 JRE compressed archives only for anything non SunOS, Temurin 8 for SunOS, JRE for all due to being small, around 40MB

jgstew · December 17, 2021, 1:32pm

Same.

You don’t need to do this because you can put the JRE in the client utility cache and then it would not be redownloaded even if you don’t leave it behind. Any solution I end up cobbling together will not leave JRE behind and will not give the option to leave it behind. Leave nothing behind that is a future problem.

You will see the client utility cache is being used for the existing Windows scan task for the Unzip.exe file. I recommend setting all clients to have a client utility cache that is at least 200mb, 500mb preferred. The default is a useless 10mb.

Any small utility prefetch that is shared across multiple fixlets/tasks should be placed into the Utility Cache. You must use the utility actionscript command referencing the exact file that was prefetched.

Example:

// Add Unzip to Utility Cache:
utility __Download\unzip.exe

Client Setting Info:

set _BESClient_Download_UtilitiesCacheLimitMB to 500
List of settings and detailed descriptions
bigfix-content/fixlet/clientsettings/Set __BESClient_Download_UtilitiesCacheLimitMB_ to _500_ - Universal.bes at main · jgstew/bigfix-content · GitHub

I just finished automating this.

My fixlets/tasks are being automatically generated using templates that automatically get the newest logpresso utility prefetches inserted.

In my testing JRE 11 is the maximum. I think JRE 7 is the minimum but I have not tested this.

Mario · December 17, 2021, 1:42pm

Ah yes, it did occur to me but for some reason I have not tested how it deals with updates, I should have tested it. This will ahve to keep running possibly forever into the future (updates, inadvertent old installs, you name it).
So it needs to deal with JRE updates. I don’t mind keeping this way as an option (till i get some proper sleep to think about it), but the utility is probably the most correct method.

Mario · December 17, 2021, 1:44pm

Ah yes, that is very nice indeed! There is a great many updates, which is excellent.

Suspected this may be the case but definitely not enough testing on my end yet

jgstew · December 17, 2021, 1:46pm

From: Release 2.1.0 Release · logpresso/CVE-2021-44228-Scanner · GitHub

“From now on, JDK7 is the minimum supported version.”

For my first pass at this, I planned to actually have a task that is only relevant on machines that already have a JAVA binary in the PATH env var and reference the first JAVA found in PATH. This will not work if that JAVA is not > 7 and < 11 so that is not the best solution. Actually downloading a portable JRE to run it is more likely to work consistently.

Mario · December 17, 2021, 1:52pm

This is precisely the reason I went with this method instead of hunting java around the system.

My original thought was similar to yours tho, to see if its in the path, then to do a file based search for java, then finally to check the version of it.

However… this is madness perhaps… a lot of problems potentially. Even if the path once found is stored in a variable for reuse, would need re-testing it is still there, still version is correct, still scavenger hunt to find other versions if its not there, then end up at the download portion anyways after all that.

The download of a non installable java version just struck me as less problematic, problematic from perhaps a change management perspective but this problem will have priority clearance at all times I guess.
Not to mention the java would have to be downloaded anyways even in the scavenger hunt version if its not on a system at all, so not much better from change perspective.

JasonWalker · December 17, 2021, 1:55pm

I honestly don’t see much of a problem in sticking with a specific JRE version (for each platform) and using that version indefinitely – I don’t think it’s necessary to download the latest available JRE at any given time.

The issues with Java vulnerabilities stem from running untrusted code or allowing applications to interact with the JRE, ie network sockets. In this case we are only using the JRE to execute one specific application, that does not provide interaction with other processes, and we are removing the JRE when finished.

Mario · December 17, 2021, 1:58pm

From what I read, if I remember correctly in this state of mind is that Temurin/OpenJDK 8 is supported to 2026, 8 is the only one supporting SunOS

jgstew · December 17, 2021, 1:59pm

This clean up part is actually kind of complicated on Windows, easier on non-windows if we can use /tmp but that has other problems. I would probably use /tmp at first on non-windows.

We can’t rely on the bigfix client clean up of the __Download folder without using a wait command for the scan, which has the drawback of holding up the client for an unknown amount of time.

Agreed. I would recommend using the newest LTS JRE that supports the utility.

Mario · December 17, 2021, 2:00pm

Eh? If im not mistaken, my bes file I posted up removes it on completion (if you dont tick to keep it) both the download and the extracted versions.

jgstew · December 17, 2021, 2:01pm

We are speaking theoretically at the moment. I don’t think we have looked at your BES files yet.

Mario · December 17, 2021, 2:02pm

Oh right, check it out when you have time (wait lol!)

nicksberger · December 17, 2021, 2:28pm

@jgstew how do i grab your latest scanner with updated logpresso binary. Just checked Github, still has 1.7.0

bigfixforum · December 17, 2021, 2:31pm

This is what I did to scan the jar for AIX/SunOS/ Linux.

If the server has NFS/NFS4 or bind, “find” command takes forever and NAS chokes. we have tons of server which has CPU spike due find perusing the NAS share . “locate” is more efficient way of finding any file if “mlocate db” is present otherwise “find” is alternative.
And sample output looks like this

AIX:
/app/IBM/oracle/product/12.1.0/client_1/oui/jlib/jlib/log4j-core.jar,log4j-core
/app/IBM/oracle/product/12.1.0/client_1/oui/jlib/jlib/log4j-core.jar,log4j-core

SunOS:
/usr/lib/rda/da/lib/log4j-core.jar,log4j-core
/var/tmp/10_Recommended/patches/150618-02/SUNWocmu/reloc/ocm/ccr/lib/log4j-core.jar,log4j-core
/opt/ocm/ccr/lib/log4j-core.jar,log4j-core

Linux:
/opt/tomcat8/webapps/ROOT/WEB-INF/lib/log4j-core-2.14.0.jar,2.14.0
/opt/tomcat8/webapps/api/WEB-INF/lib/log4j-core-2.15.0.jar,2.15.0
/opt/tomcat8/webapps/cybertest/WEB-INF/lib/log4j-core-2.14.0.jar,2.14.0
/opt/tomcat8/webapps/page/WEB-INF/lib/log4j-core-2.15.0.jar,2.15.0

if [ -f /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt ]
then
                /bin/rm /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt
fi

if [ `uname -s` = "SunOS" ]
then
       for mtpt in `egrep 'zfs|ufs' /etc/mnttab|awk ' { print $2}'`
        do
                 for path in `find $mtpt -name "log4j-core*" 2> /dev/null`
                 do
                        filename=`echo $path |awk -F/ ' {print $NF}'`
                        ver=`echo $filename|sed 's/log4j-core-//g'|sed 's/\.jar//g'`
                        echo "$path,$ver" >> /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt
                 done
        done
elif [ `uname -s` = "AIX" ]
then
        for mtpt in `mount |grep jfs|awk ' {print $2}'`
        do
                 for path in `find $mtpt -name "log4j-core*" 2> /dev/null`
                 do
                        filename=`echo $path |awk -F/ ' {print $NF}'`
                        ver=`echo $filename|sed 's/log4j-core-//g'|sed 's/\.jar//g'`
                        echo "$path,$ver" >> /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt
                 done
        done


elif [ `uname -s` = "Linux" ]
then
        lct=`which locate 2>/dev/null`
        if [ ! -z "$lct" ]
        then
                /etc/cron.daily/mlocate
                for file in `$lct log4j-core`
                do
                filename=`echo $file |awk -F/ ' {print $NF}'`
                #ver=`echo $filename|sed 's/log4j-core-//g'|awk -F. '{ print $1"."$2"."$3 }'`
                ver=`echo $filename|sed 's/log4j-core-//g'|sed 's/\.jar//g'`

                echo "$file,$ver" >> /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt
                done
        else
          for mtpt in `egrep 'xfs|ext' /etc/fstab |awk ' {print $2}'`
          do
                 for path in `find $mtpt -name "log4j-core*" 2> /dev/null`
                 do
                        filename=`echo $path |awk -F/ ' {print $NF}'`
                        ver=`echo $filename|sed 's/log4j-core-//g'|sed 's/\.jar//g'`
                        echo "$path,$ver" >> /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt
                 done
          done
                        #echo "locate notFound" > /var/opt/BESClient/BPS-Scans/CVE-2021-44228.txt
        fi

fi

jgstew · December 17, 2021, 2:43pm

Many changes incoming. Haven’t posted latest to github yet, but working on it now.

The Logpresso scan utility works for JRE 7 through JRE 17+, so basically JRE 7+. My earlier failures building it on JRE 16+ were my fault. See here: Maximum Supported JRE version? · Issue #124 · logpresso/CVE-2021-44228-Scanner · GitHub

jgstew · December 17, 2021, 2:53pm

Newest versions posted now. The Linux one now supports custom path exclusions thanks to @JasonWalker but that has not been added to the Windows one yet.

bigfix-content/fixlet/Logpresso log4j2-scan - Linux x64.bes at main · jgstew/bigfix-content · GitHub
bigfix-content/fixlet/Logpresso log4j2-scan - Windows x64.bes at main · jgstew/bigfix-content · GitHub

jgstew · December 17, 2021, 3:00pm

We have basically abandoned this approach in favor of the Logpresso Log4J2 scan utility because it finds much more vulnerabilities than looking for log4j-core JAR files because it also finds the vulnerability in JAR files that contain many different libraries as well as EAR and WER files. The utility also offers a command parameter to fix the vulnerability automatically.

The Logpresso utility should automatically avoid all symlinks and all network shares as we have the command crafted in our existing fixlets.

log4j2-scan --scan-log4j1 --no-symlink --no-empty-report --exclude-config file-with-custom-excludes

The newest log4j2-scan utility will avoid NFS / CIFS automatically as well as some others, but our fixlets build a list of path excludes that are more expansive than what the log4j2scan should exclude by default, plus allow you to specify custom excludes.

Right now we don’t have a solution for running Logpresso utility on AIX/SunOS etc, but that is being looked into by either using an existing JRE on the system, or by prefetching a portable JRE to use for the scan then clean up after.