Log4j CVE-2021-44228 Detection and Mitigation

jgstew · December 21, 2021, 4:41pm

ah, so that is one that should be excluded? interesting. We can add that, but also provide feedback to logpresso utility that it should be excluded. Do you know if it shows up as just afs in the mount -l command and/or relevance? I am assuming so?

jgstew · December 21, 2021, 4:43pm

2.16.0 is still vulnerable, just not to the original CVE. You need to be 2.17 to be in the clear. Even 2.15 should be clear of the original very bad remote code execution vulnerability.

If you are on anything other than 2.17 then you have known CVEs you are still vulnerable to, just not as many as 2.14 or 2.15.

keepthemomentum · December 21, 2021, 5:38pm

Thank you.
Does this relevance look for versions below 2.16? Could you please help with that.

it whose (not exists (following texts of last “log4j-core” of it) whose (it as version >= version “2.16.0”)) of (it as string) of (if exists property “locked lines” then locked lines of it else lines of it) whose (it does not start with “SCAN_COMPLETE” and it as lowercase does not end with “-javadoc.jar” and it as lowercase does not end with “-sources.jar” and it as lowercase does not end with “-tests.jar”) of files “BPS-Scans/CVE-2021-44228.txt” of folder(pathname of parent folder of parent folder of client folder of site “actionsite”)

JasonWalker · December 21, 2021, 6:56pm

That looks like what it’s trying to do, but also it’s an older version of what I had in the analysis, and also that’s deprecated in favor of the LogPresso scans and analysis

jgstew · December 22, 2021, 12:43am

FYI this setting to configure the client utility cache to be at least 300BM will likely be a prereq to running the Logpresso scan using the JAR and temporary downloaded JRE on all platforms in the future. It also just a best practice in general to set all clients to have this setting.

See this task to do so:

https://github.com/jgstew/bigfix-content/blob/master/fixlet/clientsettings/Set%20__BESClient_Download_UtilitiesCacheLimitMB_%20to%20_300_%20if%20lower%20%20-%20Universal.bes

Mario · December 22, 2021, 7:35am

Hi all, I made progress, been not working since Friday so this is first day I’m working on this… on my annual leave… worked out plan of attack yesterday and this afternoon coded up the relevance for the action for the currently all JAVA version of Logpresso.
Here is link to current progress: https://bigfix.me/fixlet/details/26903

This is very much in development still so its not production ready but its testing fine so far on what I have tested both Windows and Linux

Notes/brain dump since my last update

Windows version needs updating to specify Server 2012R2+
Windows version will get 1.7JRE for less then Server 2012R2 versions
Replace windows versions once redone with openjdk 8 instead of 11
All Windows relevance needs to be redone based on above 2 things
All task relevance needs to be still done… This will never work on them 3 or 4 for example, unlike the conventional earlier native find command version
This is not using “utility” but I think I will change it over to use it, just been busy coding this and making it work plus researching linux/jre issues/dependencies
All versions for each OS use around a 40MB JRE download to function obviously…
This does not work yet with Debian/Ubuntu but it will be really easy to do so, just havent had the minutes yet…
Fedora will be added in too as I found the dependencies to specify
No Mac yet…
Few Linux versions, SunOS and AIX in here, using only JRE1.8/OpenJDK 8/11 supporting multiple architectures
Linux will likewise also receive 1.7/OpenJDK 7 versions so it works down to RHEL 5.11/glibc 2.4+ and this is lightly pre provisioned already
I have included glibc checks for linux where needed including some light pre provisioning for earlier versions of JRE.
I would like to borrow @jgstew line for detecting the shell command, but not done it yet
It works straight away with the logpresso analysis as I changed the folder to use BPS-Scans
It appears all linux, aix and sunos versions at least that support jre1.7/1.8 can run the same tar commands to extract jre as well as to run the logpresso scanner. So that will be tidied up eventually, currently the extraction for the JRE it is set up to be able to use different commands… just seems to be unnecessary thus far
I would also like to attempt a detection of java from the path for *NIX if possible to save downloading a JRE needlessly, which would need a version check for >= 1.7
Considering and most likely will add more options to description sctren, like the force fixing to strip jndi class exclusions and other useful things

Lots more work here but its a start

nicksberger · December 22, 2021, 9:53am

If anyone is using this, I’ve added a 5 minute timeout to terminate the LSOF process.

override wait
timeout_seconds=300
disposition=terminate
wait /bin/sh -c “(lsof / | grep log4j > /var/opt/BESClient/BPS-Scans/lsof_results.txt)”

jgstew · December 22, 2021, 2:34pm

Definitely a good idea to add a timeout to things like this.

jgstew · December 22, 2021, 2:36pm

We are actually hard at work on this and hope to make significant progress by end of day.

Only part of the progress is public at the moment: bigfix-content/fixlet/Logpresso Log4j2-scan - Universal - JRE.bes at main · jgstew/bigfix-content · GitHub

We actually need to find a source for JRE/JDK downloads for IBM Processors and HP-UX. This is the main thing we are missing, plus some smaller relevance and actionscript enhancements.

The earlier relevance I had for this had a bug that Jason found and have since fixed.

Some versions of AIX cannot all use the same command it seems and require using gunzip first. We aren’t sure if any other older Unix will need the same, but likely. Hard to figure out what the requirement is and how to write relevance to handle the case where some need it and some don’t.

We are preferring to handle this in a separate fixlet/task. See here: bigfix-content/fixlet/Logpresso log4j2-scan - Universal.bes at main · jgstew/bigfix-content · GitHub

The goal is for this fixlet/task to be relevant on all systems that have a JRE/JDK already installed and detect it through a combination of inspecting the PATH variable folders for java as well as some likely folders on many OSes.

To help mitigate that the lookup for JAVA on the system is a bit slow, we have added this MIME field that prevents the fixlet/task applicability relevance from being evaluated every loop:

		<MIMEField>
			<Name>x-relevance-evaluation-period</Name>
			<Value>01:00:00</Value>
		</MIMEField>

I would consider this a best practice for any applicability that could take more than a fraction of a second in general.

strawgate · December 22, 2021, 2:44pm

My enhanced content is all written in an OS Agnostic way (it doesnt scan the full drive which removes the need for OS-Specific exceptions) – the only OS-specific content is the prefetch and expansion of the JRE so if anyone can contribute which openjdk JRE worked for them and what command they used to expand it, my content makes it trivial to support additional OSes.

Once I have those I can wire them up into the existing content and everything “just works”. Feel free to message me or make an issue here with the info: https://github.com/VerveIndustrialProtection/CVE-2021-44228-Log4j/issues

All I need is this for your desired platform:

if {architecture of operating system = "x86_64"}
	prefetch openjre.tar.gz sha1:6A30C541FB7D141F8D34A1F2D5CA3B70FCF437FF size:41045860 https://builds.openlogic.com/downloadJDK/openlogic-openjdk-jre/8u292-b10/openlogic-openjdk-jre-8u292-b10-linux-x64.tar.gz sha256:B51EF522D35B23EE2469BB2324B365FFDD9773420329ECE675BDFCF1B9365AE4
else
	prefetch openjre.tar.gz sha1:2F9A0AC58A8A212473521321242E07CC76BD0EFE size:40585042 https://builds.openlogic.com/downloadJDK/openlogic-openjdk-jre/8u292-b10/openlogic-openjdk-jre-8u292-b10-linux-x32.tar.gz sha256:57211788D6B1BEA3AE0C47F61A97F3259FFA91019A29AB3055AD7C580ADF02CE
endif

wait tar -flags "__Download/openjre.tar.gz" -C "__Download"

continue if {exit code of action = 0}

Just let me know!

jgstew · December 22, 2021, 2:47pm

We have found that we need to add a check for properties availability for some of the required architecture checks.

Not all of these work without adding the property check guards around them, and they haven’t been fully tested and validated yet, but that is in progress:

Solaris x86:

((exists match (regex "SunOS") of it ) of name of operating system) AND (type of main processor contains "i386")

Solaris Sparc:

((exists match (regex "SunOS") of it ) of name of operating system) AND (type of main processor contains "sparc")

SuSe SLES x64:

(exists match (regex "SuSE") of name of operating system) AND ("x86_64" = architecture of operating system)

SuSe SLES x86:

(exists match (regex "SuSE") of name of operating system) AND (exists match (regex "^i[6-9]86$") of architecture of operating system)

SuSe SLES ppc64be:

(exists match (regex "SuSE") of name of operating system) AND ((if ("ppc64" = architecture of it) then (big endian of it) else (false)) of operating system)

SuSe SLES ppc64le:

(exists match (regex "SuSE") of name of operating system) AND ((if ("ppc64le" = architecture of it) then (little endian of it) else (false)) of operating system)

SuSe SLES s390x (Z):

(exists match (regex "SuSE") of name of operating system) AND ("s390x" = architecture of operating system)

AIX: (no arch check needed it seems)

(exists match (regex "AIX") of name of operating system)

Raspbian:

(exists match (regex "Raspbian [a-zA-Z/]* (9|10)") of name of operating system) AND (((it = "armv7l") AND (family of main processor >= 4)) of architecture of operating system)

RHEL Family / Ubuntu Family / Linux on ppc64le:

(name of operating system as lowercase contains "linux") AND ((architecture of it = "ppc64le") of operating system)

RHEL Family Linux (including IBM PowerKVM) on ppc64:

(name of operating system as lowercase contains "linux") AND (architecture of it = "ppc64" AND big endian of it) of operating system

RHEL Family on s390x (Z):

(name of operating system as lowercase contains "linux") AND ((architecture of it = "s390x") of operating system)

These relevances are pulled from the BigFix client upgrade fixlets, but then made more generic to better match the case of the JRE/JDK needs as far as I know. Also, the BigFix client upgrade fixlets are actually more restrictive on relevance than they need to be even for BigFix itself. Our client technically works on more platforms that we “officially” support so the preference is for these to error on the side of being relevant than unrelevant.

strawgate · December 22, 2021, 2:49pm

Thanks for the relevance! I’m not intending to create universal detection Fixlets (just universal results / analyses) which makes this a little easier

Mario · December 22, 2021, 3:15pm

Bit to unpack here, I was aware you guys are developing something but I dont know what and what stage of course. I believe I have the basis and the working out and general structure to support multiuple OS and architectures in place quite easily and well (some cleanup and de-duplication required that I am fixing now).

I have additionally glibc relevance checks that I feel should be handy, as they are requirements to at least the Adoptium builds of OpenJDK JRE’s. I am not sure how you are handling this part but as mentioned I feel like my way is very appropriate and also easy to expand (check the non windows section) expecially as many Linux OS can fit into the same mold.

Ah right, is the one in the Universal JRE working I guess? I do like the concept of it

I had the same concerns which is why I left the non windows section as is currently till confirm/add more OS types and testing is completed. This method should be able to reduce the command execution to some number, instead of having duplicates.

Im not disagreeing but to me it seems it would be beneficial to run less actions, and less choosing, sorting computer lists into separate tasks. Unfortunately BigFix skills are not superb with the teams I may be dealing with and this would result in more confusion then it already is.

Additionally all results feed into one analysis, so all data is created into one report and consumed as one. Despite it having OS data in it, its just easier to handle a scan task in one hit and let the relevance/script take care of what gets processed where. It just seems like it may be less problematic overall.

jgstew:

To help mitigate that the lookup for JAVA on the system is a bit slow, we have added this MIME field that prevents the fixlet/task applicability relevance from being evaluated every loop:
	&lt;MIMEField&gt;
		&lt;Name&gt;x-relevance-evaluation-period&lt;/Name&gt;
		&lt;Value&gt;01:00:00&lt;/Value&gt;
	&lt;/MIMEField&gt;
I would consider this a best practice for any applicability that could take more than a fraction of a second in general.

That is pretty handy, was not aware of it as didnt look it up, always suspected there may be such a “setting”

I need to take a good look at the JRE version you guys have in github

jgstew · December 22, 2021, 3:21pm

I prefer many fixlets than a single fixlet that tries to do too much, but I do take your point here.

If you run any of them and get results, then it should make all of them non-relevant, so the relevance helps prevent more than one being run at the same time. The other issue is we are finding that the JRE method works in places where the windows binary does not.

I want to differentiate between a run that is doing mitigation with a run that is doing detection only by examining the results file so you could run mitigation right after detection. I have not done this yet.

Mario · December 22, 2021, 3:34pm

Its a tough world haha, there is no right or wrong way here, I am easily open to either idea if its executed well

Yeah I realised this early on, its a bugger but a fact, the binary files are not uber reliable in every case.
Perhaps a binary run should be made and the dependency error (or other errors) detected and a client setting “set” to prevent the binary task being available to that endpoint?

Still this increases maintenance of the actions especially if ad hoc currently, one would have to know which task to handle per server, this still makes me uneasy about having multiple tasks for tye same job essentially.

I am just off to bed but tomorrow I need to make a mitigation specific task, my plan is to actually have a separate fixlet that can be loaded up with a site file CSV that basically has the endoint name, path to remediate.
It will almost be a copy of the task I presented here: https://bigfix.me/fixlet/details/26896
Of course modified to suit plus site file usage which is pretty trivial.

This list would come from the results analysis thats been through review and change process, its not something that should be done automatically, even tho it could.
Idea is then this task creates a “include path” list that can be fed into the mitigation --force-fix task command.
It would use this input path file sitting somewhere in the BPS-Scan folder and only be relevant if that file exists.

Advantage being that if the endpoint has 7 log4j vulnerabilities, and 2 have valid exemptions from Oracle for example, then only 5 log4j’s need the mitigation. This would save time on scans but also ensure only the necessary log4j’s are backed up and mitigated.

jgstew · December 22, 2021, 3:37pm

we already have mitigation specific tasks. They are all generated from the same template as the detection tasks, but with slight modification to include the remediation command line and title change. They are already on the github.

When we build them, they get the newest logpresso utility included or the newest JAR for the replacement one automatically.

In general I would hope you could mitigate everything everywhere in most cases, and only have to go through this effort if there are known issues with applications that you consider low risk.

Mario · December 22, 2021, 3:39pm

Ill check it out cheers! Just need to see about the last part of my earlier post about specifying paths to include, ie, not all will be mitigated potentially, at least at this stage anyways.

Got told there is some specific products from Oracle that do not have the vulnerability and should not be touched. However I do not have first hand information, only what was passed down to me quickly over the phone.

jgstew · December 22, 2021, 3:40pm

Technically you could already do this with our existing mitigation ones by specifying paths to exclude and exclude the ones you know you don’t want to mitigate.

There is an HTML form in the description of the fixlet you can enter what you wish to exclude.

Mario · December 22, 2021, 3:42pm

Agreed, however the teams are requesting a bulk task.
So specify 20 or 100 or even more servers at a time as they get change approval, instead of working action by action for each endpoint.

jgstew · December 22, 2021, 3:43pm

True, but if they have common exclusions, they could be mitigated all with the same task. It is only if one server has a path that should be mitigated and another server has the same path that should not be where you need to do anything specific.