Log4j CVE-2021-44228, CVE-2021-45046 Summary Page

Please see the disclaimer on our Community Content efforts in detecting and mitigating these Log4j vulnerabilities at Log4j Vulnerability Identification and 3rd Party Remediation Solution Testing Statement

Posting this Summary page to link to latest approaches and content. Comments here will be locked, but refer to the megathread below for the latest discussion points:

Megathread with ongoing discussion:

Latest notes from Apache:
https://logging.apache.org/log4j/2.x/security.html

CVE Links:

https://nvd.nist.gov/vuln/detail/CVE-2021-45105

https://nvd.nist.gov/vuln/detail/CVE-2021-4104

US CISA Guidance:

Affected BigFix Products:

Current guidance:

Upgrade to: Log4j-core-2.17.0.jar
or
Remove the affected JNDI class from earlier versions of Log4j-core-2.x.x.jar

…in any release other than 2.16.0, you may remove the JndiLookup class from the classpath:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

( from Log4j – )

Note: Details are emerging, but CVE-2021-45105 may not be remeditated by removing the JndiLookup.class. That workaround is not listed on Apache’s site, and given that it affects 2.16.0 that already had the lookup removed it is likely only remediated by upgraded to 2.17.0. CVE-2021-45105 is a Denial-of-Service with CVSS score 7.5.

(edits)
2021-12-18 Added CVE-2021-45105

4 Likes

Poll on which workarounds, if any, you’d like to see us develop:

BigFix Inventory Team has created custom BFI Signatures. If you have BigFix Inventory, you may import these custom signatures into your server, run an Import to generate a new catalog, and then scan your endpoints using the new Catalog.

This method has the benefit of centralized reporting in Inventory, as well as respecting all processor throttling and remote mount exclusions.

https://bigfix.me/signature/details/1244
https://bigfix.me/signature/details/1247
https://bigfix.me/signature/details/1248

3 Likes

BigFix Compliance team has produced Fixlets to mitigate the issue in BigFix Compliance. See

1 Like

Latest Logpresso-based content with thanks to @jgstew.

All Logpresso / Log4jscan content is use at-your-own-risk

Logpresso has produced an open-source scanner. HCL does not review the source code or the compiled binaries provided by Logpresso. Use Logpresso tools at your own risk.
That said, Logpresso is a powerful tool to scan systems, identify the affected Java libaries, and optionally can mitigate the vulnerability by removing the JNDI class from the JAR file. A backup of the original file is preserved in case there is need to rollback the change.

While HCL cannot endorse Logpresso, for your own evaluation it is worth noting the tool has been referenced by the wider security community, including at https://www.cisecurity.org/log4j-zero-day-vulnerability-response/ and https://docs.microsoft.com/en-us/azure/databricks/kb/libraries/verify-log4j-version

Expand for Details (deprecated)

Here is an analysis to collect the results of the Logpresso scan utility:

Here is the task to run the utility on Windows:

Here is the task to run the utility on Linux: (I am not certain I am sufficiently excluding network drives!!!)

Right now, these do not make changes, these are reporting only. You can modify them to do the fix by adding to the command, but proceed with caution!
And again, specifically on Linux, I don’t think I am excluding all possible network drives! You should proceed with caution if you know your linux devices have remote mounts / network drives!

Please provide feedback on your testing, especially any refinements you would recommend for excluding obvious network drives or other folders you think should be excluded on Linux or Windows here: Log4j CVE-2021-44228 Detection and Mitigation

Edit: For tips on how to download files from GitHub, see https://forum.bigfix.com/t/tips-downloading-files-from-github/40283
2 Likes

Shell script scan for log4j-core-X.jar

Expand for Details (deprecated)

WARNING: The following content is deprecated, outdated, or problematic but retained here for reference

Warning
The ‘find’ search on UNIX/Linux systems may not completely eliminate scans of NFS systems (especially where Automount is in use), and does not exclude remote CIFS mounts. Scanning a large number of machines could overwhelm NFS/CIFS servers.

Warning
This early version of detection method has known limitations. It does not scan content inside WAR, JAR, or EAR archives, it only finds the “log4j-core-X.jar” file visible in the filesystem list. This content is deprecated in favor of the Logpresso scans linked below.

This CVE can affect Log4j components earlier than version 2.16.0. Log4j is embedded in many Java products.
We have developed an alpha Scan Task and Results Analysis to attempt to identify problematic Log4j components.

Considering that this method has not yet been widely tested, I’d welcome anyone who can help test, provide feedback or improvements. The Scan should run on Windows, Mac, or Linux systems, though I haven’t been able to test on all OS flavors yet. Looking for community improvements as we go.
Newest Versions:

Scan Task: https://bigfix.me/fixlet/details/26897
Analysis Results: https://bigfix.me/analysis/details/2998668

Known issues:

The Scan Task on Linux/UNIX systems excludes NFS mounts, but CIFS mounts are not excluded and may produce heavy load on CIFS servers when multiple clients execute scans. Workarounds under investigation.

12/16/2021 21:05 UTC - Updated Analysis and link here. Add hashes for 2.12.2 and 2.16.0. Add 2.15.0 as “Potentially Vulnerable”. Exclude 2.12.2 as “Potentially Vulnerable”.

12/16/2021 23:20 UTC - Updated Analysis and link here. Results for “Potentially Vulnerable” were incorrect on some platforms due to differences in the <version> of <string> cast. Replaced with a Regular Expression that should work better on more platforms.

  • Note this property is likely to be removed in a later version, in favor of just displaying the files found without attempting to judge which are “vulnerable” as new CVEs are being reported.

Setting system-wide environment variables for Windows:

Expand for Details (deprecated)

WARNING: The following content is deprecated, outdated, or problematic but retained here for reference

I cannot stress enough that this is “use at your own risk”. Any individual Java startup script could remove or override the environment settings. Even where these environment variables remain in effect, it is clear that these settings do not completely remediate the risk

I consider this workaround Deprecated, but based on customers asking for the content anyway, here is a link to setting the particular system-wide variables for Windows systems.

This Fixlet applies the following Environment Variable values on Windows:

JAVA_TOOL_OPTIONS = -Dlog4j.formatMsgNoLookups=true (preserving any existing entries)
JDK_JAVA_OPTIONS = -Dlog4j.formatMsgNoLookups=true (preserving any existing entries)
_JAVA_OPTIONS = -Dlog4j.formatMsgNoLookups=true (preserving any existing entries)
LOG4J_FORMAT_MSG_NO_LOOKUPS = true (overwriting any existing entry)

available here: https://bigfix.me/fixlet/details/26898

Light weight active log4j use detection

Longtime community member @strawgate has published BigFix content that takes a novel approach to detecting vulnerable Log4j libraries that are in active-use. This is a much less resource-intensive approach than full filesystem scans. While this does not remove the need for full system scans, this approach appears to be light enough to watch the running system over time with much less resource impact to the system.

The same caveats as earlier apply - this is not BigFix-managed content, etc.

We are linking with our thanks to his content at the following location:

For discussion on this method, please see the following forum thread:

3 Likes

Latest Logpresso-based content with thanks to @jgstew and @strawgate
Update: Monday, 12/20/2021 15:45 UTC

Good Monday morning, BigFix’ers, it’s been a busy weekend for us as I’m sure it has been for many of you. We have lots of content updates to announce and summarize today.

All Logpresso / Log4jscan content is use at-your-own-risk

Analysis to collect the results of the Logpresso scan utility:

Significant updates on the Analysis to retrieve results of the logpresso-log4jscan process were made over the weekend.

The property list is expanded to include

  • log4j Scan - Vulnerable File List
  • log4j Scan - Scan Technology
  • log4j Scan - Last Scan Time
  • log4j Scan - Vulnerable File Count
  • log4j Scan - Potentially Vulnerable File Count
  • log4j Scan - Mitigated File Count
  • log4j Scan - File Result Details
  • log4j Scan - File Scan Count
  • log4j Scan - Scan Duration

The “File Scan Count” and “Scan Duration” are useful in identifying failed scans or scans that may have included network paths - a scan with 0 files or empty result probably failed to execute, and a scan with a long duration may have crossed network paths or scanned very large filesystems.

The “Scan Technology” entry includes the Logpresso-Log4jscan version that was executed.

The “File result details” entry contains the full line from the logfile, including the version of Log4j-core that the Logpresso scanner recognized, if any.

Scanning Tasks

The Scan Task falls into three categories:

Common to all these methods are some known challenges & risks. While both log4jscan itself, and our Fixlet Content, are making efforts to avoid scanning network-mounted paths and symlinks, we cannot guarantee that those exclusions are completely effective. Especially on UNIX & Mac systems, it can be difficult to determine whether a filesystem is local or remote in a generalized way, as the OS hides some of those details from the user/process; and in the case of technologies like Automount, those remote filesystems may not even appear until they are accessed. Please Note that the Linux, Mac, and Java-based scans include the ability to add your own filesystem exclusion paths in the Description tab of the fixlet. These paths are added to our own list of generated exclusions when executing the scan.

More Community Content

In parallel, longtime guru and contributor @strawgate and Verve Industrial are producing content for low-impact scans. Rather than scanning the entire filesystem, their unique approach starts by checking which files are in use by a process - which can perform much faster/more-frequent scans that can help show where you’re most vulnerable. That content is described at Free Tools: Multiple Log4j Detection Mechanisms + Remediation . Be sure to check that out and evaluate whether Verve’s solutions are right for your. In particular in at least some areas he’s running ahead of our content on Remediation.

Forward Plans

  • We will be refreshing the Scan content frequently as new versions of Log4jscan are released. The current content is based on Log4jscan 2.3.6; 2.3.7 has already been released, adding even more network filesystems to the default exclusions, and we will be building for that version shortly. There have been five version releases in the last 24 hours.

  • We are asking for community members to let us know about their experiences with the scanning content that is available today. In particular, we need to hear if any of this is still scanning over network paths where we want to prevent scans from overwhelming shared fileservers. We have not had reports of scans traversing the network since much earlier releases.

  • We are planning to add more JRE runtimes for the “Download a Temporary JRE”-based scans. Please let us know in the Megathread which OS’s you need to see the most.

  • Planning to add use-at-your-own-risk remediation. Log4jscan itself can open the vulnerable JAR file and remove the vulnerable JndiLookup.class file, leaving the vulnerabilities at least partially remediated. We are also planning content to replace the log4j-core-X.jar file entirely with the latest Apache Log4j version (while retaining the old version’s filename).

    – Either of these methods carries significant risks of breaking your applications, and where possible you should follow guidance from your software providers on remediating their products; however taking no action at all can also expose serious vulnerabilities. Whether to patch or replace the Log4j libraries is a risk-based decision that we all must make for our own platforms.

Please provide feedback on your testing, especially any refinements you would recommend for excluding obvious network drives or other folders you think should be excluded on Linux or Windows here: Log4j CVE-2021-44228 Detection and Mitigation

For tips on how to download files from GitHub, see Tips: Downloading files from GitHub

3 Likes

Mitigation fixlets are now on the github, will update the above post with the links soon.

See here: https://github.com/jgstew/bigfix-content/tree/master/fixlet

1 Like

Major content refresh and usage instructions

All Logpresso / Log4jscan content is use at-your-own-risk. Please see Log4j Vulnerability Identification and 3rd Party Remediation Solution Testing Statement

Updates: 2022-01-05 16:00 UTC

  • Updated the Java versions of LogPresso scans to use scanner 2.7.1. (Only the Java versions, either with a temporary JRE download or system JRE, are updated; standalone binary scanner updates coming soon)
  • Added Undo-Remediation Task for LogPresso-based remediation ( restore the original files where LogPresso removed JndiLookup.class )
  • Updated “With Temporary JRE” Scan, Remediation, and Undo-Remediation tasks to explicitly remove the temporary JRE, Logpresso, and Unzip downloads from the __BESData\sitename\__Download folder
  • Add JSON report output to Java-based Scan/Remediation tasks (updated Analysis coming soon)
  • Java-based Remediation and Undo-Remediation Tasks no longer have a Default Action (must choose the Action explicitly)

Usage Instructions:

While improvements are very much ongoing, we wanted to make the community aware of progress and provide some general usage and troubleshooting instructions for using this all together.

The general process is

Download, Import, and Activate the Analysis to report Logpresso Log4j-scan results.
Download and Import one or more of the Log4j-scan Tasks.
Execute the scans that are appropriate for your environment to collect results.

  • Run with care. Start in small groups and watch for system impacts. When scanner larger groups, use the ‘Stagger Action Start Times’ option to avoid overloading servers or shared storage systems.
    Evaluate the results and determine where mitigations are needed.
    Perform mitigations using one of the community fixlets, or apply vendor patches, or perform manual remediations.

Which Scans to run?

We have currently developed five different scans, and six different Remediation fixlets. All report results in the same way, all generate the same outputs. Which ones should you use?

OS-Specific Binary Scans

These three scans download a prebuilt binary for a limited set of operating systems. These may have issues if you don’t have certain prerequisites installed - Visual C++ Runtime on Windows, or specific glibc versions on Linux. More recent versions of Logpressso Log4j-scan are “static-compiled” which reduces these compatibility issues but may not eliminate them entirely. If you need the smallest possible downloads, and your operating systems support these binary scans, feel free to use them. If you do encounter any compatibility issues, switch to one of the Java-based scans below.

Logpresso Scanner - Linux

Logpresso Scanner - Mac

Logpresso Scanner - Windows

Java-based scanner, with existing Java installation

If your systems already have Java installed, and it is at least JRE 7, you may use the “Universal Java” Scan. This provides the widest range of compatibility, able to handle many operating systems for which we have not created specific JRE downloads, such as 32-bit Linux, Solaris, AIX, and even Raspbian.

Logpresso Scanner - Universal
(updated 2022-01-05)

Java-based scanner, with a downloaded JRE

For systems that do not have Java installed, or the Java is too old to run Logpresso Log4j-scan, we have ongoing work on a Task that downloads a temporary Java Runtime to execute the scanner. Currently we provide downloads for 32-bit Windows, 64-bit Windows, 64-bit Linux and Mac. We will be focusing work on this over the next few days to add more Java runtimes. 32-bit Linux, AIX, Solaris, and HP-UX are intended.

Logpresso Scanner - Universal with JRE
(updated 2022-01-05)


Checking the results

Regardless of which of the above scans you execute, we can retrieve the results in a single analysis. Download, import, and activate this Analysis:
LogPresso Log4j-scan results

The property list from this Analysis includes

  • log4j Scan - Vulnerable File List
  • log4j Scan - Scan Technology
  • log4j Scan - Last Scan Time
  • log4j Scan - Vulnerable File Count
  • log4j Scan - Potentially Vulnerable File Count
  • log4j Scan - Mitigated File Count
  • log4j Scan - File Result Details
  • log4j Scan - File Scan Count
  • log4j Scan - Scan Duration

Some things to watch for in the Results - since the scanner and our Tasks are rapidly-evolving, watch for Scan Results that either have really long durations (which might indicate the machine is crossing network parths) or have a 0 “File Scan Count” (indicating nothing was actually scanned on the machine).

Also note that by default, each of these Analysis Properties is set to evaluate “Every 6 hours”. During testing, if you are running scans more frequently on a small set of systems, you may want to reduce that interval to “Every hour” or “Every 15 minutes”. Because the log parsing relevance can be expensive, avoid using ‘Every Report’ for these properties.


Remediation

We are excited to announce our first set of Remediation tasks based on Logpresso Log4j-scan results. These are all very much use at your own risk. Test. It is impossible for us to predict which applications may be broken by any of these remediations. However given the severity of these issues we felt it important to provide options while we all wait for patches from individual software vendors.

Mitigations are split into two categories:

Logpresso Log4j-scan remediation

The Logpresso Log4j-scan utility itself can perform some remediations. Specifically it opens the log4j-core-2.x.jar file and removes the JndiLookup.class from the file, following the guidance as recommended at the Apache security page. This mitigates the worst of the CVEs but may not mitigate the more recent, denial-of-service based vulnerabilities. Still this can be a very effective step to take to provide the most protection while maintaining the best backward-compatibility with existing applications.
For each of our Logpresso scans, we provide a Mitigation task:

Logpresso Remediation - Windows

Logpresso Remediation - Linux

Logpresso Remediation - Mac

Logpresso Remediation - Java - Universal
(updated 2022-01-05)

Logpresso Remediation - Java - With Temporary JRE
(updated 2022-01-05 )

Log4j-core-2.17.0.jar replacement

Additionally, we provide a Task that, instead of modifying Log4j-core, will replace the Log4j-core-2.x.jar file with the latest 2.17.0 version. This closes all of the known CVEs, but could be more likely to introduce compatibility problems with the applications that depend on Log4j.
This task parses the output log of a previous Logpresso Log4j-scan task, and where possible replaces the Log4j-core-2.x.jar file.
When we replace the file, we keep the original version’s filename, which can help reduce compatibility problems. The original file is backed up so it can be restored later if necessary.
Limitations:

  • Does not descend within embedded JAR, WAR, EAR files. Only replaces the Log4j-core-2.x.jar where it is directly accessible in the filesystem.
  • Does not replace Log4j 1.x versions.

Log4j Mitigation - Replace Log4j with 2.17.0


General Process

The general process we’re recommending is to execute one or more of the Scans, observe the results in the Analysis, and then perform one or more of the Remediations.

Start at small scales and then work up to larger deployments. Engage your operations and application teams, if possible.

Where a software vendor provides a fix or updated version, use those in preference to the mitigation techniques described here. That’s the safest way to maintain compatibility.


Troubleshooting and Rollback

How the Tasks work

Each of the Scanning tasks creates a folder relative to the BigFix Client. On Linux, by default at /var/opt/BESClient/BPS-Scans, and on Windows at c:\Program Files (x86)\BigFix Enterprise\BES Client\BPS-Scans

Rollback LogPresso-based remediation:

When taking a Remediation action via Logpresso Log4j, a .ZIP archive is stored here to allow restoring the original versions, if necessary.

After executing a LogPresso-based remediation, a Task is available to rollback the changes that were made by restoring the original versions of the affected JAR files. When rolling back, every remediated file is restored - rollback is an all-or-nothing proposition. If only specific applications need to be rolled back, either expand “Manual Rollback” details below, or use this Task to rollback all of the updates, and then perform a new Remediation scan while excluding the specific directories that need to remain at the original log4j version.

With a temporary Java Runtime download:
Logpresso Log4j2-scan - Universal - JRE - UNDO Remediation

With existing system Java Runtime:
Logpresso Log4j2-scan - Universal - UNDO Remediation

Manual Rollback

The archive contains the full path to the original version of any files it modified, as shown in this shell snippet (similar exists for Windows):

[root@cent-1 BPS-Scans]# pwd
/var/opt/BESClient/BPS-Scans

[root@cent-1 BPS-Scans]# ls -l
-rw-------. 1 root root     292 Dec 21 13:06 log4j2-path-exclusions.txt
-rw-------. 1 root root 1574810 Dec 21 13:06 log4j2_scan_backup_20211221_130611.zip
-rw-------. 1 root root   59877 Dec 21 13:06 logpresso-log4j2-scan.jar
-rw-------. 1 root root    1450 Dec 21 13:06 results-log4j2-scan.txt

[root@cent-1 BPS-Scans]# unzip -l log4j2_scan_backup_20211221_130611.zip
Archive:  log4j2_scan_backup_20211221_130611.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
  1762733  12-21-2021 13:06   /tmp/test/apache-log4j-2.14.0-bin/log4j-core-2.14.0.jar
---------                     -------
  1762733                     1 file

Rollback log4j-core-2.x.jar file replacement:

When we perform a replacement with the log4j-core-2.17.0.jar file, we make backup copies of the original file, and store copies of the file replacement scripts on the client as a reference for which replacements we performed:

[root@cent-1 BPS-Scans]# ls -l
total 1612
-rw-------. 1 root root     292 Dec 21 13:06 log4j2-path-exclusions.txt
-rw-------. 1 root root   59877 Dec 21 13:06 logpresso-log4j2-scan.jar
-rw-------. 1 root root       0 Dec 20 22:20 replace_log4j-1646_output.txt
-rw-------. 1 root root     444 Dec 20 22:20 replace_log4j-1646.txt
-rw-------. 1 root root    1450 Dec 21 13:06 results-log4j2-scan.txt

[root@cent-1 BPS-Scans]# cat replace_log4j-1646.txt
#!/bin/sh
# log4j-core-2.x.jar replacement script Mon, 20 Dec 2021 22:20:11 -0500
# \cp is used to prevent the common 'cp -i' alias from being used
# -n is used to prevent clobbering destination.  We only want to backup the first, original version, of log4j-core
\cp -n "/tmp/test/log4j-core-2.12.2.jar" "/tmp/test/log4j-core-2.12.2.jar-disabled"
\cp "__Download/apache-log4j-2.17.0-bin/log4j-core-2.17.0.jar" "/tmp/test/log4j-core-2.12.2.jar"

Our script and output file are named to include the Action ID of the action that performed the replacement, so multiple executions would create separate files. In this case, from the batch we can see that /tmp/test/log4j-core-2.12.2.jar was replaced with the 2.17.0 version, with original file being copied to /tmp/test/log4j-core-2.12.2.jar-disabled. That ‘-disabled’ file could be restored if this replacement caused a compatibility problem.


As always, we are thankful for the great collaboration we’ve seen among the BigFix Community as we all work toward finding and remediating this vulnerability as we have every other challenge that doesn’t fit neatly into patch/replacement category. Continue to watch this space for the latest updates. Future plans include adding more JRE runtimes to our “Use Temp JRE” Task, and potentially enhancing some of the reporting aspects (such as CSV or JSON reports from the Logpresso utility).

4 Likes

As mentioned in yesterday’s Release Announcement, we now have official, supported content in the “BES License and Inventory” Site. This includes the release version of Scan fixlet, as well as the Analysis to retrieve the scan results. As noted in the Release Announcement, “A BigFix task that can also do mitigation of Log4j vulnerabilities using the same technology is available if you contact us or unofficially through the BigFix community”.

If you’ve been using our Community version of the Analysis, note that the Analysis and Property names are duplicated, which could cause some confusion when you use the properties as filters or columns in Web Reports. Once you activate the Analysis in the “BES License and Inventory” Site, I’d recommend saving off any existing Web Reports results from the previous analysis, and then either rename or remove any custom copies of the Community version of the analysis.

Some changes to note yesterday afternoon on the Community and included in the Release version of the scan task, are enablement of scans on AIX and Solaris systems.

4 Likes

New updates added to the “Major content refresh and usage instructions” above.

  • Updated the Java versions of LogPresso scans to use scanner 2.7.1. (Only the Java versions, either with a temporary JRE download or system JRE, are updated; standalone binary scanner updates coming soon)
  • Added Undo-Remediation Task for LogPresso-based remediation ( restore the original files where LogPresso removed JndiLookup.class )
  • Updated “With Temporary JRE” Scan, Remediation, and Undo-Remediation tasks to explicitly remove the temporary JRE, Logpresso, and Unzip downloads from the __BESData\sitename__Download folder
  • Add JSON report output to Java-based Scan/Remediation tasks (updated Analysis coming soon)
    Java-based Remediation and Undo-Remediation Tasks no longer have a Default Action (must choose the Action explicitly)
6 Likes

The BigFix team has published an update to the “BES Inventory and License” Site, enabling updated Scans, Mitigation, and Mitigation Rollback using the 2.7.1 version of Logpresso Scanner

1 Like

I have added a web report based on the “log4j2-scan results” analysis in the “BES Inventory and License” site here:

3 Likes