Major content refresh and usage instructions
Updates: 2022-01-05 16:00 UTC
- Updated the Java versions of LogPresso scans to use scanner 2.7.1. (Only the Java versions, either with a temporary JRE download or system JRE, are updated; standalone binary scanner updates coming soon)
- Added Undo-Remediation Task for LogPresso-based remediation ( restore the original files where LogPresso removed JndiLookup.class )
- Updated “With Temporary JRE” Scan, Remediation, and Undo-Remediation tasks to explicitly remove the temporary JRE, Logpresso, and Unzip downloads from the
__BESData\sitename\__Download folder
- Add JSON report output to Java-based Scan/Remediation tasks (updated Analysis coming soon)
- Java-based Remediation and Undo-Remediation Tasks no longer have a Default Action (must choose the Action explicitly)
Usage Instructions:
While improvements are very much ongoing, we wanted to make the community aware of progress and provide some general usage and troubleshooting instructions for using this all together.
The general process is
Download, Import, and Activate the Analysis to report Logpresso Log4j-scan results.
Download and Import one or more of the Log4j-scan Tasks.
Execute the scans that are appropriate for your environment to collect results.
- Run with care. Start in small groups and watch for system impacts. When scanner larger groups, use the ‘Stagger Action Start Times’ option to avoid overloading servers or shared storage systems.
Evaluate the results and determine where mitigations are needed.
Perform mitigations using one of the community fixlets, or apply vendor patches, or perform manual remediations.
Which Scans to run?
We have currently developed five different scans, and six different Remediation fixlets. All report results in the same way, all generate the same outputs. Which ones should you use?
OS-Specific Binary Scans
These three scans download a prebuilt binary for a limited set of operating systems. These may have issues if you don’t have certain prerequisites installed - Visual C++ Runtime on Windows, or specific glibc versions on Linux. More recent versions of Logpressso Log4j-scan are “static-compiled” which reduces these compatibility issues but may not eliminate them entirely. If you need the smallest possible downloads, and your operating systems support these binary scans, feel free to use them. If you do encounter any compatibility issues, switch to one of the Java-based scans below.
Logpresso Scanner - Linux
Logpresso Scanner - Mac
Logpresso Scanner - Windows
Java-based scanner, with existing Java installation
If your systems already have Java installed, and it is at least JRE 7, you may use the “Universal Java” Scan. This provides the widest range of compatibility, able to handle many operating systems for which we have not created specific JRE downloads, such as 32-bit Linux, Solaris, AIX, and even Raspbian.
Logpresso Scanner - Universal
(updated 2022-01-05)
Java-based scanner, with a downloaded JRE
For systems that do not have Java installed, or the Java is too old to run Logpresso Log4j-scan, we have ongoing work on a Task that downloads a temporary Java Runtime to execute the scanner. Currently we provide downloads for 32-bit Windows, 64-bit Windows, 64-bit Linux and Mac. We will be focusing work on this over the next few days to add more Java runtimes. 32-bit Linux, AIX, Solaris, and HP-UX are intended.
Logpresso Scanner - Universal with JRE
(updated 2022-01-05)
Checking the results
Regardless of which of the above scans you execute, we can retrieve the results in a single analysis. Download, import, and activate this Analysis:
LogPresso Log4j-scan results
The property list from this Analysis includes
- log4j Scan - Vulnerable File List
- log4j Scan - Scan Technology
- log4j Scan - Last Scan Time
- log4j Scan - Vulnerable File Count
- log4j Scan - Potentially Vulnerable File Count
- log4j Scan - Mitigated File Count
- log4j Scan - File Result Details
- log4j Scan - File Scan Count
- log4j Scan - Scan Duration
Some things to watch for in the Results - since the scanner and our Tasks are rapidly-evolving, watch for Scan Results that either have really long durations (which might indicate the machine is crossing network parths) or have a 0 “File Scan Count” (indicating nothing was actually scanned on the machine).
Also note that by default, each of these Analysis Properties is set to evaluate “Every 6 hours”. During testing, if you are running scans more frequently on a small set of systems, you may want to reduce that interval to “Every hour” or “Every 15 minutes”. Because the log parsing relevance can be expensive, avoid using ‘Every Report’ for these properties.
Remediation
We are excited to announce our first set of Remediation tasks based on Logpresso Log4j-scan results. These are all very much use at your own risk. Test. It is impossible for us to predict which applications may be broken by any of these remediations. However given the severity of these issues we felt it important to provide options while we all wait for patches from individual software vendors.
Mitigations are split into two categories:
Logpresso Log4j-scan remediation
The Logpresso Log4j-scan utility itself can perform some remediations. Specifically it opens the log4j-core-2.x.jar file and removes the JndiLookup.class from the file, following the guidance as recommended at the Apache security page. This mitigates the worst of the CVEs but may not mitigate the more recent, denial-of-service based vulnerabilities. Still this can be a very effective step to take to provide the most protection while maintaining the best backward-compatibility with existing applications.
For each of our Logpresso scans, we provide a Mitigation task:
Logpresso Remediation - Windows
Logpresso Remediation - Linux
Logpresso Remediation - Mac
Logpresso Remediation - Java - Universal
(updated 2022-01-05)
Logpresso Remediation - Java - With Temporary JRE
(updated 2022-01-05 )
Log4j-core-2.17.0.jar replacement
Additionally, we provide a Task that, instead of modifying Log4j-core, will replace the Log4j-core-2.x.jar file with the latest 2.17.0 version. This closes all of the known CVEs, but could be more likely to introduce compatibility problems with the applications that depend on Log4j.
This task parses the output log of a previous Logpresso Log4j-scan task, and where possible replaces the Log4j-core-2.x.jar file.
When we replace the file, we keep the original version’s filename, which can help reduce compatibility problems. The original file is backed up so it can be restored later if necessary.
Limitations:
- Does not descend within embedded JAR, WAR, EAR files. Only replaces the Log4j-core-2.x.jar where it is directly accessible in the filesystem.
- Does not replace Log4j 1.x versions.
Log4j Mitigation - Replace Log4j with 2.17.0
General Process
The general process we’re recommending is to execute one or more of the Scans, observe the results in the Analysis, and then perform one or more of the Remediations.
Start at small scales and then work up to larger deployments. Engage your operations and application teams, if possible.
Where a software vendor provides a fix or updated version, use those in preference to the mitigation techniques described here. That’s the safest way to maintain compatibility.
Troubleshooting and Rollback
How the Tasks work
Each of the Scanning tasks creates a folder relative to the BigFix Client. On Linux, by default at /var/opt/BESClient/BPS-Scans, and on Windows at c:\Program Files (x86)\BigFix Enterprise\BES Client\BPS-Scans
Rollback LogPresso-based remediation:
When taking a Remediation action via Logpresso Log4j, a .ZIP archive is stored here to allow restoring the original versions, if necessary.
After executing a LogPresso-based remediation, a Task is available to rollback the changes that were made by restoring the original versions of the affected JAR files. When rolling back, every remediated file is restored - rollback is an all-or-nothing proposition. If only specific applications need to be rolled back, either expand “Manual Rollback” details below, or use this Task to rollback all of the updates, and then perform a new Remediation scan while excluding the specific directories that need to remain at the original log4j version.
With a temporary Java Runtime download:
Logpresso Log4j2-scan - Universal - JRE - UNDO Remediation
With existing system Java Runtime:
Logpresso Log4j2-scan - Universal - UNDO Remediation
Manual Rollback
The archive contains the full path to the original version of any files it modified, as shown in this shell snippet (similar exists for Windows):
[root@cent-1 BPS-Scans]# pwd
/var/opt/BESClient/BPS-Scans
[root@cent-1 BPS-Scans]# ls -l
-rw-------. 1 root root 292 Dec 21 13:06 log4j2-path-exclusions.txt
-rw-------. 1 root root 1574810 Dec 21 13:06 log4j2_scan_backup_20211221_130611.zip
-rw-------. 1 root root 59877 Dec 21 13:06 logpresso-log4j2-scan.jar
-rw-------. 1 root root 1450 Dec 21 13:06 results-log4j2-scan.txt
[root@cent-1 BPS-Scans]# unzip -l log4j2_scan_backup_20211221_130611.zip
Archive: log4j2_scan_backup_20211221_130611.zip
Length Date Time Name
--------- ---------- ----- ----
1762733 12-21-2021 13:06 /tmp/test/apache-log4j-2.14.0-bin/log4j-core-2.14.0.jar
--------- -------
1762733 1 file
Rollback log4j-core-2.x.jar file replacement:
When we perform a replacement with the log4j-core-2.17.0.jar file, we make backup copies of the original file, and store copies of the file replacement scripts on the client as a reference for which replacements we performed:
[root@cent-1 BPS-Scans]# ls -l
total 1612
-rw-------. 1 root root 292 Dec 21 13:06 log4j2-path-exclusions.txt
-rw-------. 1 root root 59877 Dec 21 13:06 logpresso-log4j2-scan.jar
-rw-------. 1 root root 0 Dec 20 22:20 replace_log4j-1646_output.txt
-rw-------. 1 root root 444 Dec 20 22:20 replace_log4j-1646.txt
-rw-------. 1 root root 1450 Dec 21 13:06 results-log4j2-scan.txt
[root@cent-1 BPS-Scans]# cat replace_log4j-1646.txt
#!/bin/sh
# log4j-core-2.x.jar replacement script Mon, 20 Dec 2021 22:20:11 -0500
# \cp is used to prevent the common 'cp -i' alias from being used
# -n is used to prevent clobbering destination. We only want to backup the first, original version, of log4j-core
\cp -n "/tmp/test/log4j-core-2.12.2.jar" "/tmp/test/log4j-core-2.12.2.jar-disabled"
\cp "__Download/apache-log4j-2.17.0-bin/log4j-core-2.17.0.jar" "/tmp/test/log4j-core-2.12.2.jar"
Our script and output file are named to include the Action ID of the action that performed the replacement, so multiple executions would create separate files. In this case, from the batch we can see that /tmp/test/log4j-core-2.12.2.jar was replaced with the 2.17.0 version, with original file being copied to /tmp/test/log4j-core-2.12.2.jar-disabled. That ‘-disabled’ file could be restored if this replacement caused a compatibility problem.
As always, we are thankful for the great collaboration we’ve seen among the BigFix Community as we all work toward finding and remediating this vulnerability as we have every other challenge that doesn’t fit neatly into patch/replacement category. Continue to watch this space for the latest updates. Future plans include adding more JRE runtimes to our “Use Temp JRE” Task, and potentially enhancing some of the reporting aspects (such as CSV or JSON reports from the Logpresso utility).
