Trying to include some JSON input inside another JSON element

1

We are trying to automate ServiceNow ticketing of Log4j vulnerabilities found using the Logpresso Scan4j tool.

The idea is that we can use the JSON output from the tool and feed that to the ServiceNow API, then our ServiceNow Developers can pickup the resulting data and create/close tickets automatically based on the content.

Getting the LogPresso tool to generate the results of the scan as JSON was easy, but to include it in the “additional_info” field in the JSON to be submitted to ServiceNow, I need to escape all the Quotes in the Results JSON file.

I’ve made progress, but I’ve run into a problem the I don’t see the source of.

The Relevance I used is …

  • concatenation "\%22" of substrings separated by "%22" of ((it & "%0d%0a") of (lines of file "log4j2_scan_report_20220119_173304.json" of folder "<location of Results JSON file>") as string)
  • concatenation "\%22" of substrings separated by "%22" of (("%0d%0a" & it ) of (lines of file "log4j2_scan_report_20220119_173304.json" of folder "<ReportFolder>") as string)

I have a couple problems,

  1. The Relevance is adding an extra escaped " character at the beginning of each line starting with the 2nd line of the file.
  2. The escaped CR/LF characters are not being converted to their ASCII equivalent characters. I’ve actually getting %0d%0a in the output text.
  3. If I prepend the CR/LF characters, the extra escaped " characters end up at the end of each line.

Sample JSON output fragment …

{
    "summary": {
        "scanner_banner": "Logpresso CVE-2021-44228 Vulnerability Scanner 2.7.2 (2022-01-11)",
        "scanner_version": "2.7.2",
        "scanner_release_date": "2022-01-11",
        "scanner_args": [
            "--drives",
            "c",
            "--report-json"

Fixlet Debugger Output from Item #2 …

Q: concatenation "\%22" of substrings separated by "%22" of ((it & "%0d%0a") of (lines of file "log4j2_scan_report_20220119_173304.json" of folder "<ReportFolder>") as string)
A: {%0d%0a\"    \"summary\": {%0d%0a\"        \"scanner_banner\": \"Logpresso CVE-2021-44228 Vulnerability Scanner 2.7.2 (2022-01-11)\",%0d%0a\"        \"scanner_version\": \"2.7.2\",%0d%0a\"        \"scanner_release_date\": \"2022-01-11\",%0d%0a\"        \"scanner_args\": [%0d%0a\"            \"--drives\",%0d%0a\"            \"c\",%0d%0a\"            \"--report-json\"

Fixlet Debugger Output from Item #3 …

Q: concatenation "\%22" of substrings separated by "%22" of (("%0d%0a" & it) of (lines of file "log4j2_scan_report_20220119_173304.json" of folder "<ReportFolder>") as string)
A: %0d%0a{\"%0d%0a    \"summary\": {\"%0d%0a        \"scanner_banner\": \"Logpresso CVE-2021-44228 Vulnerability Scanner 2.7.2 (2022-01-11)\",\"%0d%0a        \"scanner_version\": \"2.7.2\",\"%0d%0a        \"scanner_release_date\": \"2022-01-11\",\"%0d%0a        \"scanner_args\": [\"%0d%0a            \"--drives\",\"%0d%0a            \"c\",\"%0d%0a            \"--report-json\"\"

Where is that extra escaped quote character coming from, and why are the CR/LF characters showing up like this?

2

I can dig into this a bit later in the day, but a few observations -

  • The Fixlet Debugger will always display the CR/LF as %0d%0a in the output - but that’s just in the Debugger, if you use it in a createfile or other statement in the client, the literal CR/LF is embedded.
  • You may not need to care about the newlines at all - putting it all in one line is perfectly valid JSON format.
  • The double quotes may not be all that you need to escape - I think at least the backslashes and possibly other special characters will need their own escaping as well.
  • This would be much easier in a higher-level language like Python or PowerShell or possibly even VBScript - would it be possibly to leverage a higher-level language, likely whatever you’re using with your ServiceNow API already - to do this for you?
3

So, if I’m understanding correctly, you want to embed the entire logpresso scan JSON data embedded as a string value into another JSON file. (if you just wanted to embedded the JSON as a child JSON object that’s easier and less escaping needed).

Here’s a shot Python snippet I used to build an example, to show what a properly-escaped embedded JSON would look like -

import json
with open("C:\\Temp\\log4j2-report.json", "r") as input_file:
    orig_data = json.load(input_file)
new_data = {"report_data": json.dumps(orig_data)}
with open("c:\\Temp\\report-output.json", "w") as output_file:
    output_file.write(json.dumps(new_data))

JSON input snippet:

{
    "summary": {
        "scanner_banner": "Logpresso CVE-2021-44228 Vulnerability Scanner 2.7.1 (2022-01-02)",
        "scanner_version": "2.7.1"
    },
    "files": [
        {
            "path": "C:\\Program Files\\BigFix Enterprise\\SCA\\wlp\\usr\\servers\\server1\\lib\\log4j-core-2.14.0 - Copy.jar",
            "status": "MITIGATED"
        }]
}

Sample output snippet:

{"report_data": "{\"summary\": {\"scanner_banner\": \"Logpresso CVE-2021-44228 Vulnerability Scanner 2.7.1 (2022-01-02)\", \"scanner_version\": \"2.7.1\"}, \"files\": [{\"path\": \"C:\\\\Program Files\\\\BigFix Enterprise\\\\SCA\\\\wlp\\\\usr\\\\servers\\\\server1\\\\lib\\\\log4j-core-2.14.0 - Copy.jar\", \"status\": \"MITIGATED\"}}

So it looks like both doublequotes and backslashes get escaped, I’m not sure whether any other special characters are escaped in there as well; but at least curly- and square-brackets did not need escaping inside the string.

4

The goal of trying to incorporate the CR/LF characters was to make the output more human readable during testing. Since it’s not going to help me in the FixletDebugger, I can substitute some other character there during testing and replace that character with the CR/LF using something like Notepad ++ or Textpad while testing.

The reason to do as much of this work inside BigFix is that I have to cover +50k endpoints including Windows (Servers and end user devices), Mac end user devices, and Linux (servers and end user devices). Fortunately I don’t have to worry about our AIX devices for this project.

The politics of getting permission to install Python on all these devices would be a nightmare. To complicate matters, some of them are “restricted” in terms of what we are allowed to install on them. And our Linux team usually pares the OS to the absolute bare minimum required to function. I had to submit a request to add several modules the BES Client needed that they were not originally including in their image.

This sounds like it will be a multi-step procedure. Something like "escape the “/” characters first, then escape the " characters, then strip the odd extra " character from the lines. Since this is all going to be done in an Action script from a Task, I can perform each step on it’s own outputting to a file to let the next step feed off the results of the previous step. Then when we’re finished, clean up the files we created.

The plan is to use CURL to submit the request to the ServiceNow API since that function is available on all the targeted platforms (I may have to have it added to the Linux hosts, but that should be a minimal effort request.

BTW: Thank you for pointing out that I needed to escape the \ characters as well. I totally forgot about those!

5

Ok, I think I have an example that may work here. This embeds the structure of the log4j-scan JSON result as a string inside the report_value key of a new JSON file, you’d need to substitute with whatever key ServiceNow is expecting.
The CR/LF codes %0d%0a are actually not valid inside of JSON value, instead we should use the escaped newline string ‘\n’

Because of the escaping of curly-brackets in actionscript, at the start of the appendfile we include {{ to put a literal { symbol into the file, then the "report_value": " is a literal string, then the relevance query surrounded by { } symbols, and then finally a literal doublequote and } to the end of the JSON file.

In the Relevance query, I’m replacing \ with \\ and " with \" for each line, and then concatenating the lines with the JSON newline \n string

In Python I am able to open the resulting file as JSON, and from the “report_value” key I am able to parse the string as its own JSON object, so I think this should work.

delete __appendfile
appendfile {{"report_value": "{ concatenation "\n" of (concatenation "\%22" of substrings separated by "%22" of concatenation "\\" of substrings separated by "\" of it) of lines of file "c:\temp\log4j2-report.json"}"}

delete c:\temp\output.json
copy __appendfile c:\temp\output.json

Python to read this resulting file (for testing only, no need to deploy Python everyone, and this should be fairly straightforward to convert to PowerShell if you prefer:

import json
# Load wrapper JSON from file
with open("C:\\Temp\\output.json", "r") as test_output:
    output_data = json.load(test_output)
print(json.dumps(output_data))

# Load embedded JSON string from the report_value key
scan_results = json.loads(output_data["report_value"])
print(json.dumps(scan_results))