Compare process to file is failing because of the size of the file 50MB and larger

Content Authoring
1

I am running a relevance statement that runs the code below and then compares to a text file, for each line and only outputs what does not match. This works fine for text files that are less than 10MB in size but files reach anywhere from 40MB to as large as 177MB. Just looking for your thoughts as how one might solve this problem.

The code I am running is

 {concatenation ";" of elements of (set of ((if (it does not start with "-") then it else nothing) of ((item 4 of it & ":" & following text of last ":" of item 1 of it & ":" & item 0 of it) of ( (if (tcp of it) then ("tcp") else ("udp")), local address of it as string & ":" & local port of it as string, (if (exists remote address of it) then (remote address of it as string & ":" & remote port of it as string) else (":")), (if (exists tcp state of it) then (tcp state of it as string) else ("-")), (if (exists process of it and exists name of process of it) then (name of process of it) else ("-")) ) of sockets of network)) - set of (preceding text of last ":" of it as string & ":" & (following text of last ":" of it as string as lowercase)) of lines of file (parent folder of client as string & "\services.txt"))}

the file that is being compared is services.txt and the format per line is shown below. The reason for the size of the file is the range has to be what the vendor specifics for their service, such as the DNS for both tcp and udp as an example.

services:port:protocol

alarm_arb.exe:1025:TCP
alarm_arb.exe:1026:TCP
alarm_arb.exe:1027:TCP
alarm_arb.exe:1028:TCP
alarm_arb.exe:1029:TCP
alarm_arb.exe:1030:TCP
alarm_arb.exe:1031:TCP
alarm_arb.exe:1032:TCP
alarm_arb.exe:1033:TCP
alarm_arb.exe:1034:TCP
alarm_arb.exe:1035:TCP
alarm_arb.exe:1036:TCP

The error code I get is
Command Failed (Relevance substitution failed )

2

Just to confirm, if you run the given relevance manually against such large files via Fixlet Debugger, do you get a result (without any errors)? If so, how long does it take to evaluate (you can enable evaluation time output via View -> Show Evaluation Time)?

1 Like
3

For a file that is 17MB - evaluation time is = 1.246ms
For a file that is 177MB - error: singular expression refers to nonexistent object

I am just renaming the files in the same directory.

the vm is a 2008, 64bit, 6GB Ram system.

Is it possible to have the process take more time to run before it stops? The file takes several seconds to open in notepad for example.

4

Given the error (singular expression refers to nonexistent object), I don’t believe the issue to be associated with the evaluation duration, but rather with the contents of the file (i.e. there is a condition within the file that the given relevance is not currently expecting and handling properly).

1 Like
5

Aram,

Good catch as that was the problem. The file does appear to have some anomaly that was causing the error. I used other files and increased the size to 177MB and ran on system with 2GB Ram with no issue. Thanks for the suggestions.

1 Like
6

Using something like this can help:

lines of files (parent folder of client as string & "\services.txt") whose(exists lines of it)