Look at two files simultaneously

Content Authoring
1

(imported topic written by jeremytoo)

I’m examining a unix system, and want to see if there are any accounts with the following criteria:

password is set

password expiration is > 90 days

account has shell of /bin/bash

I can get the first two, since passwords are password age are recorded in /etc/shadow. My relevance looks like this:

concatenation “;” of ((items 0 of it, items 1 of it) whose (item 1 of it > “90”) of ( (preceding texts of first “:” of it, ( preceding texts of first “:” of (following texts of first “:” of ( following texts of first “:” of ( following texts of first “:” of ( (following texts of first “:” of it) )) )) ) ) of lines whose (it contains regex "^

a-zA-Z0-9

*:

$a-zA-Z0-9

a-zA-Z0-9

" ) of file “/etc/shadow”) as string)

this works beautifully.

However, I’m not sure how to proceed with checking item 0 of it (the username) against the first field in /etc/passwd, and then looking at the last field in /etc/passwd (field 7).

basically, I need to do a join on the first field of each file. Does anyone have any idea how I would accomplish this?

2

(imported comment written by NoahSalzman)

I’m trying to restate your problem. Does this simplification accurately describe what you are trying to do?

file_1:

foo:1234:purple

bar:2345:grey

file_2:

foo:####:hot

bar:$$$$:cold

And you want an output of:

foo;1234;hot

bar;2345;cold

Right?

3

(imported comment written by jeremytoo)

yep, that’s an accurate analogy to what I’m looking for.

4

(imported comment written by jeremytoo)

To solve this problem, I had to create a regex of all the usernames from /etc/passwd that did not have /bin/false or /sbin/nologin as their login shell. (this is only mildly more complex than the original statement of simply requiring /bin/bash).

It’s taken me time, but here is the working code:

Working code to get all usernames with passwords set, and do not have /bin/false or bin nologin set:

concatenation “;” of ((items 0 of it, items 1 of it) whose (item 1 of it > “90” AND item 0 of it contains regex (concatenation “|” of preceding texts of firsts “:” of (lines whose (not (it contains regex “nologin$|false$”)) of file “/etc/passwd”)) ) of ( (preceding texts of first “:” of it, ( preceding texts of first “:” of (following texts of first “:” of ( following texts of first “:” of ( following texts of first “:” of ( (following texts of first “:” of it) )) )) ) ) of lines whose (it contains regex "^

a-zA-Z0-9

*:

$a-zA-Z0-9

a-zA-Z0-9

" ) of file “/etc/shadow”) as string)

this will produce a report like this:

root, 99999;sampleuser, 1500

If there are commented lines in /etc/shadow (why would you have those?), it’ll give a false positive, but it’s not hard to exclude that.

The trick seemed to be this: once I had a username to compare, I could then compare it to a regex made of all the valid usernames from /etc/passwd.

Any idea how much of a CPU/disk hit this thing is? Will the inspector cache the results of the /etc/passwd read? or is it going to hammer the disk for EVERY username it evaluates?

5

(imported comment written by blibbons91)

OK, I’ve been trying to wrap my mind around how andersje solved his problem, and I’m just not digesting it. I have a somewhat similar need to compare both the passwd and shadow files and output elements from each, but I try as I might\ I cannot figure out how to leverage the parsing/comparison logic used by andersje for my own devious uses.

Here’s what I’d like to do:

  1. Parse each line of the passwd file

  2. Separate each line of the passwd file based on :

  3. Use element 1 (username), 3 (UID), 4 (GID), 5 (User ID Info) and 6 (Home directory)

  4. Iterate through the lines in the Shadow file searching for the username (element 1 in the previous step)

  5. Separate the line in the shadow file identified with the username based on :

  6. Use element 5 (max password valid) in the shadow file

  7. Create a line of the combined data from both the Shadow and Passwd files

For example, if the passwd file has the following entries:

root:x:0:0:Admin Team:/root:/bin/bash;

bin:x:1:1:Admin Team:/bin:/sbin/nologin;

And the Shadow file has the following entries:

root:$1$TGKPzwUN$Z:14669:0:90:7:::;

bin:$1$TGKPzwUN$Z:14669:0:90:7:::;

I’d like to generate the following comma-separated output:

root,0,0,Admin Team,/root,90

bin,1,1,Admin Team,/bin,90

Unlike andersje, I’m not concerned with password expiration > 90 days. I just need a full inventory. andersje’s solution only returns user ID and password expiration if it’s greater than 90 days. I tried playing around with it to get more of the data I need, but no luck.

Any insight would be extremely appreciated!

Thanks,

Rob

—Slightly modified code below, but still only returns User ID & Expiration Days------

((items 0 of it, items 1 of it) whose (item 0 of it contains regex (concatenation “|” of preceding texts of firsts “:” of (lines of file “c:\passwd.txt”)) ) of ( (preceding texts of first “:” of it, ( preceding texts of first “:” of (following texts of first “:” of ( following texts of first “:” of ( following texts of first “:” of ( (following texts of first “:” of it) )) )) ) ) of lines whose (it contains regex "^

a-zA-Z0-9

*:

$a-zA-Z0-9

a-zA-Z0-9

" ) of file “c:\shadow.txt”) as string)

6

(imported comment written by SystemAdmin)

You have the right plan, which is to think about what you’re trying to do and separate out each specific task. The way I did it was different from andersje and also happens to take care of the concern about doing multiple reads of the files.

For the passwd file, you are pretty much just taking a large beginning chunk of each line, the first element we’ll use to match and the second we will need to discard. So, lets create a relevance that will pull out the data we need:

Q: ( (preceding text of first “:” of it &":"& following text of first “:” of following text of first “:” of it)of preceding text of last “:” of it ) of lines of file “c:\temp\passwd.txt”

A: root:0:0:Admin Team:/root

A: bin:1:1:Admin Team:/bin

Then follow the same process for the shadow file, but because we just need the first part of the line and then one other item (buried in the middle of the string), rather than use multiple “preceding text/following text” statements to try to strip it out, we’ll use a regex to filter out just the parts we need.

Q: ( (parenthesized part 1 of it &":"& parenthesized part 2 of it) of match (regex "

(.):.:.:.:(.):.:.:.:.;") of it) of lines whose (it contains regex "

a-zA-Z0-9

:

$a-zA-Z0-9

a-zA-Z0-9

" ) of file “c:\temp\shadow.txt”

A: root:90

A: bin:90

Once we know how to derive the sub-elements of the lines of the files, we want to pull the matching lines out of each file in turn.

Q: (item 0 of it, item 1 of it) whose (preceding text of first “:” of item 0 of it = preceding text of first “:” of item 1 of it) of (lines of file “c:\temp\passwd.txt”, lines whose (it contains regex "^

a-zA-Z0-9

*:

$a-zA-Z0-9

a-zA-Z0-9

" ) of file “c:\temp\shadow.txt”)

We then add the relevance statements we created earlier to the “item 0 of it, item 1 of it” code, which gets us very close to the final result.

For the last changes, we only need to append “:” and the second half of our “item 1 of it” (which is the number of days from the shadow file) and then finally adjust the format by splitting the string on “:” and combining it again using “,”.

Q: (concatenation “,” of substrings separated by “:” of (item 0 of it &":"& following text of first “:” of item 1 of it)) of (((preceding text of first “:” of it &":"& following text of first “:” of following text of first “:” of it)of preceding text of last “:” of item 0 of it), ( (parenthesized part 1 of it &":"& parenthesized part 2 of it) of match (regex "

(.):.:.:.:(.):.:.:.:.;") of item 1 of it) ) whose (preceding text of first “:” of item 0 of it = preceding text of first “:” of item 1 of it) of (lines of file “c:\temp\passwd.txt”, lines whose (it contains regex "

a-zA-Z0-9

:

$a-zA-Z0-9

a-zA-Z0-9

" ) of file “c:\temp\shadow.txt”)

A: root,0,0,Admin Team,/root,90

A: bin,1,1,Admin Team,/bin,90

T: 2.807 ms

I: plural string

Hope this helps.

-Jim

7

(imported comment written by NoahSalzman)

Nice! The breakdown is very helpful for readers of the Forum.

8

(imported comment written by blibbons91)

Jim, your explanation is phenomenal!

Thanks so much for the walk through, it helps immensely. Even beyond this challenge.

Thanks,

Rob