BFI Software discovery - Catalog deficient or am I missing something?

Usage and Config BFI
1

Hi again, everyone. After a little bit of extra configuration (detailed in my posts over the last few days), I’ve managed to successfully scan about 20 TB comprised of 48 remote shared disks on one of my Linux hosts. Scan time took just under 46 hours, which is actually quite a bit better than I had feared.

After processing the scan upload from the client, re-running the BFI Import, I was suprised to see the scan found…nothing.

I can’t find any software installations matching a path from one of my NFS mounts, and when I look at the “Software Installations” report for the host that executed the scan, I only see four products (and those are all installed on local storage, not NFS).

I really expected to find thousands of products on the NFS mounts. I know there are at least several dozen different paths for Java Runtime Environment, for example, that should be rather easy to detect.

So, am I missing something in how the product works? I had expected it to know the names/sizes/sha1/version of a large catalog of binaries, and to find those regardless of where they were installed.

So, I started perusing the Catalog. Surely there’s a signature for Java 6…I browsed Reports->Software Catalog->Publishers->Oracle->Java Platform SE->Java Platform SE, Version 6 -> Java Platform SE, Release 6.0, and found “Java Platform SE, Component Version 6.0”. There is one Signature present, and it appears to only be a signature for Windows. Does BFI really not have a signature for Java 6 on Linux? AIX? Other? Or am I misunderstanding this?

So then I found Reports-> Catalog -> Signatures. Filtering the signatures list to “Platform = Linux”, it looks like there are only 3014 signatures. Running that through Excel, I’m only finding 41 unique Vendores and 646 unique Products (hundreds of which are just the OS name).

Is there some other catalog I need?

2

Ah, after exporting to CSV, I can find there is a signature for “Java Runtime Environment” (for versions 1.6 and 1.8, but not for 1.7). It wasn’t showing up in the browser’s “Find in Page” because I guess the data hadn’t loaded.

However, the scan still did not detect my dozens of instances of JRE 1.6 and JRE 1.8 installations on remote shared disks…

3

Signature info:

Definition
A computer running Linux has:
A signature that uses: IBM Common Inventory Tool 2.5
<Variable export="true" name="IS_INSTALLED"><PackageInfoMatch name="jre" vendor="Oracle*" version="1.6*"/></Variable><Condition withVariable="IS_INSTALLED"><Action do="SKIP" onValue="false" /></Condition>

Does this mean that Java will be found only when it’s installed via the package manager (RPM), and not detected based on the java binary files embedded in other software installations?

4

@JasonWalker Take a look at the Scanned File Data report. You can create a signature from this data. After the next import, you’ll find these discoveries in the Software Installations report.

Does this mean that Java will be found only when it’s installed via the package manager (RPM), and not detected based on the java binary files embedded in other software installations?

I think that BFI is designed to find “Installed” software, which in Linux would be from an RPM.

3 Likes
5

In that case, I’m not sure why so much attention/effort is put into the Remote Shared Disk scanning? Wouldn’t all of the RPMs show up anyway regardless of installation path…if very little of the BFI Catalog handles software detection on NFS mounts, I don’t see the value in that effort.

1 Like
6

@JasonWalker, it’s scary that I actually have the exact same problem, for which I opened a ticket with IBM on last week. We have some instances of Java on Linux where we copy the files onto the system but don’t install their package. The files show up in the scanned file data but since the only Java 1.8 signature in the catalog (for example) requires that the PackageInfo match, it’s not matching up against these file system results.

The concern here is that there is Java on the system and it is being used, but not being matched and reported on and I don’t want to have to create custom signatures for all the java versions. In my ticket with IBM, the ask was for them to create signatures for all Java results with the criteria of file system results and not just package matches, but I haven’t heard yet if it has been accepted.

7

I recently faced similar issues detecting installations of Oracle Goldengate software on RHEL servers. Oracle GG is installed by copying over zip files and extracting it.
On Windows, signature used to look for the presence of ggsci.exe file and detect the installation. But on Linux, the executable is named “ggsci” and by default, BFI scanned file data only contains files with extensions sh,pl,bin,ear,war.

I am assuming this is the reason why software installations detected on *nix servers are relatively less because not everything is installed using rpms.

2 Likes
8

Should I see data from scanned remote shared disks / nfs mounts in the “Scanned file data” report?
For my Linux hosts, I am seeing local scanned file data (limited to the .sh, .pl, .bin, .ear, and .war file extensions as mentioned by @rohit, which is in fact very limited).

However I’m not seeing any matches for any NFS paths in the Scanned File Data report. Although I do see in my BES server’s UploadManagerData that there are uploads containing the file data for NFS share paths, with files named in the form “shared_itsitsearch_0_-[mount_number]-_[BES_Client_ID].xml.bz2”, and when I extract them the results do list NFS file paths.

9

Hmm - I would have expected it to be there. Perhaps another option to consider is the Unrecognized Files report. This would require changing the Advanced Server Setting recompute_unrecognized_scan_data to 1, which would result in a list of the top 1000 files most commonly encountered after the next import. I do have some Linux machines, so I will see what it provides.

1 Like
10

Look forward to your findings.

I’ve done some more digging myself, and I think I’ve decoded just a bit about how it works internally.
Each Remote Shared Disk is assigned as a negative computer_ID value while real hosts are assigned positive values.
In the database, I can find files that were detected on NFS paths via the query

SELECT TOP (1000) [id]
      ,[scan_file_id]
      ,[computer_id]
      ,[path]
      ,[name]
      ,[full_path_sha1]
      ,[size]
      ,[version]
      ,[md5]
      ,[sha256]
      ,[valid_from]
      ,[valid_to]
  FROM [BFInventory].[sam].[file_facts_scd]
  WHERE computer_id < 0

Selecting the max([id]) I find 378,179 file detections. The ‘path’ values are in the from ‘server:\mounted\path\name’ (interesting the forwardslashes were converted to backslashes)

Unfortunately, no searches for either the NFS server name, the path with forwardslashes, or the path with backslashes give me any results in the “Scanned File Data” report. Nor do the results look particularly valuable - as posted earlier, the Linux scans behave kind of like Windows scans, expecting a ‘filename extension’ to mean something. All I really have in this table are a bunch of .sh and .pl scripts, where what I really need are ‘executable binaries’.

I’ve submitted a series of RFE’s this morning, if you think these might be helpful then please give us a vote everyone! Thanks in advance…

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=124800
BFI: Detect and Scan automount NFS filesystems
Currently BigFix Inventory ignores any NFS filesystems that are managed by automount, but are not actively mounted at the time of the software scan. Additionally any prior scan of the filesystem is marked as ‘deleted’ when a subsequent scan occurs with the filesystem inactive/dismounted.

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=124801
BFI: Include Linux Shared Filesystem (NFS) paths in “Scanned File Data” report or in a new report. The ‘Scanned File Data’ report should list all files found in a scan, allowing us to create custom signatures. However files that were located on NFS paths appear to be missing from the list (perhaps because the files are not associated to any particular computer?). This is a request to add NFS paths to the “Scanned File Data” report, or to create a new report specific to NFS paths.

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=124802
BFI: Enhance Linux/UNIX Inventory Scan to include all ‘Executable’ files
Currently the options for scanning files are based on filename/extension (which are meaningful on Windows but less so on Linux/UNIX). Requesting an enhancement to include ‘Executable’ files in Linux/UNIX scans. ‘Executable’ files are determined by file permissions. This would allow detection of many more Linux/UNIX products. A simple example would include ‘Java Runtimes’. The executable file ‘java’ is not detectable based on filename extension (it doesn’t have one) but would be detectable as a file with the Execute bit set.

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=124804
BFI: Provide a ‘Software Template’ Report to provide data for remote shared disks
Currently the only way to via the software inventory results for products found on an NFS mount are to create a computer group to apply the resulting ‘Software Template’ from an NFS scan against a set of target computers, and then review the results from one of those members. However if one of the group members happened to be dismounted from the NFS path at the time of that computers scan, it would no longer be a member of the group and not have the Software Template applied on it. Directly exposing the NFS mount’s scan in a report would make it much easier to identify products installed on an NFS path and to identify unsanctioned software installations.

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=124806
BFI: Catalog enhancements for file scans on Linux/UNIX. The current software signatures for Linux/UNIX platforms are based almost entirely on ‘OS Package Installed’ (presence of RPM / DEB / BFF package). There are few or none of the file-based signatures. Software that is simply ‘extracted’ to a local filesystem or remote NFS path goes largely undetected by BigFix Inventory.

3 Likes
11

Just to confirm, files from Linux were returned, but as you noted, only those with executable extensions (.sh,.bin,.ear).

I definitely will be voting for your RFE’s. Scanning for files with the execute bit set seems like a no brainer.

1 Like
12

@JasonWalker Have you seen these two RFE’s RFE, RFE? It might have some bearing on your situation.

13

I had not seen that one, thanks. I haven’t had a problem with it yet, I don’t think I have any snapshot directories available via NFS.

When you say that you had results in the file scan details for Linux machines, were any of those results from NFS paths?

14

I have a pretty simple Linux deployment, so I don’t have any NFS mounts. I might play around with it this weekend and see if the results change.

Keep in mind that the scanned files represent a rollup of found data by commonality across machines. It’s likely that your supposition about not having any machine association would prevent those files from being visible to this report in any case.

15

So i was going through the following link
https://www.ibm.com/support/knowledgecenter/en/SSKLLW_9.5.0/com.ibm.bigfix.inventory.doc/Inventory/planinconf/c_types_software_scans.html

It states

Starting from application update 9.2.8, the file system scan additionally reports files that are based on processes that have been run on the computers in your infrastructure, regardless of their extension. To view these files, you need to meet the following criteria:

The BigFix server and client are in version 9.5.5 or higher.
The operating system is Linux or Solaris.
Application Usage Statistics analysis is activated.

The reported files, with supplementing information such as path and size, are available on the Scanned File Data report and can be used to create custom discovery and use signatures for software components. If the process has been active at least once since the feature was turned on and the associated file exists on the computer, this file is displayed on the report.

“regardless of the extension” is what caught my eye. Since my deployment is still 9.2, I am not able to verify this but worth a look.

16

You can configure the extensions, look at:
https://www.ibm.com/support/knowledgecenter/en/SSKLLW_9.5.0/com.ibm.bigfix.inventory.doc/Inventory/planinconf/optimizing_scanned_file_data.html

17

@michalpaluch most of *nix executables do not have an extension like .sh or .pl. So the ask here is for enhancement to the scanner to discover all files with executable bit set along with the files with extension.