I successfully ran the fixlet “Upgrade to the latest version of IBM License Metric Tool (9.2.28.0)” in our BigFix DEV environment yesterday. Last night I attempted to run the same fixlet in our BigFix PROD environment. The progress shows the script is still “Running” at this step in the script:
waithidden “{parameter “temp_folder” as string}\setup-server-windows-x86_64.bat” -f “{parameter “temp_folder” as string}{parameter “resp_file”}” -i silent >> “{parameter “log_file” as string}” 2<&1
I’ve started combing the BigFix Forum for answers as the process has been “Running” for coming up on 24 hours. I am at a loss as to how to troubleshoot this. I appreciate any assistance I can get.
We’re running all-in-one BigFix and ILMT on one Windows 2019 server.
Troubleshooting will start with that log file - any errors appearing in it? What are the last few lines?
Thank you @ssakunala for this information about what and where to find the logs.
No error messages…
All I’m seeing in the ILMT_upgrade.log is this:
“The product is being installed…”
Nothing is logged in the msg_servers.log or trace_servers.log in the following directory:
C:\Windows\System32\config\systemprofile\LMT9.2.28.0\
What do you see in the BigFIx Client Log for the day you started the install?
Look in C:\Program Files (x86)\BigFix Enterprise\BES Client__BESData__Global\Logs\
You’ll see up to 10 log files with the naming convention: yyyymmdd.log.
Search for references to the Fixlet by name or ID (1002).
Log file 20220706.log indicates the commands succeeded:
| 3360 |
Relevant - Upgrade to the latest version of IBM License Metric Tool (9.2.28.0) (fixlet:3454) |
9.2.28.0 |
| 3372 |
ActionLogMessage: (action:3454) Download url: ‘http://software.bigfix.com/download/lmt/server/9.2.28.0_GA/LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip’ |
9.2.28.0 |
| 3381 |
Command succeeded (Prefetch download manager collected file) prefetch LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip sha1:e0a7a781a7df9bbc93c189aca3192813b6231b0f size:1239507824 http://software.bigfix.com/download/lmt/server/9.2.28.0_GA/LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip sha256:c98fef1f997b0778c4e40e2face07b9d1cd37ad3e0e50801a68b811efb352fff (action:3454) |
9.2.28.0 |
| 3383 |
Command succeeded parameter “pkg_name” = “LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip” (action:3454) |
9.2.28.0 |
| 3396 |
Command succeeded move “__Download\LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip” “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip” (action:3454) |
9.2.28.0 |
| 3400 |
Command started - waithidden cmd.exe /c ““E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\unzip.exe” -oq “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip” -d “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade”” (action:3454) |
9.2.28.0 |
| 3402 |
Command succeeded (Exit Code=0) waithidden cmd.exe /c ““E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\unzip.exe” -oq “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip” -d “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade”” (action:3454) |
9.2.28.0 |
| 3403 |
Command succeeded delete “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\LMT-server-9.2.28.0-20220614-1645-windows-x86_64.zip” (action:3454) |
9.2.28.0 |
Is that where the data related to action 3454 ends?
No, it ends at the following line item. Every command above it succeeded.
3416 Command started - waithidden “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\setup-server-windows-x86_64.bat” -f “E:\BigFixEnterprise\BESClient__BESData\IBM License Reporting__Download\ILMT_upgrade\fixlet_upgrade_response.txt” -i silent >> “E:\BigFixEnterprise\BESClient\LMT\ILMT_upgrade.log” 2<&1 (action:3454) action:3454
I’m checking the Windows Event logs now…
Perhaps you should open a case with IBM to get direct assistance.
It looks like the service terminated unexpectedly.
System Events Log
7/6/2022 4:26:37 PM Event ID: 7034 | Service Control Manager
The IBM License Metric Tool 9.2.28.0 service service terminated unexpectedly. It has done this 1 time(s).
Have HCL engineers seen similar issues?
UPDATE
I’ve opened a case with IBM for assistance.
Thanks everyone for your help to get me this far. I appreciate it!
1 Like
Any success with the upgrade? Is any information logged to E:\BigFixEnterprise\BESClient\LMT\ILMT_upgrade.log?
So far, no, @ssakunala. I’ve combed through the logs and nothing stands out. It seems something stopped the process but it wasn’t logged. I’m still waiting for IBM to acknowledge the logs I uploaded to the case on 7/12/22. My coworker asked for a status update this morning. Hopefully they will respond.
@ssakunala … apologies for the dup post. I was having issues with my account this morning and I created a new account to respond to your question.
It maybe a long shot, but I recall Microsoft’s EMET tool with ASLR would cause Java applications to fail to launch. Those protections are now rolled into the OS on Server 2019 as part of Defender APT.
Do you have Defender APT turned on, and if so, could you consider turning it off to see whether that’s the issue? If that’s it, we can look at adding exceptions for the ILMT Java Runtime.
1 Like
You can also try manual upgrade by downloading the installation package. This may give more information about the issues with virus/security scanner mentioned by Jason.
1 Like
@JasonWalker That’s great information to know. We turned on Defender APT a few months ago and also recently upgraded the OS on that Windows server to v2019. I will reach out to our teams to coordinate the effort of turning Defender APT off for a short period and then I will attempt the ILMT v9.2.28.0 upgrade again manually as @ssakunala suggested. Thank you both so much!
As a follow up to this thread, I worked with our network admins and we ruled out Defender APT.
As I understand it, we’re not running Defender APT and our development environment is configured the same as our production environment. But that didn’t explain why the installation in development succeeded and the installation in production had issues. We chalked it up to an anomaly and moved on because the update appeared to be successful, at least from the ILMT Client perspective (not so much according to BigFix).
On September 20 I attempted to install ILMT v9.2.29.0, fixlet “Upgrade to the latest version of IBM License Metric Tool (9.2.29.0)” and the same thing happened.
Here are my observations:
-
We’ve not seen any other issues with updates until recently.
-
The log directories for the update (in %USERPROFILE%\LMT) for both the v9.2.28.0 and v9.2.29.0 updates are empty (file size is 1 kb).
-
In BigFix, the updates both still show “running” on the root server (have screenshots if needed).
-
In the ILMT Client, the “Update Schema” portion of the upgrade for the database was successful for both v9.2.28.0 and v9.2.29.0 and the “About” window showed both updates were successful to the latest version.
-
In the master.tag file I noted the entry “ALREADY_INSTALLED_VERSION=9.2.28.0” so on some level the v9.2.28.0 had some measure of success.
-
There are ffdc logs in %USERPROFILE%\LMT\wlp\usr\servers\server1\logs\ffdc_archiveLogs (all have a date/time stamp of 9/20/2022 3:13 PM but the most recent date in the files is 3/14/22 and earliest date is 5/17/18.
I don’t know enough about the software to know where to go from here. IBM was absolutely no help at all. Any thoughts?
You may want to consider Reinstalling License Metric Tool with an existing database. This will allow you to replace the IBM WebSphere Liberty Server files, and hopefully, allow a successful update by Fixlet next time around.
1 Like