BigFix server has not started since db2 was updated

Hello, all!
I have a all-in-one ILMT server on Linux. ILMT-9.2.23, BigFix - 9.5.14.73, db2 - 10.5. This configurations works perfectly.
After the db was upgraded to the ver. 11.5, the BigFix server fails to start.

ilmt-t1:~# /etc/init.d/besserver start
Starting BigFix: BESRootServer: [ OK ]
ilmt-t1:~# /etc/init.d/besserver status
BESRootServer dead but subsys locked

If db2 is stopped, BigFix server runs and tries to connect to the db. It can be seen in logs.
Upgrading of the BigFix to the ver.10.0 didn’t help.
I think that upgrading of the db should be a trivial procedure, and maybe I missed something.
The steps I have done:

  • [db2inst1@ilmt-t1 ~]$ db2 force application all
  • [db2inst1@ilmt-t1 ~]$ db2 terminate
  • [db2inst1@ilmt-t1 ~]$ db2stop force
  • ilmt-t1:/opt/ibm/db2/V11.5/instance# ./db2iupgrade -u db2fenc1 db2inst1
  • [db2inst1@ilmt-t1 ~]$ db2start
  • [db2inst1@ilmt-t1 ~]$ db2 “UPGRADE DATABASE besrepor user db2inst1 using mypassword”
  • [db2inst1@ilmt-t1 ~]$ db2 “UPGRADE DATABASE bent user db2inst1 using mypassword”
  • [db2inst1@ilmt-t1 ~]$ db2 “UPGRADE DATABASE temadb user db2inst1 using mypassword”

Any ideas? I will appreciate any help.

Hello Sergey,
Were there any errors with any of the above DB2 update steps?

Can you check for dasmigr under /opt/ibm/db2/V11.5/instance? If it is present, run dasmigr as root.
Then try to connect to each database to verify successful connection.

Once everything is running, check DB2 logs and BigFix logs for any errors.

I’m not sure I understand how you could have upgraded BigFix to 10.0, if it was already in a broken state and unable to connect to the database?
In any case I think you should open a support incident, so our team can analyze your environment and help with recovery.

I’m not sure I understand how you could have upgraded BigFix to 10.0, if it was already in a broken state and unable to connect to the database?

I have done it in such way:
ilmt-t1:/install/ServerInstaller_10.0.0.133-rhe6.x86_64# ./install.sh -upgrade -opt BES_LICENSE_PVK=/install/license_files/T1-2/licenses/license.pvk -opt BES_LICENSE_PVK_PWD=mypassword
Welcome to the upgrade of BigFix version 10.0.0.133
Warning: Site license credentials have been provided, but they will not be used in this upgrade.
Info: Stopping server services …
Info: The rpm ‘./repos/BESRootServer-10.0.0.133-rhel.x86_64.rpm’ was installed successfully.
Info: Configuring the database for the server component, please wait …
Info: The database for the server component was configured successfully.
Info: Stopping the client service …
Info: The rpm ‘./repos/BESAgent-10.0.0.133-rhe6.x86_64.rpm’ was installed successfully.
Info: Stopping Web Reports service …
Info: The rpm ‘./repos/BESWebReportsServer-10.0.0.133-rhel.x86_64.rpm’ was installed successfully.
Info: Configuring the database for the Web Reports component, please wait …
Info: The database for the Web Reports component was configured successfully.
Redirecting to /bin/systemctl status iptables.service
Error: An error occurred while starting the service: 'BESRootServer’
Info: The service ‘BESFillDB’ started successfully.
Info: The service ‘BESGatherDB’ started successfully.
Info: The service ‘BESClient’ started successfully.
Info: The service ‘BESWebReportsServer’ started successfully.
The ‘BigFix’ upgrade completed successfully.
ilmt-t1:/install/ServerInstaller_10.0.0.133-rhe6.x86_64#

Logs of BESRelay.log after the trying to start the besserver were:
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - BES Root Server version 10.0.0.133 built for RedHat 6 x86_64 running on sysname:Linux release:3.10.0-957.el7.x86_64 arch:x86_64, starting
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - Soft Limit for Max Open Files: 1024.
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - Using cURL library - 7.64.0-DEV
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - OpenSSL Initialized (Non-FIPS Mode)
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - Using OpenSSL crypto library libBEScrypto - OpenSSL 1.0.2u-fips 20 Dec 2019
Wed, 12 May 2021 09:50:25 +0300 - Main Thread (2860222592) - Successfully connected to database
Wed, 12 May 2021 09:50:26 +0300 - Main Thread (2860222592) - Signature Algorithms: sha256, sha1
Wed, 12 May 2021 09:50:26 +0300 - Main Thread (2860222592) - Download Algorithms: sha256, sha1
Wed, 12 May 2021 09:50:26 +0300 - Main Thread (2860222592) - TLS Cipher List: HIGH:!ADH:!AECDH:!kDH:!kECDH:!PSK:!SRP
Wed, 12 May 2021 09:50:27 +0300 - Main Thread (2860222592) - Successfully read server signing key

In any case I think you should open a support incident, so our team can analyze your environment and help with recovery.
I have never done such tickets, can you help me with a link how I can apply correctly?

1 Like

Thank you for the details…I don’t see a database connection problem in the log.

In any case, the support registration page is at https://support.hcltechsw.com/csm?id=csm_registration

Once registered you can “Create a product case” from the link at https://support.hcltechsw.com/csm?id=csm_index or, if you have trouble finding your Customer ID, you can use the Contact Us - Guest Form at the bottom of this page as well.

Hello Sergey,

Were there any errors with any of the above DB2 update steps?

No errors, all goes successfully. I’ve run db2val - there were no errors.

Can you check for dasmigr under /opt/ibm/db2/V11.5/instance? If it is present, run dasmigr as root.
I have a dasmigr, but I haven’t the administration server, do I need it?
ilmt-t1:/opt/ibm/db2/V11.5/instance# ./dasmigr
DBI1074E Administration server does not exist.

Explanation:

An attempt was made to update or upgrade an administration server. An
administration server does not exist.

User response:

Create an administration server using the dascrt command.
ilmt-t1:/opt/ibm/db2/V11.5/instance#

Then try to connect to each database to verify successful connection.
No problem.
[db2inst1@ilmt-t1 ~]$ db2 connect to besrepor user db2inst1 using mypassword

Database Connection Information

Database server = DB2/LINUXX8664 11.5.5.0
SQL authorization ID = DB2INST1
Local database alias = BESREPOR

[db2inst1@ilmt-t1 ~]$

Once everything is running, check DB2 logs and BigFix logs for any errors.
This is my first work with a db2, I don’t know how to work with their logs yet. I don’t see an errors in BigFix.
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - BES Root Server version 10.0.0.133 built for RedHat 6 x86_64 running on sysname:Linux release:3.10.0-957.el7.x86_64 arch:x86_64, starting
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - Soft Limit for Max Open Files: 1024.
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - Using cURL library - 7.64.0-DEV
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - OpenSSL Initialized (Non-FIPS Mode)
Wed, 12 May 2021 09:50:24 +0300 - Main Thread (2860222592) - Using OpenSSL crypto library libBEScrypto - OpenSSL 1.0.2u-fips 20 Dec 2019
Wed, 12 May 2021 09:50:25 +0300 - Main Thread (2860222592) - Successfully connected to database
Wed, 12 May 2021 09:50:26 +0300 - Main Thread (2860222592) - Signature Algorithms: sha256, sha1
Wed, 12 May 2021 09:50:26 +0300 - Main Thread (2860222592) - Download Algorithms: sha256, sha1
Wed, 12 May 2021 09:50:26 +0300 - Main Thread (2860222592) - TLS Cipher List: HIGH:!ADH:!AECDH:!kDH:!kECDH:!PSK:!SRP
Wed, 12 May 2021 09:50:27 +0300 - Main Thread (2860222592) - Successfully read server signing key

you might need to chek to see if db2 is set to restart on a reboot as the instance owner run the command "db2set BD2AUTOSTART -i bd2inst1. If you get YES as a response then DB2 is set to auto start

To configure DB2 to autostart run db2iauto -on db2inst1

If you are using DB2 version 11.5 then DB2 is set to autostart out of the box