WebUI stuck at initializing

Usage and Config WebUI
1

Short Version:

1.) WebUI update on 2018-05-11 was released
2.) WebUI is now stuck at ‘Initializing’ (found 2018-05-12)
3.) Did not have a chance to look at it until approximately 2018-06-18
4.) Rechecked permissions and connection to remote database and it seems fine.
5.) Logs all see fine minus the autopatch.log repeating that there is a DB Issue of ‘DB initialization in progress’

webui initializing
webui initializing936×154 35.9 KB

Longer Version:
Including the above I started troubleshooting what was the issue. I believe we were utilizing the old ETL process before this update. We are now switched over to a direct connect with our Remote Database that I believe to be set up correctly. Admittedly we have not had a chance to look into this until around 2018-06-18. Form what i could tell we did have a couple issues, but from pouring over the forums and various IBM Docs I believe we should be set up correctly. Unfortunately the WebUI is still just initializing.

While troubleshooting I started WebUI with a clean slate by uninstalling the WebUI service and setting it back up. I did not touch our database as it is already established and in use. During this time we also upgraded our BigFix Server to 9.5.9 from 9.5.6.

Currently after my troubleshooting we stand at:
BigFix Server is at 9.5.9. We utilize a remote MS SQL Database. The database is accessed by BigFix via NT Auth utilizing a domain account dedicated to BigFix. As far as I can tell the account can access the BF_Enterprise database and meets the minimum requirements from the WebUI Remote Database Guide. I am seeing minimal traffic in the various WebUI Logs, except for the AutoPatch.log, service-app.log and login.log. I will provide snippets from those logs shortly. I did wait for the WebUI to initialize over night so it has had almost 24 hours now to initialize. From what I have seen it shouldn’t take this long?

I also have a laundry list of links I have looked through that I can add for future reference too. Any one have any ideas? Let me know if there is anything else I may have missed.

1 Like
2

Obviously without error details from logs, the following are suggestions on where I’d start troubleshooting.

  1. Have you confirmed the SQL credentials in question have the necessary login and related permissions to the BFEnterprise database? For example, logged in via MSSQL management stuido.
  2. Is your SQL instance running on standard port configurations? Do you Compliance or Inventory running in your BigFix environment where you can double-check the BFEnterprise database instance credentials.
  3. Have you tried running the “Deploy/UPdate WebUI Database Configuration” fixlet in the “BES Support” site with the known good SQL instance and credential details?
3

Thanks for responding. I am working on approval for the logs still, hope to have them soon.

  1. The Credentials have the necessary login and related permissions as far as I can tell. I can login via MSSQL Management Studio with the aforementioned account. The Domain Account is the DBO for BF_Enterprise.

  2. Yes it is running standard ports, I checked the SQL Server Configuration Manager and the TCP/IP Properties and the Port is standard.

  3. I have as far as I can tell. When I use credentials I know are bad (I.E. known bad password) I receive the appropriate errors in the logs, mainly referencing unable to login.

4

The only logs that update more than once after running “Deploy/UPdate WebUI Database Configuration” are the below (snippets for now. I will have to scrub & validate for the entire logs):

Autopatch.log - This repeats over and over:
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug Engine starting
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug Initialize DB pool.
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug DB pool initialized.
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug Initialize Dependency.
2018-06-26T11:26:38.526Z bf:autopatch:debug doing CustomTableConnector auth
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug Engine error handler triggered.
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug intermittent error is
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug { error: 50,
internalError: ‘dbissue’,
message: ‘DB initialization in progress.’ }
2018-06-26T11:26:38.526Z bf:autopatch:engine:debug Engine will retry after 2 minutes.

Service-app.log - [FQDN] is the correct name for our BigFix Server:
Tue, 26 Jun 2018 11:26:24 GMT bf:bfquery:debug GET https://[FQDN]:52315/api/webui-sites
Tue, 26 Jun 2018 11:26:24 GMT bf:bfquery:debug GET https://[FQDN]:52315/api/webui-sites
Tue, 26 Jun 2018 11:26:24 GMT bf:appmonitor:debug sync pending sites
Tue, 26 Jun 2018 11:26:24 GMT bf:appmonitor:debug GET https://[FQDN]:52315/api/webui-sites
Tue, 26 Jun 2018 11:26:24 GMT bf:appmonitor:debug For Autopatch app, nodeArgs:
Tue, 26 Jun 2018 11:26:24 GMT bf:appmonitor:debug [ ‘–expose_gc’, ‘–max-old-space-size=8192’ ]
Tue, 26 Jun 2018 11:27:34 GMT bf:bfquery:debug GET https://[FQDN]:52315/api/webui-sites
Tue, 26 Jun 2018 11:27:34 GMT bf:bfquery:debug GET https://[FQDN]:52315/api/webui-sites
Tue, 26 Jun 2018 11:27:34 GMT bf:appmonitor:debug sync pending sites
Tue, 26 Jun 2018 11:27:34 GMT bf:appmonitor:debug GET https://[FQDN]:52315/api/webui-sites
Tue, 26 Jun 2018 11:27:34 GMT bf:appmonitor:debug For Autopatch app, nodeArgs:
Tue, 26 Jun 2018 11:27:34 GMT bf:appmonitor:debug [ ‘–expose_gc’, ‘–max-old-space-size=8192’ ]

Right now I am trying to see about confirming if there is any traffic from our BigFix Server to/from the database in regards to the WebUI. I know the Root Server is definitely creating traffic.

5

I have the exact same issue in our environment and have a working case with IBM. I will update you if I have a solution to this issue.

Update: 06/29

We figured the issue was with the port 5000 being used by another application, once we sort that out. WebUI started working!

1 Like
6

We get this a lot, but usually if you’re running into initialization problems for more than 2 hours or so, something has gone wrong.

I’ve written an internal facing guide on how to fix this and I’m working with our documentation team to come up with a more public facing document… but TYPICALLY the stuff that you see when you run into the initialization problem are:

  • Connectivity to the database from the remote WebUI box isn’t good (ports, firewalls, proxies, IPSes)
  • The database credentials are wrong
  • The database credentials were correct, but they’ve changed recently
  • The database credentials are correct but don’t have the necessary privileges for the WebUI to do everything
  • You’re running MSSQL and you have encryption turned on
    ** If that’s you, just turn on this client setting on the WebUI box: _WebUIAppEnv_MSSQL_CXN_ENCRYPT = 1

A lot of this you can verify yourself by testing on something like ODBC from the remote WebUI box. We’re working on a database connectivity tester tool built in to the WebUI so that when you run into the initialization problems, you aren’t off wondering what’s up.

Let me know if those suggestions help you out. If there’s something else going on, then we have something really interesting on our hands and i’m happy to help.

3 Likes
7

Apologies for the delay in getting back to you…

Here is what I did to test the credentials we are utilizing for WebUI:

  1. Login to the WebUI Box (same box as our BES Root Server) with the Windows Credentials that will be utilized for the WebUI.
  2. Utilizing ODBC connect to the BFEnterprise on our remote MSSQL Server. It properly populates the available databases so I know it can read at least.
  3. Testing the Data Source completes successfully. It does mention Data encryption was enforced (thanks for that setting I’m not sure how I missed it, but I’ll get into that setting in a second) though.

So I believe that takes care of db creds being incorrect or recently changed. For the privileges I can launch MS SQL Server Management Studio as the same credentials I utilized for the ODBC and access the BFEnterprise database. Looking the permissions for that account it has the sysadmin role assigned and for user mapping it has the dbo schema for BFEnterprise. Based off the Deployment Requirements I believe those requirements have been met?

We are using MSSQL and encryption is turned on. I am honestly not sure how I missed that setting. I set it then restarted the WebUI Service. This changed the errors I am getting from the above errors I provided to the below:
"ConnectionError: Login failed."
or
"Unable to establish a database connection, please ensure database configuration has been deployed correctly. Failed with error: Login failed."
which does lend to what you said. I am looking into and reverifiying everything above. One of the above two errors started showing up after adding the “_WebUIAppEnv_MSSQL_CXN_ENCRYPT = 1” settings to these logs:

  • datasync.log
  • pps.log
  • prfmgr.log
  • autpatch.log

I’ll work on getting more logs posted if I do not have much luck with re-verifying the login.