Unable to properly activate/enable WebUI

mxc0bbn · January 24, 2016, 2:54am

So the basic situation is as follows:

Updated 9.2.5.130 to 9.2.6
Enabled WebUI through the Fixlet
I can see the WebUI Apps tab in the operator details
I can see the 3 WebUI apps in the navigation pane.

The problem is that I just can’t get it to finish doing the “WebUI initialization” process. I first enabled everything last night and I know from SUA that ETL can sometimes take a while so I left it overnight…This morning when I tried to logon I was still getting

Looking at the ETL log I can see it’s processing data. I see things like:

Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug Updated SITE_USERS 24 rows in 0.079 seconds (303 rows per second)
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug GET https://localhost:52315/api/etl/users
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug Updated USERS 3 rows in 0.093 seconds (32 rows per second)
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug GET https://localhost:52315/api/etl/user-roles
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug Updated USER_ROLES 2 rows in 0.094 seconds (21 rows per second)
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug Computed all fixlets in 0.016 seconds
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug Loaded patches 19658 rows in 0.015 seconds (1310533 rows per second)
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:debug Loaded critical patches 1688 rows in 0.235 seconds (7182 rows per second)

However from time to time I also see an error pop-up:
Sun, 24 Jan 2016 02:54:14 GMT bf:bfetl:error Child process exited unexpectedly.

Correspondingly in the service-app.log I’m getting constant errors like this:

Sun, 24 Jan 2016 02:54:20 GMT bf:appmonitor:debug Starting etl 1.0.1 on port 5001
Sun, 24 Jan 2016 02:54:41 GMT bf:appmonitor:debug etl 1.0.1 exited unexpectedly.
Sun, 24 Jan 2016 02:54:42 GMT bf:appmonitor:debug Starting etl 1.0.1 on port 5001
Sun, 24 Jan 2016 02:55:03 GMT bf:appmonitor:debug etl 1.0.1 exited unexpectedly.
Sun, 24 Jan 2016 02:55:04 GMT bf:appmonitor:debug Starting etl 1.0.1 on port 5001
Sun, 24 Jan 2016 02:55:16 GMT bf:appmonitor:debug GET https://localhost:52315/api/webui-sites
Sun, 24 Jan 2016 02:55:25 GMT bf:appmonitor:debug etl 1.0.1 exited unexpectedly.

I tried stopping the Root Server Service, deleting everything in the WebUI folder and restarting, but the same starts happening again.

From time to time the logon screen appears allowing me to attempt to logon, but if I hit the browser’s refresh button it almost always goes back to the “Initializing” page…when I do manage to try to log on it sometimes tells me:
“Your administrator has not given you access to any WebUI applications…”. Other times it just goes to “Website cannot display” or if I do it through Firefox I’ll sometimes get a blank page with this at the top:
“{“code”:“ECONNRESET”,“errno”:“ECONNRESET”,“syscall”:“read”}”

Now, I’m trying this with the primary master console operator…which according to the guide should override any permissions not granted to the WebUI apps…BUT…just in case I granted him full access to all the apps (Explicitly since Global was already set as granted).

So…going on 24 hours and still can’t get WebUI to come up.

Anyone have any ideas or suggestions?

Thanks,
Mike.

dexdexdex · January 24, 2016, 7:38pm

This kind of behavior seems awfully suspicious. I think the best way to start debugging is if you post the full contents of the ETL log for us (this will help us narrow down which part of the ETL may be dying for you). Honestly, while you’re at it, might as well get us all of the WebUI logs and zip it up and send it to us so we can check if there’s anything weird going on there too.

The default location on Windows for these logs is in C:\Program Files(x86)\BigFix Enterprise\BES Server\WebUI\logs (just get us everything inside that directory)!

-Dex

mxc0bbn · January 25, 2016, 3:27am

Thanks Dex. I’ll start up the server and post the logs tomorrow. I will tell you though, the logs look pretty much like the snippets I posted except the etl log rolls over at 5MB and just becomes a new log…other than that it all pretty much looks the same…

I’ll post in the morning and see if there’s something you can see that I didn’t.

Thanks,
Mike

mxc0bbn · January 25, 2016, 4:29am

So I decided to just fire up the server and get the logs tonight. I uploaded them to my Google Drive Public Folder here:

Something I noticed. I hadn’t gone into the “common.log” before, but when I just opened it I saw something that could be a clue to the problem I’m having:

Sun, 24 Jan 2016 02:39:22 GMT bf:bfapp:debug Common 0.1.0 listening on port 5003 in production mode
Sun, 24 Jan 2016 03:00:05 GMT bf:bfquery:error POST https://localhost:5001/etl/refresh/users failed: Error: read ECONNRESET
Sun, 24 Jan 2016 03:00:05 GMT bf:error Error: Error: read ECONNRESET
Sun, 24 Jan 2016 03:01:28 GMT bf:bfquery:error POST https://localhost:5001/etl/refresh/users failed: Error: read ECONNRESET
Sun, 24 Jan 2016 03:01:28 GMT bf:error Error: Error: read ECONNRESET
Sun, 24 Jan 2016 03:01:33 GMT bf:error Error: Error: no such user: 1

This could be a non-issue because the BES Root Server service is being run by an AD Domain user, yet since I was just going to pull the logs and nothing more I didn’t bother to start up my AD DC box…so it could just mean that it was trying to authenticate that AD user and it had nothing to authenticate it against.

Or it could mean something different. I’ll fire up all the servers tomorrow and check this log again…

Other than that I didn’t see anything that could tell me what’s going on. Hopefully you will.

Thanks,
Mike

mxc0bbn · January 25, 2016, 7:38pm

So, it looks like I was right about the error above. I started up the environment with the AD server online and the common.log doesn’t show those errors anymore.

Mon, 25 Jan 2016 19:42:17 GMT bf:bfapp:debug Common 0.1.0 listening on port 5003 in production mode

dexdexdex · January 26, 2016, 2:55am

@mxc0bbn: It looks like you got to this before me. Yeah, if you’re authenticating against AD it makes sense that etl/refresh/users isn’t able to get anything.

We could possibly make this more clear in the user documentation / admin guide?

mxc0bbn · January 27, 2016, 10:56pm

Hey Dex,

Yes the particular issue was resolved by firing up my AD server, but I’m stil having the problem in the original post. Still no resolution on that as it’s still saying “etl exited unepectedly” and “Child process terminated abnormally” yada yada…

dexdexdex · January 28, 2016, 3:27am

So I dug around a little bit more… it looks like your ETL process dies right after “Computed Baseline applicability” but right before the next message which should be “Computed devices for master operators”…

This looks like a bug / PMR at this point, but it would really help us out if we could get at the cached copy of data that’s causing you this problem in the first place. The location of this file by default is at C:\Program Files (x86)\BigFix Enterprise|BigFix Server\WebUI\webui.db.

If you can send me a PM with a link to this cached database (only if you are comfortable with this!) that’d help expedite this case significantly.

-Dex

dexdexdex · February 10, 2016, 4:46am

We isolated @mxc0bbn’s problem after a webex session + some debugging on our part. Basically, our ETL caching binary relies a particular instruction POP-CNT for optimization reasons. That particular instruction wasn’t supported in @mxc0bbn’s server.

If people are seeing their ETL processes bomb out and aren’t sure why, there’s a utility which will tell you whether your processor supports the POP-CNT instruction or not: https://technet.microsoft.com/en-us/sysinternals/cc835722.aspx

This is actually a bug that was already open that we’re hoping to fix soon.