TEM Console freezes

(imported topic written by AviE91)

Hey,

i’m experiencing freezes when i approve a task / action after selecting all the options, even the simplest action such as unlock or edit a computer settings issued to number of computers

causes the console to get stuck even stopping an action causes this, it gets stuck at the stage “sending to clients” then it stops responding,

it started 1 day ago after we received a new license that removed some sites we didn’t use, following we deleted an operator account (all done from BESADMIN.exe),

when we pressed ok the BESADMIN.exe took long time to propogate to the point where we needed to terminate the process after waiting 1+ hour,

is there anything we can do check everything is consistent now or enable logging on the console / server side interface for the console to see where it’s stuck??

(imported comment written by MikeOttum)

What version of the Console/Server are you using?

(imported comment written by SystemAdmin)

I would recommend opening a PMR with Technical Support on this issue so that we can help try to debug this issue.

The following link is to the world wide Support directory. http://www.ibm.com/planetwide/region.html

Please be sure to have your IBM Customer Number ready when you call.

(imported comment written by MattBoyd)

We opened a support case a month ago from tomorrow regarding console/root server process performance issues and have gotten the run around. I think our issue is (at least partially) being caused by a huge increase in the size of signed data that each action has (as well as several tables in the database). The size of our actionsite quadrupled! In our case, I think there’s a bug in BESAdmin related to how it re-signs content - the base64 encoded signature of afxm files in our actionsite is huge, and decoding it shows that there’s a lot of duplicate text. The same goes for signing data stored in various areas in the database.

(imported comment written by SystemAdmin)

I know you are now actively engaged with Support on this but send me a private message if you get stalled again. Opening a PMR is the correct choice on this sort of issue in general.

Thanks,

Kevin

(imported comment written by MattBoyd)

correction… fxf (action) files, not axfm.

(imported comment written by lavaliant91)

We are experiencing the same issue with console freezes and have opened a PMR about 2 weeks ago, sending logs and process dumps. I’m hoping for resolution soon, as we are in the middle of a large scale out of the service for our organization.

(imported comment written by AviE91)

Hey,

To update we have opened a PMR and after exchanging logs and doing besadmin /resetdatabaseepoch it’s still frozen at the moment we are at a point we no login can be done and we sent the database to IBM, I will update when there’s progress on the matter.

(imported comment written by AviE91)

Hey Again,

just wanted to update that apparently during one of our upgrades, the devs think it’s the one from 8.0 to 8.1 which is our current version, the DB version stayed at 8.0 while all the rest of the components were upgraded, so they suggested we run the following command

NT Authentication

runas /user:<NTAccountName> “BESAdmin.exe /upgrade /dsn:enterprise_setup”

we did that and opened besadmin gui and all users were painted in red “not exist in database” so we tried repair on one, but all our PVKs were revoked by that process, so now we’re waiting to hear from devs about how to continue.

(imported comment written by MattBoyd)

Wow! That’s the exact opposite of support!

Our case remains unresolved. I can’t even get a follow-up from devs. In case it helps someone else struggling with this in their 8.2 environment, I was able to somewhat alleviate our issues. In particular, one TEM query that retrievies CUSTOM_SITE_WRITER records seems to be the most significant bottleneck in our deployment. To give you an idea of what kind of impact this has, I used SQL Management Studio to measure the amount of data returned from this

query:

select CSW.Sitename,

U.MastheadUsername,

CSW.UserID,

CSW.IsOwner,

CSW.AuthorizingMasterOperatorID,

CSW.AuthorizingOwnerID,

CSW.ServerID,

CSW.Certificate,

CSW.SupportingCerts,

CSW.SignedData

from CUSTOM_SITE_WRITERS CSW

INNER JOIN USERINFO U

on ( CSW.UserID = U.UserID and U.IsDeleted = 0 )

where CSW.IsDeleted = 0

Each time I ran the query, the result set was around 120 MB in size, which is quite large (in my opinion) for a result set that runs frequently. The query runs 20-30 times per minute on average (as I mentioned previously) depending on the number of operators connected! And, to give you a better baseline, the size of the result set from this query prior to migrating servers was only 24 MB!

Then, I updated the site writer permissions on roughly 20 console operators (I wrote a query to determine which console operators had the largest sum of SiteData in CUSTOM_SITE_WRITERS and performed this operation on them) and measured the result set again. The result set is now 62 MB.

Here’s what I did in the console that reduced the size of SignedData field records associated with particular console operators:

  1. Go to Operators in the All Content pane

  2. Select the operator, then go to the Sites pane

  3. Sort by Explicit permissions, then select all entries in the list with explicit 'Writer Permissions"

  4. Click “Reader” to change the permissions then click “Writer” again (without saving) to change them back. Click Save Changes

When I did this, the size of the SignedData field in each associated record from the CUSTOM_SITE_WRITERS table would shrink from ~100 KB to ~12 KB. I also noticed that the certificate and SupportingCerts fields were set to an empty string (in most cases, they contained data prior to performing this operation). However, it appears that the size of your signing key is a factor in the final size of the SignedData field, so your results may vary. The lack of control over this is very frustrating!

The console is somewhat more responsive and retrieving action results is faster, though there’s definitely room for improvement. Also, this doesn’t address the issue with our Actionsite quadrupling in size, though I have a feeling that they’re related.

Here’s the query I wrote to check the size of SignedData in the CUSTOM_SITE_WRITERS table - please draw your own conclusions, doin so at your own risk. In our environment, the total size for some console operators was 4000+ KB before updating the site writer permissions. Prior to migrating, the the size for most console operators was under 100 KB. Obviously, there’s other factors involved, especially the number of sites you have and the number of sites the console operator has explicit writer permissions for.

select U.MastheadUsername,

CSW.UserID,

U.Username,

(SUM(DATALENGTH(CSW.SignedData))/(1024)) as

Total Size of SignedData fields in KB

from CUSTOM_SITE_WRITERS CSW

INNER JOIN USERINFO U

on ( CSW.UserID = U.UserID and U.IsDeleted = 0 )

where CSW.IsDeleted = 0

GROUP BY U.Username, U.MastheadUsername, CSW.UserID

ORDER by

Total Size of SignedData fields in KB

desc

Good luck and please keep sharing anything you find.

(imported comment written by MattBoyd)

Ah yes, the wonderful IBM forum bbcode. Normally, I’d click the preview tab before posting, but now the preview tab throws a JavaScript error and I can’t back to the plain text tab. Rad!

select  U.MastheadUsername, CSW.UserID, U.Username, (SUM(DATALENGTH(CSW.SignedData))/(1024)) as [Total Size of SignedData fields in KB] from    CUSTOM_SITE_WRITERS CSW INNER JOIN USERINFO U on ( CSW.UserID = U.UserID and U.IsDeleted = 0 ) where   CSW.IsDeleted = 0 GROUP BY U.Username, U.MastheadUsername, CSW.UserID ORDER by [Total Size of SignedData fields in KB] desc

(imported comment written by AviE91)

Hey,

Glad it’s over, we restored the DB to prior activating the new license and re activated the new license, the propagation went much faster this time and it was clear that on the 1st time something was wrong, corruption is all i can pinpoint the issue to.