We had been using a custom site with security checklist fixlets. I deleted and recreated that custom site on my IEM server and now when I try to do an import on the SCA server I get this message “contain the same Sentinel ID” for several of the fixlets in the site. I deleted the site altogether and still get the messages (I read somewhere that when a custom site is deleted that it is just marked as such and not actually removed from the IEM DB.)
I am thinking that the SCA server is trying to import the same fixlets as new (rows in the DB) but the old fixlets are still there in the SCA DB with the same “sentinel ID”. How do I solve this?
In the Server Settings -> Data Retention setting I enabled “Discard data older than” and set to 1 day and clicked the Restart Service button. When does this purge actually occur and does it actually delete records from the DB? I still see in the GUI that it thinks there is data going back several months.
So if I can either remove old data from the SCA server or figure out how to physically delete data from the IEM server that is marked as deleted, it get me working again. Anyone know how to do this?
Yes the log does tell me which Fixlets. I already tried just deleting those fixlets and reimporting, but the import still flags them. That is when I deleted the site completely from the IEM server and reimported, still gives the same error messages about the same fixlets and same (now removed) site!
Well, I discovered the BESAuditTrailCleaner utility, downloaded the latest version 3.0.10.94 and ran it to remove everything. I still get duplicate Sentinel errors but no longer do I get the list of Fixlet names. Here is the complete import log:
Import Log:
# Logfile created on 2015-09-01 21:20:11 +0000 by logger.rb/v1.2.7
2015-09-01 21:20:11 (+0:00:00.000) INFO: TEMA version: 1.5.78
2015-09-01 21:20:17 (+0:00:06.064) INFO: Calling Model.before_snapshot: Success
2015-09-01 21:20:18 (+0:00:00.648) INFO: Initialize datasource Data Source: Success
2015-09-01 21:20:18 (+0:00:00.149) INFO: ETL from Data Source - DatasourceSite (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:20:19 (+0:00:00.748) INFO: ETL from Data Source - RawDatasourceAnalysisActivation (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:20:26 (+0:00:06.962) INFO: ETL from Data Source - RawDatasourceAnalysis (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:20:32 (+0:00:05.727) INFO: ETL from Data Source - DatasourceAnalysis (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:21:08 (+0:00:36.790) INFO: ETL from Data Source - DatasourceProperty (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:21:09 (+0:00:00.063) INFO: ETL from Data Source - RawComputerId (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:21:09 (+0:00:00.782) INFO: ETL from Data Source - Computer (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:21:10 (+0:00:00.221) INFO: ETL from Data Source - ComputerPropertyValue (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:21:10 (+0:00:00.292) INFO: ETL from Data Source - ComputerDimension (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:21:52 (+0:00:42.040) INFO: ETL from Data Source - RawDatasourceFixlet (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:22:43 (+0:00:51.579) INFO: ETL from Data Source - DatasourceFixlet (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:22:52 (+0:00:08.536) INFO: ETL from Data Source - DatasourceGroupFixletResult (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:23:01 (+0:00:08.680) INFO: ETL from Data Source - DatasourceGroup (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:23:01 (+0:00:00.029) INFO: ETL from Data Source - RawDatasourceGroupMembership (0x000000000029C10E - 0x00000000002AC02E): Success
2015-09-01 21:23:01 (+0:00:00.127) INFO: ETL from Data Source - DatasourceGroupMembership (0x000000000029C10E - 0x00000000002AC02E): Success
So I guess I will try this again in the morning, perhaps by then the setting to remove data older than 1 day will have been processed and it will be like no data has been imported yet… there should be no Sentinel ID’s at all to be duplicates of anything.
I don’t recommend running the audit trail cleaner if your SCA is not fully imported. The audit trail cleaner removes records from BFEnterprise which can cause problems in SCA if SCA didn’t manage to pull updated info about something before it was removed. I’d refrain from using it until you’ve unborked SCA, and hopefully you haven’t cleaned any rows that were generated since SCA imports started failing.
The sentinel problem itself is somewhat related. Your conclusions so far are more or less correct. The problem is due to it trying to insert the new fixlets before the old fixlets were removed from the system. Except in this case it’s a bug / incorrect handling, since doing both at once is a valid operation.
In the future it’s probably safer to not recreate the sentinel fixlets (assuming they didn’t change, and they probably shouldn’t have but I can’t fully guarantee this…) and only recreate the checks.
If you bump to SCA 1.6.139 there is better handling of duplicate sentinels. Otherwise there is a “duplicate sentinel fix” I floated around previously that might work for you, could check bugzilla to get help with that.
Thanks for the advise Karl. By the time I read your post I had already run the audit trail cleaner, but that did not work anyway. I did just now upgrade SCA to v1.6.139, then from the SCA server itself I browsed and clicked the button to upgrade the schema. However when that was complete and I tried to log into SCA normally I simply get the “We are sorry, but something went wrong. Contact your IBM Endpoint Manager Analytics administrator if the problem persists.”
Well it turns out this problem is only when trying to log in with an LDAP ID, I can log in with a local ID fine. Testing the LDAP connection works, but I cannot create new LDAP users. I simply get an “Unknown Error” popup when trying to add a new LDAP user.
Previously I defined a Directory Server to use IBM BluePages for authentication:
LDAP Server: Other
User Filter: (objectclass=person)
Login Attribute: mail
Group Filter: (objectclass=person) (I don’t care about groups)
Membership Attribute: uniqueMember
Search Base: o=ibm.com
SSL
Anonymous Bind
Host: bluepages.ibm.com
Port: 636
Security Protocol: TLS 1.0
The Test Connection function works just fine with the above settings:
Connecting: Success
Querying Root DSE: Naming Context:
Searching Groups: Found at least one group
Searching Users: Found at least one user
There is an unconfirmed bug regarding upgrading of Directory Servers. Try inspecting the directory_servers table in the SCA DB. Then for the non-working directory server, I suspect security_protocol would be blank despite the SCA UI saying it is TLS 1.0. Try manually inputting “TLSv1” into that field.
It may be helpful to check the tail of tema.log to see what errors are being thrown right when you trigger the “something went wrong” page.
Karl, thanks again for pointing me in the right direction. I thought I had monkeyed with the security protocol choices already. But, this time I changed from TLS 1.0 to TLS 1.1, saved and was able to create a new LDAP ID (and log in with it). Then I changed back to TLS 1.0 and I can still log in. So I guess editing and changing forced it to write the value into the DB table properly.
More recent versions of SCA will have some level of built-in protection against the issue, but it is something that can be caused by various things and is in general indicative of an unexpected state for BigFix data. A good preventative measure is to ensure SCA imports are always up-to-date when attempting any kind of DB cleanup.
In SCA 1.7 there is a special import button called “Remediate” under Server Settings, that may help to resolve the issue. The idea is to re-synchronize SCA with the BigFix server’s data. Unfortunately this is similar in scope to a “full import”. If you use this function, I highly recommend a SCA DB backup beforehand, as well as knowing it will take several times longer than a scheduled import.
Thanks, I upgraded to 1.7 but it didn’t help. What I ended up having to do was go into SQL and open the table in Design and select “Manage Indexes and Keys” option. Selected the key indicated in the import error log and changed the Yes/No drop down for the “Is Unique” property to allow duplicates and only then would SCA import run. Scheduled imports have run normally since.
I don’t recommend changing the DB schema, maybe it worked for now but you’ve put the DB into an abnormal state that may cause problems in the future.
Now that you have everything in the table, though, you should be able to easily identify all the sentinels which are duplicated. I would collect a list of the sentinels and figure out which ones should not be there.
If the Remediate function did not resolve the issue, it’s entirely possible that there indeed exist duplicate sentinels in your BigFix deployment, track down a few of the duplicate sentinels and see whether they exist in BigFix.
PLEASE ADHERE TO THE HOTFIX INSTRUCTIONS IN THE READ ME SECTION BELOW (THE README IS ALSO INCLUDED IN THE ATTACHED .RAR FILE)
GTS Sentinel Fix Readme
Running the cleanup ETL:
Backup both the IEM and SCA databases.
In the SCA database “TEM_Analytics”, increase log size and limited to 100GB (note that currently is set to ~50GB and it’s not enough for the growth of transaction log)
Backup and replace TEMA\work\tema\webapp\WEB-INF\app\models\datasource_fixlet.rb
Backup and replace TEMA\work\tema\webapp\WEB-INF\domains\scm\app\models\scm\sentinel.rb
Restart SCA service.
Server Settings => Untick “Discard data older than…” => Save
Imports => Disable scheduled imports for the moment
Imports => Import Now
When import completes, it is recommended to restore original datasource_fixlet.rb and sentinel.rb.
Remember that you must restart the server whenever replacing files in SCA.
After the ETL:
IMPORT LOG:
There should be two sets of extra log statements in the cleanup ETL’s import log. The first set is which fixlets were deleted. This is mostly for informational purposes, although some manual inspection / sanity checks should be performed.
The second set is duplicate sentinels in the sentinels table. THIS IS IMPORTANT! Look at the sets of duplicates, decide which sentinel you want to keep going forward, then soft-delete the other one in IEM. Failure to do so may result in more failed ETLs once you revert sentinel.rb.
SETTINGS:
You can safely re-enable automatic imports once you’ve soft-deleted duplicates in IEM.
You can try re-enabling discarding old data as well. If this ends up causing transaction log full errors, steps to manually prune data should be taken until it is at a state handleable by the automatic pruning.
SCA REPORTS:
Even if the import succeeds, you should note changes to the state of the reports. Are there any large discrepencies in reports before/after the cleanup ETL? Missing checks? Missing checklists? Do the graphs look funky? Etc.
“Unable to process metadata from” ERRORS:
It is doubtful that this is due to any changes we made. It is likely it is because fixlets other than sentinel fixlets are suffering from the same zombie problem. Someone should investigate these one by one and see what their deal is.