Does IEM have any logs for the API requests so that we can figure out whats happening server side when there is a new request?
There were some logging enhancements in that are, maybe…a year ago? What version are you running?
You can find main information about activities with REST API in the server_audit.log file
Some additional information mainly supposed to be used for troubleshooting purpose from the support team and so not so ‘immediate’ to be consumed can be found in the BESRelay.log file enabling the tracing for the server and filtering for the string: "/api/"
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0023445
thanks @gpoliafico @JasonWalker
BigFix API was not returning computer details at specific point of time for 30-45 minutes. by setting up webreports logging i see below error.
Sun, 26 Apr 2020 13:35:34 +0000 – Background (1912) – CAUGHT EXCEPTION ON BACKGROUND THREAD: HTTP Error 28: Timeout was reached
Sun, 26 Apr 2020 13:35:34 +0000 – 11428 – A background server data exception occurred for datasource Hostname* (22945679658492) : HTTP Error 28: Timeout was reached.
What my might the reason fid the HTTP 28 error?
Need to have a look to the logs, other … not easy to say what can cause the behavior, could be the server need a deeper cleanup, a performance issue on the server, something need to be handle at db level … for this type of issues better involve the support team opening a serviceNow ticket
Error 28 Timeout means the server was not responding. Check the BESRelay.log on the server for that same time period. Also check Windows Event Log, see whether the network went offline of something like that.
And note the logs you’ve shown are in UTC time zone, you may need to translate to the server’s local time zone to correlate other logs.
Are you calling the API from the same machine or a remote host ? If the latter check if there a firewall somewhere. Just try to ping the remote host.
@rgangemi API execution is from remote host and bigfix server is able to ping the remote host.
And the API was not returning data ina particular time and post that it was able to fetch the data.
and in the below message i have BESRelay.log SSL error. but it am not sure if the SSL is the issue as API is able to connect now
@JasonWalker
I have tried check the BESRelay.log.
Sun, 26 Apr 2020 16:15:46 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 17:16:16 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 17:32:04 +0000 - /api/computer/10108538 (8992) - REQUIREMENT FAILURE: false (GetPropertyResults.cpp:130)
Sun, 26 Apr 2020 18:16:47 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 19:17:17 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 20:17:48 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 21:18:19 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Well, this excerpt from besrelay.log is almost three hours later than the problem… what was going on the the besrelay log during the API connection error?
I don’t see any error in the BESRelay log at the specific time on the 26th APR i only see this in the log.
Sun, 26 Apr 2020 00:06:34 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 01:07:05 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 02:07:35 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 02:26:08 +0000 - /api/computer/10108538 (6168) - REQUIREMENT FAILURE: false (GetPropertyResults.cpp:130)
Sun, 26 Apr 2020 03:08:05 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 04:08:36 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 05:09:06 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 06:09:44 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 07:10:15 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 08:10:56 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 09:01:27 +0000 - DatabasePropagator (9176) - Database Error: [Microsoft][SQL Server Native Client 11.0]Shared Memory Provider: Timeout error [258]. (08001: 258)
[Microsoft][SQL Server Native Client 11.0]Unable to complete login process due to delay in prelogin response (08001: 258)
[Microsoft][SQL Server Native Client 11.0]Login timeout expired (S1T00: 0)
Sun, 26 Apr 2020 09:01:29 +0000 - DatabasePropagator (9176) - Disconnecting from database; will reconnect in 10 seconds.
Sun, 26 Apr 2020 09:11:37 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 09:36:24 +0000 - DatabasePropagator (9176) - Database Error: [Microsoft][SQL Server Native Client 11.0]Shared Memory Provider: Timeout error [258]. (08001: 258)
[Microsoft][SQL Server Native Client 11.0]Unable to complete login process due to delay in prelogin response (08001: 258)
[Microsoft][SQL Server Native Client 11.0]Login timeout expired (S1T00: 0)
Sun, 26 Apr 2020 09:36:26 +0000 - DatabasePropagator (9176) - Disconnecting from database; will reconnect in 10 seconds.
Sun, 26 Apr 2020 10:12:19 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 11:12:52 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 12:13:32 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 13:14:15 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 14:14:45 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 15:15:16 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 16:15:46 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 17:16:16 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 17:32:04 +0000 - /api/computer/10108538 (8992) - REQUIREMENT FAILURE: false (GetPropertyResults.cpp:130)
Sun, 26 Apr 2020 18:16:47 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 19:17:17 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 20:17:48 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 21:18:19 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 22:18:49 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Sun, 26 Apr 2020 22:25:37 +0000 - /api/computer/10108538 (14944) - REQUIREMENT FAILURE: false (GetPropertyResults.cpp:130)
Sun, 26 Apr 2020 23:19:20 +0000 - LicenseUpdater (5808) - HTTPS failure on {https://gatherer.bigfix.com/cgi-bin/LicenseServerFrontend.pl}; Message {HTTP Error 60: SSL peer certificate or SSH remote key was not OK: SSL certificate verification failed}; retrying using HTTP
Looks like you lost connection to the database server a few hours earlier; is that expected (were you patching or rebooting the database server?). Or did you maybe have some network issues that day.
Unrelated, but see Important note on LicenseUpdater or Airgap Errors due to certificate about those license update errors.
SSL certificate for SQL has expired on the server. The license update failure might be due to this.