Tivoli Remote Control login page failing with database connection error

PraveenV · August 31, 2018, 2:30pm

Hi Team,

We are running TRC version 9.1.2.00.20 with standalone WAS and remote DB2, we have RC on Linux server.

Now when we try to run our RC web portal to we getting DB connection error and sometime even webpage doesn’t starts. After checking RC server log we can see below error :

25-Aug-2018,08:14:37 [WebContainer : 41] [ERROR] - Threw SQLException whilst getting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,000 seconds.
25-Aug-2018,08:14:37 [WebContainer : 41] [ERROR] -
An unexpected application error occurred. Details are shown below. There may be additional information in the log files.

Status: 500 - (Internal server error) The server encountered an error and can’t fulfill the request.

Exception occurred: class java.lang.NullPointerException

Message: java.lang.NullPointerException

java.lang.NullPointerException

java.lang.NullPointerException
at com.ibm.uk.greenock.db.PromptedQuery.(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.hwKeyMatch(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.getHWKey(Unknown Source)
at com.ibm.uk.greenock.callHome.CallHomeDAO.processCallHome(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.handleRequest(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.uk.greenock.filter.SecureRequestFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.uk.greenock.filter.UTF8Filter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

25-Aug-2018,08:20:49 [WebContainer : 44] [INFO ] - Attempting match with the old target data
25-Aug-2018,08:20:56 [WebContainer : 42] [ERROR] - Threw SQLException whilst getting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,000 seconds.
25-Aug-2018,08:20:56 [WebContainer : 42] [INFO ] - Database connection error
25-Aug-2018,08:21:08 [WebContainer : 43] [ERROR] - Threw SQLException whilst getting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,000 seconds.
25-Aug-2018,08:21:08 [WebContainer : 43] [ERROR] -
An unexpected application error occurred. Details are shown below. There may be additional information in the log files.

Status: 500 - (Internal server error) The server encountered an error and can’t fulfill the request.

Exception occurred: class java.lang.NullPointerException

Message: java.lang.NullPointerException

java.lang.NullPointerException

java.lang.NullPointerException
at com.ibm.uk.greenock.db.PromptedQuery.(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.hwKeyMatch(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.getHWKey(Unknown Source)
at com.ibm.uk.greenock.callHome.CallHomeDAO.processCallHome(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.handleRequest(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.uk.greenock.filter.SecureRequestFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.uk.greenock.filter.UTF8Filter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)

25-Aug-2018,08:21:41 [WebContainer : 14] [ERROR] - Threw SQLException whilst getting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,000 seconds.
25-Aug-2018,08:21:41 [WebContainer : 14] [ERROR] -
An unexpected application error occurred. Details are shown below. There may be additional information in the log files.

Status: 500 - (Internal server error) The server encountered an error and can’t fulfill the request.

Exception occurred: class java.lang.NullPointerException

Message: java.lang.NullPointerException

java.lang.NullPointerException

java.lang.NullPointerException
at com.ibm.uk.greenock.db.PromptedQuery.(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.perfectMatch(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.getHWKey(Unknown Source)
at com.ibm.uk.greenock.callHome.CallHomeDAO.processCallHome(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.handleRequest(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.uk.greenock.filter.SecureRequestFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.uk.greenock.filter.UTF8Filter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:949)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)

25-Aug-2018,08:21:41 [WebContainer : 14] [ERROR] -
An unexpected application error occurred. Details are shown below. There may be additional information in the log files.

Status: 500 - (Internal server error) The server encountered an error and can’t fulfill the request.

Exception occurred: class java.lang.NullPointerException

Message: java.lang.NullPointerException

java.lang.NullPointerException

java.lang.NullPointerException
at com.ibm.uk.greenock.db.PromptedQuery.(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.perfectMatch(Unknown Source)
at com.ibm.uk.greenock.asset.RCAsset.getHWKey(Unknown Source)
at com.ibm.uk.greenock.callHome.CallHomeDAO.processCallHome(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.handleRequest(Unknown Source)
at com.ibm.uk.greenock.servlet.CallHomeExec.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1233)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:782)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.uk.greenock.filter.SecureRequestFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.uk.greenock.filter.UTF8Filter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebApp

29-Aug-2018,09:17:05 [WebContainer : 25] [ERROR] - Threw SQLException whilst get ting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,000 seconds.
29-Aug-2018,09:17:05 [WebContainer : 25] [INFO ] - Database connection error
29-Aug-2018,09:17:26 [WebContainer : 48] [ERROR] - Threw SQLException whilst get ting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,001 seconds.
29-Aug-2018,09:17:26 [WebContainer : 48] [INFO ] - Database connection error

After that if we reboot our WAS and DB instance will be able to connect the webpage again.

Please help us to resolve this issue.

f.pezzotti · August 31, 2018, 4:49pm

According to the log file the WAS is not able to connect to the datasource (database):

[WebContainer : 41] [ERROR] - Threw SQLException whilst getting a connection from a datasource :J2CA1010E: Connection not available; timed out waiting for 180,000 seconds.

What WAS version do you have ? Have you installed the latest fixpack ?

I would suggest you to restart only the WAS when the error occur again. If this solve the problem then there is a problem in the configuration of the WAS.

The maxPoolSize might be limiting the number of connection that the server is allowed to open with the database. For the WAS Liberty embedded with the product the maxPoolSize value is set to 173:

<server> 
<connectionManager maxPoolSize="173"  maxIdleWait="3600" /> 
<webContainer  deferServletLoad="false" 
channelWriteType="sync"  
/>  
<executor maxThreads="148" coreThreads="54" keepAlive="3600" />  
</server>

This is the memory.xml default content.
Please make sure that those values are aligned with your WAS configuration.

PraveenV · September 4, 2018, 9:55am

Hi Thanks for your reply…

We are doing the same also I updated the MAX connection to 173 but still facing the same issue on daily basis.
We using Standalon WAS with version 8.5.5.13, we planning to update it with latest one

PraveenV · September 4, 2018, 10:37am

please find the snap of the settings in our environment