Wednesday, December 24, 2014

weblogic.common.ResourceException: Could not create pool connection.The DBMS driver exception was: IO Error: Got minus one from a read call

Problem

The Database is not allowing new connections and getting the "ResourceException: Could not create pool connection.The DBMS driver exception was: IO Error: Got minus one from a read call" when starting up weblogic server instacnes

Error Message


                at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doPrepare(ActivateOperation.java:98)
                at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
                at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
                at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
                at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:218)
                at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
                at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
                at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.prepare(DeploymentReceiverCallbackDeliverer.java:41)
                at weblogic.deploy.service.internal.statemachines.targetserver.AwaitingContextUpdateCompletion.callDeploymentReceivers(AwaitingContextUpdateCompletion.java:164)
                at weblogic.deploy.service.internal.statemachines.targetserver.AwaitingContextUpdateCompletion.handleContextUpdateSuccess(AwaitingContextUpdateCompletion.java:66)
                at weblogic.deploy.service.internal.statemachines.targetserver.AwaitingContextUpdateCompletion.contextUpdated(AwaitingContextUpdateCompletion.java:32)
                at weblogic.deploy.service.internal.targetserver.TargetDeploymentService.notifyContextUpdated(TargetDeploymentService.java:225)
                at weblogic.deploy.service.internal.DeploymentService$1.run(DeploymentService.java:189)
                at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: Got minus one from a read call
                at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(Unknown Source)
                at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)
                at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)
                at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:250)
                at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1329)
                at weblogic.jdbc.common.internal.ConnectionPool.start(ConnectionPool.java:176)
                at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:507)
                at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:428)
                at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:280)

                ... 28 more


Cause

From the above error message there is nothing wrong from WebLogic side, and we need to check if there is any thing wrong at Database end. Database team has to investigate to find out the exact root cause of the issue. The immediate cause of the problem is that the JDBC driver has attempted to read from a network Socket that has been closed by "the other end" (database). The possible causes for this issue are

(1) Hung sessions which are maxing out the processes on Database server
(2) Database is in inconsistent state or may be it is under maintenance
(3) Database Could have reached maximum number of connections
(4) Check there may be network issues
(5) If there are too many database connections opened to the database service, it could refuse new connections.


Solution

If you don't see any of the above causes, restarting or bouncing the database will fix the issue.
If it is related to insufficient resources at Database end, you may need to increase processes or sessions.
If there are any hung sessions, Database team has to kill all hung sessions and start the database server to fix the issue.