org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager#createConnections leaks connection if replay throws exception

[Nick-The-Uncharted]

Bug Report

For English only, other languages will not accept.

Before report a bug, make sure you have:

org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager#createConnections(java.lang.String, java.lang.String, javax.sql.DataSource, int, org.apache.shardingsphere.infra.executor.sql.execute.engine.ConnectionMode) leaks connection if replay throws exception.

Thread stack:

java.sql.SQLNonTransientConnectionException: (conn=2848) Socket error
	at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:300)
	at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:366)
	at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:938)
	at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:854)
	at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:773)
	at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:697)
	at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:553)
	at org.mariadb.jdbc.Connection.setAutoCommit(Connection.java:204)
	at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:699)
	at com.alibaba.druid.filter.FilterAdapter.connection_setAutoCommit(FilterAdapter.java:964)
	at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:694)
	at com.alibaba.druid.filter.logging.LogFilter.connection_setAutoCommit(LogFilter.java:467)
	at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:694)
	at com.alibaba.druid.filter.FilterAdapter.connection_setAutoCommit(FilterAdapter.java:964)
	at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:694)
	at com.alibaba.druid.filter.FilterAdapter.connection_setAutoCommit(FilterAdapter.java:964)
	at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:694)
	at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.setAutoCommit(ConnectionProxyImpl.java:416)
	at com.alibaba.druid.pool.DruidPooledConnection.setAutoCommit(DruidPooledConnection.java:732)
	at org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.lambda$setAutoCommit$0(DriverDatabaseConnectionManager.java:165)
	at org.apache.shardingsphere.driver.jdbc.adapter.invocation.MethodInvocationRecorder.replay(MethodInvocationRecorder.java:53)
	at org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.createConnections(DriverDatabaseConnectionManager.java:386)
	at org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.getConnections(DriverDatabaseConnectionManager.java:359)
	at org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.getRandomConnection(DriverDatabaseConnectionManager.java:335)
	at xxx.ShardingDBConfiguration$SmartConnection.prepareStatement(ShardingDBConfiguration.java:50)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.instantiateStatement(PreparedStatementHandler.java:88)
	at org.apache.ibatis.executor.statement.BaseStatementHandler.prepare(BaseStatementHandler.java:90)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.prepare(RoutingStatementHandler.java:60)
	at org.apache.ibatis.executor.BatchExecutor.doUpdate(BatchExecutor.java:71)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
	at jdk.internal.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
	at xxx.MysqlInterceptor.intercept(MysqlInterceptor.java:58)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:59)
	at jdk.proxy2/jdk.proxy2.$Proxy273.update(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at jdk.proxy2/jdk.proxy2.$Proxy273.update(Unknown Source)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
	... 44 more
Caused by: java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source)
	at java.base/java.net.Socket$SocketInputStream.read(Unknown Source)
	at java.base/java.io.BufferedInputStream.fill(Unknown Source)
	at java.base/java.io.BufferedInputStream.read1(Unknown Source)
	at java.base/java.io.BufferedInputStream.read(Unknown Source)
	at org.mariadb.jdbc.client.socket.impl.PacketReader.readReusablePacket(PacketReader.java:62)
	at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:153)
	at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:915)
	... 85 more

Which version of ShardingSphere did you use?

5.5.0

Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?

ShardingSphere-JDBC

Expected behavior

Connection is closed even some exception throwed during replay.

Actual behavior

Connection is never closed when some exception throwed during replay.

Reason analyze (If you can)

org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager#createConnections(java.lang.String, java.lang.String, javax.sql.DataSource, int, org.apache.shardingsphere.infra.executor.sql.execute.engine.ConnectionMode) didn't handle exception while calling replay method.

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

Example codes for reproduce this issue (such as a github link).

[terrymanu]

Could you submit a pull request to fix it?

[linghengqian]

• Will this be fixed by Alleviate connection leaks caused by Seata Client throwing exceptions #34463 ? However, the master branch actually does not have nativeTest for the mariadb jdbc driver under the GraalVM Native Image. The current issue looks a bit like a need for more information to me.

[Nick-The-Uncharted]

@terrymanu @linghengqian Sorry for being too busy to reply, I am unable to push code to github due to the security policy of my company. I believe #34463 didn't fix this issue because it didn't change code of
org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.createConnections(DriverDatabaseConnectionManager.java:386)

[linghengqian]

@Nick-The-Uncharted Can you share a git with minimal reproducible unit tests? I'm surprised that the handling of #34463 didn't fix this issue.

[linghengqian]

• So what you are talking about is private List<Connection> createConnections(final String databaseName, final String dataSourceName, final DataSource dataSource, final int connectionSize, final TransactionConnectionContext transactionConnectionContext) throws SQLException instead of private List<Connection> createConnections(final String databaseName, final String dataSourceName, final DataSource dataSource, final int connectionSize, final ConnectionMode connectionMode) throws SQLException?

[Nick-The-Uncharted]

@linghengqian Sorry for misreading the code, the change in #34463 do fixed my problem, but methodInvocationRecorder.replay(connection); in createConnections(final String databaseName, final String dataSourceName, final DataSource dataSource, final int connectionSize, final TransactionConnectionContext transactionConnectionContext) do still have a chance of leaking connection if relay throws.

[linghengqian]

• @Nick-The-Uncharted The interesting thing here is that I can actually only reproduce the connection leak bug mentioned in Alleviate connection leaks caused by Seata Client throwing exceptions #34463 . I need someone to show me how to make the connection leak happen in another method, because it seems that there is a higher-level method that handles the exception.