HIVE-29031: Setup Derby database in qtests via qt:database option

[zabetak]

What changes were proposed in this pull request?

1. Add Derby support in QTestDatabaseHandler and implement proper setup and cleaup logic
2. Refactor docker instructions in separate class and update class hierarchy
3. Update all .q files using 'jdbc:derby' URLs to exploit the new functionality
4. Re-enable previously disabled tests since flakiness is resolved.
5. Rework dataconnector.q test exploiting the init script functionality and drop all Derby binary files.

Why are the changes needed?

1. Stabilize CI and avoid flakiness due to improper Derby init/cleanup
2. Facilitate writing unit tests for Derby and improve their readability (e.g., dataconnector.q)

Does this PR introduce any user-facing change?

No

How was this patch tested?

mvn clean test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=authorization_privilege_objects.q,dataconnector.q,explain_jdbc.q,external_jdbc_auth.q,external_jdbc_join_mv.q,external_jdbc_rowcount.q,external_jdbc_table.q,external_jdbc_table3.q,external_jdbc_table4.q,external_jdbc_table_partition.q,external_jdbc_table_perf.q,external_jdbc_table_typeconversion.q,external_jdbc_view.q,jdbc_handler.q,jdbc_split_filter.q,qt_database_derby.q 

mvn test -Dtest=TestNegativeLlapLocalCliDriver.java -Dqfile=authorization_jdbc_keystore.q,dataconnector_authfail_alter.q,dataconnector_authfail_drop.q,dataconnector_authfail_remoteDB.q,external_jdbc_negative.q

mvn clean test -Dtest=TestContribCliDriver -Dqfile=dboutput.q

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
18 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[zabetak]

A clean flaky run with all jdbc related tests can also be found here: https://ci.hive.apache.org/job/hive-flaky-check/904/

[Aggarwal-Raghav]

LGTM +1 (non-binding)

[saihemanth-cloudera]

I feel JDBC URL changes are required in these files as well:
https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/metastore/hooks/SampleURLHook.java#L37
https://github.com/apache/hive/blob/master/beeline/src/test/resources/hive-site.xml#L39
https://github.com/apache/hive/blob/master/data/conf/hive-site-old.xml#L58
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPackageJdo.java#L44-L45
https://github.com/apache/hive/blob/master/itests/hive-unit/src/test/java/org/apache/hadoop/hive/UtilsForTest.java#L40-L42

[zabetak]

Thanks for reviewing @saihemanth-cloudera ! All the files that you highlighted are setting the JDBC URL for the metastore. The qt:database option is not used for the metastore so changing these files is out of scope for the current PR.

[saihemanth-cloudera]

LGTM +1

[zabetak]

Many thanks for the reviews @Aggarwal-Raghav and @saihemanth-cloudera !