[Fix][Connector-V2] Await privilege grant to fix flaky Paimon tests

[dybyte]

Purpose of this pull request

The Paimon connectors CI occasionally fails. In these cases, the tests hang until they timeout, which can take up to 2 hours and is a very cumbersome issue. The failure occurs because the jobs attempt to access the database before the necessary privileges have been granted.

Example error logs:

2025-09-03 15:06:32,239 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel] [5.1] task 1000300000000 error with exception: [org.apache.paimon.privilege.NoPrivilegeException: User paimon doesn't have privilege SELECT on table seatunnel_namespace11.st_test], cancel other task in taskGroup TaskGroupLocation{jobId=1015638881762017281, pipelineId=1, taskGroupId=3}.

2025-09-03T16:36:38.8565633Z ##[error]The operation was canceled.

To address this, a utility class has been added to ensure that the required database privileges are granted before job execution. This should prevent the flaky test behavior and avoid long-running test hang-ups.

For details, see: https://github.com/dybyte/seatunnel/actions/runs/17436786971/job/49509156864

Does this PR introduce any user-facing change?

No

How was this patch tested?

Covered by existing test

Check list

• If any new Jar binary package adding in your PR, please add License Notice according
  New License Guide
• If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
• If you are contributing the connector code, please check that the following files are updated:
  1. Update plugin-mapping.properties and add new connector information in it
  2. Update the pom file of seatunnel-dist
  3. Add ci label in label-scope-conf
  4. Add e2e testcase in seatunnel-e2e
  5. Update connector plugin_config

[Hisoka-X]

cc @hawk9821 as well.

[zhangshenghang]

Hi，why was only SELECT/INSERT checked here?

[dybyte]

Currently, the tests only check for insert and select privileges. I thought it would be fine to add checks for other privileges when we introduce corresponding tests for them. What do you think?

[zhangshenghang]

Currently, the tests only check for insert and select privileges. I thought it would be fine to add checks for other privileges when we introduce corresponding tests for them. What do you think?

@dybyte Sure, but it would be better if we could improve it when adding methods. See your schedule.

[dybyte]

Done, thanks for the suggestion!

[hawk9821]

The grant operation of Paimon is synchronous, and this behavior is quite unusual. I think we should pay attention to why the job does not exit (fail fast) after throw exception.

[Hisoka-X]

The grant operation of Paimon is synchronous, and this behavior is quite unusual. I think we should pay attention to why the job does not exit (fail fast) after throw exception.

@hawk9821 Should we create another issue to track it? Does it block on this PR be merged?

[zhangshenghang]

The grant operation of Paimon is synchronous, and this behavior is quite unusual. I think we should pay attention to why the job does not exit (fail fast) after throw exception.

@hawk9821 Should we create another issue to track it? Does it block on this PR be merged?

Yes, we can fix it first and analyze the cause of the problem as a separate issue

[dybyte]

I’m not entirely certain why the permission check still throws an exception even after the retry. However, it seems that the reason the job does not terminate afterward may be related to #9749 .

Typically, tests verify behavior with assertions after the job has successfully completed. In this particular case, however, an exception occurs during the job execution itself. As a result, the job is likely not removed from pendingJobMasterMap and continues to be counted in runningJobCount.

For more details, please see the end of this log:
https://github.com/dybyte/seatunnel/actions/runs/17436786971/job/49509156864

2025-09-03T15:41:40.5697311Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: ***********************************************
2025-09-03T15:41:40.5698018Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT:                 Job info detail
2025-09-03T15:41:40.5698734Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: ***********************************************
2025-09-03T15:41:40.5699653Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: createdJobCount           :                   0
2025-09-03T15:41:40.5700308Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: pendingJobCount           :                   0
2025-09-03T15:41:40.5700922Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: scheduledJobCount         :                   0
2025-09-03T15:41:40.5701606Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: runningJobCount           :                   1
2025-09-03T15:41:40.5702402Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: failingJobCount           :                   0
2025-09-03T15:41:40.5702986Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: failedJobCount            :                   0
2025-09-03T15:41:40.5703563Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: cancellingJobCount        :                   0
2025-09-03T15:41:40.5704134Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: canceledJobCount          :                   0
2025-09-03T15:41:40.5704791Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: finishedJobCount          :                   0
2025-09-03T15:41:40.5705342Z [] 2025-09-03 15:41:40,568 INFO  tc.seatunnel-engine:openjdk:8 - STDOUT: ***********************************************

This is just my hypothesis based on the logs, please correct me if I’m wrong.
@Hisoka-X @hawk9821 @zhangshenghang

[dybyte]

After reviewing the CI logs again I noticed Slow operation detected messages for CheckpointErrorReportOperation.
In the CI environment, multiple jobs run concurrently and resources may be constrained, which can delay Hazelcast’s OperationThreads. This could potentially cause pipeline state updates or checkpoint error handling to be postponed, leaving the job stuck in the “running” state..

[zhangshenghang]

After reviewing the CI logs again I noticed Slow operation detected messages for CheckpointErrorReportOperation. In the CI environment, multiple jobs run concurrently and resources may be constrained, which can delay Hazelcast’s OperationThreads. This could potentially cause pipeline state updates or checkpoint error handling to be postponed, leaving the job stuck in the “running” state..

If it's the problem you mentioned, not only does Paimon have this issue, but many places have the same problem? Can we reproduce this problem? Or should we wait until # 9749 is merged before verifying?

[dybyte]

If it's the problem you mentioned, not only does Paimon have this issue, but many places have the same problem? Can we reproduce this problem? Or should we wait until # 9749 is merged before verifying?

Since this only happens occasionally in CI, it's hard to pinpoint the exact cause.
My previous comments are just guesses based on the logs, nothing is confirmed yet.
If anyone can find a way to reproduce it consistently, please let me know.

[Hisoka-X]

We can waiting #9749 be merged, then see what would happend in this case.