Skip to content

[Bug][Connector-v2][MYQL-CDC] If the primary key IDs are not consecutive during the MySQL full data stage, a split error will occur #10270

@zyhui98

Description

@zyhui98

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

运行一段时间后,提示位点提交超时,没有其他错误。

trigger checkpoint failed
org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml or jobConfig env.
at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:176) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$20(CheckpointCoordinator.java:810) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:808) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:296) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:688) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202

SeaTunnel Version

2.3.12

SeaTunnel Config

env {
"job.mode"=STREAMING
"job.name"="20260104160319"
"checkpoint.timeout"=60000
"checkpoint.interval"=120000
"job.retry.times"=0
"job.retry.interval.seconds"=30
"parallelism"=3
"read_limit.rows_per_second"=5000
tag_filter {
   group = "2000455856282628097_group"
   team = "2000455856282628097_team"
}
"savemode.execute.location"=CLUSTER
}
source {
MySQL-CDC {
      plugin_output = "source"
      username = "******"
      password = "******"
      table-names = ["aist_geofence.t_aist_geofence_info"]
      base-url = "jdbc:mysql://******:******/aist_geofence"
      format = "DEFAULT"
      startup.mode = "initial"
      snapshot.split.size = 1000
    }}
transform {
}
sink {
ElasticsearchHB {
      plugin_input = "source"
      username =  "esclient_user"
      password = "******"
      tls_verify_certificate = false
      tls_verify_hostname = false
      index = "test_poi_aist_geofence_info_20260104160319"
      job_id = 2007724509605326850
      max_batch_size = 1000
      primary_keys_hb = ["guid"]
    }}

Running Command

正常启动任务

Error Exception

trigger checkpoint failed
org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml or jobConfig env.
at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:176) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$20(CheckpointCoordinator.java:810) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:808) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:296) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:688) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) ~[seatunnel-starter.jar:2.3.12.hb-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions