Airbyte Sync Issues (Self-hosted via abctl)

[malinosqui]

Hi everyone!

I'm running into an issue with my self-hosted Airbyte installation using the abctl installation method (version 1.4.1). The problem occurs when I try to start a sync. The source (PostgreSQL) and destination (BigQuery) connections successfully pass the connection tests in the UI, but the syncs themselves don’t execute. Upon checking the logs, I found the following:

Logs (Partial):

1. Workloads are being created, but no pods are running:

No pod found running for workload 30db08bb-bab9-4f16-8b0b-3136ed79e6a3_5_0_sync

2. The system attempts to clean up and recreate pods but doesn’t proceed:

Mutex key: 30db08bb-bab9-4f16-8b0b-3136ed79e6a3 specified for workload. Attempting to delete existing pods...
Existing pods for mutex key: 30db08bb-bab9-4f16-8b0b-3136ed79e6a3 deleted.

3. Containers are set up but fail to execute the replication:

[source] image: airbyte/source-postgres:3.6.28 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1})
[destination] image: airbyte/destination-bigquery:2.9.2 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1})

4. System resources appear sufficient, with enough memory and disk space:

• Memory available: ~9.8 GiB
• Disk space available: ~43 GiB

What I’ve Tried So Far:

• Tested connections via the Airbyte UI – both the source and destination are working fine.
• Restarted services using abctl stop/start, but the issue persists.
• Checked abctl logs, but no critical errors appear other than what’s mentioned above.

My Setup:

• Airbyte Version: 1.4.1
• Installation Method: abctl (Kubernetes using kind)
• System Resources (AWS EC2):
• Memory: 15 GiB
• Disk: 97 GiB (43 GiB free)