[source-hubspot] Issue: Incremental sync for line_items stalls

[LucasOrellanaPrecision]

Connector Name

source-hubspot

Connector Version

4.4.9

What step the error happened?

During the sync

Relevant information

Issue Summary
I am experiencing a significant delay when syncing line_items from HubSpot to BigQuery using Airbyte. The process takes an excessive amount of time to start, even though the number of records is small. Logs indicate that the sync remains idle for at least an hour before actually beginning.

Key Observations
Other HubSpot tables sync without issues – This problem is specific to line_items.
Sync hangs before it officially starts – The workload is created and marked as RUNNING, but no actual data transfer occurs for a long time.
The logs repeatedly show idle messages, such as:

Pool queue size: 0, Active threads: 0

This suggests that the process is not actively fetching or writing data.
The buffer remains empty, as indicated by:

Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0

No explicit errors appear in the logs, meaning there is no immediate failure but rather a stall in execution.

hs_bq_line_items_logs.txt

Possible Causes Considered
API rate limits? → Unlikely, since other tables sync fine.
Permissions? → Not an issue, as full refresh works.
Schema-related issues? → There might be a problem with how line_items is processed.
Pagination or cursor (updatedAt) issues? → Airbyte may be stuck waiting for changes.

Questions for Support
Is there a known issue with syncing line_items from HubSpot?
Could this be related to how Airbyte handles incremental syncs on deeply nested or structured data?
Are there specific configurations or workarounds to speed up incremental syncs for this stream?

Logs
I have attached the full logs for further investigation. Let me know if you need additional information.

Thanks for your help! 🚀

Relevant log output

2025-02-05 18:37:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:38:31 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-05 18:38:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:39:31 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-05 18:39:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:40:14 source INFO Marking stream line_items as RUNNING
2025-02-05 18:40:14 replication-orchestrator INFO Stream status TRACE received of status: RUNNING for stream line_items
2025-02-05 18:40:31 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 20 MB (20.0 MB), %% used: 0.026939428379004083 | Queue `line_items`, num records: 3, num bytes: 20.35 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-05 18:40:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:41:31 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 20 MB (20.0 MB), %% used: 0.026939428379004083 | Queue `line_items`, num records: 29, num bytes: 195.93 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-05 18:41:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:42:31 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 20 MB (20.0 MB), %% used: 0.026939428379004083 | Queue `line_items`, num records: 47, num bytes: 317.76 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-05 18:42:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:43:31 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 20 MB (20.0 MB), %% used: 0.026939428379004083 | Queue `line_items`, num records: 63, num bytes: 425.93 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-02-05 18:43:31 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-02-05 18:44:25 source INFO Advancing bookmark for line_items stream from 2025-02-05T18:12:46.464462+00:00 to 2025-02-05T21:41:24.020000+00:00
2025-02-05 18:44:25 source INFO Read 824 records from line_items stream
2025-02-05 18:44:25 source INFO Marking stream line_items as STOPPED
2025-02-05 18:44:25 source INFO Finished syncing line_items
2025-02-05 18:44:25 replication-orchestrator INFO Stream status TRACE received of status: COMPLETE for stream line_items
2025-02-05 18:44:25 source INFO SourceHubspot runtimes:
Syncing stream line_items 1:14:03.164786
2025-02-05 18:44:25 source INFO Finished syncing SourceHubspot
2025-02-05 18:44:25 replication-orchestrator INFO Total records read: 829 (5 MB)

Contribute

• Yes, I want to contribute