Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[source-microsoft-dataverse] fails at schema discovery (out of memory?) #52678

Open
1 task
Mosaba opened this issue Jan 31, 2025 · 3 comments
Open
1 task

[source-microsoft-dataverse] fails at schema discovery (out of memory?) #52678

Mosaba opened this issue Jan 31, 2025 · 3 comments
Labels
area/connectors Connector related issues autoteam community connectors/source/microsoft-dataverse team/use type/bug Something isn't working

Comments

@Mosaba
Copy link

Mosaba commented Jan 31, 2025

Connector Name

source-microsoft-dataverse

Connector Version

0.1.21

What step the error happened?

Configuring a new connector

Relevant information

Microsoft Dataverse connector passes connection test but then fails on the step "We are fetching the schema of your data source.
This should take less than a minute, but may take a few minutes on slow internet connections or data sources with a large amount of streams."

Discovering schema failed
The main container of the DISCOVER operation returned an exit code 137

Learn more >
Failure origin: source
Failure type: system_error

The same Dataverse credentials work fine on Fivetran but not in Airbyte Cloud

Relevant log output

025-01-31 08:43:21 info APPLY Stage: CLAIM — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover) — (dataplaneId = aws-eu-paris-queue-1)
2025-01-31 08:43:22 info Claimed: true for 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover via API for aws-eu-paris-queue-1
2025-01-31 08:43:22 info APPLY Stage: CHECK_STATUS — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover) — (dataplaneId = aws-eu-paris-queue-1)
2025-01-31 08:43:22 info No pod found running for workload 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover
2025-01-31 08:43:22 info APPLY Stage: BUILD — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover) — (dataplaneId = aws-eu-paris-queue-1)
2025-01-31 08:43:22 info APPLY Stage: MUTEX — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover) — (dataplaneId = aws-eu-paris-queue-1)
2025-01-31 08:43:22 info No mutex key specified for workload: 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover. Continuing...
2025-01-31 08:43:22 info APPLY Stage: LAUNCH — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover) — (dataplaneId = aws-eu-paris-queue-1)
2025-01-31 08:43:22 info [initContainer] image: airbyte/workload-init-container:build-edefda00d3-21055-master resources: ResourceRequirements(claims=[], limits={memory=1000Mi, cpu=4}, requests={memory=1000Mi, cpu=4}, additionalProperties={})
2025-01-31 08:43:35 info Attempting to update workload: 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover to LAUNCHED.
2025-01-31 08:43:35 info Pipeline completed for workload: 9220e3de-3b60-4bb2-a46f-046d59ea235a_59f73996-94a7-4040-8cd0-72221acd2e6e_0_discover.
2025-01-31 08:43:36 info 
2025-01-31 08:43:36 info ----- START DISCOVER -----
2025-01-31 08:43:36 info

Contribute

  • Yes, I want to contribute
@Mosaba Mosaba added area/connectors Connector related issues needs-triage type/bug Something isn't working labels Jan 31, 2025
@natikgadzhi
Copy link
Contributor

Would you be willing to share testing credentials with us so we could verify this? We don't have test accounts, so it would be difficult to fix quickly.

@Mosaba
Copy link
Author

Mosaba commented Feb 1, 2025

Unfortunately this is a client's Dynamics instance so I can't share the credentials they have set up for me. If there is anything further I can provide from Airbyte Cloud in additional to the logs I can have a go. Thanks.

@Mosaba
Copy link
Author

Mosaba commented Feb 3, 2025

I don't know why this is the case but trying the same thing today the log is slightly different:

2025-02-03 15:13:22 info APPLY Stage: CLAIM — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover) — (dataplaneId = prod-dataplane-aws-paris-0)
2025-02-03 15:13:22 info Claimed: true for 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover via API for prod-dataplane-aws-paris-0
2025-02-03 15:13:22 info APPLY Stage: CHECK_STATUS — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover) — (dataplaneId = prod-dataplane-aws-paris-0)
2025-02-03 15:13:22 info No pod found running for workload 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover
2025-02-03 15:13:22 info APPLY Stage: BUILD — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover) — (dataplaneId = prod-dataplane-aws-paris-0)
2025-02-03 15:13:22 info APPLY Stage: MUTEX — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover) — (dataplaneId = prod-dataplane-aws-paris-0)
2025-02-03 15:13:22 info No mutex key specified for workload: 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover. Continuing...
2025-02-03 15:13:22 info APPLY Stage: LAUNCH — (workloadId = 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover) — (dataplaneId = prod-dataplane-aws-paris-0)
2025-02-03 15:13:22 info [initContainer] image: airbyte/workload-init-container:build-5a409cffbe-21058-master resources: ResourceRequirements(claims=[], limits={memory=1000Mi, cpu=4}, requests={memory=1000Mi, cpu=4}, additionalProperties={})
2025-02-03 15:13:37 info Attempting to update workload: 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover to LAUNCHED.
2025-02-03 15:13:37 info 
2025-02-03 15:13:37 info Pipeline completed for workload: 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover.
2025-02-03 15:13:39 info ----- START DISCOVER -----
2025-02-03 15:13:39 info 
2025-02-03 15:14:47 info Connector exited, processing output
2025-02-03 15:14:47 info Output file jobOutput.json found
2025-02-03 15:14:47 info Connector exited with exit code 137
2025-02-03 15:14:47 info Reading messages from protocol version 0.2.0
2025-02-03 15:14:47 warn MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2025-02-03 15:14:47 info Checking for optional control message...
2025-02-03 15:14:47 info Writing output of 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover to the doc store
2025-02-03 15:14:48 info Marking workload 9220e3de-3b60-4bb2-a46f-046d59ea235a_68d6e323-33b4-478e-b038-db1ffbf2f44e_0_discover as successful
2025-02-03 15:14:48 info 
2025-02-03 15:14:48 info Deliberately exiting process with code 0.
2025-02-03 15:14:48 info ----- END DISCOVER -----
2025-02-03 15:14:48 info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/connectors Connector related issues autoteam community connectors/source/microsoft-dataverse team/use type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@natikgadzhi @Mosaba @marcosmarxm @octavia-squidington-iii