Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vector DBs adapter examples how to use it with source containing multiple resources #1663

Closed
dat-a-man opened this issue Aug 3, 2024 · 0 comments · Fixed by #1662
Closed
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@dat-a-man
Copy link
Collaborator

https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1721995419954429

@VioletM we should document how to adapt sources. the example below IMO is not the best pattern:

from sql_database import sql_database
from dlt.destinations.adapters import weaviate_adapter
import dlt

products_table = sql_database().with_resources("products")

pipeline = dlt.pipeline(
        pipeline_name="postgres_to_weaviate_pipeline",
        destination="weaviate",
    )

products_table.table_name = "Products"

info = pipeline.run(
    weaviate_adapter(products_table.products, vectorize="description"),
)

print(info)

beacuse it takes the resource out of source and adapts it and loads it. what user should do:

products_tables = sql_database().with_resources("products", "customers")

pipeline = dlt.pipeline(
        pipeline_name="postgres_to_weaviate_pipeline",
        destination="weaviate",
    )

# adapt the resource within the source
weaviate_adapter(products_tables.products, vectorize="description")
weaviate_adapter(products_tables.customers, vectorize="bio")

info = pipeline.run(products_tables)
@dat-a-man dat-a-man converted this from a draft issue Aug 3, 2024
@dat-a-man dat-a-man added the documentation Improvements or additions to documentation label Aug 3, 2024
@dat-a-man dat-a-man self-assigned this Aug 3, 2024
@dat-a-man dat-a-man moved this from In Progress to In Review in Documentation Aug 4, 2024
@github-project-automation github-project-automation bot moved this from In Review to Done in Documentation Aug 23, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
Status: Done
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants