-
Notifications
You must be signed in to change notification settings - Fork 186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs/issue 1661 add tip to source docs and update weaviate docs #1662
Changes from 3 commits
c0fa9d8
44854dc
b2759c6
662a437
3e275d4
0506c3a
2dcb8d7
4079287
2c0d6b5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -116,6 +116,22 @@ weaviate_adapter( | |||||
tokenization={"title": "word", "description": "whitespace"}, | ||||||
) | ||||||
``` | ||||||
When adapting resources, it's important to implement changes directly to the data source itself rather than extracting resources separately. Here's how: | ||||||
|
||||||
```py | ||||||
products_tables = sql_database().with_resources("products", "customers") | ||||||
|
||||||
pipeline = dlt.pipeline( | ||||||
pipeline_name="postgres_to_weaviate_pipeline", | ||||||
destination="weaviate", | ||||||
) | ||||||
|
||||||
# adapt the resource within the source | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||||||
weaviate_adapter(products_tables.products, vectorize="description") | ||||||
weaviate_adapter(products_tables.customers, vectorize="bio") | ||||||
|
||||||
info = pipeline.run(products_tables) | ||||||
``` | ||||||
|
||||||
:::tip | ||||||
|
||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -187,6 +187,12 @@ Several data sources are prone to contain semi-structured documents with very de | |
MongoDB databases. Our practical experience is that setting the `max_nesting_level` to 2 or 3 | ||
produces the clearest and human-readable schemas. | ||
|
||
:::tip TIP | ||
The `max_table_nesting` parameter at the source level doesn't automatically apply to individual | ||
resources when accessed directly (e.g., using `source.resources["resource_1"])`. To make sure it | ||
works, either use `source.with_resources("resource_1")` or set the parameter directly on the resource. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please, add an example how parameter could be set directly in the resource There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
::: | ||
|
||
### Modify schema | ||
|
||
The schema is available via `schema` property of the source. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done