Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs/issue 1661 add tip to source docs and update weaviate docs #1662

Merged
merged 9 commits into from
Aug 23, 2024
16 changes: 16 additions & 0 deletions docs/website/docs/dlt-ecosystem/destinations/weaviate.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,22 @@ weaviate_adapter(
tokenization={"title": "word", "description": "whitespace"},
)
```
When adapting resources, it's important to implement changes directly to the data source itself rather than extracting resources separately. Here's how:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When adapting resources, it's important to implement changes directly to the data source itself rather than extracting resources separately. Here's how:
When using the `weaviate_adapter`, it's important to apply it directly to resources, not to the whole source. Here's an example:

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


```py
products_tables = sql_database().with_resources("products", "customers")

pipeline = dlt.pipeline(
pipeline_name="postgres_to_weaviate_pipeline",
destination="weaviate",
)

# adapt the resource within the source
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# adapt the resource within the source
# apply adapter to the needed resources

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

weaviate_adapter(products_tables.products, vectorize="description")
weaviate_adapter(products_tables.customers, vectorize="bio")

info = pipeline.run(products_tables)
```

:::tip

Expand Down
6 changes: 6 additions & 0 deletions docs/website/docs/general-usage/source.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,12 @@ Several data sources are prone to contain semi-structured documents with very de
MongoDB databases. Our practical experience is that setting the `max_nesting_level` to 2 or 3
produces the clearest and human-readable schemas.

:::tip TIP
The `max_table_nesting` parameter at the source level doesn't automatically apply to individual
resources when accessed directly (e.g., using `source.resources["resource_1"])`. To make sure it
works, either use `source.with_resources("resource_1")` or set the parameter directly on the resource.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please, add an example how parameter could be set directly in the resource

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

:::

### Modify schema

The schema is available via `schema` property of the source.
Expand Down
Loading