+
@@ -121,9 +128,13 @@ Use `is_partition: True` to show that a dimension exists over a specific time wi
You can also use `is_partition` for [categorical](#categorical) dimensions as well.
-MetricFlow enables metric aggregation during query time. For example, you can aggregate the `messages_per_month` measure. If you originally had a `time_granularity` for the time dimensions `metric_time`, you can specify a yearly granularity for aggregation in your CLI query:
+MetricFlow enables metric aggregation during query time. For example, you can aggregate the `messages_per_month` measure. If you originally had a `time_granularity` for the time dimensions `metric_time`, you can specify a yearly granularity for aggregation in your query:
```bash
+# dbt Cloud users
+dbt sl query --metrics messages_per_month --dimensions metric_time --order metric_time --time-granularity year
+
+# dbt Core users
mf query --metrics messages_per_month --dimensions metric_time --order metric_time --time-granularity year
```
@@ -344,7 +355,11 @@ In the sales tier example, if sales_person_id 456 is Tier 2 from 2022-03-08 onwa
The following command or code represents how to return the count of transactions generated by each sales tier per month:
-```
+```bash
+# dbt Cloud users
+dbt sl query --metrics transactions --dimensions metric_time__month,sales_person__tier --order metric_time__month --order sales_person__tier
+
+# dbt Core users
mf query --metrics transactions --dimensions metric_time__month,sales_person__tier --order metric_time__month --order sales_person__tier
```
diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md
index ba82a4aa4a5..e06b5046976 100644
--- a/website/docs/docs/build/measures.md
+++ b/website/docs/docs/build/measures.md
@@ -234,6 +234,15 @@ metrics:
We can query the semi-additive metrics using the following syntax:
+For dbt Cloud:
+
+```bash
+dbt sl query --metrics mrr_by_end_of_month --dimensions metric_time__month --order metric_time__month
+dbt sl query --metrics mrr_by_end_of_month --dimensions metric_time__week --order metric_time__week
+```
+
+For dbt Core:
+
```bash
mf query --metrics mrr_by_end_of_month --dimensions metric_time__month --order metric_time__month
mf query --metrics mrr_by_end_of_month --dimensions metric_time__week --order metric_time__week
diff --git a/website/docs/docs/build/metricflow-cli.md b/website/docs/docs/build/metricflow-commands.md
similarity index 62%
rename from website/docs/docs/build/metricflow-cli.md
rename to website/docs/docs/build/metricflow-commands.md
index 2650b2215ae..049b415d40e 100644
--- a/website/docs/docs/build/metricflow-cli.md
+++ b/website/docs/docs/build/metricflow-commands.md
@@ -1,69 +1,142 @@
---
-title: MetricFlow CLI
-id: metricflow-cli
-description: "Query metrics and metadata in your dbt project with the metricflow cli"
-sidebar_label: "MetricFlow CLI commands"
+title: MetricFlow commands
+id: metricflow-commands
+description: "Query metrics and metadata in your dbt project with the MetricFlow commands."
+sidebar_label: "MetricFlow commands"
tags: [Metrics, Semantic Layer]
---
-Once you define metrics in your dbt project, you can query metrics, dimensions, dimension values, and validate your configs using the MetricFlow command line (CLI).
+Once you define metrics in your dbt project, you can query metrics, dimensions, dimension values, and validate your configs using the MetricFlow commands.
-# Installation
+MetricFlow allows you to define and query metrics in your dbt project in the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), or [dbt Core](/docs/core/installation). To experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dynamically query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.
-You can install the [MetricFlow CLI](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to install the MetricFlow CLI on Windows or Linux operating systems:
+MetricFlow is compatible with Python versions 3.8, 3.9, 3.10 and 3.11.
-1. Create or activate your virtual environment.`python -m venv venv`
+
+## MetricFlow
+
+MetricFlow is a dbt package that allows you to define and query metrics in your dbt project. You can use MetricFlow to query metrics in your dbt project in the dbt Cloud CLI, dbt Cloud IDE, or dbt Core.
+
+**Note** — MetricFlow commands aren't supported in dbt Cloud jobs yet. However, you can add MetricFlow validations with your git provider (such as GitHub Actions) by installing MetricFlow (`pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs.
+
+
+
+
+
+MetricFlow commands are embedded in the dbt Cloud CLI, which means you can immediately run them once you install the dbt Cloud CLI.
+
+A benefit to using the dbt Cloud CLI or dbt Cloud IDE is that you won't need to manage versioning — your dbt Cloud account will automatically manage the versioning for you.
+
+
+
+
+
+:::info
+You can create metrics using MetricFlow in the dbt Cloud IDE. However, support for running MetricFlow commands in the IDE will be available soon.
+:::
+
+
+
+
+
+
+:::info Use dbt Cloud CLI for semantic layer development
+
+Use the dbt Cloud CLI for the experience in defining and querying metrics in your dbt project on dbt Cloud or dbt Core with MetricFlow.
+
+A benefit to using the dbt Cloud CLI or dbt Cloud IDE is that you won't need to manage versioning — your dbt Cloud account will automatically manage the versioning for you.
+:::
+
+
+You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to instal MetricFlow on Windows or Linux operating systems:
+
+1. Create or activate your virtual environment`python -m venv venv`
2. Run `pip install dbt-metricflow`
+ * You can install MetricFlow using PyPI as an extension of your dbt adapter in the command line. To install the adapter, run `pip install "dbt-metricflow[your_adapter_name]"` and add the adapter name at the end of the command. For example, for a Snowflake adapter run `pip install "dbt-metricflow[snowflake]"`
- * You can install MetricFlow using PyPI as an extension of your dbt adapter in the CLI. To install the adapter, run `pip install "dbt-metricflow[your_adapter_name]"` and add the adapter name at the end of the command. For example, for a Snowflake adapter run `pip install "dbt-metricflow[snowflake]"`
+**Note**, you'll need to manage versioning between dbt Core, your adapter, and MetricFlow.
-The MetricFlow CLI is compatible with Python versions 3.8, 3.9, 3.10 and 3.11
+
+
+
-# CLI commands
-The MetricFlow CLI provides the following commands to retrieve metadata and query metrics.
+## MetricFlow commands
-To execute the commands, use the `mf` prefix before the command name. For example, to list all metrics, run `mf list metrics`:
+MetricFlow provides the following commands to retrieve metadata and query metrics.
+
+
+
+
+Use the `dbt sl` prefix before the command name to execute them in dbt Cloud. For example, to list all metrics, run `dbt sl list metrics`.
- [`list`](#list) — Retrieves metadata values.
- [`list metrics`](#list-metrics) — Lists metrics with dimensions.
- [`list dimensions`](#list) — Lists unique dimensions for metrics.
- [`list dimension-values`](#list-dimension-values) — List dimensions with metrics.
- [`list entities`](#list-entities) — Lists all unique entities.
+- [`query`](#query) — Query metrics and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started.
+
+
+
+
-## List
+
+
+Use the `mf` prefix before the command name to execute them in dbt Core. For example, to list all metrics, run `mf list metrics`.
+
+- [`list`](#list) — Retrieves metadata values.
+- [`list metrics`](#list-metrics) — Lists metrics with dimensions.
+- [`list dimensions`](#list) — Lists unique dimensions for metrics.
+- [`list dimension-values`](#list-dimension-values) — List dimensions with metrics.
+- [`list entities`](#list-entities) — Lists all unique entities.
+- [`validate-configs`](#validate-configs) — Validates semantic model configurations.
+- [`health-checks`](#health-checks) — Performs data platform health check.
+- [`tutorial`](#tutorial) — Dedicated MetricFlow tutorial to help get you started.
+- [`query`](#query) — Query metrics and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started.
+
+
+
+
+### List
This command retrieves metadata values related to [Metrics](/docs/build/metrics-overview), [Dimensions](/docs/build/dimensions), and [Entities](/docs/build/entities) values.
-## List metrics
+### List metrics
```bash
-mf list
-
+dbt sl list # In dbt Cloud
+mf list # In dbt Core
+```
This command lists the metrics with their available dimensions:
```bash
-mf list metrics
+dbt sl list metrics # In dbt Cloud
+
+mf list metrics # In dbt Core
+
Options:
--search TEXT Filter available metrics by this search term
--show-all-dimensions Show all dimensions associated with a metric.
--help Show this message and exit.
```
-## List dimensions
+### List dimensions
This command lists all unique dimensions for a metric or multiple metrics. It displays only common dimensions when querying multiple metrics:
```bash
-mf list dimensions --metrics
+dbt sl list dimensions --metrics # In dbt Cloud
+
+mf list dimensions --metrics # In dbt Core
+
Options:
- --metrics SEQUENCE List dimensions by given metrics (intersection). Ex.
- --metrics bookings,messages
+ --metrics SEQUENCE List dimensions by given metrics (intersection). Ex. --metrics bookings,messages
--help Show this message and exit.
```
@@ -72,7 +145,10 @@ Options:
This command lists all dimension values with the corresponding metric:
```bash
-mf list dimension-values --metrics --dimension
+dbt sl list dimension-values --metrics --dimension # In dbt Cloud
+
+mf list dimension-values --metrics --dimension # In dbt Core
+
Options:
--dimension TEXT Dimension to query values from [required]
--metrics SEQUENCE Metrics that are associated with the dimension
@@ -83,24 +159,30 @@ Options:
of the data (inclusive)
--help Show this message and exit.
```
-## List entities
+
+### List entities
This command lists all unique entities:
```bash
-mf list entities --metrics
+dbt sl list entities --metrics # In dbt Cloud
+
+mf list entities --metrics # In dbt Core
+
Options:
- --metrics SEQUENCE List entities by given metrics (intersection). Ex.
- --metrics bookings,messages
+ --metrics SEQUENCE List entities by given metrics (intersection). Ex. --metrics bookings,messages
--help Show this message and exit.
```
-## Validate-configs
+### Validate-configs
This command performs validations against the defined semantic model configurations:
```bash
-mf validate-configs
+dbt sl validate-configs # In dbt Cloud
+
+mf validate-configs # In dbt Core
+
Options:
--dw-timeout INTEGER Optional timeout for data warehouse
validation steps. Default None.
@@ -118,28 +200,34 @@ Options:
--help Show this message and exit.
```
-## Health checks
+### Health checks
This command performs a health check against the data platform you provided in the configs:
```bash
-mf health-checks
+dbt sl health-checks #in dbt Cloud
+
+mf health-checks #in dbt Core
```
-## Tutorial
+### Tutorial
Follow the dedicated MetricFlow tutorial to help you get started:
```bash
-mf tutorial
+dbt sl tutorial # In dbt Cloud
+
+mf tutorial # In dbt Core
```
-## Query
+### Query
Create a new query with MetricFlow, execute that query against the user's data platform, and return the result:
```bash
-mf query --metrics --group-by
+dbt sl query --metrics --group-by # In dbt Cloud
+
+mf query --metrics --group-by # In dbt Core
Options:
@@ -170,8 +258,9 @@ Options:
--csv FILENAME Provide filepath for data frame output to csv
- --explain In the query output, show the query that was
- executed against the data warehouse
+ --compile (dbt Cloud) In the query output, show the query that was
+ --explain (dbt Core) executed against the data warehouse
+
--show-dataflow-plan Display dataflow plan in explain output
@@ -186,7 +275,7 @@ Options:
```
-## Query examples
+### Query examples
The following tabs present various different types of query examples that you can use to query metrics and dimensions. Select the tab that best suits your needs:
@@ -198,7 +287,9 @@ Use the example to query metrics by dimension and return the `order_total` metri
**Query**
```bash
-mf query --metrics order_total --group-by metric_time
+dbt sl query --metrics order_total --group-by metric_time # In dbt Cloud
+
+mf query --metrics order_total --group-by metric_time # In dbt Core
```
**Result**
@@ -221,7 +312,9 @@ You can include multiple dimensions in a query. For example, you can group by th
**Query**
```bash
-mf query --metrics order_total --group-by metric_time, is_food_order
+dbt sl query --metrics order_total --group-by metric_time, is_food_order # In dbt Cloud
+
+mf query --metrics order_total --group-by metric_time, is_food_order # In dbt Core
```
**Result**
@@ -248,7 +341,11 @@ You can add order and limit functions to filter and present the data in a readab
**Query**
```bash
-mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time
+# In dbt Cloud
+dbt sl query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time
+
+# In dbt Core
+mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time
```
**Result**
@@ -273,7 +370,11 @@ You can further filter the data set by adding a `where` clause to your query.
**Query**
```bash
-mf query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True"
+# In dbt Cloud
+dbt sl query --metrics order_total --group-by metric_time --where "{{ Dimension('order_id__is_food_order') }} = True"
+
+# In dbt Core
+mf query --metrics order_total --group-by metric_time --where "{{ Dimension('order_id__is_food_order') }} = True"
```
**Result**
@@ -301,7 +402,12 @@ To filter by time, there are dedicated start and end time options. Using these o
**Query**
```bash
- mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27'
+
+# In dbt Cloud
+dbt sl query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27'
+
+# In dbt Core
+mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27'
```
**Result**
@@ -331,20 +437,24 @@ The following tabs present additional query examples, like exporting to a CSV. S
-
+
-Add `--explain` to your query to view the SQL generated by MetricFlow.
+Add `--compile` (or `--explain` for dbt Core users) to your query to view the SQL generated by MetricFlow.
**Query**
```bash
- mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --explain
+# In dbt Cloud
+dbt sl query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --compile
+
+# In dbt Core
+mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --explain
```
**Result**
```bash
✔ Success 🦄 - query completed after 0.28 seconds
-🔎 SQL (remove --explain to see data or add --show-dataflow-plan to see the generated dataflow plan):
+🔎 SQL (remove --compile to see data or add --show-dataflow-plan to see the generated dataflow plan):
SELECT
metric_time
, is_food_order
@@ -374,6 +484,10 @@ Add the `--csv file_name.csv` flag to export the results of your query to a csv.
**Query**
```bash
+# In dbt Cloud
+dbt sl query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --csv query_example.csv
+
+# In dbt Core
mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --csv query_example.csv
```
@@ -386,14 +500,16 @@ mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 -
-## Time granularity
+### Time granularity
Optionally, you can specify the time granularity you want your data to be aggregated at by appending two underscores and the unit of granularity you want to `metric_time`, the global time dimension. You can group the granularity by: `day`, `week`, `month`, `quarter`, and `year`.
Below is an example for querying metric data at a monthly grain:
```bash
-mf query --metrics revenue --group-by metric_time__month
+dbt sl query --metrics revenue --group-by metric_time__month # In dbt Cloud
+
+mf query --metrics revenue --group-by metric_time__month # In dbt Core
```
## FAQs
@@ -403,7 +519,7 @@ mf query --metrics revenue --group-by metric_time__month
To add a dimension filter to a where filter, you have to indicate that the filter item is part of your model and use a template wrapper:
Before using the template wrapper, however, you will need to set up your terminal to escape curly braces for the filter template to work. +Here's an example query:
Before using the template wrapper, however, you will need to set up your terminal to escape curly braces for the filter template to work.
`dependencies.yml` or `packages.yml` to your dbt project. This should be at the same level as your `dbt_project.yml` file.
2. Specify the package(s) you wish to add using one of the supported syntaxes, for example:
@@ -366,3 +372,4 @@ packages:
```
+
diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md
new file mode 100644
index 00000000000..39a4b2e52fd
--- /dev/null
+++ b/website/docs/docs/build/saved-queries.md
@@ -0,0 +1,43 @@
+---
+title: Saved queries
+id: saved-queries
+description: "Saved queries are a way to save commonly used queries in MetricFlow. They can be used to save time and avoid writing the same query over and over again."
+sidebar_label: "Saved queries"
+tags: [Metrics, Semantic Layer]
+---
+
+:::info Saved queries coming soon
+Saved queries isn't currently available in MetricFlow but support is coming soon.
+:::
+
+Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query.
+
+To define a saved query, refer to the following specification:
+
+ Parameter | Description | Type |
+| --------- | ----------- | ---- |
+| `name` | The name of the metric. | Required |
+| `description` | The description of the metric. | Optional |
+| `metrics` | The metrics included in the saved query. | Required |
+| `group_bys` | The value displayed in downstream tools. | Required |
+| `where` | Filter applied to the query. | Optional |
+
+The following is an example of a saved query:
+
+```yaml
+saved_query:
+ name: p0_booking
+ description: Booking-related metrics that are of the highest priority.
+ metrics:
+ - bookings
+ - instant_bookings
+ group_bys:
+ - TimeDimension('metric_time', 'day')
+ - Dimension('listing__capacity_latest')
+ where:
+ - "{{ Dimension('listing__capacity_latest') }} > 3"
+```
+
+### FAQs
+
+* All metrics in a saved query need to use the same dimensions in the `group_by` or `where` clauses.
diff --git a/website/docs/docs/build/sl-getting-started.md b/website/docs/docs/build/sl-getting-started.md
index c0bf59ae0c2..34c3479cbc0 100644
--- a/website/docs/docs/build/sl-getting-started.md
+++ b/website/docs/docs/build/sl-getting-started.md
@@ -8,38 +8,33 @@ meta:
api_name: dbt Semantic Layer APIs
---
-import InstallMetricFlow from '/snippets/_sl-install-metricflow.md';
import CreateModel from '/snippets/_sl-create-semanticmodel.md';
import DefineMetrics from '/snippets/_sl-define-metrics.md';
import ConfigMetric from '/snippets/_sl-configure-metricflow.md';
import TestQuery from '/snippets/_sl-test-and-query-metrics.md';
+import ConnectQueryAPI from '/snippets/_sl-connect-and-query-api.md';
+import RunProdJob from '/snippets/_sl-run-prod-job.md';
-This getting started page presents a sample workflow to help you create your first metrics in dbt Cloud or the command-line interface (CLI). It uses the [Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as the project data source and is available for you to use.
+This getting started page presents a sample workflow to help you create your first metrics in dbt Cloud or the command line interface (CLI). It uses the [Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as the project data source and is available for you to use.
If you prefer, you can create semantic models and metrics for your own dbt project. This page will guide you on how to:
- [Create a semantic model](#create-a-semantic-model) using MetricFlow
- [Define metrics](#define-metrics) using MetricFlow
-- [Test and query metrics locally](#test-and-query-metrics) using MetricFlow
+- [Test and query metrics](#test-and-query-metrics) using MetricFlow
- [Run a production job](#run-a-production-job) in dbt Cloud
- [Set up dbt Semantic Layer](#set-up-dbt-semantic-layer) in dbt Cloud
- [Connect to and query the API](#connect-and-query-api) with dbt Cloud
-
-MetricFlow allows users to define metrics in their dbt project whether in dbt Cloud or in dbt Core. dbt Core users can use the [MetricFlow CLI](/docs/build/metricflow-cli) to define metrics in their local dbt Core project.
+MetricFlow allows you to define metrics in your dbt project and query them whether in dbt Cloud or dbt Core with [MetricFlow commands](/docs/build/metricflow-commands).
However, to experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.
## Prerequisites
-- Have an understanding of key concepts in [MetricFlow](/docs/build/about-metricflow), which powers the revamped dbt Semantic Layer.
-- Have both your production and development environments running dbt version 1.6 or higher. Refer to [upgrade in dbt Cloud](/docs/dbt-versions/upgrade-core-in-cloud) for more info.
-- Use Snowflake, BigQuery, Databricks, Redshift, or Postgres (Postgres available in the CLI only, dbt Cloud support coming soon).
-- Create a successful run in the environment where you configure the Semantic Layer.
- - **Note:** Semantic Layer currently supports the Deployment environment for querying. (_development querying experience coming soon_)
-- Set up the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions.
- - **Note:** To access the API and query metrics in downstream tools, you must have a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account. dbt Core or Developer accounts can define metrics using [MetricFlow CLI](/docs/build/metricflow-cli) or the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud).
-- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the revamped dbt Semantic Layer. +import SetUp from '/snippets/_v2-sl-prerequisites.md'; + +
- - * To authenticate, you need to [generate a service token](/docs/dbt-cloud-apis/service-tokens) with Semantic Layer Only and Metadata Only permissions. - * Refer to the [SQL query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the API. - -- To learn more about the sophisticated integrations that connect to the dbt Semantic Layer, refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more info. +
+
+ ```yml
+dever-env-id: '123456'
+```
+
+
+
+
+
+
+```yml
+dbt_cloud:
+ dever-env-id: '123456'
+```
+
+
diff --git a/website/docs/docs/cloud/about-cloud-develop.md b/website/docs/docs/cloud/about-cloud-develop.md
new file mode 100644
index 00000000000..9f864ede5ca
--- /dev/null
+++ b/website/docs/docs/cloud/about-cloud-develop.md
@@ -0,0 +1,33 @@
+---
+title: About developing in dbt Cloud
+id: about-cloud-develop
+description: "Learn how to develop your dbt projects using dbt Cloud."
+sidebar_label: "About developing in dbt Cloud"
+pagination_next: "docs/cloud/cloud-cli-installation"
+hide_table_of_contents: true
+---
+
+dbt Cloud offers a fast and reliable way to work on your dbt project. It runs dbt Core in a hosted (single or multi-tenant) environment. You can develop in your browser using an integrated development environment (IDE) or in a dbt Cloud-powered command line interface (CLI):
+
+
+ +The following sections provide detailed instructions on setting up the dbt Cloud CLI and dbt Cloud IDE. To get started with dbt development, you'll need a [developer](/docs/cloud/manage-access/seats-and-users) account. For a more comprehensive guide about developing in dbt, refer to our [quickstart guides](/quickstarts). + + +--------- +**Note**: The dbt Cloud CLI and the open-sourced dbt Core are both command line tools that let you run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features). + diff --git a/website/docs/docs/cloud/about-cloud-setup.md b/website/docs/docs/cloud/about-cloud-setup.md index d65c2aead2c..7b68b52a45a 100644 --- a/website/docs/docs/cloud/about-cloud-setup.md +++ b/website/docs/docs/cloud/about-cloud-setup.md @@ -13,6 +13,8 @@ dbt Cloud is the fastest and most reliable way to deploy your dbt jobs. It conta - Configuring access to [GitHub](/docs/cloud/git/connect-github), [GitLab](/docs/cloud/git/connect-gitlab), or your own [git repo URL](/docs/cloud/git/import-a-project-by-git-url). - [Managing users and licenses](/docs/cloud/manage-access/seats-and-users) - [Configuring secure access](/docs/cloud/manage-access/about-user-access) +- Configuring the [dbt Cloud IDE](/docs/cloud/about-cloud-develop) +- Installing and configuring the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) These settings are intended for dbt Cloud administrators. If you need a more detailed first-time setup guide for specific data platforms, read our [quickstart guides](/quickstarts). diff --git a/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md b/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md index 67ad9778fe1..71f3175a108 100644 --- a/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md +++ b/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md @@ -8,84 +8,91 @@ pagination_next: "docs/cloud/about-cloud/architecture" pagination_prev: null --- -dbt Cloud is the fastest and most reliable way to deploy dbt. Develop, test, schedule, document, and investigate data models all in one browser-based UI. In addition to providing a hosted architecture for running dbt across your organization, dbt Cloud comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring & alerting, and an integrated development environment (IDE). +dbt Cloud is the fastest and most reliable way to deploy dbt. Develop, test, schedule, document, and investigate data models all in one browser-based UI. + +In addition to providing a hosted architecture for running dbt across your organization, dbt Cloud comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring and alerting, an integrated development environment (IDE), and allows you to develop and run dbt commands from your local command line interface (CLI) or code editor. dbt Cloud's [flexible plans](https://www.getdbt.com/pricing/) and features make it well-suited for data teams of any size — sign up for your [free 14-day trial](https://www.getdbt.com/signup/)!
*These features are available on [selected plans](https://www.getdbt.com/pricing/). diff --git a/website/docs/docs/cloud/cloud-cli-installation.md b/website/docs/docs/cloud/cloud-cli-installation.md index 44d411bbf2d..f46c795a2ce 100644 --- a/website/docs/docs/cloud/cloud-cli-installation.md +++ b/website/docs/docs/cloud/cloud-cli-installation.md @@ -1,16 +1,14 @@ --- -title: Installing the dbt Cloud CLI (Alpha) +title: Install dbt Cloud CLI +sidebar_label: "Install dbt Cloud CLI" id: cloud-cli-installation description: "Instructions for installing and configuring dbt Cloud CLI" +pagination_next: "docs/cloud/configure-cloud-cli" --- -:::warning Alpha functionality +import CloudCLIFlag from '/snippets/_cloud-cli-flag.md'; -The following installation instructions are for the dbt Cloud CLI, currently in Alpha (actively in development and being tested). - -These instructions are not intended for general audiences at this time. - -::: +
+
+
+
+:::info Use native packages or a virtual environment to avoid overriding dbt Core
+
+Installing the dbt Cloud CLI with pip replaces dbt Core. This change can be avoided by using the native install method and configuring your PATH or by creating a new virtual environment.
+
+Otherwise, to switch back to dbt Core, uninstall the dbt Cloud CLI and follow the dbt Core installation instructions.
+
+:::
+
+Before installing the dbt Cloud CLI, make sure you have Python installed and your virtual environment venv or pyenv . If you already have a Python environment configured, you can skip to the [pip installation step](#install-dbt-cloud-cli-in-pip).
+
+### Install a virtual environment
+
+We recommend using virtual environments (venv) to namespace `cloud-cli`.
+
+1. Create a new venv:
+ ```shell
+ python3 -m venv dbt-cloud
+ ```
+
+2. Activate the virtual environment each time you create a shell window or session:
+ ```shell
+ source dbt-cloud/bin/activate # activate the environment for Mac and Linux OR
+ dbt-env\Scripts\activate # activate the environment for Windows
+ ```
+
+3. (Mac and Linux only) Create an alias to activate your dbt environment with every new shell window or session. You can add the following to your shell's configuration file (for example, $HOME/.bashrc, $HOME/.zshrc) while replacing `` with the path to your virtual environment configuration:
+ ```shell
+ alias env_dbt='source /bin/activate'
+ ```
+
+### Install dbt Cloud CLI in pip
+
+1. (Optional) If you already have dbt Core installed, this installation will override that package. Note your dbt Core version in case you need to reinstall it later:
+
+ ```bash
+ dbt --version
+ ```
+
+2. Make sure you're in your virtual environment and run the following command to install the dbt Cloud CLI:
+
+ ```bash
+ pip3 install dbt
+ ```
+
+3. (Optional) To revert back to dbt Core, first uninstall both the dbt Cloud CLI and dbt Core
+4. Reinstall dbt Core using the version from Step 2.
+
+ ```bash
+ pip3 uninstall dbt-core dbt
+ pip3 install dbt-core==VERSION
+ ```
+
+
-
+
+
+Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.
-Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. If your operating system runs into path conflicts, refer to the [FAQs](#faqs).
1. Run the following command to verify that there is no conflict with a dbt Core installation on your system:
-```bash
-which dbt
-```
+ ```bash
+ which dbt
+ ```
- This should return a `dbt not found`. If the dbt help text appears, use `pip uninstall dbt` to deactivate dbt Core from your machine.
2. Install the dbt Cloud CLI with Homebrew:
-```bash
-brew tap dbt-labs/dbt-cli
-brew install dbt-cloud-cli
-```
+ ```bash
+ brew untap dbt-labs/dbt
+ brew tap dbt-labs/dbt-cli
+ brew install dbt
+ ```
3. Verify the installation by running `dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.
-
-* You no longer need to use the `dbt deps` command. Previously, you had to run that command.
+ * You don't have to run the `dbt deps` command when your environment starts. Previously, you had to do it during initialization. However, you'll still need to run `dbt deps` if you make changes to your `packages.yml` file.
-
+
-If your operating system runs into path conflicts, refer to the [FAQs](#faqs).
+Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.
1. Download the latest Windows release for your platform from [GitHub](https://github.com/dbt-labs/dbt-cli/releases).
-2. Extract the `dbt.exe` executeable into the same folder as your dbt project.
+2. Extract the `dbt.exe` executable into the same folder as your dbt project.
:::info
Advanced users can configure multiple projects to use the same dbt Cloud CLI by placing the executable in the Program Files folder and [adding it to their Windows PATH environment variable](https://medium.com/@kevinmarkvi/how-to-add-executables-to-your-path-in-windows-5ffa4ce61a53).
-Note that if you are using VS Code, you'll need to restart it to pick up modified environment variables.
+Note that if you are using VS Code, you must restart it to pick up modified environment variables.
:::
3. Verify the installation by running `./dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.
-
-* You no longer need to use the `dbt deps` command. Previously, you had to run that command.
+ * You don't have to run the `dbt deps` command when your environment starts. Previously, you had to do it during initialization. However, you'll still need to run `dbt deps` if you make changes to your `packages.yml` file.
-
+
Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.
@@ -83,10 +138,10 @@ Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.
2. Extract the `dbt-cloud-cli` binary to the same folder as your dbt project.
-```bash
-tar -xf dbt_0.29.9_linux_amd64.tar.gz
-./dbt --version
-```
+ ```bash
+ tar -xf dbt_0.29.9_linux_amd64.tar.gz
+ ./dbt --version
+ ```
:::info
@@ -95,89 +150,57 @@ Advanced users can configure multiple projects to use the same Cloud CLI executa
:::
3. Verify the installation by running `./dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.
-
-* You no longer need to use the `dbt deps` command. Previously, you had to run that command.
+ * You don't have to run the `dbt deps` command when your environment starts. Previously, you had to do it during initialization. However, you'll still need to run `dbt deps` if you make changes to your `packages.yml` file.
+
## Update dbt Cloud CLI
The following instructions explain how to update the dbt CLoud CLI to the latest version depending on your operating system.
-
-
-
+During the public preview period, we recommend updating before filing a bug report. This is because the API is subject to breaking changes.
-To update the dbt Cloud CLI, run `brew upgrade dbt-cloud-cli`.
-
-
-
-To update, follow the same process explained in [Install manually (Windows)](/docs/cloud/cloud-cli-installation?install=windows#install-dbt-cloud-cli) and replace the existing `dbt.exe` executable with the new one.
+
+
+To update:
+- Make sure you're in your virtual environment
+- Run `pip install --upgrade dbt`.
+
-
-
-## Configure the dbt Cloud CLI
-
-After installation, you can configure the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, you can execute the following command to compile a project using dbt Cloud:
+
-```bash
-dbt compile
-```
+To update the dbt Cloud CLI, run `brew upgrade dbt`. (You can also use `brew install dbt`).
-**Prerequisites**
-
-- You must set up a project in dbt Cloud.
-- You must have your [personal development credentials](/docs/dbt-cloud-environments#set-developer-credentials) set for that project. The dbt Cloud CLI will use these credentials, stored securely in dbt Cloud, to communicate with your data platform.
-- You must [enroll](/docs/dbt-versions/experimental-features) in the dbt Cloud beta features.
- - To enroll, navigate to your **Profile Settings** and enable the **Beta** flag under **Experimental Features**.
-
-Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt Cloud project.
-
-1. Ensure you meet the prerequisites above.
-2. Create an environment variable with your [dbt Cloud API key](/docs/dbt-cloud-apis/user-tokens):
- - On MacOS, Linux, or Windows add an environment variable:
-
- ```bash
- export DBT_CLOUD_API_KEY="1234" # Replace 1234 with your API key
- ```
-
- - In Powershell, add an environment variable:
- - Note that this variable resets if you restart your shell. To add an environment variable permanently, add a system environment variable in your platform.
+
-3. Navigate to a dbt project in your terminal:
+
-```bash
-cd ~/dbt-projects/jaffle_shop
-```
+To update, follow the same process explained in [Windows](/docs/cloud/cloud-cli-installation?install=windows#install-dbt-cloud-cli) and replace the existing `dbt.exe` executable with the new one.
-4. In your `dbt_project.yml` file, ensure there is a section titled `dbt-cloud`. This section is required to have a `project-id` field with a valid project ID.
+
-```yaml
-# dbt_project.yml
-name:
+
-version:
-...
+To update, follow the same process explained in [Windows](/docs/cloud/cloud-cli-installation?install=linux#install-dbt-cloud-cli) and replace the existing `dbt` executable with the new one.
-dbt-cloud:
- project-id: PROJECT_ID
-```
+
-- To find your project ID, go to **Develop** in the navigation menu. Select the dbt Cloud project URL, such as `https://cloud.getdbt.com/develop/26228/projects123456`, where the project ID is `123456`.
+
-## Use the dbt Cloud CLI
+## Next steps
-The dbt Cloud CLI shares the same set of commands as dbt Core. When you invoke a dbt command, that command is sent to dbt Cloud for processing.
+After installation, you can [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core.
-The dbt Cloud CLI supports [project dependencies](/docs/collaborate/govern/project-dependencies), which is an exciting way to depend on another project using the metadata service in dbt Cloud. It instantly resolves references (or `ref`) to public models defined in other projects. You don't need to execute or analyze these upstream models yourself. Instead, you treat them as an API that returns a dataset.
+For example, you can execute `dbt compile` to compile a project using dbt Cloud.
-Share feedback or request features you'd like to see on the [dbt community Slack](https://getdbt.slack.com/archives/C05M77P54FL).
+Note, that if you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file.
## FAQs
@@ -190,11 +213,16 @@ The dbt Cloud CLI and dbt Core
diff --git a/website/docs/docs/cloud/configure-cloud-cli.md b/website/docs/docs/cloud/configure-cloud-cli.md
new file mode 100644
index 00000000000..952afad2066
--- /dev/null
+++ b/website/docs/docs/cloud/configure-cloud-cli.md
@@ -0,0 +1,92 @@
+---
+title: Configure dbt Cloud CLI
+id: configure-cloud-cli
+description: "Instructions on how to configure the dbt Cloud CLI"
+sidebar_label: "Configure dbt Cloud CLI"
+pagination_next: null
+---
+
+import CloudCLIFlag from '/snippets/_cloud-cli-flag.md';
+
+"
+ active-host: ""
+ defer-env-id: ""
+ projects:
+ - project-id: ""
+ account-host: ""
+ api-key: ""
+
+ - project-id: ""
+ account-host: ""
+ api-key: ""
+
+ ```
+
+4. After downloading the config file, navigate to a dbt project in your terminal:
+
+ ```bash
+ cd ~/dbt-projects/jaffle_shop
+ ```
+
+5. In your `dbt_project.yml` file, ensure you have or include a `dbt-cloud` section with a `project-id` field. The `project-id` field contains the dbt Cloud project ID you want to use.
+
+ ```yaml
+ # dbt_project.yml
+ name:
+
+ version:
+ ...
+
+ dbt-cloud:
+ project-id: PROJECT_ID
+ ```
+
+ - To find your project ID, select **Develop** in the dbt Cloud navigation menu. You can use the URL to find the project ID. For example, in `https://cloud.getdbt.com/develop/26228/projects/123456`, the project ID is `123456`.
+
+## Use the dbt Cloud CLI
+
+- The dbt Cloud CLI shares the same set of [dbt commands](/reference/dbt-commands) as dbt Core and processes the commands you invoke.
+- It allows you to use automatic deferral of build artifacts to your Cloud project's production environment.
+- It also supports [project dependencies](/docs/collaborate/govern/project-dependencies), which allows you to depend on another project using the metadata service in dbt Cloud.
+ - Project dependencies instantly connects to and references (or `ref`) public models defined in other projects. This means you don't need to execute or analyze these upstream models yourself. Instead, you treat them as an API that returns a dataset.
+
+:::tip Use the
+- `dbt run --help`: Lists the flags available for the `run` command +::: diff --git a/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md b/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md index f0c011659f9..9fc382f0217 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md @@ -1,24 +1,31 @@ --- -title: "Develop in the IDE" +title: "About the dbt Cloud IDE" id: develop-in-the-cloud description: "Develop, test, run, and build in the Cloud IDE. With the Cloud IDE, you can compile dbt code into SQL and run it against your database directly" -sidebar_label: Develop in the IDE +sidebar_label: About the IDE tags: [IDE] pagination_next: "docs/cloud/dbt-cloud-ide/ide-user-interface" pagination_prev: null --- -The dbt Cloud integrated development environment (IDE) is a single interface for building, testing, running, and version-controlling dbt projects from your browser. With the Cloud IDE, you can compile dbt code into SQL and run it against your database directly. +The dbt Cloud integrated development environment (IDE) is a single web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database. -## Prerequisites +The dbt Cloud IDE offers several [editing features](/docs/cloud/dbt-cloud-ide/ide-user-interface#editing-features) for faster and more efficient data platform development and governance: -To develop in the Cloud IDE, make sure you have the following: +- Syntax highlighting for SQL: Makes it easy to distinguish different parts of your code, reducing syntax errors and enhancing readability. +- Auto-completion: Suggests table names, arguments, and column names as you type, saving time and reducing typos. +- Code [formatting and linting](/docs/cloud/dbt-cloud-ide/lint-format): Help standardize and fix your SQL code effortlessly. +- Navigation tools: Easily move around your code, jump to specific lines, find and replace text, and navigate between project files. +- Version control: Manage code versions with a few clicks. -- A [dbt Cloud account](https://cloud.getdbt.com/) and [Developer seat license](/docs/cloud/manage-access/seats-and-users) -- A git repository set up and git provider must have `write` access enabled. See [Connecting your GitHub Account](/docs/cloud/git/connect-github) or [Importing a project by git URL](/docs/cloud/git/import-a-project-by-git-url) for detailed setup instructions -- A dbt project connected to a [data platform](/docs/cloud/connect-data-platform/about-connections) -- A [development environment and development credentials](#access-the-cloud-ide) set up -- The environment must be on dbt version 1.0 or higher +These [features](#dbt-cloud-ide-features) create a powerful editing environment for efficient SQL coding, suitable for both experienced and beginner developers. + +
+
+
:::tip Disable ad blockers
@@ -26,21 +33,16 @@ To improve your experience using dbt Cloud, we suggest that you turn off ad bloc
:::
-## Develop in the Cloud IDE
-
-The Cloud IDE is a powerful tool that can help streamline and govern your data platform development process. It offers a range of [editing features](/docs/cloud/dbt-cloud-ide/ide-user-interface#editing-features) that can help make your data platform development process faster and more efficient. Some of the editing features include:
-
-- The IDE has syntax highlighting for SQL. This makes it easy to visually distinguish between different parts of your code. This helps prevent syntax errors and improve readability.
-- Use the IDE built-in auto-completion, which suggests table names, arguments, and column names as you type. This saves time and reduces the likelihood of typos or errors in your code.
-- The code [formatting and linting](/docs/cloud/dbt-cloud-ide/lint-format) tools allow you to standardize and fix your SQL code with ease.
-- The IDE has a range of navigation tools, making it easy to move around your code with ease. You can quickly jump to specific lines of code, find and replace text, and navigate between different files in your project.
-- Use the version control menu and features to version-control your code with just a few clicks.
+## Prerequisites
-All of these [features](#cloud-ide-features) work together to create a powerful editing environment that can help you write and maintain high-quality SQL code in less time. Whether you're a seasoned developer or just starting out, the Cloud IDE has everything you need to be productive, collaborative, and efficient.
+- A [dbt Cloud account](https://cloud.getdbt.com/) and [Developer seat license](/docs/cloud/manage-access/seats-and-users)
+- A git repository set up and git provider must have `write` access enabled. See [Connecting your GitHub Account](/docs/cloud/git/connect-github) or [Importing a project by git URL](/docs/cloud/git/import-a-project-by-git-url) for detailed setup instructions
+- A dbt project connected to a [data platform](/docs/cloud/connect-data-platform/about-connections)
+- A [development environment and development credentials](#access-the-cloud-ide) set up
+- The environment must be on dbt version 1.0 or higher
-
{{Dimension('primary_entity__dimension_name')}}.
-Here's an example query: mf query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True".Before using the template wrapper, however, you will need to set up your terminal to escape curly braces for the filter template to work. +Here's an example query:
dbt sl query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True".Before using the template wrapper, however, you will need to set up your terminal to escape curly braces for the filter template to work.
How to set up your terminal to escape curly braces?
diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 74e25262994..8d18a55e949 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -3,7 +3,7 @@ title: "Packages" id: "packages" --- -## What is a package? + Software engineers frequently modularize code into libraries. These libraries help programmers operate with leverage: they can spend more time focusing on their unique business logic, and less time implementing code that someone else has already spent the time perfecting. In dbt, libraries like these are called _packages_. dbt's packages are so powerful because so many of the analytic problems we encountered are shared across organizations, for example: @@ -22,13 +22,19 @@ dbt _packages_ are in fact standalone dbt projects, with models and macros that * Models in the package will be materialized when you `dbt run`. * You can use `ref` in your own models to refer to models from the package. * You can use macros in the package in your own project. +* It's important to note that defining and installing dbt packages is different from [defining and installing Python packages](/docs/build/python-models#using-pypi-packages) -:::note Using Python packages -Defining and installing dbt packages is different from [defining and installing Python packages](/docs/build/python-models#using-pypi-packages). +:::info `dependencies.yml` has replaced `packages.yml` +Starting from dbt v1.6, `dependencies.yml` has replaced `packages.yml`. This file can now contain both types of dependencies: "package" and "project" dependencies. +- "Package" dependencies lets you add source code from someone else's dbt project into your own, like a library. +- "Project" dependencies provide a different way to build on top of someone else's work in dbt. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies) for more info. +- +You can rename `packages.yml` to `dependencies.yml`, _unless_ you need to use Jinja within your packages specification. This could be necessary, for example, if you want to add an environment variable with a git token in a private git package specification. ::: + ## How do I add a package to my project? 1. Add a file named-- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the revamped dbt Semantic Layer. +import SetUp from '/snippets/_v2-sl-prerequisites.md'; + +
- - * To authenticate, you need to [generate a service token](/docs/dbt-cloud-apis/service-tokens) with Semantic Layer Only and Metadata Only permissions. - * Refer to the [SQL query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the API. - -- To learn more about the sophisticated integrations that connect to the dbt Semantic Layer, refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more info. +
+
+
+ +The following sections provide detailed instructions on setting up the dbt Cloud CLI and dbt Cloud IDE. To get started with dbt development, you'll need a [developer](/docs/cloud/manage-access/seats-and-users) account. For a more comprehensive guide about developing in dbt, refer to our [quickstart guides](/quickstarts). + + +--------- +**Note**: The dbt Cloud CLI and the open-sourced dbt Core are both command line tools that let you run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features). + diff --git a/website/docs/docs/cloud/about-cloud-setup.md b/website/docs/docs/cloud/about-cloud-setup.md index d65c2aead2c..7b68b52a45a 100644 --- a/website/docs/docs/cloud/about-cloud-setup.md +++ b/website/docs/docs/cloud/about-cloud-setup.md @@ -13,6 +13,8 @@ dbt Cloud is the fastest and most reliable way to deploy your dbt jobs. It conta - Configuring access to [GitHub](/docs/cloud/git/connect-github), [GitLab](/docs/cloud/git/connect-gitlab), or your own [git repo URL](/docs/cloud/git/import-a-project-by-git-url). - [Managing users and licenses](/docs/cloud/manage-access/seats-and-users) - [Configuring secure access](/docs/cloud/manage-access/about-user-access) +- Configuring the [dbt Cloud IDE](/docs/cloud/about-cloud-develop) +- Installing and configuring the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) These settings are intended for dbt Cloud administrators. If you need a more detailed first-time setup guide for specific data platforms, read our [quickstart guides](/quickstarts). diff --git a/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md b/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md index 67ad9778fe1..71f3175a108 100644 --- a/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md +++ b/website/docs/docs/cloud/about-cloud/about-dbt-cloud.md @@ -8,84 +8,91 @@ pagination_next: "docs/cloud/about-cloud/architecture" pagination_prev: null --- -dbt Cloud is the fastest and most reliable way to deploy dbt. Develop, test, schedule, document, and investigate data models all in one browser-based UI. In addition to providing a hosted architecture for running dbt across your organization, dbt Cloud comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring & alerting, and an integrated development environment (IDE). +dbt Cloud is the fastest and most reliable way to deploy dbt. Develop, test, schedule, document, and investigate data models all in one browser-based UI. + +In addition to providing a hosted architecture for running dbt across your organization, dbt Cloud comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring and alerting, an integrated development environment (IDE), and allows you to develop and run dbt commands from your local command line interface (CLI) or code editor. dbt Cloud's [flexible plans](https://www.getdbt.com/pricing/) and features make it well-suited for data teams of any size — sign up for your [free 14-day trial](https://www.getdbt.com/signup/)!
+
*These features are available on [selected plans](https://www.getdbt.com/pricing/). diff --git a/website/docs/docs/cloud/cloud-cli-installation.md b/website/docs/docs/cloud/cloud-cli-installation.md index 44d411bbf2d..f46c795a2ce 100644 --- a/website/docs/docs/cloud/cloud-cli-installation.md +++ b/website/docs/docs/cloud/cloud-cli-installation.md @@ -1,16 +1,14 @@ --- -title: Installing the dbt Cloud CLI (Alpha) +title: Install dbt Cloud CLI +sidebar_label: "Install dbt Cloud CLI" id: cloud-cli-installation description: "Instructions for installing and configuring dbt Cloud CLI" +pagination_next: "docs/cloud/configure-cloud-cli" --- -:::warning Alpha functionality +import CloudCLIFlag from '/snippets/_cloud-cli-flag.md'; -The following installation instructions are for the dbt Cloud CLI, currently in Alpha (actively in development and being tested). - -These instructions are not intended for general audiences at this time. - -::: +
-
+1. Install using [pip](/docs/cloud/cloud-cli-installation?install=pip#install-dbt-cloud-cli).
+2. Install natively, but ensure that you deactivate your Python environment or uninstall it using `pip uninstall dbt` before proceeding.
+
+3. (Advanced users) Install natively, but modify the $PATH environment variable to correctly point to the dbt Cloud CLI binary to use both dbt Cloud CLI and dbt Core together.
+
+You can always uninstall the dbt Cloud CLI to return to using dbt Core.
+How do I solve for path conflicts
-For compatibility, both the dbt Cloud CLI and dbt Core are invoked by running `dbt`. This can create path conflicts if your operating system selects one over the other based on your $PATH environment variable (settings). +How do I run both the dbt Cloud CLI and dbt Core?
+For compatibility, both the dbt Cloud CLI and dbt Core are invoked by runningdbt. This can create path conflicts if your operating system selects one over the other based on your $PATH environment variable (settings).
-If you have dbt Core installed locally, ensure that you deactivate your Python environment or uninstall it using `pip uninstall dbt` before proceeding. Alternatively, advanced users can modify the $PATH environment variable to correctly point to the dbt Cloud CLI binary to use both dbt Cloud CLI and dbt Core together.
+If you have dbt Core installed locally, either:
-You can always uninstall the Cloud CLI to return to using dbt Core.
---help flag
+As a tip, most command-line tools have a `--help` flag to show available commands and arguments. Use the `--help` flag with dbt in two ways:
+- `dbt --help`: Lists the commands available for dbt+- `dbt run --help`: Lists the flags available for the `run` command +::: diff --git a/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md b/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md index f0c011659f9..9fc382f0217 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md @@ -1,24 +1,31 @@ --- -title: "Develop in the IDE" +title: "About the dbt Cloud IDE" id: develop-in-the-cloud description: "Develop, test, run, and build in the Cloud IDE. With the Cloud IDE, you can compile dbt code into SQL and run it against your database directly" -sidebar_label: Develop in the IDE +sidebar_label: About the IDE tags: [IDE] pagination_next: "docs/cloud/dbt-cloud-ide/ide-user-interface" pagination_prev: null --- -The dbt Cloud integrated development environment (IDE) is a single interface for building, testing, running, and version-controlling dbt projects from your browser. With the Cloud IDE, you can compile dbt code into SQL and run it against your database directly. +The dbt Cloud integrated development environment (IDE) is a single web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database. -## Prerequisites +The dbt Cloud IDE offers several [editing features](/docs/cloud/dbt-cloud-ide/ide-user-interface#editing-features) for faster and more efficient data platform development and governance: -To develop in the Cloud IDE, make sure you have the following: +- Syntax highlighting for SQL: Makes it easy to distinguish different parts of your code, reducing syntax errors and enhancing readability. +- Auto-completion: Suggests table names, arguments, and column names as you type, saving time and reducing typos. +- Code [formatting and linting](/docs/cloud/dbt-cloud-ide/lint-format): Help standardize and fix your SQL code effortlessly. +- Navigation tools: Easily move around your code, jump to specific lines, find and replace text, and navigate between project files. +- Version control: Manage code versions with a few clicks. -- A [dbt Cloud account](https://cloud.getdbt.com/) and [Developer seat license](/docs/cloud/manage-access/seats-and-users) -- A git repository set up and git provider must have `write` access enabled. See [Connecting your GitHub Account](/docs/cloud/git/connect-github) or [Importing a project by git URL](/docs/cloud/git/import-a-project-by-git-url) for detailed setup instructions -- A dbt project connected to a [data platform](/docs/cloud/connect-data-platform/about-connections) -- A [development environment and development credentials](#access-the-cloud-ide) set up -- The environment must be on dbt version 1.0 or higher +These [features](#dbt-cloud-ide-features) create a powerful editing environment for efficient SQL coding, suitable for both experienced and beginner developers. + +
-
What is the difference between developing on the Cloud IDE and on the CLI?
+What is the difference between developing on the dbt Cloud IDE, the dbt Cloud CLI, and dbt Core?
-
@@ -67,7 +92,7 @@ When you're building on top of another team's work, resolving the references in
- You don't need to mirror any conditional configuration of the upstream project such as `vars`, environment variables, or `target.name`. You can reference them directly wherever the Finance team is building their models in production. Even if the Finance team makes changes like renaming the model, changing the name of its schema, or [bumping its version](/docs/collaborate/govern/model-versions), your `ref` would still resolve successfully.
- You eliminate the risk of accidentally building those models with `dbt run` or `dbt build`. While you can select those models, you can't actually build them. This prevents unexpected warehouse costs and permissions issues. This also ensures proper ownership and cost allocation for each team's models.
-### Usage
+### How to use ref
**Writing `ref`:** Models referenced from a `project`-type dependency must use [two-argument `ref`](/reference/dbt-jinja-functions/ref#two-argument-variant), including the project name:
@@ -88,6 +113,8 @@ with monthly_revenue as (
**Cycle detection:** Currently, "project" dependencies can only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, in turn, on `jaffle_marketing.roi_by_channel`. dbt will check for cycles across projects and raise errors if any are detected. We are considering support for this pattern in the future, whereby dbt would still check for node-level cycles while allowing cycles at the project level.
+For more guidance on how to use dbt Mesh, refer to the dedicated [dbt Mesh guide](/guides/best-practices/how-we-mesh/mesh-1-intro).
+
### Comparison
If you were to instead install the `jaffle_finance` project as a `package` dependency, you would instead be pulling down its full source code and adding it to your runtime environment. This means:
@@ -100,7 +127,7 @@ There are a few cases where installing another internal project as a package can
- Unified deployments — In a production environment, if the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, they could use dbt's [selection syntax](/reference/node-selection/syntax) to create a new "passthrough" project that installed both projects as packages.
- Coordinated changes — In development, if you wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`) on a downstream model (`jaffle_marketing.roi_by_channel`) _before_ introducing changes to a staging or production environment, you can install the `jaffle_finance` package as a package within `jaffle_marketing`. The installation can point to a specific git branch, however, if you find yourself frequently needing to perform end-to-end testing across both projects, we recommend you re-examine if this represents a stable interface boundary.
-These are the exceptions, rather than the rule. Installing another team's project as a package adds complexity, latency, and risk of unnecessary costs. By defining clear interface boundaries across teams, by serving one team's public models as "APIs" to another, and by enabling practitioners to develop with a more narrowly-defined scope, we can enable more people to contribute, with more confidence, while requiring less context upfront.
+These are the exceptions, rather than the rule. Installing another team's project as a package adds complexity, latency, and risk of unnecessary costs. By defining clear interface boundaries across teams, by serving one team's public models as "APIs" to another, and by enabling practitioners to develop with a more narrowly defined scope, we can enable more people to contribute, with more confidence, while requiring less context upfront.
## FAQs
@@ -109,3 +136,7 @@ These are the exceptions, rather than the rule. Installing another team's projec
If you're using private packages with the [git token method](/docs/build/packages#git-token-method), you must define them in the `packages.yml` file instead of the `dependencies.yml` file. This is because conditional rendering (like Jinja-in-yaml) is not supported.
+
+
+## Related docs
+- Refer to the [dbt Mesh](/guides/best-practices/how-we-mesh/mesh-1-intro) guide for more guidance on how to use dbt Mesh.
diff --git a/website/docs/docs/connect-adapters.md b/website/docs/docs/connect-adapters.md
index f45da732abb..77ead34e51d 100644
--- a/website/docs/docs/connect-adapters.md
+++ b/website/docs/docs/connect-adapters.md
@@ -11,9 +11,9 @@ This section provides more details on different ways you can connect dbt to an a
Explore the fastest and most reliable way to deploy dbt using dbt Cloud, a hosted architecture that runs dbt Core across your organization. dbt Cloud lets you seamlessly [connect](/docs/cloud/about-cloud-setup) with a variety of [verified](/docs/supported-data-platforms) data platform providers directly in the dbt Cloud UI.
-### Install using the CLI
+### Install with dbt Core
-Install dbt Core, which is an open-source tool, locally using the CLI. dbt communicates with a number of different data platforms by using a dedicated adapter plugin for each. When you install dbt Core, you'll also need to install the specific adapter for your database, [connect to dbt Core](/docs/core/about-core-setup), and set up a `profiles.yml` file.
+Install dbt Core, an open-source tool, locally using the command line. dbt communicates with a number of different data platforms by using a dedicated adapter plugin for each. When you install dbt Core, you'll also need to install the specific adapter for your database, [connect to dbt Core](/docs/core/about-core-setup), and set up a `profiles.yml` file.
With a few exceptions [^1], you can install all [Verified adapters](/docs/supported-data-platforms) from PyPI using `pip install adapter-name`. For example to install Snowflake, use the command `pip install dbt-snowflake`. The installation will include `dbt-core` and any other required dependencies, which may include both other dependencies and even other adapter plugins. Read more about [installing dbt](/docs/core/installation).
diff --git a/website/docs/docs/core/about-core-setup.md b/website/docs/docs/core/about-core-setup.md
index d119a10f8b4..a4d5ff09ee3 100644
--- a/website/docs/docs/core/about-core-setup.md
+++ b/website/docs/docs/core/about-core-setup.md
@@ -3,7 +3,7 @@ title: About dbt Core setup
id: about-core-setup
description: "Configuration settings for dbt Core."
sidebar_label: "About dbt Core setup"
-pagination_next: "docs/core/about-the-cli"
+pagination_next: "docs/core/about-dbt-core"
pagination_prev: null
---
@@ -11,7 +11,7 @@ dbt Core is an [open-source](https://github.com/dbt-labs/dbt-core) tool that ena
This section of our docs will guide you through various settings to get started:
-- [About the CLI](/docs/core/about-the-cli)
+- [About dbt Core](/docs/core/about-dbt-core)
- [Installing dbt](/docs/core/installation)
- [Connecting to a data platform](/docs/core/connect-data-platform/profiles.yml)
- [How to run your dbt projects](/docs/running-a-dbt-project/run-your-dbt-projects)
diff --git a/website/docs/docs/core/about-dbt-core.md b/website/docs/docs/core/about-dbt-core.md
new file mode 100644
index 00000000000..a35d92420f3
--- /dev/null
+++ b/website/docs/docs/core/about-dbt-core.md
@@ -0,0 +1,25 @@
+---
+title: "About dbt Core"
+id: "about-dbt-core"
+sidebar_label: "About dbt Core"
+---
+
+[dbt Core](https://github.com/dbt-labs/dbt-core) is an open sourced project where you can develop from the command line and run your dbt project.
+
+To use dbt Core, your workflow generally looks like:
+
+1. **Build your dbt project in a code editor —** popular choices include VSCode and Atom.
+
+2. **Run your project from the command line —** macOS ships with a default Terminal program, however you can also use iTerm or the command line prompt within a code editor to execute dbt commands.
+
+:::info How we set up our computers for working on dbt projects
+
+We've written a [guide](https://discourse.getdbt.com/t/how-we-set-up-our-computers-for-working-on-dbt-projects/243) for our recommended setup when running dbt projects using dbt Core.
+
+:::
+
+If you're using the command line, we recommend learning some basics of your terminal to help you work more effectively. In particular, it's important to understand `cd`, `ls` and `pwd` to be able to navigate through the directory structure of your computer easily.
+
+You can find more information on installing and setting up the dbt Core [here](/docs/core/installation).
+
+**Note** — dbt supports a dbt Cloud CLI and dbt Core, both command line interface tools that enable you to run dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features).
diff --git a/website/docs/docs/core/about-the-cli.md b/website/docs/docs/core/about-the-cli.md
deleted file mode 100644
index d05fb514dfa..00000000000
--- a/website/docs/docs/core/about-the-cli.md
+++ /dev/null
@@ -1,22 +0,0 @@
----
-title: "About the CLI"
-id: "about-the-cli"
-sidebar_label: "About the CLI"
----
-
-dbt ships with a command line interface (CLI) for running your dbt project. This way of running dbt and a dbt project is free and open source.
-
-To use the CLI, your workflow generally looks like:
-1. **Build your dbt project in a code editor —** popular choices include VSCode and Atom.
-
-1. **Run your project from the command line —** macOS ships with a default Terminal program, however you can also use iTerm or the command line prompt within a code editor to execute dbt commands.
-
-:::info How we set up our computers for working on dbt projects
-
-We've written a [guide](https://discourse.getdbt.com/t/how-we-set-up-our-computers-for-working-on-dbt-projects/243) for our recommended setup when running dbt projects using the CLI.
-
-:::
-
-If you're using the CLI, we recommend learning some basics of your terminal to help you work more effectively. In particular, it's important to understand `cd`, `ls` and `pwd` to be able to navigate through the directory structure of your computer easily.
-
-You can find more information on installing and setting up the dbt CLI [here](/dbt-cli/cli-overview).
diff --git a/website/docs/docs/core/connect-data-platform/profiles.yml.md b/website/docs/docs/core/connect-data-platform/profiles.yml.md
index 67b0eb15fbe..97254dda1c4 100644
--- a/website/docs/docs/core/connect-data-platform/profiles.yml.md
+++ b/website/docs/docs/core/connect-data-platform/profiles.yml.md
@@ -3,7 +3,7 @@ title: "About profiles.yml"
id: profiles.yml
---
-If you're using dbt from the [command line (CLI)](/docs/core/about-the-cli), you'll need a `profiles.yml` file that contains the connection details for your data platform. When you run dbt from the CLI, it reads your `dbt_project.yml` file to find the `profile` name, and then looks for a profile with the same name in your `profiles.yml` file. This profile contains all the information dbt needs to connect to your data platform.
+If you're using [dbt Core](/docs/core/about-dbt-core), you'll need a `profiles.yml` file that contains the connection details for your data platform. When you run dbt Core from the command line, it reads your `dbt_project.yml` file to find the `profile` name, and then looks for a profile with the same name in your `profiles.yml` file. This profile contains all the information dbt needs to connect to your data platform.
For detailed info, you can refer to the [Connection profiles](/docs/core/connect-data-platform/connection-profiles).
diff --git a/website/docs/docs/core/installation-overview.md b/website/docs/docs/core/installation-overview.md
index 47b4269eb15..25628b76358 100644
--- a/website/docs/docs/core/installation-overview.md
+++ b/website/docs/docs/core/installation-overview.md
@@ -13,6 +13,14 @@ You can install dbt Core on the command line by using one of these methods:
- [Use a Docker image to install dbt](/docs/core/docker-install)
- [Install dbt from source](/docs/core/source-install)
+:::tip Pro tip: Using the --help flag
+
+Most command-line tools, including dbt, have a `--help` flag that you can use to show available commands and arguments. For example, you can use the `--help` flag with dbt in two ways:
+— `dbt --help`: Lists the commands available for dbt
+— `dbt run --help`: Lists the flags available for the `run` command + +::: + ## Upgrading dbt Core dbt provides a number of resources for understanding [general best practices](/blog/upgrade-dbt-without-fear) while upgrading your dbt project as well as detailed [migration guides](/guides/migration/versions/upgrading-to-v1.4) highlighting the changes required for each minor and major release, and [core versions](/docs/dbt-versions/core) diff --git a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md index a401dc54094..3ddbf76d152 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md +++ b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md @@ -32,10 +32,8 @@ You can use the dbt Semantic Layer for a variety of tools and applications of da import Features from '/snippets/_sl-plan-info.md'
There are two main ways to develop with dbt: using the web-based IDE in dbt Cloud or using the command-line interface (CLI) in dbt Core:
- — - dbt Cloud IDE dbt Cloud is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.
- — - dbt Core CLI The command line interface (CLI) uses dbt Core, an open-source software that’s freely available. You can build your dbt project in a code editor, like Jetbrains or VSCode, and run dbt commands from the command line. +
diff --git a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md
index de643413a8a..05910b23e7f 100644
--- a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md
+++ b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md
@@ -36,11 +36,13 @@ The IDE streamlines your workflow, and features a popular user interface layout
* Added (A) — The IDE detects added files
* Deleted (D) — The IDE detects deleted files.
-- — - dbt Cloud IDE dbt Cloud is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.
- — - dbt Core CLI The command line interface (CLI) uses dbt Core, an open-source software that’s freely available. You can build your dbt project in a code editor, like Jetbrains or VSCode, and run dbt commands from the command line. +
You can develop dbt using the web-based IDE in dbt Cloud or on the command line interface using the dbt Cloud CLI or open-source dbt Core, all of which enable you to execute dbt commands. The key distinction between the dbt Cloud CLI and dbt Core is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its features.
+ — + dbt Cloud IDE: dbt Cloud is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.
+ — + dbt Cloud CLI: The dbt Cloud CLI allows you to run dbt commands against your dbt Cloud development environment from your local command line or code editor. It supports cross-project ref, speedier, lower-cost builds, automatic deferral of build artifacts, and more.
+ — + dbt Core: dbt Core is an open-sourced software that’s freely available. You can build your dbt project in a code editor, and run dbt commands from the command line.
+ — + dbt Cloud IDE: dbt Cloud is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.
+ — + dbt Cloud CLI: The dbt Cloud CLI allows you to run dbt commands against your dbt Cloud development environment from your local command line or code editor. It supports cross-project ref, speedier, lower-cost builds, automatic deferral of build artifacts, and more.
+ — + dbt Core: dbt Core is an open-sourced software that’s freely available. You can build your dbt project in a code editor, and run dbt commands from the command line.
+— `dbt --help`: Lists the commands available for dbt
+— `dbt run --help`: Lists the flags available for the `run` command + +::: + ## Upgrading dbt Core dbt provides a number of resources for understanding [general best practices](/blog/upgrade-dbt-without-fear) while upgrading your dbt project as well as detailed [migration guides](/guides/migration/versions/upgrading-to-v1.4) highlighting the changes required for each minor and major release, and [core versions](/docs/dbt-versions/core) diff --git a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md index a401dc54094..3ddbf76d152 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md +++ b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md @@ -32,10 +32,8 @@ You can use the dbt Semantic Layer for a variety of tools and applications of da import Features from '/snippets/_sl-plan-info.md'
diff --git a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md
index 5a8da0a3f48..4d0d4f6d1a2 100644
--- a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md
+++ b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md
@@ -174,7 +174,7 @@ To query metric values, here are the following parameters that are available:
| `grain` | A parameter specific to any time dimension and changes the grain of the data from the default for the metric. | `group_by=[Dimension('metric_time')`
`grain('week\|day\|month\|quarter\|year')]` | Optional | | `where` | A where clause that allows you to filter on dimensions and entities using parameters. This takes a filter list OR string. Inputs come with `Dimension`, and `Entity` objects. Granularity is required if the `Dimension` is a time dimension | `"{{ where=Dimension('customer__country') }} = 'US')"` | Optional | | `limit` | Limit the data returned | `limit=10` | Optional | -|`order` | Order the data returned | `order_by=['-order_gross_profit']` (remove `-` for ascending order) | Optional | +|`order` | Order the data returned by a particular field | `order_by=['order_gross_profit']`, use `-` for descending, or full object notation if the object is operated on: `order_by=[Metric('order_gross_profit').descending(True)`] | Optional | | `compile` | If true, returns generated SQL for the data platform but does not execute | `compile=True` | Optional | @@ -281,18 +281,62 @@ where=[{{ Dimension('metric_time').grain('month') }} >= '2017-03-09', {{ Dimensi }} ``` -### Query with a limit and order by +### Query with a limit Use the following example to query using a `limit` or `order_by` clauses: +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time')], + limit=10) + }} +``` +### Query with Order By Examples + +Order By can take a basic string that's a Dimension, Metric, or Entity and this will default to ascending order + +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time')], + limit=10, + order_by=['order_gross_profit'] + }} +``` + +For descending order, you can add a `-` sign in front of the object. However, you can only use this short hand notation if you aren't operating on the object or using the full object notation. + ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10, - order_by=['order_gross_profit']) + order_by=[-'order_gross_profit'] }} ``` +If you are ordering by an object that's been operated on (e.g., change granularity), or you are using the full object notation, descending order must look like: + +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time').grain('week')], + limit=10, + order_by=[Metric('order_gross_profit').descending(True), Dimension('metric_time').grain('week').descending(True) ] + }} +``` + +Similarly, this will yield ascending order: + +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time').grain('week')], + limit=10, + order_by=[Metric('order_gross_profit'), Dimension('metric_time').grain('week')] + }} +``` + ### Query with compile keyword diff --git a/website/docs/docs/dbt-cloud-environments.md b/website/docs/docs/dbt-cloud-environments.md index 55bec30be92..8fa4522d47c 100644 --- a/website/docs/docs/dbt-cloud-environments.md +++ b/website/docs/docs/dbt-cloud-environments.md @@ -5,7 +5,7 @@ description: "Learn about dbt Cloud's development environment to execute your pr pagination_next: null --- -An environment determines how dbt Cloud will execute your project in both the dbt Cloud IDE (for development) and scheduled jobs (for deployment). +An environment determines how dbt Cloud will execute your project in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) (for development) and scheduled jobs (for deployment). Critically, in order to execute dbt, environments define three variables: @@ -35,7 +35,7 @@ To create a new dbt Cloud development environment: ### Set developer credentials -To use the IDE, each developer will need to set up [personal development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide) to your warehouse connection in their **Profile Settings**. This allows you to set separate target information and maintain individual credentials to connect to your warehouse via the dbt Cloud IDE. +To use the dbt Cloud IDE or dbt Cloud CLI, each developer will need to set up [personal development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide) to your warehouse connection in their **Profile Settings**. This allows you to set separate target information and maintain individual credentials to connect to your warehouse.
* Install these adapters using the CLI as they're not currently supported in dbt Cloud.
+
* Install these adapters using dbt Core as they're not currently supported in dbt Cloud.
diff --git a/website/docs/docs/trusted-adapters.md b/website/docs/docs/trusted-adapters.md index e19bb40785f..08191e8ea42 100644 --- a/website/docs/docs/trusted-adapters.md +++ b/website/docs/docs/trusted-adapters.md @@ -6,7 +6,7 @@ hide_table_of_contents: true Trusted adapters are adapters not maintained by dbt Labs, that we feel comfortable recommending to users for use in production. -Free and open-source tools for the data professional are increasingly abundant. This is by-and-large a *good thing*, however it requires due dilligence that wasn't required in a paid-license, closed-source software world. As a user, there are questions to answer important before taking a dependency on an open-source project. The trusted adapter designation is meant to streamline this process for end users. +Free and open-source tools for the data professional are increasingly abundant. This is by-and-large a *good thing*, however it requires due diligence that wasn't required in a paid-license, closed-source software world. As a user, there are questions to answer important before taking a dependency on an open-source project. The trusted adapter designation is meant to streamline this process for end users.
-import NewSLChanges from '/snippets/_new-sl-changes.md';
-
-
-import NewSLChanges from '/snippets/_new-sl-changes.md';
-
-
diff --git a/website/snippets/_cloud-cli-flag.md b/website/snippets/_cloud-cli-flag.md new file mode 100644 index 00000000000..523591a438c --- /dev/null +++ b/website/snippets/_cloud-cli-flag.md @@ -0,0 +1,5 @@ +:::info Public preview functionality + +The dbt Cloud CLI is currently in [public preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Share feedback or request features you'd like to see on the [dbt community Slack](https://getdbt.slack.com/archives/C05M77P54FL). + +::: diff --git a/website/snippets/_cloud-environments-info.md b/website/snippets/_cloud-environments-info.md index 5388379dc34..b0c8e86a2b5 100644 --- a/website/snippets/_cloud-environments-info.md +++ b/website/snippets/_cloud-environments-info.md @@ -3,17 +3,17 @@ In dbt Cloud, there are two types of environments: - Deployment environment — Determines the settings used when jobs created within that environment are executed. -- Development environment — Determines the settings used in the dbt Cloud IDE for that particular dbt Cloud project. +- Development environment — Determines the settings used in the dbt Cloud IDE or dbt Cloud CLI, for that particular project. Each dbt Cloud project can only have a single development environment but can have any number of deployment environments. | | Development Environments | Deployment Environments | | --- | --- | --- | -| Determines settings for | dbt Cloud IDE | dbt Cloud Job runs | +| Determines settings for | dbt Cloud IDE or dbt Cloud CLI | dbt Cloud Job runs | | How many can I have in my project? | 1 | Any number | :::note -For users familiar with development on the CLI, each environment is roughly analogous to an entry in your `profiles.yml` file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments [here](/docs/core/dbt-core-environments). +For users familiar with development on dbt Core, each environment is roughly analogous to an entry in your `profiles.yml` file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments [here](/docs/core/dbt-core-environments). ::: ## Common environment settings @@ -38,7 +38,7 @@ Both development and deployment environments have a section called **General Set By default, all environments will use the default branch in your repository (usually the `main` branch) when accessing your dbt code. This is overridable within each dbt Cloud Environment using the **Default to a custom branch** option. This setting have will have slightly different behavior depending on the environment type: -- **Development**: determines which branch in the dbt Cloud IDE developers create branches from and open PRs against +- **Development**: determines which branch in the dbt Cloud IDE or dbt Cloud CLI developers create branches from and open PRs against. - **Deployment:** determines the branch is cloned during job executions for each environment. For more info, check out this [FAQ page on this topic](/faqs/Environments/custom-branch-settings)! @@ -59,7 +59,7 @@ Something to note, Extended Attributes doesn't mask secret values. We recommend
-If you're developing in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [orchestrating job runs](/docs/deploy/deployments), Extended Attributes parses through the provided YAML and extracts the `profiles.yml` attributes. For each individual attribute: +If you're developing in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), or [orchestrating job runs](/docs/deploy/deployments), Extended Attributes parses through the provided YAML and extracts the `profiles.yml` attributes. For each individual attribute: - If the attribute exists in another source (such as your project settings), it will replace its value (like environment-level values) in the profile. It also overrides any custom environment variables. diff --git a/website/snippets/_enterprise-permissions-table.md b/website/snippets/_enterprise-permissions-table.md index c89835c8437..3eb313e0f5b 100644 --- a/website/snippets/_enterprise-permissions-table.md +++ b/website/snippets/_enterprise-permissions-table.md @@ -7,7 +7,7 @@ Key: Permissions: * Account-level permissions — Permissions related to management of the dbt Cloud account. For example, billing and account settings. -* Project-level permissions — Permissions related to the projects in dbt Cloud. For example, repos and access to the IDE. +* Project-level permissions — Permissions related to the projects in dbt Cloud. For example, repos and access to the IDE or dbt Cloud CLI. ### Account roles Account roles enable you to manage the dbt Cloud account and manage the account settings (for example, generating service tokens, inviting users, configuring SSO). They also provide project-level permissions. The **Account Admin** role is the highest level of access you can assign. @@ -38,7 +38,7 @@ Account roles enable you to manage the dbt Cloud account and manage the account | Credentials | W | | W | | R | | Custom env. variables | W | | W | | R | | dbt adapters | W | | W | | R | -| Develop (IDE) | W | | W | | | +| Develop (IDE or dbt Cloud CLI) | W | | W | | | | Environments | W | | W | | R | | Jobs | W | | W | | R | | Metadata | R | | R | | R | @@ -52,7 +52,7 @@ Account roles enable you to manage the dbt Cloud account and manage the account ### Project role permissions -The project roles enable you to work within the projects in various capacities. They primarily provide access to project-level permissions such as repos and the IDE, but may also provide some account-level permissions. +The project roles enable you to work within the projects in various capacities. They primarily provide access to project-level permissions such as repos and the IDE or dbt Cloud CLI, but may also provide some account-level permissions. #### Account permissions for project roles @@ -78,7 +78,7 @@ The project roles enable you to work within the projects in various capacities. | Credentials | W | W | W | W | R | W | | | | R | R | | | Custom env. variables | W | W | W | W | W | W | R | | | R | W | | | dbt adapters | W | W | W | W | R | W | | | | R | R | | -| Develop (IDE) | W | W | | W | | | | | | | | | +| Develop (IDE or dbt Cloud CLI) | W | W | | W | | | | | | | | | | Environments | W | R | R | R | R | W | R | | | R | R | | | Jobs | W | R | R | W | R | W | R | | | R | R | | | Metadata | R | R | R | R | R | R | R | R | | R | R | | diff --git a/website/snippets/_new-sl-setup.md b/website/snippets/_new-sl-setup.md index b802db9c5ae..ad248bc3ca9 100644 --- a/website/snippets/_new-sl-setup.md +++ b/website/snippets/_new-sl-setup.md @@ -1,13 +1,13 @@ You can set up the dbt Semantic Layer in dbt Cloud at the environment and project level. Before you begin: -- You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America. +- You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment. Single-tenant coming soon. - You must be part of the Owner group, and have the correct [license](/docs/cloud/manage-access/seats-and-users) and [permissions](/docs/cloud/manage-access/self-service-permissions) to configure the Semantic Layer: * Enterprise plan — Developer license with Account Admin permissions. Or Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. * Team plan — Owner with a Developer license. - You must have a successful run in your new environment. :::tip -If you're using the legacy Semantic Layer, we **highly** recommend you [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to dbt v1.6 or higher to use the new dbt Semantic Layer. Refer to the dedicated [migration guide](/guides/migration/sl-migration) for more info. +If you're using the legacy Semantic Layer, dbt Labs strongly recommends that you [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to dbt version 1.6 or newer to use the latest dbt Semantic Layer. Refer to the dedicated [migration guide](/guides/migration/sl-migration) for details. ::: 1. In dbt Cloud, create a new [deployment environment](/docs/deploy/deploy-environments#create-a-deployment-environment) or use an existing environment on dbt 1.6 or higher. diff --git a/website/snippets/_sl-connect-and-query-api.md b/website/snippets/_sl-connect-and-query-api.md new file mode 100644 index 00000000000..429f41c3bf6 --- /dev/null +++ b/website/snippets/_sl-connect-and-query-api.md @@ -0,0 +1,10 @@ +You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer. + +You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment. Single-tenant coming soon. + +- To learn how to use the JDBC or GraphQL API and what tools you can query it with, refer to [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview). + + * To authenticate, you need to [generate a service token](/docs/dbt-cloud-apis/service-tokens) with Semantic Layer Only and Metadata Only permissions. + * Refer to the [SQL query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the API. + +- To learn more about the sophisticated integrations that connect to the dbt Semantic Layer, refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more info. diff --git a/website/snippets/_sl-create-semanticmodel.md b/website/snippets/_sl-create-semanticmodel.md index bc4276efcb6..6e0376ab10b 100644 --- a/website/snippets/_sl-create-semanticmodel.md +++ b/website/snippets/_sl-create-semanticmodel.md @@ -1,4 +1,4 @@ -The following steps will walk you through setting up semantic models, which you can do with the dbt Cloud IDE or the CLI. Semantic models consist of [entities](/docs/build/entities), [dimensions](/docs/build/dimensions), and [measures](/docs/build/measures). +The following steps describe how to set up semantic models. Semantic models consist of [entities](/docs/build/entities), [dimensions](/docs/build/dimensions), and [measures](/docs/build/measures). We highly recommend you read the overview of what a [semantic model](/docs/build/semantic-models) is before getting started. If you're working in the [Jaffle shop example](https://github.com/dbt-labs/jaffle-sl-template), delete the `orders.yml` config or delete the .yml extension so it's ignored during parsing. **We'll be rebuilding it step by step in this example.** diff --git a/website/snippets/_sl-define-metrics.md b/website/snippets/_sl-define-metrics.md index 29af3f5b7c3..3734e819c1b 100644 --- a/website/snippets/_sl-define-metrics.md +++ b/website/snippets/_sl-define-metrics.md @@ -1,4 +1,4 @@ -Now that you've created your first semantic model, it's time to define your first metric! You can define metrics with the dbt Cloud IDE or CLI. +Now that you've created your first semantic model, it's time to define your first metric! You can define metrics with the dbt Cloud IDE or command line. MetricFlow supports different metric types like [simple](/docs/build/simple), [ratio](/docs/build/ratio), [cumulative](/docs/build/cumulative), and [derived](/docs/build/derived). It's recommended that you read the [metrics overview docs](/docs/build/metrics-overview) before getting started. diff --git a/website/snippets/_sl-install-metricflow.md b/website/snippets/_sl-install-metricflow.md deleted file mode 100644 index 73e60d34e85..00000000000 --- a/website/snippets/_sl-install-metricflow.md +++ /dev/null @@ -1,8 +0,0 @@ -Install the [MetricFlow CLI](/docs/build/metricflow-cli) as an extension of a dbt adapter from PyPI. The MetricFlow CLI is compatible with Python versions 3.8, 3.9, 3.10 and 3.11 - -Use pip install `metricflow` and your [dbt adapter](/docs/supported-data-platforms): - -- Create or activate your virtual environment. `python -m venv venv` or `source your-venv/bin/activate` -- Run `pip install "dbt-metricflow[your_adapter_name]"` - - You must specify `[your_adapter_name]`. - - For example, run `pip install "dbt-metricflow[snowflake]"` if you use a Snowflake adapter. diff --git a/website/snippets/_sl-partner-links.md b/website/snippets/_sl-partner-links.md index fa8146c6418..59e5b54b7c0 100644 --- a/website/snippets/_sl-partner-links.md +++ b/website/snippets/_sl-partner-links.md @@ -2,26 +2,61 @@ The following tools integrate with the dbt Semantic Layer:
-Before you connect to these tools, you'll need to first [set up the dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl) and [generate a service token](/docs/dbt-cloud-apis/service-tokens) to create **Semantic Layer Only** and **Metadata Only** permissions. - +Before you connect to these tools, you'll need to first [set up the dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl) and [generate a service token](/docs/dbt-cloud-apis/service-tokens) to create **Semantic Layer Only** and **Metadata Only** permissions. diff --git a/website/snippets/_sl-plan-info.md b/website/snippets/_sl-plan-info.md index 5fba18de6bb..083ab2209bc 100644 --- a/website/snippets/_sl-plan-info.md +++ b/website/snippets/_sl-plan-info.md @@ -1,2 +1,2 @@ -To define and query metrics with the {props.product}, you must be on a {props.plan} multi-tenant plan, {props.instance} (Additional region support coming soon).
The re-released dbt Semantic Layer is available on dbt v1.6 or higher. dbt Core users can use the MetricFlow CLI to define metrics in their local project, but won't be able to dynamically query them with integrated tools.
+To define and query metrics with the {props.product}, you must be on a {props.plan} multi-tenant plan .
diff --git a/website/snippets/_sl-run-prod-job.md b/website/snippets/_sl-run-prod-job.md new file mode 100644 index 00000000000..a637b0b431e --- /dev/null +++ b/website/snippets/_sl-run-prod-job.md @@ -0,0 +1,7 @@ +Once you’ve defined metrics in your dbt project, you can perform a job run in your deployment environment in dbt Cloud to materialize your metrics. The deployment environment is only supported for the dbt Semantic Layer currently. + +1. Select **Deploy** from the top navigation bar. +2. Select **Jobs** to rerun the job with the most recent code in the deployment environment. +3. Your metric should appear as a red node in the dbt Cloud IDE and dbt directed acyclic graphs (DAG). + +
-Before you begin, you'll need to install the [MetricFlow CLI](/docs/build/metricflow-cli) package and make sure you run at least one model.
-### Install MetricFlow
-import InstallMetricFlow from '/snippets/_sl-install-metricflow.md';
+
- --group-by ` to query the metrics and dimensions. For example, `mf query --metrics order_total --group-by metric_time`
-4. Verify that the metric values are what you expect. To further understand how the metric is being generated, you can view the generated SQL if you type `--explain` in the CLI.
-5. Run `mf validate-configs` to run validation on your semantic models and metrics.
-6. Commit and merge the code changes that contain the metric definitions.
+To get started:
+
+1. Make sure you've installed the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation).
+2. Navigate to your dbt project directory.
+3. Run a dbt command, such as `dbt parse`, `dbt run`, `dbt compile`, or `dbt build`. If you don't, you'll receive an error message that begins with: "ensure that you've ran an artifacts...."
+ - MetricFlow builds a semantic graph and generates a `semantic_manifest.json` file in dbt Cloud, which is stored in the `/target` directory. If using the Jaffle shop example, run `dbt seed && dbt run` to ensure the required data is in your data platform before proceeding.
+
+4. Run `dbt sl --help` to confirm you have MetricFlow installed and that you can view the available commands.
+5. Run `dbt sl query --metrics --group-by ` to query the metrics and dimensions. For example, `dbt sl query --metrics order_total --group-by metric_time`
+6. Verify that the metric values are what you expect. To further understand how the metric is being generated, you can view the generated SQL if you type `--compile` in the command line.
+7. Commit and merge the code changes that contain the metric definitions.
To streamline your metric querying process, you can connect to the [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) to access your metrics programmatically. For SQL syntax, refer to [Querying the API for metric metadata](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the API.
+
+
+
+
+
+
+
+This step is for dbt Core users only. MetricFlow is compatible with Python versions 3.8, 3.9, 3.10 and 3.11. You need to use `pip` to install MetricFlow on Windows or Linux operating systems:
+
+:::note
+The dbt Cloud CLI is strongly recommended to define and query metrics for your dbt project in dbt Cloud or dbt Core with MetricFlow. If you're using dbt Core, you'll need to manage versioning between dbt Core, your adapter, and MetricFlow.
+:::
+
+
+1. Install [MetricFlow](/docs/build/metricflow-commands) as an extension of a dbt adapter from PyPI.
+2. Create or activate your virtual environment with `python -m venv venv` or `source your-venv/bin/activate`.
+3. Run `pip install dbt-metricflow`.
+ - You can install MetricFlow using PyPI as an extension of your dbt adapter in the command line. To install the adapter, run `pip install "dbt-metricflow[your_adapter_name]"` and add the adapter name at the end of the command. As an example for a Snowflake adapter, run `pip install "dbt-metricflow[snowflake]"`.
+ - You'll need to manage versioning between dbt Core, your adapter, and MetricFlow.
+4. Run `dbt parse`. This allows MetricFlow to build a semantic graph and generate a `semantic_manifest.json`.
+ - This creates the file in your `/target` directory. If you're working from the Jaffle shop example, run `dbt seed && dbt run` before proceeding to ensure the data exists in your warehouse.
+5. Run `mf --help` to confirm you have MetricFlow installed and that you can view the available commands.
+6. Run `mf query --metrics --group-by ` to query the metrics and dimensions. For example, `mf query --metrics order_total --group-by metric_time`.
+7. Verify that the metric values are what you expect. To further understand how the metric is being generated, you can view the generated SQL if you type `--explain` in the command line.
+8. Run `mf validate-configs` to run validation on your semantic models and metrics.
+9. Commit and merge the code changes that contain the metric definitions.
+
+To streamline your metric querying process, you can connect to the [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) to access your metrics programmatically. For SQL syntax, refer to [Querying the API for metric metadata](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the API.
+
+
+
+
+
+
diff --git a/website/snippets/_v2-sl-prerequisites.md b/website/snippets/_v2-sl-prerequisites.md
index 9fdc3b53143..852730604d0 100644
--- a/website/snippets/_v2-sl-prerequisites.md
+++ b/website/snippets/_v2-sl-prerequisites.md
@@ -1,17 +1,14 @@
-To use the Semantic Layer, you must:
-
-- Have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America.
+- Have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment. Single-tenant coming soon.
- Have both your production and development environments running dbt version 1.6 or higher. Refer to [upgrade in dbt Cloud](/docs/dbt-versions/upgrade-core-in-cloud) for more info.
-- Use Snowflake, BigQuery, Databricks, or Redshift (dbt Cloud Postgres support coming soon).
+- Use Snowflake, BigQuery, Databricks, or Redshift.
- Create a successful run in the environment where you configure the Semantic Layer.
- **Note:** Semantic Layer currently supports the Deployment environment for querying. (_development querying experience coming soon_)
- Set up the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions.
- - **Note:** To access the API and query metrics in downstream tools, you must have a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account. dbt Core or Developer accounts can define metrics with the [MetricFlow CLI](/docs/build/metricflow-cli) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) but won't be able to dynamically query them.
-- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the revamped dbt Semantic Layer. - + - To access the API and query metrics in downstream tools, you must have a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account. dbt Core or Developer accounts can define metrics but won't be able to dynamically query them.
+- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the latest dbt Semantic Layer.
diff --git a/website/src/css/custom.css b/website/src/css/custom.css
index 972b86a48ca..fc51ef8a8ef 100644
--- a/website/src/css/custom.css
+++ b/website/src/css/custom.css
@@ -2022,6 +2022,18 @@ html[data-theme="dark"] .theme-doc-sidebar-container>div>button.button:hover {
display: none;
}
+.card-container {
+ position: relative;
+}
+
+.external-link {
+ position: absolute;
+ top: 0;
+ right: 0;
+ margin: 10px;
+ color: #818589; /* You can adjust the color as needed */
+}
+
@media (max-width: 996px) {
.quickstart-container {
flex-direction: column;
diff --git a/website/static/img/docs/dbt-cloud/cloud-ide/ide-command-bar.jpg b/website/static/img/docs/dbt-cloud/cloud-ide/ide-command-bar.jpg
new file mode 100644
index 00000000000..fe60ddd7e03
Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-ide/ide-command-bar.jpg differ
diff --git a/website/static/img/docs/dbt-cloud/defer-toggle.jpg b/website/static/img/docs/dbt-cloud/defer-toggle.jpg
new file mode 100644
index 00000000000..7bd5a1c1283
Binary files /dev/null and b/website/static/img/docs/dbt-cloud/defer-toggle.jpg differ
diff --git a/website/static/img/icons/google-sheets-logo-icon.svg b/website/static/img/icons/google-sheets-logo-icon.svg
new file mode 100644
index 00000000000..d080c1dd53d
--- /dev/null
+++ b/website/static/img/icons/google-sheets-logo-icon.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/hex.svg b/website/static/img/icons/hex.svg
new file mode 100755
index 00000000000..00431ffe299
--- /dev/null
+++ b/website/static/img/icons/hex.svg
@@ -0,0 +1,5 @@
+
diff --git a/website/static/img/icons/lightdash.svg b/website/static/img/icons/lightdash.svg
new file mode 100644
index 00000000000..96f4676e7ee
--- /dev/null
+++ b/website/static/img/icons/lightdash.svg
@@ -0,0 +1,96 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/mode.svg b/website/static/img/icons/mode.svg
new file mode 100644
index 00000000000..269c182cd8b
--- /dev/null
+++ b/website/static/img/icons/mode.svg
@@ -0,0 +1,165 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/tableau-software.svg b/website/static/img/icons/tableau-software.svg
new file mode 100644
index 00000000000..28996f1dadd
--- /dev/null
+++ b/website/static/img/icons/tableau-software.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/white/google-sheets-logo-icon.svg b/website/static/img/icons/white/google-sheets-logo-icon.svg
new file mode 100644
index 00000000000..d080c1dd53d
--- /dev/null
+++ b/website/static/img/icons/white/google-sheets-logo-icon.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/white/hex.svg b/website/static/img/icons/white/hex.svg
new file mode 100644
index 00000000000..00431ffe299
--- /dev/null
+++ b/website/static/img/icons/white/hex.svg
@@ -0,0 +1,5 @@
+
diff --git a/website/static/img/icons/white/lightdash.svg b/website/static/img/icons/white/lightdash.svg
new file mode 100644
index 00000000000..96f4676e7ee
--- /dev/null
+++ b/website/static/img/icons/white/lightdash.svg
@@ -0,0 +1,96 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/white/mode.svg b/website/static/img/icons/white/mode.svg
new file mode 100644
index 00000000000..269c182cd8b
--- /dev/null
+++ b/website/static/img/icons/white/mode.svg
@@ -0,0 +1,165 @@
+
\ No newline at end of file
diff --git a/website/static/img/icons/white/tableau-software.svg b/website/static/img/icons/white/tableau-software.svg
new file mode 100644
index 00000000000..28996f1dadd
--- /dev/null
+++ b/website/static/img/icons/white/tableau-software.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/website/vercel.json b/website/vercel.json
index a9187933980..14b3a0a6af0 100644
--- a/website/vercel.json
+++ b/website/vercel.json
@@ -2,6 +2,21 @@
"cleanUrls": true,
"trailingSlash": false,
"redirects": [
+ {
+ "source": "/docs/build/metricflow-cli",
+ "destination": "/docs/build/metricflow-commands",
+ "permanent": true
+ },
+ {
+ "source": "/docs/core/about-the-cli",
+ "destination": "/docs/core/about-dbt-core",
+ "permanent": true
+ },
+ {
+ "source": "/docs/cloud/about-cloud/about-cloud-ide",
+ "destination": "/docs/cloud/about-cloud-develop",
+ "permanent": true
+ },
{
"source": "/faqs/models/reference-models-in-another-project",
"destination": "/docs/collaborate/govern/project-dependencies",
`grain('week\|day\|month\|quarter\|year')]` | Optional | | `where` | A where clause that allows you to filter on dimensions and entities using parameters. This takes a filter list OR string. Inputs come with `Dimension`, and `Entity` objects. Granularity is required if the `Dimension` is a time dimension | `"{{ where=Dimension('customer__country') }} = 'US')"` | Optional | | `limit` | Limit the data returned | `limit=10` | Optional | -|`order` | Order the data returned | `order_by=['-order_gross_profit']` (remove `-` for ascending order) | Optional | +|`order` | Order the data returned by a particular field | `order_by=['order_gross_profit']`, use `-` for descending, or full object notation if the object is operated on: `order_by=[Metric('order_gross_profit').descending(True)`] | Optional | | `compile` | If true, returns generated SQL for the data platform but does not execute | `compile=True` | Optional | @@ -281,18 +281,62 @@ where=[{{ Dimension('metric_time').grain('month') }} >= '2017-03-09', {{ Dimensi }} ``` -### Query with a limit and order by +### Query with a limit Use the following example to query using a `limit` or `order_by` clauses: +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time')], + limit=10) + }} +``` +### Query with Order By Examples + +Order By can take a basic string that's a Dimension, Metric, or Entity and this will default to ascending order + +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time')], + limit=10, + order_by=['order_gross_profit'] + }} +``` + +For descending order, you can add a `-` sign in front of the object. However, you can only use this short hand notation if you aren't operating on the object or using the full object notation. + ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10, - order_by=['order_gross_profit']) + order_by=[-'order_gross_profit'] }} ``` +If you are ordering by an object that's been operated on (e.g., change granularity), or you are using the full object notation, descending order must look like: + +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time').grain('week')], + limit=10, + order_by=[Metric('order_gross_profit').descending(True), Dimension('metric_time').grain('week').descending(True) ] + }} +``` + +Similarly, this will yield ascending order: + +```bash +select * from {{ +semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], + group_by=[Dimension('metric_time').grain('week')], + limit=10, + order_by=[Metric('order_gross_profit'), Dimension('metric_time').grain('week')] + }} +``` + ### Query with compile keyword diff --git a/website/docs/docs/dbt-cloud-environments.md b/website/docs/docs/dbt-cloud-environments.md index 55bec30be92..8fa4522d47c 100644 --- a/website/docs/docs/dbt-cloud-environments.md +++ b/website/docs/docs/dbt-cloud-environments.md @@ -5,7 +5,7 @@ description: "Learn about dbt Cloud's development environment to execute your pr pagination_next: null --- -An environment determines how dbt Cloud will execute your project in both the dbt Cloud IDE (for development) and scheduled jobs (for deployment). +An environment determines how dbt Cloud will execute your project in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) (for development) and scheduled jobs (for deployment). Critically, in order to execute dbt, environments define three variables: @@ -35,7 +35,7 @@ To create a new dbt Cloud development environment: ### Set developer credentials -To use the IDE, each developer will need to set up [personal development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide) to your warehouse connection in their **Profile Settings**. This allows you to set separate target information and maintain individual credentials to connect to your warehouse via the dbt Cloud IDE. +To use the dbt Cloud IDE or dbt Cloud CLI, each developer will need to set up [personal development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide) to your warehouse connection in their **Profile Settings**. This allows you to set separate target information and maintain individual credentials to connect to your warehouse.
* Install these adapters using the CLI as they're not currently supported in dbt Cloud.
+
* Install these adapters using dbt Core as they're not currently supported in dbt Cloud.
diff --git a/website/docs/docs/trusted-adapters.md b/website/docs/docs/trusted-adapters.md index e19bb40785f..08191e8ea42 100644 --- a/website/docs/docs/trusted-adapters.md +++ b/website/docs/docs/trusted-adapters.md @@ -6,7 +6,7 @@ hide_table_of_contents: true Trusted adapters are adapters not maintained by dbt Labs, that we feel comfortable recommending to users for use in production. -Free and open-source tools for the data professional are increasingly abundant. This is by-and-large a *good thing*, however it requires due dilligence that wasn't required in a paid-license, closed-source software world. As a user, there are questions to answer important before taking a dependency on an open-source project. The trusted adapter designation is meant to streamline this process for end users. +Free and open-source tools for the data professional are increasingly abundant. This is by-and-large a *good thing*, however it requires due diligence that wasn't required in a paid-license, closed-source software world. As a user, there are questions to answer important before taking a dependency on an open-source project. The trusted adapter designation is meant to streamline this process for end users.
Considerations for depending on an open-source project
diff --git a/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md b/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md index aa464aced4a..a398ace164e 100644 --- a/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md +++ b/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md @@ -11,10 +11,6 @@ meta:
@@ -59,8 +56,8 @@ instance="hosted in North America"
icon="dbt-bit"/>
-import NewSLChanges from '/snippets/_new-sl-changes.md';
-import InstallMetricFlow from '/snippets/_sl-install-metricflow.md';
+
import CreateModel from '/snippets/_sl-create-semanticmodel.md';
import DefineMetrics from '/snippets/_sl-define-metrics.md';
import ConfigMetric from '/snippets/_sl-configure-metricflow.md';
import TestQuery from '/snippets/_sl-test-and-query-metrics.md';
+import ConnectQueryAPI from '/snippets/_sl-connect-and-query-api.md';
+import RunProdJob from '/snippets/_sl-run-prod-job.md';
-
-
diff --git a/website/docs/reference/node-selection/putting-it-together.md b/website/docs/reference/node-selection/putting-it-together.md
index 8faf02e6cc9..48fc5188b32 100644
--- a/website/docs/reference/node-selection/putting-it-together.md
+++ b/website/docs/reference/node-selection/putting-it-together.md
@@ -4,16 +4,16 @@ title: "Putting it together"
```bash
- $ dbt run --select my_package.*+ # select all models in my_package and their children
- $ dbt run --select +some_model+ # select some_model and all parents and children
+dbt run --select "my_package.*+" # select all models in my_package and their children
+dbt run --select "+some_model+" # select some_model and all parents and children
- $ dbt run --select tag:nightly+ # select "nightly" models and all children
- $ dbt run --select +tag:nightly+ # select "nightly" models and all parents and children
+dbt run --select "tag:nightly+" # select "nightly" models and all children
+dbt run --select "+tag:nightly+" # select "nightly" models and all parents and children
- $ dbt run --select @source:snowplow # build all models that select from snowplow sources, plus their parents
+dbt run --select "@source:snowplow" # build all models that select from snowplow sources, plus their parents
- $ dbt test --select config.incremental_strategy:insert_overwrite,test_name:unique # execute all `unique` tests that select from models using the `insert_overwrite` incremental strategy
- ```
+dbt test --select "config.incremental_strategy:insert_overwrite,test_name:unique" # execute all `unique` tests that select from models using the `insert_overwrite` incremental strategy
+```
@@ -22,8 +22,8 @@ and feed exports, while _excluding_ the biggest incremental models (and one othe
```bash
- $ dbt run --select @source:snowplow,tag:nightly models/export --exclude package:snowplow,config.materialized:incremental export_performance_timing
- ```
+dbt run --select "@source:snowplow,tag:nightly models/export" --exclude "package:snowplow,config.materialized:incremental export_performance_timing"
+```
This command selects all models that:
diff --git a/website/docs/reference/node-selection/set-operators.md b/website/docs/reference/node-selection/set-operators.md
index 7d6b6c2411c..af399b9cad5 100644
--- a/website/docs/reference/node-selection/set-operators.md
+++ b/website/docs/reference/node-selection/set-operators.md
@@ -11,7 +11,7 @@ Run snowplow_sessions, all ancestors of snowplow_sessions, fct_orders, and all a
```bash
- $ dbt run --select +snowplow_sessions +fct_orders
+dbt run --select "+snowplow_sessions +fct_orders"
```
### Intersections
@@ -22,15 +22,15 @@ Run all the common ancestors of snowplow_sessions and fct_orders:
```bash
- $ dbt run --select +snowplow_sessions,+fct_orders
- ```
+dbt run --select "+snowplow_sessions,+fct_orders"
+```
Run all the common descendents of stg_invoices and stg_accounts:
```bash
- $ dbt run --select stg_invoices+,stg_accounts+
+dbt run --select "stg_invoices+,stg_accounts+"
```
@@ -38,5 +38,5 @@ Run models that are in the marts/finance subdirectory *and* tagged nightly:
```bash
- $ dbt run --select marts.finance,tag:nightly
- ```
+dbt run --select "marts.finance,tag:nightly"
+```
diff --git a/website/docs/reference/node-selection/state-comparison-caveats.md b/website/docs/reference/node-selection/state-comparison-caveats.md
index baeeb7e4c75..73947c80a66 100644
--- a/website/docs/reference/node-selection/state-comparison-caveats.md
+++ b/website/docs/reference/node-selection/state-comparison-caveats.md
@@ -27,8 +27,8 @@ The command `dbt test -s state:modified` will include both:
As long as you're adding or changing tests at the same time that you're adding or changing the resources (models, seeds, snapshots) they select from, all should work the way you expect with "simple" state selection:
```shell
-$ dbt run -s state:modified
-$ dbt test -s state:modified
+dbt run -s "state:modified"
+dbt test -s "state:modified"
```
This can get complicated, however. If you add a new test without modifying its underlying model, or add a test that selects from a new model and an old unmodified one, you may need to test a model without having first run it.
@@ -36,8 +36,8 @@ This can get complicated, however. If you add a new test without modifying its u
In v0.18.0, you needed to handle this by building the unmodified models needed for modified tests:
```shell
-$ dbt run -s state:modified @state:modified,1+test_type:data
-$ dbt test -s state:modified
+dbt run -s "state:modified @state:modified,1+test_type:data"
+dbt test -s "state:modified"
```
In v0.19.0, dbt added support for deferring upstream references when testing. If a test selects from a model that doesn't exist as a database object in your current environment, dbt will look to the other environment instead—the one defined in your state manifest. This enables you to use "simple" state selection without risk of query failure, but it may have some surprising consequences for tests with multiple parents. For instance, if you have a `relationships` test that depends on one modified model and one unmodified model, the test query will select from data "across" two different environments. If you limit or sample your data in development and CI, it may not make much sense to test for referential integrity, knowing there's a good chance of mismatch.
@@ -45,8 +45,8 @@ In v0.19.0, dbt added support for deferring upstream references when testing. If
If you're a frequent user of `relationships` tests or data tests, or frequently find yourself adding tests without modifying their underlying models, consider tweaking the selection criteria of your CI job. For instance:
```shell
-$ dbt run -s state:modified
-$ dbt test -s state:modified --exclude test_name:relationships
+dbt run -s "state:modified"
+dbt test -s "state:modified" --exclude "test_name:relationships"
```
### False positives
@@ -58,7 +58,7 @@ State comparison works by identifying discrepancies between two manifests. Thos
dbt will do its best to capture *only* changes that are the result of modifications made in development. In projects with intricate env-aware logic, dbt will err on the side of running too many models (i.e. false positives). Over the next several versions of dbt, we're working on:
- iterative improvements to dbt's built-in detective abilities
-- better options for more complex projects, in the form of more-specific subselectors (see [this issue](https://github.com/dbt-labs/dbt-core/issues/2704))
+- better options for more complex projects, in the form of more-specific sub-selectors (see [this issue](https://github.com/dbt-labs/dbt-core/issues/2704))
State comparison is now able to detect env-aware config in `dbt_project.yml`. For instance, this target-based config would register as a modification in v0.18.0, but in v0.19.0 it no longer will:
diff --git a/website/docs/reference/node-selection/syntax.md b/website/docs/reference/node-selection/syntax.md
index 924347492a1..616bcfe6447 100644
--- a/website/docs/reference/node-selection/syntax.md
+++ b/website/docs/reference/node-selection/syntax.md
@@ -25,6 +25,8 @@ We use the terms "
By default, `dbt run` executes _all_ of the models in the dependency graph; `dbt seed` creates all seeds, `dbt snapshot` performs every snapshot. The `--select` flag is used to specify a subset of nodes to execute.
+To follow [POSIX standards](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html) and make things easier to understand, we recommend CLI users use quotes when passing arguments to the `--select` or `--exclude` option (including single or multiple space-delimited, or comma-delimited arguments). Not using quotes might not work reliably on all operating systems, terminals, and user interfaces. For example, `dbt run --select "my_dbt_project_name"` runs all models in your project.
+
### How does selection work?
1. dbt gathers all the resources that are matched by one or more of the `--select` criteria, in the order of selection methods (e.g. `tag:`), then graph operators (e.g. `+`), then finally set operators ([unions](/reference/node-selection/set-operators#unions), [intersections](/reference/node-selection/set-operators#intersections), [exclusions](/reference/node-selection/exclude)).
@@ -52,28 +54,28 @@ Examples:
```bash
- $ dbt run --select my_dbt_project_name # runs all models in your project
- $ dbt run --select my_dbt_model # runs a specific model
- $ dbt run --select path.to.my.models # runs all models in a specific directory
- $ dbt run --select my_package.some_model # run a specific model in a specific package
- $ dbt run --select tag:nightly # run models with the "nightly" tag
- $ dbt run --select path/to/models # run models contained in path/to/models
- $ dbt run --select path/to/my_model.sql # run a specific model by its path
+dbt run --select "my_dbt_project_name" # runs all models in your project
+dbt run --select "my_dbt_model" # runs a specific model
+dbt run --select "path.to.my.models" # runs all models in a specific directory
+dbt run --select "my_package.some_model" # run a specific model in a specific package
+dbt run --select "tag:nightly" # run models with the "nightly" tag
+dbt run --select "path/to/models" # run models contained in path/to/models
+dbt run --select "path/to/my_model.sql" # run a specific model by its path
```
dbt supports a shorthand language for defining subsets of nodes. This language uses the characters `+`, `@`, `*`, and `,`.
```bash
- # multiple arguments can be provided to --select
- $ dbt run --select my_first_model my_second_model
+# multiple arguments can be provided to --select
+ dbt run --select "my_first_model my_second_model"
- # these arguments can be projects, models, directory paths, tags, or sources
- $ dbt run --select tag:nightly my_model finance.base.*
+# these arguments can be projects, models, directory paths, tags, or sources
+dbt run --select "tag:nightly my_model finance.base.*"
- # use methods and intersections for more complex selectors
- $ dbt run --select path:marts/finance,tag:nightly,config.materialized:table
- ```
+# use methods and intersections for more complex selectors
+dbt run --select "path:marts/finance,tag:nightly,config.materialized:table"
+```
As your selection logic gets more complex, and becomes unwieldly to type out as command-line arguments,
consider using a [yaml selector](/reference/node-selection/yaml-selectors). You can use a predefined definition with the `--selector` flag.
@@ -151,7 +153,7 @@ After issuing one of the above commands, you can reference the results by adding
```bash
# You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag.
-$ dbt run --select result: --defer --state path/to/prod/artifacts
+dbt run --select "result: --defer --state path/to/prod/artifacts"
```
The available options depend on the resource (node) type:
@@ -170,7 +172,7 @@ The available options depend on the resource (node) type:
The state and result selectors can also be combined in a single invocation of dbt to capture errors from a previous run OR any new or modified models.
```bash
-$ dbt run --select result:+ state:modified+ --defer --state ./
+dbt run --select "result:+ state:modified+ --defer --state ./"
```
### Fresh rebuilds
@@ -184,7 +186,7 @@ As example:
```bash
# Command step order
dbt source freshness
-dbt build --select source_status:fresher+
+dbt build --select "source_status:fresher+"
```
@@ -203,6 +205,6 @@ After issuing one of the above commands, you can reference the source freshness
```bash
# You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag.
-$ dbt source freshness # must be run again to compare current to previous state
-$ dbt build --select source_status:fresher+ --state path/to/prod/artifacts
+dbt source freshness # must be run again to compare current to previous state
+dbt build --select "source_status:fresher+" --state path/to/prod/artifacts
```
diff --git a/website/docs/reference/node-selection/test-selection-examples.md b/website/docs/reference/node-selection/test-selection-examples.md
index 478aac932c5..ace886df258 100644
--- a/website/docs/reference/node-selection/test-selection-examples.md
+++ b/website/docs/reference/node-selection/test-selection-examples.md
@@ -19,14 +19,14 @@ Run generic tests only:
```bash
- dbt test --select test_type:generic
+ dbt test --select "test_type:generic"
```
Run singular tests only:
```bash
- dbt test --select test_type:singular
+ dbt test --select "test_type:singular"
```
In both cases, `test_type` checks a property of the test itself. These are forms of "direct" test selection.
@@ -87,8 +87,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in
In this mode, any test that depends on unbuilt resources will raise an error.
```shell
- dbt test --select orders
- dbt build --select orders
+dbt test --select "orders"
+dbt build --select "orders"
```
@@ -102,8 +102,10 @@ It will only include tests whose references are each within the selected nodes.
Put another way, it will prevent tests from running if one or more of its parents is unselected.
```shell
- dbt test --select orders --indirect-selection=cautious
- dbt build --select orders --indirect-selection=cautious
+
+dbt test --select "orders" --indirect-selection=cautious
+dbt build --select "orders" --indirect-selection=cautious
+
```
@@ -122,8 +124,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in
In this mode, any test that depends on unbuilt resources will raise an error.
```shell
- dbt test --select orders
- dbt build --select orders
+dbt test --select "orders"
+dbt build --select "orders"
```
@@ -137,8 +139,10 @@ It will only include tests whose references are each within the selected nodes.
Put another way, it will prevent tests from running if one or more of its parents is unselected.
```shell
- dbt test --select orders --indirect-selection=cautious
- dbt build --select orders --indirect-selection=cautious
+
+dbt test --select "orders" --indirect-selection=cautious
+dbt build --select "orders" --indirect-selection=cautious
+
```
@@ -152,8 +156,9 @@ It will only include tests whose references are each within the selected nodes (
This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input).
```shell
- dbt test --select orders --indirect-selection=buildable
- dbt build --select orders --indirect-selection=buildable
+dbt test --select "orders" --indirect-selection=buildable
+dbt build --select "orders" --indirect-selection=buildable
+
```
@@ -172,8 +177,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in
In this mode, any test that depends on unbuilt resources will raise an error.
```shell
- dbt test --select orders
- dbt build --select orders
+dbt test --select "orders"
+dbt build --select "orders"
```
@@ -187,8 +192,9 @@ It will only include tests whose references are each within the selected nodes.
Put another way, it will prevent tests from running if one or more of its parents is unselected.
```shell
- dbt test --select orders --indirect-selection=cautious
- dbt build --select orders --indirect-selection=cautious
+dbt test --select "orders" --indirect-selection=cautious
+dbt build --select "orders" --indirect-selection=cautious
+
```
@@ -202,8 +208,8 @@ It will only include tests whose references are each within the selected nodes (
This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input).
```shell
-dbt test --select orders --indirect-selection=buildable
-dbt build --select orders --indirect-selection=buildable
+dbt test --select "orders" --indirect-selection=buildable
+dbt build --select "orders" --indirect-selection=buildable
```
@@ -213,8 +219,10 @@ dbt build --select orders --indirect-selection=buildable
This mode will only include tests whose references are each within the selected nodes and will ignore all tests from attached nodes.
```shell
- dbt test --select orders --indirect-selection=empty
- dbt build --select orders --indirect-selection=empty
+
+dbt test --select "orders" --indirect-selection=empty
+dbt build --select "orders" --indirect-selection=empty
+
```
@@ -234,25 +242,25 @@ The following examples should feel somewhat familiar if you're used to executing
```bash
# Run tests on a model (indirect selection)
- dbt test --select customers
-
+ dbt test --select "customers"
+
# Run tests on two or more specific models (indirect selection)
- dbt test --select customers orders
+ dbt test --select "customers orders"
# Run tests on all models in the models/staging/jaffle_shop directory (indirect selection)
- dbt test --select staging.jaffle_shop
+ dbt test --select "staging.jaffle_shop"
# Run tests downstream of a model (note this will select those tests directly!)
- dbt test --select stg_customers+
+ dbt test --select "stg_customers+"
# Run tests upstream of a model (indirect selection)
- dbt test --select +stg_customers
+ dbt test --select "+stg_customers"
# Run tests on all models with a particular tag (direct + indirect)
- dbt test --select tag:my_model_tag
+ dbt test --select "tag:my_model_tag"
# Run tests on all models with a particular materialization (indirect selection)
- dbt test --select config.materialized:table
+ dbt test --select "config.materialized:table"
```
@@ -261,19 +269,20 @@ The following examples should feel somewhat familiar if you're used to executing
```bash
# tests on all sources
- dbt test --select source:*
- # tests on one source
- dbt test --select source:jaffle_shop
+ dbt test --select "source:*"
+ # tests on one source
+ dbt test --select "source:jaffle_shop"
+
# tests on two or more specific sources
- dbt test --select source:jaffle_shop source:raffle_bakery
+ dbt test --select "source:jaffle_shop source:raffle_bakery"
# tests on one source table
- dbt test --select source:jaffle_shop.customers
+ dbt test --select "source:jaffle_shop.customers"
# tests on everything _except_ sources
- dbt test --exclude source:*
+ dbt test --exclude "source:*"
```
### More complex selection
@@ -282,10 +291,12 @@ Through the combination of direct and indirect selection, there are many ways to
```bash
- dbt test --select assert_total_payment_amount_is_positive # directly select the test by name
- dbt test --select payments,test_type:singular # indirect selection, v1.2
- dbt test --select payments,test_type:data # indirect selection, v0.18.0
- dbt test --select payments --data # indirect selection, earlier versions
+
+ dbt test --select "assert_total_payment_amount_is_positive" # directly select the test by name
+ dbt test --select "payments,test_type:singular" # indirect selection, v1.2
+ dbt test --select "payments,test_type:data" # indirect selection, v0.18.0
+ dbt test --select "payments" --data # indirect selection, earlier versions
+
```
@@ -294,13 +305,14 @@ Through the combination of direct and indirect selection, there are many ways to
```bash
# Run tests on all models with a particular materialization
- dbt test --select config.materialized:table
+ dbt test --select "config.materialized:table"
# Run tests on all seeds, which use the 'seed' materialization
- dbt test --select config.materialized:seed
+ dbt test --select "config.materialized:seed"
# Run tests on all snapshots, which use the 'snapshot' materialization
- dbt test --select config.materialized:snapshot
+ dbt test --select "config.materialized:snapshot"
+
```
Note that this functionality may change in future versions of dbt.
@@ -328,7 +340,8 @@ models:
```bash
- dbt test --select tag:my_column_tag
+ dbt test --select "tag:my_column_tag"
+
```
Currently, tests "inherit" tags applied to columns, sources, and source tables. They do _not_ inherit tags applied to models, seeds, or snapshots. In all likelihood, those tests would still be selected indirectly, because the tag selects its parent. This is a subtle distinction, and it may change in future versions of dbt.
@@ -356,5 +369,6 @@ models:
```bash
- dbt test --select tag:my_test_tag
+ dbt test --select "tag:my_test_tag"
+
```
diff --git a/website/docs/reference/node-selection/yaml-selectors.md b/website/docs/reference/node-selection/yaml-selectors.md
index 78342e32779..1e3f8d8d1e2 100644
--- a/website/docs/reference/node-selection/yaml-selectors.md
+++ b/website/docs/reference/node-selection/yaml-selectors.md
@@ -34,6 +34,7 @@ Each `definition` is comprised of one or more arguments, which can be one of the
Use the `union` and `intersection` operator-equivalent keywords to organize multiple arguments.
### CLI-style
+
```yml
definition:
'tag:nightly'
@@ -42,6 +43,7 @@ definition:
This simple syntax supports use of the `+`, `@`, and `*` [graph](/reference/node-selection/graph-operators) operators, but it does not support [set](/reference/node-selection/set-operators) operators or `exclude`.
### Key-value
+
```yml
definition:
tag: nightly
@@ -317,7 +319,7 @@ selectors:
Then in our job definition:
```bash
-$ dbt run --selector nightly_diet_snowplow
+dbt run --selector nightly_diet_snowplow
```
## Default
@@ -325,6 +327,7 @@ $ dbt run --selector nightly_diet_snowplow
Selectors may define a boolean `default` property. If a selector has `default: true`, dbt will use this selector's criteria when tasks do not define their own selection criteria.
Let's say we define a default selector that only selects resources defined in our root project:
+
```yml
selectors:
- name: root_project_only
@@ -338,16 +341,18 @@ selectors:
```
If I run an "unqualified" command, dbt will use the selection criteria defined in `root_project_only`—that is, dbt will only build / freshness check / generate compiled SQL for resources defined in my root project.
+
```
-$ dbt build
-$ dbt source freshness
-$ dbt docs generate
+dbt build
+dbt source freshness
+dbt docs generate
```
If I run a command that defines its own selection criteria (via `--select`, `--exclude`, or `--selector`), dbt will ignore the default selector and use the flag criteria instead. It will not try to combine the two.
-```
-$ dbt run --select model_a
-$ dbt run --exclude model_a
+
+```bash
+dbt run --select "model_a"
+dbt run --exclude model_a
```
Only one selector may set `default: true` for a given invocation; otherwise, dbt will return an error. You may use a Jinja expression to adjust the value of `default` depending on the environment, however:
diff --git a/website/sidebars.js b/website/sidebars.js
index b4da9d911d7..fc87c1142c7 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -24,7 +24,6 @@ const sidebarSettings = {
"docs/cloud/about-cloud/architecture",
"docs/cloud/about-cloud/tenancy",
"docs/cloud/about-cloud/regions-ip-addresses",
- "docs/cloud/about-cloud/about-cloud-ide",
"docs/cloud/about-cloud/browsers",
],
}, // About dbt Cloud directory
@@ -123,19 +122,33 @@ const sidebarSettings = {
}, // Supported Git providers
{
type: "category",
- label: "Develop in the IDE",
- link: {
- type: "doc",
- id: "docs/cloud/dbt-cloud-ide/dbt-cloud-ide",
- },
+ label: "Develop in dbt Cloud",
+ link: { type: "doc", id: "docs/cloud/about-cloud-develop" },
items: [
- "docs/cloud/dbt-cloud-ide/dbt-cloud-ide",
- "docs/cloud/dbt-cloud-ide/develop-in-the-cloud",
- "docs/cloud/dbt-cloud-ide/ide-user-interface",
- "docs/cloud/dbt-cloud-ide/lint-format",
- "docs/cloud/dbt-cloud-ide/dbt-cloud-tips",
+ "docs/cloud/about-cloud-develop",
+ "docs/cloud/about-cloud-develop-defer",
+ {
+ type: "category",
+ label: "dbt Cloud CLI",
+ link: { type: "doc", id: "docs/cloud/cloud-cli-installation" },
+ items: [
+ "docs/cloud/cloud-cli-installation",
+ "docs/cloud/configure-cloud-cli",
+ ],
+ },
+ {
+ type: "category",
+ label: "dbt Cloud IDE",
+ link: { type: "doc", id: "docs/cloud/dbt-cloud-ide/develop-in-the-cloud" },
+ items: [
+ "docs/cloud/dbt-cloud-ide/develop-in-the-cloud",
+ "docs/cloud/dbt-cloud-ide/ide-user-interface",
+ "docs/cloud/dbt-cloud-ide/lint-format",
+ "docs/cloud/dbt-cloud-ide/dbt-cloud-tips",
+ ],
+ },
],
- }, // dbt Cloud IDE directory
+ }, // dbt Cloud develop directory
{
type: "category",
label: "Secure your tenant",
@@ -146,8 +159,9 @@ const sidebarSettings = {
"docs/cloud/secure/about-privatelink",
"docs/cloud/secure/snowflake-privatelink",
"docs/cloud/secure/databricks-privatelink",
- "docs/cloud/secure/redshift-privatelink",
- "docs/cloud/secure/postgres-privatelink",
+ "docs/cloud/secure/redshift-privatelink",
+ "docs/cloud/secure/postgres-privatelink",
+ "docs/cloud/secure/ip-restrictions",
],
}, // PrivateLink
"docs/cloud/billing",
@@ -160,7 +174,7 @@ const sidebarSettings = {
link: { type: "doc", id: "docs/core/about-core-setup" },
items: [
"docs/core/about-core-setup",
- "docs/core/about-the-cli",
+ "docs/core/about-dbt-core",
"docs/core/dbt-core-environments",
{
type: "category",
@@ -283,9 +297,10 @@ const sidebarSettings = {
"docs/build/about-metricflow",
"docs/build/join-logic",
"docs/build/validation",
+ "docs/build/saved-queries",
"docs/build/metricflow-time-spine",
- "docs/build/metricflow-cli",
- ]
+ "docs/build/metricflow-commands",
+ ],
},
{
type: "category",
@@ -295,8 +310,8 @@ const sidebarSettings = {
"docs/build/semantic-models",
"docs/build/dimensions",
"docs/build/entities",
- "docs/build/measures"
- ]
+ "docs/build/measures",
+ ],
},
{
type: "category",
@@ -308,7 +323,7 @@ const sidebarSettings = {
"docs/build/derived",
"docs/build/ratio",
"docs/build/simple",
- ]
+ ],
},
],
},
@@ -442,8 +457,9 @@ const sidebarSettings = {
type: "category",
label: "Use the dbt Semantic Layer",
collapsed: true,
- link: { type: "doc", id: "docs/use-dbt-semantic-layer/quickstart-sl" },
+ link: { type: "doc", id: "docs/use-dbt-semantic-layer/dbt-sl" },
items: [
+ "docs/use-dbt-semantic-layer/dbt-sl",
"docs/use-dbt-semantic-layer/quickstart-sl",
"docs/use-dbt-semantic-layer/setup-sl",
"docs/use-dbt-semantic-layer/sl-architecture",
@@ -453,7 +469,8 @@ const sidebarSettings = {
link: { type: "doc", id: "docs/use-dbt-semantic-layer/avail-sl-integrations" },
items: [
"docs/use-dbt-semantic-layer/avail-sl-integrations",
- //"docs/use-dbt-semantic-layer/gsheets",
+ "docs/use-dbt-semantic-layer/gsheets",
+ "docs/use-dbt-semantic-layer/tableau",
],
},
],
@@ -516,7 +533,10 @@ const sidebarSettings = {
{
type: "category",
label: "Job",
- link: { type: "doc", id: "docs/dbt-cloud-apis/discovery-schema-job" },
+ link: {
+ type: "doc",
+ id: "docs/dbt-cloud-apis/discovery-schema-job",
+ },
items: [
"docs/dbt-cloud-apis/discovery-schema-job",
"docs/dbt-cloud-apis/discovery-schema-job-model",
@@ -978,7 +998,19 @@ const sidebarSettings = {
},
{
type: "category",
- label: "Materializations best practices",
+ label: "How we build our dbt Mesh projects",
+ link: {
+ type: "doc",
+ id: "guides/best-practices/how-we-mesh/mesh-1-intro",
+ },
+ items: [
+ "guides/best-practices/how-we-mesh/mesh-2-structures",
+ "guides/best-practices/how-we-mesh/mesh-3-implementation",
+ ],
+ },
+ {
+ type: "category",
+ label: "Materialization best practices",
link: {
type: "doc",
id: "guides/best-practices/materializations/materializations-guide-1-guide-overview",
diff --git a/website/snippets/_adapters-trusted.md b/website/snippets/_adapters-trusted.md
index 96f62a925eb..7747ce16dec 100644
--- a/website/snippets/_adapters-trusted.md
+++ b/website/snippets/_adapters-trusted.md
@@ -2,19 +2,18 @@
Install with dbt Core
"
icon="materialize"/>
Install with dbt Core
"
icon="oracle"/>
diff --git a/website/snippets/_adapters-verified.md b/website/snippets/_adapters-verified.md
index 7caf099b7d1..3cc1e800448 100644
--- a/website/snippets/_adapters-verified.md
+++ b/website/snippets/_adapters-verified.md
@@ -2,61 +2,60 @@
-* Install these adapters using the CLI as they're not currently supported in dbt Cloud.
@@ -90,16 +81,7 @@ import SlSetUp from '/snippets/_new-sl-setup.md';
## Connect and query API
-You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer.
-
-You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America (Additional region support coming soon).
-
-- To learn how to use the JDBC or GraphQL API and what tools you can query it with, refer to the {frontMatter.meta.api_name}.
- - * To authenticate, you need to [generate a service token](/docs/dbt-cloud-apis/service-tokens) with Semantic Layer Only and Metadata Only permissions. - * Refer to the [SQL query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the APIs. - -- To learn more about the sophisticated integrations that connect to the dbt Semantic Layer, refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more info. +
-import NewSLChanges from '/snippets/_new-sl-changes.md';
-
-
@@ -28,7 +24,7 @@ The dbt Semantic Layer includes the following components:
| **[MetricFlow](/docs/build/about-metricflow)** | MetricFlow in dbt allows users to centrally define their semantic models and metrics with YAML specifications. | ✅ | ✅ | ✅ | BSL package (code is source available) |
| **MetricFlow Server**| A proprietary server that takes metric requests and generates optimized SQL for the specific data platform. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise)|
| **Semantic Layer Gateway** | A service that passes queries to MetricFlow server and executes the SQL generated by MetricFlow against the data platform|
❌| ✅ | ✅ | Proprietary, Cloud (Team & Enterprise) | -| **Semantic Layer API** | The interfaces that allow users to submit metric queries include the MetricFlow CLI and JDBC API. They also serve as the foundation for building first-class integrations with various tools. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise)| +| **Semantic Layer APIs** | The interfaces that allow users to submit metric queries using the GraphQL and JDBC APIs. They also serve as the foundation for building first-class integrations with various tools. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise)| ## Related questions diff --git a/website/docs/docs/use-dbt-semantic-layer/tableau.md b/website/docs/docs/use-dbt-semantic-layer/tableau.md new file mode 100644 index 00000000000..c505ea323f4 --- /dev/null +++ b/website/docs/docs/use-dbt-semantic-layer/tableau.md @@ -0,0 +1,67 @@ +--- +title: "Tableau (beta)" +description: "Use Tableau worksheets to query the dbt Semantic Layer and produce dashboards with trusted date." +tags: [Semantic Layer] +sidebar_label: "Tableau (beta)" +--- + +:::info Beta functionality +The Tableau integration with the dbt Semantic Layer is a [beta feature](/docs/dbt-versions/product-lifecycles#dbt-cloud). +::: + + +The Tableau integration allows you to use worksheets to query the Semantic Layer directly and produce your dashboards with trusted data. This integration provides a live connection to the dbt Semantic Layer through Tableau Desktop. + +## Prerequisites + +1. You must have [Tableau Desktop](https://www.tableau.com/en-gb/products/desktop) installed +2. Authenticate with either Tableau Server or Tableau Cloud +3. You need your dbt Cloud host, [Environment ID](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) and [service token](/docs/dbt-cloud-apis/service-tokens) to log in. This account should be set up with the dbt Semantic Layer. +4. You must a dbt Cloud Team or Enterprise [account](https://www.getdbt.com/pricing) and multi-tenant [deployment](/docs/cloud/about-cloud/regions-ip-addresses). (Single-Tenant coming soon) + + +## Installing + +1. Download our [connector file](https://github.com/dbt-labs/semantic-layer-tableau-connector/releases/download/v1.0.0/dbt_semantic_layer.taco) locally and add it to your default folder: + - Windows: `C:\Users\\[Windows User]\Documents\My Tableau Repository\Connectors` + - Mac: `/Users/[user]/Documents/My Tableau Repository/Connectors` + - Linux: `/opt/tableau/connectors` +2. Install the [JDBC driver](/docs/dbt-cloud-apis/sl-jdbc) to the folder based on your operating system: + - Windows: `C:\Program Files\Tableau\Drivers` + - Mac: `~/Library/Tableau/Drivers` + - Linux: ` /opt/tableau/tableau_driver/jdbc` +3. Open Tableau Desktop and find the **dbt Semantic Layer by dbt Labs** connector on the left-hand side. +4. Connect with your Host, Environment ID, and service token information that's provided to you in your dbt Cloud Semantic Layer configuration. + + +## Using the integration + +Once you authenticate, the system will direct you to the data source page with all the metrics and dimensions configured in your Semantic Layer. + +- From there, go directly to a worksheet in the bottom left hand corner. +- Then, you'll find all the metrics and dimensions that are available to query on the left-hand side of your window. + +Visit the [Tableau documentation](https://help.tableau.com/current/pro/desktop/en-us/gettingstarted_overview.htm) to learn more about how to use Tableau worksheets and dashboards. + +## Things to note + +- All metrics use the "SUM" aggregation type, and this can't be altered. The dbt Semantic Layer controls the aggregation type and it is intentionally fixed. Keep in mind that the underlying aggregation in the dbt Semantic Layer might not be "SUM" (even though "SUM" is Tableau's default). +- Tableau surfaces all metrics and dimensions from the dbt Semantic Layer on the left-hand side. Note, that not all metrics and dimensions can be combined with one another. You will receive an error message if a particular dimension cannot be sliced with a metric (or vice versa). + - To display available metrics and dimensions, dbt Semantic Layer returns metadata for a fake table with the dimensions and metrics as 'columns' on this table. Because of this, you can't actually query this table for previews or extracts. + - Since this is treated as a table, dbt Semantic Layer can't dynamically change what is available. This means we display _all_ available metrics and dimensions even if a particular metric and dimension combination isn't available. + +- Certain Table calculations like "Totals" and "Percent Of" may not be accurate when using metrics aggregated in a non-additive way (such as count distinct) + +## Unsupported functionality + +The following Tableau features aren't supported at this time, however, the dbt Semantic Layer may support some of this functionality in a future release: + +- Updating the data source page +- Using "Extract" mode to view yur data +- Unioning Tables +- Writing Custom SQL +- Table Extensions +- Cross Database Joins +- All functions in Analysis --> Create Calculated Field +- Filtering on a Date Part time dimension for a Cumulative metric type +- Changing your date dimension to use "Week Number" diff --git a/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md b/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md index 34c0e813725..801227924dd 100644 --- a/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md +++ b/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md @@ -13,7 +13,7 @@ git clone git@github.com:dbt-labs/jaffle-sl-template.git cd path/to/project ``` -Next before we start writing code, we'll need to install the MetricFlow CLI as an extension of a dbt adapter from PyPI. The MetricFlow CLI is compatible with Python versions 3.8 through 3.11. +Next, before you start writing code, you need to install MetricFlow as an extension of a dbt adapter from PyPI (dbt Core users only). The MetricFlow is compatible with Python versions 3.8 through 3.11. We'll use pip to install MetricFlow and our dbt adapter: @@ -33,7 +33,7 @@ Lastly, to get to the pre-Semantic Layer starting state, checkout the `start-her git checkout start-here ``` -For more information you can [look at the docs](/docs/build/metricflow-cli) or checkout a [Quickstart](https://docs.getdbt.com/quickstarts) to get more familiar with setting up a dbt project. +For more information, refer to the [MetricFlow commands](/docs/build/metricflow-commands) or a [quickstart](/quickstarts) to get more familiar with setting up a dbt project. ## Basic commands diff --git a/website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md b/website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md new file mode 100644 index 00000000000..ba1660a8d82 --- /dev/null +++ b/website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md @@ -0,0 +1,39 @@ +--- +title: "Intro to dbt Mesh" +description: Getting started with dbt Mesh patterns +hoverSnippet: Learn how to get started with dbt Mesh +--- + +## What is dbt Mesh? + +Organizations of all sizes rely upon dbt to manage their data transformations, from small startups to large enterprises. At scale, it can be challenging to coordinate all the organizational and technical requirements demanded by your stakeholders within the scope of a single dbt project. To date, there also hasn't been a first-class way to effectively manage the dependencies, governance, and workflows between multiple dbt projects. + +Regardless of your organization's size and complexity, dbt should empower data teams to work independently and collaboratively; sharing data, code, and best practices without sacrificing security or autonomy. dbt Mesh provides the tooling for teams to finally achieve this. + +dbt Mesh is not a single product: it is a pattern enabled by a convergence of several features in dbt: + +- **[Cross-project references](/docs/collaborate/govern/project-dependencies#how-to-use-ref)** - this is the foundational feature that enables the multi-project deployments. `{{ ref() }}`s now work across dbt Cloud projects on Enterprise plans. +- **[dbt Explorer](/docs/collaborate/explore-projects)** - dbt Cloud's metadata-powered documentation platform, complete with full, cross-project lineage. +- **Governance** - dbt's new governance features allow you to manage access to your dbt models both within and across projects. + - **[Groups](/docs/collaborate/govern/model-access#groups)** - groups allow you to assign models to subsets within a project. + - **[Access](/docs/collaborate/govern/model-access#access-modifiers)** - access configs allow you to control who can reference models. +- **[Model Versions](/docs/collaborate/govern/model-versions)** - when coordinating across projects and teams, we recommend treating your data models as stable APIs. Model versioning is the mechanism to allow graceful adoption and deprecation of models as they evolve. +- **[Model Contracts](/docs/collaborate/govern/model-contracts)** - data contracts set explicit expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. + +## Who is dbt Mesh for? + +The multi-project architecture helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. If you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring this paradigm: + +- The **number of models** in your project is degrading performance and slowing down development. +- Teams have developed **separate workflows** and need to decouple development from each other. +- **Security and governance** requirements are increasing and would benefit from increased isolation. + +dbt Cloud is designed to coordinate the features above and simplify the complexity to solve for these problems. + +If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can _incrementally_ adopt the features in this guide as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow. + +## Learning goals + +- Understand the **purpose and tradeoffs** of building a multi-project architecture. +- Develop an intuition for various **dbt Mesh patterns** and how to design a multi-project architecture for your organization. +- Establish recommended steps to **incrementally adopt** these patterns in your dbt implementation. diff --git a/website/docs/guides/best-practices/how-we-mesh/mesh-2-structures.md b/website/docs/guides/best-practices/how-we-mesh/mesh-2-structures.md new file mode 100644 index 00000000000..937515954af --- /dev/null +++ b/website/docs/guides/best-practices/how-we-mesh/mesh-2-structures.md @@ -0,0 +1,52 @@ +--- +title: Deciding how to structure your dbt Mesh +description: Getting started with dbt Mesh patterns +hoverSnippet: Learn how to get started with dbt Mesh +--- +## Exploring mesh patterns + +When adopting a multi-project architecture, where do you draw the lines between projects? + +How should you organize data workflows in a world where instead of having a single dbt DAG, you have multiple projects speaking to each other, each comprised of their own DAG? + +Adopting the dbt Mesh pattern is not a one-size-fits-all process. In fact, it's the opposite! It's about customizing your project structure to fit _your_ team and _your_ data. Now you can mold your organizational knowledge graph to your organizational people graph, bringing people and data closer together rather than compromising one for the other. + +While there is not a single best way to implement this pattern, there are some common decision points that will be helpful for you to consider. + +At a high level, you’ll need to decide: + +- Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project? +- How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project? + +## Define your project interfaces by splitting your DAG + +The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns. + +### Vertical splits + +Vertical splits separate out layers of transformation in DAG order. Let's look at some examples. + +- **Splitting up staging and mart layers** to create a more tightly-controlled, shared set of components that other projects build on but can't edit. +- **Isolating earlier models for security and governance requirements** to separate out and mask PII data so that downstream consumers can't access it is a common use case for a vertical split. +- **Protecting complex or expensive data** to isolate large or complex models that are expensive to run so that they are safe from accidental selection, independently deployable, and easier to debug when they have issues. + +### Horizontal splits + +Horizontal splits separate your DAG based on source or domain. These splits are often based around the shape and size of the data and how it's used. Let's consider some possibilities for horizontal splitting. + +- **Team consumption patterns.** For example, splitting out the marketing team's data flow into a separate project. +- **Data from different sources.** For example, clickstream event data and transactional ecommerce data may need to be modeled independently of each other. +- **Team workflows.** For example, if two embedded groups operate at different paces, you may want to split the projects up so they can move independently. + +### Combining these strategies + +- **These are not either/or techniques**. You should consider both types of splits, and combine them in any way that makes sense for your organization. +- **Pick one type of split and focus on that first**. If you have a hub-and-spoke team topology for example, handle breaking out the central platform project before you split the remainder into domains. Then if you need to break those domains up horizontally you can focus on that after the fact. +- **DRY applies to underlying data, not just code.** Regardless of your strategy, you should not be sourcing the same rows and columns into multiple nodes. When working within a mesh pattern it becomes increasingly important that we don't duplicate logic or data. + +## Determine your git strategy + +A multi-project architecture can exist in a single repo (monorepo) or as multiple projects, with each one being in their own repository (multi-repo). + +- If you're a **smaller team** looking primarily to speed up and simplify development, a **monorepo** is likely the right choice, but can become unwieldy as the number of projects, models and contributors grow. +- If you’re a **larger team with multiple groups**, and need to decouple projects for security and enablement of different development styles and rhythms, a **multi-repo setup** is your best bet. diff --git a/website/docs/guides/best-practices/how-we-mesh/mesh-3-implementation.md b/website/docs/guides/best-practices/how-we-mesh/mesh-3-implementation.md new file mode 100644 index 00000000000..cfbbc7a1f28 --- /dev/null +++ b/website/docs/guides/best-practices/how-we-mesh/mesh-3-implementation.md @@ -0,0 +1,130 @@ +--- +title: "Implementing your mesh plan" +description: Getting started with dbt Mesh patterns +hoverSnippet: Learn how to get started with dbt Mesh +--- + +As mentioned before, the key decision in migrating to a multi-project architecture is understanding how your project is already being grouped, built, and deployed. We can use this information to inform our decision to split our project apart. + +- **Examine your jobs** - which sets of models are most often built together? +- **Look at your lineage graph** - how are models connected? +- **Look at your selectors** defined in `selectors.yml` - how do people already define resource groups? +- **Talk to teams** about what sort of separation naturally exists right now. + - Are there various domains people are focused on? + - Are there various sizes, shapes, and sources of data that get handled separately (such as click event data)? + - Are there people focused on separate levels of transformation, such as landing and staging data or building marts? + +## Add groups and access + +Once you have a sense of some initial groupings, you can first implement **group and access permissions** within a single project. + +- First you can create a [group](/docs/build/groups) to define the owner of a set of models. + +```yml +# in models/__groups.yml + +groups: + - name: marketing + owner: + - name: Ben Jaffleck + email: ben.jaffleck@jaffleshop.com +``` + +- Then, we can add models to that group using the `group:` key in the model's YAML entry. + +```yml +# in models/marketing/__models.yml + +models: + - name: fct_marketing_model + group: marketing + - name: stg_marketing_model + group: marketing +``` + +- Once you've added models to the group, you can **add [access](/docs/collaborate/govern/model-access) settings to the models** based on their connections between groups, *opting for the most private access that will maintain current functionality*. This means that any model that has *only* relationships to other models in the same group should be `private` , and any model that has cross-group relationships, or is a terminal node in the group DAG should be `protected` so that other parts of the DAG can continue to reference it. + +```yml +# in models/marketing/__models.yml + +models: + - name: fct_marketing_model + group: marketing + access: protected + - name: stg_marketing_model + group: marketing + access: private +``` + +- **Validate these groups by incrementally migrating your jobs** to execute these groups specifically via selection syntax. We would recommend doing this in parallel to your production jobs until you’re sure about them. This will help you feel out if you’ve drawn the lines in the right place. +- If you find yourself **consistently making changes across multiple groups** when you update logic, that’s a sign that **you may want to rethink your groups**. + +## Split your projects + +1. **Move your grouped models into a subfolder**. This will include any model in the selected group, it's associated YAML entry, as well as its parent or child resources as appropriate depending on where this group sits in your DAG. + 1. Note that just like in your dbt project, circular refereneces are not allowed! Project B cannot have parents and children in Project A, for example. +2. **Create a new `dbt_project.yml` file** in the subdirectory. +3. **Copy any macros** used by the resources you moved. +4. **Create a new `packages.yml` file** in your subdirectory with the packages that are used by the resources you moved. +5. **Update `{{ ref }}` functions** — For any model that has a cross-project dependency (this may be in the files you moved, or in the files that remain in your project): + 1. Update the `{{ ref() }}` function to have two arguments, where the first is the name of the source project and the second is the name of the model: e.g. `{{ ref('jaffle_shop', 'my_upstream_model') }}` + 2. Update the upstream, cross-project parents’ `access` configs to `public` , ensuring any project can safely `{{ ref() }}` those models. + 3. We *highly* recommend adding a [model contract](/docs/collaborate/govern/model-contracts) to the upstream models to ensure the data shape is consistent and reliable for your downstream consumers. +6. **Create a `dependencies.yml` file** ([docs](/docs/collaborate/govern/project-dependencies)) for the downstream project, declaring the upstream project as a dependency. + +```yml + +# in dependencies.yml +projects: + - name: jaffle_shop +``` + +### Best practices + +- When you’ve **confirmed the right groups**, it's time to split your projects. + - **Do *one* group at a time**! + - **Do *not* refactor as you migrate**, however tempting that may be. Focus on getting 1-to-1 parity and log any issues you find in doing the migration for later. Once you’ve fully migrated the project then you can start optimizing it for its new life as part of your mesh. +- Start by splitting your project within the same repository for full git tracking and easy reversion if you need to start from scratch. + + +## Connecting existing projects + +Some organizations may already be coordinating across multiple dbt projects. Most often this is via: + +1. Installing parent projects as dbt packages +2. Using `{{ source() }}` functions to read the outputs of a parent project as inputs to a child project. + +This has a few drawbacks: + +1. If using packages, each project has to include *all* resources from *all* projects in its manifest, slowing down dbt and the development cycle. +2. If using sources, there are breakages in the lineage, as there's no real connection between the parent and child projects. + +The migration steps here are much simpler than splitting up a monolith! + +1. If using the `package` method: + 1. In the parent project: + 1. mark all models being referenced downstream as `public` and add a model contract. + 2. In the child project: + 1. Remove the package entry from `packages.yml` + 2. Add the upstream project to your `dependencies.yml` + 3. Update the `{{ ref() }}` functions to models from the upstream project to include the project name argument. +1. If using `source` method: + 1. In the parent project: + 1. mark all models being imported downstream as `public` and add a model contract. + 2. In the child project: + 1. Add the upstream project to your `dependencies.yml` + 2. Replace the `{{ source() }}` functions with cross project `{{ ref() }}` functions. + 3. Remove the unnecessary `source` definitions. + +## Additional Resources +### Our example projects + +We've provided a set of example projects you can use to explore the topics covered here. We've split our [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) project into 3 separate projects in a multi-repo dbt Mesh. Note that you'll need to leverage dbt Cloud to use multi-project architecture, as cross-project references are powered via dbt Cloud's APIs. + +- **[Platform](https://github.com/dbt-labs/jaffle-shop-mesh-platform)** - containing our centralized staging models. +- **[Marketing](https://github.com/dbt-labs/jaffle-shop-mesh-marketing)** - containing our marketing marts. +- **[Finance](https://github.com/dbt-labs/jaffle-shop-mesh-finance)** - containing our finance marts. + +### dbt-meshify + +We recommend using the `dbt-meshify` [command line tool](
```
-$ dbt ls --select snowplow.* --output json --output-keys name resource_type description
+$ dbt ls --select snowplow.* --output json --output-keys "name resource_type description"
{"name": "snowplow_events", "description": "This is a pretty cool model", ...}
{"name": "snowplow_page_views", "description": "This model is even cooler", ...}
...
diff --git a/website/docs/reference/commands/seed.md b/website/docs/reference/commands/seed.md
index 8a410706842..d0cd199ea12 100644
--- a/website/docs/reference/commands/seed.md
+++ b/website/docs/reference/commands/seed.md
@@ -12,7 +12,7 @@ The `dbt seed` command will load `csv` files located in the `seed-paths` directo
Specific seeds can be run using the `--select` flag to `dbt seed`. Example:
```
-$ dbt seed --select country_codes
+$ dbt seed --select "country_codes"
Found 2 models, 3 tests, 0 archives, 0 analyses, 53 macros, 0 operations, 2 seed files
14:46:15 | Concurrency: 1 threads (target='dev')
diff --git a/website/docs/reference/commands/show.md b/website/docs/reference/commands/show.md
index 5bdcfacc1e8..a0e5d68c83f 100644
--- a/website/docs/reference/commands/show.md
+++ b/website/docs/reference/commands/show.md
@@ -16,7 +16,7 @@ The results of the preview query are not materialized in the data warehouse, or
Example:
```
-dbt show --select model_name.sql
+dbt show --select "model_name.sql"
```
or
```
@@ -26,7 +26,7 @@ dbt show --inline "select * from {{ ref('model_name') }}"
The following is an example of `dbt show` output for a model named `stg_orders`:
```bash
-dbt show --select stg_orders
+dbt show --select "stg_orders"
21:17:38 Running with dbt=1.5.0-b5
21:17:38 Found 5 models, 20 tests, 0 snapshots, 0 analyses, 425 macros, 0 operations, 3 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
21:17:38
@@ -46,7 +46,7 @@ dbt show --select stg_orders
For example, if you've just built a model that has a failing test, you can quickly preview the test failures right in the terminal, to find values of `id` that are duplicated:
```bash
-$ dbt build -s my_model_with_duplicates
+$ dbt build -s "my_model_with_duplicates"
13:22:47 Running with dbt=1.5.0
...
13:22:48 Completed with 1 error and 0 warnings:
@@ -58,7 +58,7 @@ $ dbt build -s my_model_with_duplicates
13:22:48
13:22:48 Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2
-$ dbt show -s unique_my_model_with_duplicates_id
+$ dbt show -s "unique_my_model_with_duplicates_id"
13:22:53 Running with dbt=1.5.0
13:22:53 Found 4 models, 2 tests, 0 snapshots, 0 analyses, 309 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
13:22:53
diff --git a/website/docs/reference/commands/source.md b/website/docs/reference/commands/source.md
index b29bf7dadc6..697ae2b5fcc 100644
--- a/website/docs/reference/commands/source.md
+++ b/website/docs/reference/commands/source.md
@@ -20,10 +20,10 @@ By default, `dbt source freshness` will calculate freshness information for all
```bash
# Snapshot freshness for all Snowplow tables:
-$ dbt source freshness --select source:snowplow
+$ dbt source freshness --select "source:snowplow"
# Snapshot freshness for a particular source table:
-$ dbt source freshness --select source:snowplow.event
+$ dbt source freshness --select "source:snowplow.event"
```
### Configuring source freshness output
diff --git a/website/docs/reference/commands/test.md b/website/docs/reference/commands/test.md
index a1a63729568..c050d82a0ab 100644
--- a/website/docs/reference/commands/test.md
+++ b/website/docs/reference/commands/test.md
@@ -10,22 +10,22 @@ The tests to run can be selected using the `--select` flag discussed [here](/ref
```bash
# run tests for one_specific_model
-dbt test --select one_specific_model
+dbt test --select "one_specific_model"
# run tests for all models in package
-dbt test --select some_package.*
+dbt test --select "some_package.*"
# run only tests defined singularly
-dbt test --select test_type:singular
+dbt test --select "test_type:singular"
# run only tests defined generically
-dbt test --select test_type:generic
+dbt test --select "test_type:generic"
# run singular tests limited to one_specific_model
-dbt test --select one_specific_model,test_type:singular
+dbt test --select "one_specific_model,test_type:singular"
# run generic tests limited to one_specific_model
-dbt test --select one_specific_model,test_type:generic
+dbt test --select "one_specific_model,test_type:generic"
```
For more information on writing tests, see the [Testing Documentation](/docs/build/tests).
diff --git a/website/docs/reference/dbt-commands.md b/website/docs/reference/dbt-commands.md
index 862829ef809..1448d9849d3 100644
--- a/website/docs/reference/dbt-commands.md
+++ b/website/docs/reference/dbt-commands.md
@@ -2,54 +2,58 @@
title: "dbt Command reference"
---
-dbt is typically run one of two ways:
+You can run dbt using the following tools:
-* In [dbt Cloud](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud)
-* On the [command line interface](/docs/core/about-the-cli) (CLI)
+- In your browser with the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud)
+- On the command line interface using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) or open-source [dbt Core](/docs/core/about-dbt-core), both of which enable you to execute dbt commands. The key distinction is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its [features](/docs/cloud/about-cloud/dbt-cloud-features).
The following sections outline the commands supported by dbt and their relevant flags. For information about selecting models on the command line, consult the docs on [Model selection syntax](/reference/node-selection/syntax).
### Available commands
-
-
-Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [CLI](/docs/core/about-the-cli). Use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.
-
-| Command | Description | Version |
-| ------- | ----------- | ------- |
-| [build](/reference/commands/build) | Build and test all selected resources (models, seeds, snapshots, tests) | All [supported versions](/docs/dbt-versions/core) |
-| [clean](/reference/commands/clean) | Deletes artifacts present in the dbt project | All [supported versions](/docs/dbt-versions/core) |
-| [clone](/reference/commands/clone) | Clone selected models from the specified state | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) |
-| [compile](/reference/commands/compile) | Compiles (but does not run) the models in a project | All [supported versions](/docs/dbt-versions/core) |
-| [debug](/reference/commands/debug) | Debugs dbt connections and projects | All [supported versions](/docs/dbt-versions/core) |
-| [deps](/reference/commands/deps) | Downloads dependencies for a project | All [supported versions](/docs/dbt-versions/core) |
-| [docs](/reference/commands/cmd-docs) | Generates documentation for a project | All [supported versions](/docs/dbt-versions/core) |
-| [list](/reference/commands/list) | Lists resources defined in a dbt project | All [supported versions](/docs/dbt-versions/core) |
-| [parse](/reference/commands/parse) | Parses a project and writes detailed timing info | All [supported versions](/docs/dbt-versions/core) |
-| [retry](/reference/commands/retry) | Retry the last run `dbt` command from the point of failure | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) |
-| [run](/reference/commands/run) | Runs the models in a project | All [supported versions](/docs/dbt-versions/core) |
-| [run-operation](/reference/commands/run-operation) | Invoke a macro, including running arbitrary maintenance SQL against
the database | All [supported versions](/docs/dbt-versions/core) | -| [seed](/reference/commands/seed) | Loads CSV files into the database | All [supported versions](/docs/dbt-versions/core) | -| [show](/reference/commands/show) | Preview table rows post-transformation | All [supported versions](/docs/dbt-versions/core) | -| [snapshot](/reference/commands/snapshot) | Executes "snapshot" jobs defined in a project | All [supported versions](/docs/dbt-versions/core) | -| [source](/reference/commands/source) | Provides tools for working with source data (including validating that
sources are "fresh") | All [supported versions](/docs/dbt-versions/core) | -| [test](/reference/commands/test) | Executes tests defined in a project | All [supported versions](/docs/dbt-versions/core) | -| [init](/reference/commands/init) | Initializes a new dbt project (CLI only) | All [supported versions](/docs/dbt-versions/core) | - +
+
+All commands in the table are compatible with either the dbt Cloud IDE, dbt Cloud CLI, or dbt Core.
+
+You can run dbt commands in your specific tool by prefixing them with `dbt`. For example, to run the `test` command, type `dbt test`.
+
+| Command | Description | Compatible tools | Version |
+| ------- | ----------- | ---------------- | ------- |
+| [build](/reference/commands/build) | Build and test all selected resources (models, seeds, snapshots, tests) | All | All [supported versions](/docs/dbt-versions/core) |
+| cancel | Cancels the most recent invocation.| dbt Cloud CLI | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) |
+| [clean](/reference/commands/clean) | Deletes artifacts present in the dbt project | All | All [supported versions](/docs/dbt-versions/core) |
+| [clone](/reference/commands/clone) | Clone selected models from the specified state | dbt Cloud CLI
dbt Core | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | +| [compile](/reference/commands/compile) | Compiles (but does not run) the models in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [debug](/reference/commands/debug) | Debugs dbt connections and projects | dbt Core | All [supported versions](/docs/dbt-versions/core) | +| [deps](/reference/commands/deps) | Downloads dependencies for a project | All | All [supported versions](/docs/dbt-versions/core) | +| [docs](/reference/commands/cmd-docs) | Generates documentation for a project | All | All [supported versions](/docs/dbt-versions/core) | +| help | Displays help information for any command | dbt Core
dbt Cloud CLI | All [supported versions](/docs/dbt-versions/core) | +| [list](/reference/commands/list) | Lists resources defined in a dbt project | All | All [supported versions](/docs/dbt-versions/core) | +| [parse](/reference/commands/parse) | Parses a project and writes detailed timing info | All | All [supported versions](/docs/dbt-versions/core) | +| reattach | Reattaches to the most recent invocation to retrieve logs and artifacts. | dbt Cloud CLI | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | +| [retry](/reference/commands/retry) | Retry the last run `dbt` command from the point of failure | All | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | +| [run](/reference/commands/run) | Runs the models in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [run-operation](/reference/commands/run-operation) | Invoke a macro, including running arbitrary maintenance SQL against the database | All | All [supported versions](/docs/dbt-versions/core) | +| [seed](/reference/commands/seed) | Loads CSV files into the database | All | All [supported versions](/docs/dbt-versions/core) | +| [show](/reference/commands/show) | Preview table rows post-transformation | All | All [supported versions](/docs/dbt-versions/core) | +| [snapshot](/reference/commands/snapshot) | Executes "snapshot" jobs defined in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [source](/reference/commands/source) | Provides tools for working with source data (including validating that sources are "fresh") | All | All [supported versions](/docs/dbt-versions/core) | +| [test](/reference/commands/test) | Executes tests defined in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [init](/reference/commands/init) | Initializes a new dbt project | dbt Core | All [supported versions](/docs/dbt-versions/core) |
-
+
Select the tabs that are relevant to your development workflow. For example, if you develop in the dbt Cloud IDE, select **dbt Cloud**.
-
+
Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.
- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
-- [clone](/reference/commands/clone): clone selected nodes from specified state (requires dbt 1.6 or higher)
+- [clone](/reference/commands/clone): clone selected nodes from the specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [deps](/reference/commands/deps): downloads dependencies for a project
- [docs](/reference/commands/cmd-docs) : generates documentation for a project
@@ -64,13 +68,13 @@ Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/
-
+
-Use the following dbt commands in the [CLI](/docs/core/about-the-cli) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.
+Use the following dbt commands in [dbt Core](/docs/core/about-dbt-core) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.
- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clean](/reference/commands/clean): deletes artifacts present in the dbt project
-- [clone](/reference/commands/clone): clone selected models from specified state (requires dbt 1.6 or higher)
+- [clone](/reference/commands/clone): clone selected models from the specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [debug](/reference/commands/debug): debugs dbt connections and projects
- [deps](/reference/commands/deps): downloads dependencies for a project
diff --git a/website/docs/reference/dbt_project.yml.md b/website/docs/reference/dbt_project.yml.md
index c706b57a73b..571e930d7da 100644
--- a/website/docs/reference/dbt_project.yml.md
+++ b/website/docs/reference/dbt_project.yml.md
@@ -11,6 +11,8 @@ By default, dbt will look for `dbt_project.yml` in your current working director
By default, dbt will look for `dbt_project.yml` in your current working directory and its parents, but you can set a different directory using the `--project-dir` flag or the `DBT_PROJECT_DIR` environment variable.
+Starting from dbt v1.5 and higher, you can specify your dbt Cloud project ID in the `dbt_project.yml` file using the `dbt-cloud` config, which doesn't require validation or storage in the project config class. To find your project ID, check your dbt Cloud project URL, such as `https://cloud.getdbt.com/11/projects/123456`, where the project ID is `123456`.
+
The following is a list of all available configurations in the `dbt_project.yml` file.
@@ -19,6 +21,81 @@ The following is a list of all available configurations in the `dbt_project.yml`
dbt uses YAML in a few different places. If you're new to YAML, it would be worth taking the time to learn how arrays, dictionaries and strings are represented.
:::
+
+
+
+
+
+```yml
+[name](/reference/project-configs/name): string
+
+[config-version](/reference/project-configs/config-version): 2
+[version](/reference/project-configs/version): version
+
+[profile](/reference/project-configs/profile): profilename
+
+[model-paths](/reference/project-configs/model-paths): [directorypath]
+[seed-paths](/reference/project-configs/seed-paths): [directorypath]
+[test-paths](/reference/project-configs/test-paths): [directorypath]
+[analysis-paths](/reference/project-configs/analysis-paths): [directorypath]
+[macro-paths](/reference/project-configs/macro-paths): [directorypath]
+[snapshot-paths](/reference/project-configs/snapshot-paths): [directorypath]
+[docs-paths](/reference/project-configs/docs-paths): [directorypath]
+[asset-paths](/reference/project-configs/asset-paths): [directorypath]
+
+[target-path](/reference/project-configs/target-path): directorypath
+[log-path](/reference/project-configs/log-path): directorypath
+[packages-install-path](/reference/project-configs/packages-install-path): directorypath
+
+[clean-targets](/reference/project-configs/clean-targets): [directorypath]
+
+[query-comment](/reference/project-configs/query-comment): string
+
+[require-dbt-version](/reference/project-configs/require-dbt-version): version-range | [version-range]
+
+[dbt-cloud](/docs/cloud/cloud-cli-installation):
+ project-id: project_id #Required
+ defer-env-id: 5678 #Optional
+
+[quoting](/reference/project-configs/quoting):
+ database: true | false
+ schema: true | false
+ identifier: true | false
+
+models:
+ [](/reference/model-configs)
+
+seeds:
+ [](/reference/seed-configs)
+
+snapshots:
+ [](/reference/snapshot-configs)
+
+sources:
+ [](source-configs)
+
+tests:
+ [](/reference/test-configs)
+
+vars:
+ [](/docs/build/project-variables)
+
+[on-run-start](/reference/project-configs/on-run-start-on-run-end): sql-statement | [sql-statement]
+[on-run-end](/reference/project-configs/on-run-start-on-run-end): sql-statement | [sql-statement]
+
+[dispatch](/reference/project-configs/dispatch-config):
+ - macro_namespace: packagename
+ search_order: [packagename]
+
+[restrict-access](/docs/collaborate/govern/model-access): true | false
+
+```
+
+
+
+
+
+
```yml
@@ -79,6 +156,9 @@ vars:
search_order: [packagename]
[restrict-access](/docs/collaborate/govern/model-access): true | false
+
```
+
+
diff --git a/website/docs/reference/global-configs/command-line-flags.md b/website/docs/reference/global-configs/command-line-flags.md
index 6496c92da6d..fbe89ce28f1 100644
--- a/website/docs/reference/global-configs/command-line-flags.md
+++ b/website/docs/reference/global-configs/command-line-flags.md
@@ -4,60 +4,95 @@ id: "command-line-flags"
sidebar: "Command line flags"
---
-Command line (CLI) flags immediately follow `dbt` and precede your subcommand. When set, CLI flags override environment variables and profile configs.
+For consistency, command-line interface (CLI) flags should come right after the `dbt` prefix and its subcommands. This includes "global" flags (supported for all commands). When set, CLI flags override environment variables and profile configs.
-Use this non-boolean config structure, replacing `` with the config you are enabling or disabling, `` with the new setting for the config, and `` with the command this config applies to:
+For example, instead of using:
+
+```bash
+dbt --no-populate-cache run
+```
+
+You should use:
+
+```bash
+dbt run --no-populate-cache
+```
+
+Historically, passing flags (such as "global flags") _before_ the subcommand is a legacy functionality that dbt Labs can remove at any time. We do not support using the same flag before and after the subcommand.
+
+## Using boolean and non-boolean flags
+
+You can construct your commands with boolean flags to enable or disable or with non-boolean flags that use specific values, such as strings.
+
+
+
+
+
+Use this non-boolean config structure:
+- Replacing `` with the command this config applies to.
+- `` with the config you are enabling or disabling, and
+- `` with the new setting for the config.
```text
-$ --=
+ --=
```
-Non-boolean config examples:
+### Example
```text
-dbt --printer-width=80 run
-dbt --indirect-selection=eager test
+dbt run --printer-width=80
+dbt test --indirect-selection=eager
```
-To turn on boolean configs, you would use the `--` CLI flag, and a `--no-` CLI flag to turn off boolean configs, replacing `` with the config you are enabling or disabling and `` with the command this config applies to.
+
+
+
+
+To enable or disable boolean configs:
+- Use `` this config applies to.
+- Followed by `--` to turn it on, or `--no-` to turn it off.
+- Replace `` with the config you are enabling or disabling
-Boolean config structure:
```text
-dbt --
-dbt --no-
+dbt --
+dbt --no-
```
-Boolean config example:
+### Example
```text
-dbt --version-check run
-dbt --no-version-check run
+dbt run --version-check
+dbt run --no-version-check
```
-
\ No newline at end of file
+
+
+
+
+
diff --git a/website/docs/reference/node-selection/defer.md b/website/docs/reference/node-selection/defer.md
index e13a4f6648a..03c3b2aac12 100644
--- a/website/docs/reference/node-selection/defer.md
+++ b/website/docs/reference/node-selection/defer.md
@@ -17,16 +17,16 @@ It is possible to use separate state for `state:modified` and `--defer`, by pass
### Usage
```shell
-$ dbt run --select [...] --defer --state path/to/artifacts
-$ dbt test --select [...] --defer --state path/to/artifacts
+dbt run --select [...] --defer --state path/to/artifacts
+dbt test --select [...] --defer --state path/to/artifacts
```
```shell
-$ dbt run --models [...] --defer --state path/to/artifacts
-$ dbt test --models [...] --defer --state path/to/artifacts
+dbt run --models [...] --defer --state path/to/artifacts
+dbt test --models [...] --defer --state path/to/artifacts
```
@@ -101,7 +101,7 @@ I want to test my changes. Nothing exists in my development schema, `dev_alice`.
```shell
-$ dbt run --select model_b
+dbt run --select "model_b"
```
@@ -128,7 +128,7 @@ Unless I had previously run `model_a` into this development environment, `dev_al
```shell
-$ dbt run --select model_b --defer --state prod-run-artifacts
+dbt run --select "model_b" --defer --state prod-run-artifacts
```
@@ -186,7 +186,7 @@ models:
```shell
-dbt test --select model_b
+dbt test --select "model_b"
```
@@ -211,7 +211,7 @@ The `relationships` test requires both `model_a` and `model_b`. Because I did no
```shell
-dbt test --select model_b --defer --state prod-run-artifacts
+dbt test --select "model_b" --defer --state prod-run-artifacts
```
diff --git a/website/docs/reference/node-selection/exclude.md b/website/docs/reference/node-selection/exclude.md
index 9ad4bd1cc0e..d2c140d1bb5 100644
--- a/website/docs/reference/node-selection/exclude.md
+++ b/website/docs/reference/node-selection/exclude.md
@@ -7,19 +7,19 @@ sidebar_label: "Exclude"
dbt provides an `--exclude` flag with the same semantics as `--select`. Models specified with the `--exclude` flag will be removed from the set of models selected with `--select`.
```bash
-$ dbt run --select my_package.*+ --exclude my_package.a_big_model+ # select all models in my_package and their children except a_big_model and its children
+dbt run --select "my_package".*+ --exclude "my_package.a_big_model+" # select all models in my_package and their children except a_big_model and its children
```
Exclude a specific resource by its name or lineage:
```bash
# test
-$ dbt test --exclude not_null_orders_order_id # test all models except the not_null_orders_order_id test
-$ dbt test --exclude orders # test all models except tests associated with the orders model
+dbt test --exclude "not_null_orders_order_id" # test all models except the not_null_orders_order_id test
+dbt test --exclude "orders" # test all models except tests associated with the orders model
# seed
-$ dbt seed --exclude account_parent_mappings # load all seeds except account_parent_mappings
+dbt seed --exclude "account_parent_mappings" # load all seeds except account_parent_mappings
# snapshot
-$ dbt snapshot --exclude snap_order_statuses # execute all snapshots except snap_order_statuses
+dbt snapshot --exclude "snap_order_statuses" # execute all snapshots except snap_order_statuses
```
diff --git a/website/docs/reference/node-selection/graph-operators.md b/website/docs/reference/node-selection/graph-operators.md
index 4fdc2f10628..8cba43e1b52 100644
--- a/website/docs/reference/node-selection/graph-operators.md
+++ b/website/docs/reference/node-selection/graph-operators.md
@@ -7,9 +7,9 @@ If placed at the front of the model selector, `+` will select all parents of the
```bash
- $ dbt run --select my_model+ # select my_model and all children
- $ dbt run --select +my_model # select my_model and all parents
- $ dbt run --select +my_model+ # select my_model, and all of its parents and children
+dbt run --select "my_model+" # select my_model and all children
+dbt run --select "+my_model" # select my_model and all parents
+dbt run --select "+my_model+" # select my_model, and all of its parents and children
```
@@ -20,9 +20,9 @@ to step through.
```bash
- $ dbt run --select my_model+1 # select my_model and its first-degree children
- $ dbt run --select 2+my_model # select my_model, its first-degree parents, and its second-degree parents ("grandparents")
- $ dbt run --select 3+my_model+4 # select my_model, its parents up to the 3rd degree, and its children down to the 4th degree
+dbt run --select "my_model+1" # select my_model and its first-degree children
+dbt run --select "2+my_model" # select my_model, its first-degree parents, and its second-degree parents ("grandparents")
+dbt run --select "3+my_model+4" # select my_model, its parents up to the 3rd degree, and its children down to the 4th degree
```
@@ -32,5 +32,5 @@ The `@` operator is similar to `+`, but will also include _the parents of the ch
@@ -96,10 +96,10 @@ dbt run --select some_model
The `fqn` method is used to select nodes based off their "fully qualified names" (FQN) within the dbt graph. The default output of [`dbt list`](/reference/commands/list) is a listing of FQN.
-```
-dbt run --select fqn:some_model
-dbt run --select fqn:your_project.some_model
-dbt run --select fqn:some_package.some_other_model
+```bash
+dbt run --select "fqn:some_model"
+dbt run --select "fqn:your_project.some_model"
+dbt run --select "fqn:some_package.some_other_model"
```
### The "package" method
@@ -111,10 +111,10 @@ selectors unambiguous.
```bash
# These three selectors are equivalent
- dbt run --select package:snowplow
- dbt run --select snowplow
- dbt run --select snowplow.*
- ```
+ dbt run --select "package:snowplow"
+ dbt run --select "snowplow"
+ dbt run --select "snowplow.*"
+```
### The "config" method
@@ -124,10 +124,10 @@ The `config` method is used to select models that match a specified [node config
```bash
- $ dbt run --select config.materialized:incremental # run all models that are materialized incrementally
- $ dbt run --select config.schema:audit # run all models that are created in the `audit` schema
- $ dbt run --select config.cluster_by:geo_country # run all models clustered by `geo_country`
- ```
+dbt run --select "config.materialized:incremental" # run all models that are materialized incrementally
+dbt run --select "config.schema:audit" # run all models that are created in the `audit` schema
+dbt run --select "config.cluster_by:geo_country" # run all models clustered by `geo_country`
+```
@@ -135,7 +135,8 @@ The `config` method is used to select models that match a specified [node config
While most config values are strings, you can also use the `config` method to match boolean configs, dictionary keys, and values in lists.
For example, given a model with the following configurations:
-```
+
+```bash
{{ config(
materialized = 'incremental',
unique_key = ['column_a', 'column_b'],
@@ -148,10 +149,10 @@ select ...
You can select using any of the following:
```bash
-$ dbt ls -s config.materialized:incremental
-$ dbt ls -s config.unique_key:column_a
-$ dbt ls -s config.grants.select:reporter
-$ dbt ls -s config.transient:true
+dbt ls -s config.materialized:incremental
+dbt ls -s config.unique_key:column_a
+dbt ls -s config.grants.select:reporter
+dbt ls -s config.transient:true
```
@@ -162,10 +163,10 @@ The `test_type` method is used to select tests based on their type, `singular` o
- ```bash
- $ dbt test --select test_type:generic # run all generic tests
- $ dbt test --select test_type:singular # run all singular tests
- ```
+```bash
+dbt test --select "test_type:generic" # run all generic tests
+dbt test --select "test_type:singular" # run all singular tests
+```
### The "test_name" method
@@ -176,10 +177,10 @@ that defines it. For more information about how generic tests are defined, read
```bash
- $ dbt test --select test_name:unique # run all instances of the `unique` test
- $ dbt test --select test_name:equality # run all instances of the `dbt_utils.equality` test
- $ dbt test --select test_name:range_min_max # run all instances of a custom schema test defined in the local project, `range_min_max`
- ```
+dbt test --select "test_name:unique" # run all instances of the `unique` test
+dbt test --select "test_name:equality" # run all instances of the `dbt_utils.equality` test
+dbt test --select "test_name:range_min_max" # run all instances of a custom schema test defined in the local project, `range_min_max`
+```
### The "state" method
@@ -204,9 +205,9 @@ The `state` method is used to select nodes by comparing them against a previous
```bash
- $ dbt test --select state:new # run all tests on new models + and new tests on old models
- $ dbt run --select state:modified # run all models that have been modified
- $ dbt ls --select state:modified # list all modified nodes (not just models)
+dbt test --select "state:new " # run all tests on new models + and new tests on old models
+dbt run --select "state:modified" # run all models that have been modified
+dbt ls --select "state:modified" # list all modified nodes (not just models)
```
@@ -236,18 +237,18 @@ The `exposure` method is used to select parent resources of a specified [exposur
```bash
- $ dbt run --select +exposure:weekly_kpis # run all models that feed into the weekly_kpis exposure
- $ dbt test --select +exposure:* # test all resources upstream of all exposures
- $ dbt ls --select +exposure:* --resource-type source # list all sources upstream of all exposures
- ```
+dbt run --select "+exposure:weekly_kpis" # run all models that feed into the weekly_kpis exposure
+dbt test --select "+exposure:*" # test all resources upstream of all exposures
+dbt ls --select "+exposure:*" --resource-type source # list all sources upstream of all exposures
+```
### The "metric" method
The `metric` method is used to select parent resources of a specified [metric](/docs/build/metrics). Use in conjunction with the `+` operator.
```bash
-$ dbt build --select +metric:weekly_active_users # build all resources upstream of weekly_active_users metric
-$ dbt ls --select +metric:* --resource-type source # list all source tables upstream of all metrics
+dbt build --select "+metric:weekly_active_users" # build all resources upstream of weekly_active_users metric
+dbt ls --select "+metric:*" --resource-type source # list all source tables upstream of all metrics
```
### The "result" method
@@ -255,10 +256,10 @@ $ dbt ls --select +metric:* --resource-type source # list all source tables
The `result` method is related to the `state` method described above and can be used to select resources based on their result status from a prior run. Note that one of the dbt commands [`run`, `test`, `build`, `seed`] must have been performed in order to create the result on which a result selector operates. You can use `result` selectors in conjunction with the `+` operator.
```bash
-$ dbt run --select result:error --state path/to/artifacts # run all models that generated errors on the prior invocation of dbt run
-$ dbt test --select result:fail --state path/to/artifacts # run all tests that failed on the prior invocation of dbt test
-$ dbt build --select 1+result:fail --state path/to/artifacts # run all the models associated with failed tests from the prior invocation of dbt build
-$ dbt seed --select result:error --state path/to/artifacts # run all seeds that generated errors on the prior invocation of dbt seed.
+dbt run --select "result:error" --state path/to/artifacts # run all models that generated errors on the prior invocation of dbt run
+dbt test --select "result:fail" --state path/to/artifacts # run all tests that failed on the prior invocation of dbt test
+dbt build --select "1+result:fail" --state path/to/artifacts # run all the models associated with failed tests from the prior invocation of dbt build
+dbt seed --select "result:error" --state path/to/artifacts # run all seeds that generated errors on the prior invocation of dbt seed.
```
### The "source_status" method
@@ -276,8 +277,8 @@ After issuing one of the above commands, you can reference the source freshness
```bash
# You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag.
-$ dbt source freshness # must be run again to compare current to previous state
-$ dbt build --select source_status:fresher+ --state path/to/prod/artifacts
+dbt source freshness # must be run again to compare current to previous state
+dbt build --select "source_status:fresher+" --state path/to/prod/artifacts
```
@@ -286,8 +287,8 @@ $ dbt build --select source_status:fresher+ --state path/to/prod/artifacts
```bash
# You can also set the DBT_STATE environment variable instead of the --state flag.
-$ dbt source freshness # must be run again to compare current to previous state
-$ dbt build --select source_status:fresher+ --state path/to/prod/artifacts
+dbt source freshness # must be run again to compare current to previous state
+dbt build --select "source_status:fresher+" --state path/to/prod/artifacts
```
@@ -305,9 +306,9 @@ Supported in v1.5 or newer.
The `group` method is used to select models defined within a [group](/reference/resource-configs/group).
- ```bash
- dbt run --select group:finance # run all models that belong to the finance group.
- ```
+```bash
+dbt run --select "group:finance" # run all models that belong to the finance group.
+```
@@ -324,9 +325,9 @@ Supported in v1.5 or newer.
The `access` method selects models based on their [access](/reference/resource-configs/access) property.
```bash
-dbt list --select access:public # list all public models
-dbt list --select access:private # list all private models
-dbt list --select access:protected # list all protected models
+dbt list --select "access:public" # list all public models
+dbt list --select "access:private" # list all private models
+dbt list --select "access:protected" # list all protected models
```
@@ -344,11 +345,11 @@ Supported in v1.5 or newer.
The `version` method selects [versioned models](/docs/collaborate/govern/model-versions) based on their [version identifier](/reference/resource-properties/versions) and [latest version](/reference/resource-properties/latest_version).
```bash
-dbt list --select version:latest # only 'latest' versions
-dbt list --select version:prerelease # versions newer than the 'latest' version
+dbt list --select "version:latest" # only 'latest' versions
+dbt list --select "version:prerelease" # versions newer than the 'latest' version
dbt list --select version:old # versions older than the 'latest' version
-dbt list --select version:none # models that are *not* versioned
+dbt list --select "version:none" # models that are *not* versioned
```
- - * To authenticate, you need to [generate a service token](/docs/dbt-cloud-apis/service-tokens) with Semantic Layer Only and Metadata Only permissions. - * Refer to the [SQL query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the APIs. - -- To learn more about the sophisticated integrations that connect to the dbt Semantic Layer, refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more info. +
❌| ✅ | ✅ | Proprietary, Cloud (Team & Enterprise) | -| **Semantic Layer API** | The interfaces that allow users to submit metric queries include the MetricFlow CLI and JDBC API. They also serve as the foundation for building first-class integrations with various tools. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise)| +| **Semantic Layer APIs** | The interfaces that allow users to submit metric queries using the GraphQL and JDBC APIs. They also serve as the foundation for building first-class integrations with various tools. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise)| ## Related questions diff --git a/website/docs/docs/use-dbt-semantic-layer/tableau.md b/website/docs/docs/use-dbt-semantic-layer/tableau.md new file mode 100644 index 00000000000..c505ea323f4 --- /dev/null +++ b/website/docs/docs/use-dbt-semantic-layer/tableau.md @@ -0,0 +1,67 @@ +--- +title: "Tableau (beta)" +description: "Use Tableau worksheets to query the dbt Semantic Layer and produce dashboards with trusted date." +tags: [Semantic Layer] +sidebar_label: "Tableau (beta)" +--- + +:::info Beta functionality +The Tableau integration with the dbt Semantic Layer is a [beta feature](/docs/dbt-versions/product-lifecycles#dbt-cloud). +::: + + +The Tableau integration allows you to use worksheets to query the Semantic Layer directly and produce your dashboards with trusted data. This integration provides a live connection to the dbt Semantic Layer through Tableau Desktop. + +## Prerequisites + +1. You must have [Tableau Desktop](https://www.tableau.com/en-gb/products/desktop) installed +2. Authenticate with either Tableau Server or Tableau Cloud +3. You need your dbt Cloud host, [Environment ID](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) and [service token](/docs/dbt-cloud-apis/service-tokens) to log in. This account should be set up with the dbt Semantic Layer. +4. You must a dbt Cloud Team or Enterprise [account](https://www.getdbt.com/pricing) and multi-tenant [deployment](/docs/cloud/about-cloud/regions-ip-addresses). (Single-Tenant coming soon) + + +## Installing + +1. Download our [connector file](https://github.com/dbt-labs/semantic-layer-tableau-connector/releases/download/v1.0.0/dbt_semantic_layer.taco) locally and add it to your default folder: + - Windows: `C:\Users\\[Windows User]\Documents\My Tableau Repository\Connectors` + - Mac: `/Users/[user]/Documents/My Tableau Repository/Connectors` + - Linux: `/opt/tableau/connectors` +2. Install the [JDBC driver](/docs/dbt-cloud-apis/sl-jdbc) to the folder based on your operating system: + - Windows: `C:\Program Files\Tableau\Drivers` + - Mac: `~/Library/Tableau/Drivers` + - Linux: ` /opt/tableau/tableau_driver/jdbc` +3. Open Tableau Desktop and find the **dbt Semantic Layer by dbt Labs** connector on the left-hand side. +4. Connect with your Host, Environment ID, and service token information that's provided to you in your dbt Cloud Semantic Layer configuration. + + +## Using the integration + +Once you authenticate, the system will direct you to the data source page with all the metrics and dimensions configured in your Semantic Layer. + +- From there, go directly to a worksheet in the bottom left hand corner. +- Then, you'll find all the metrics and dimensions that are available to query on the left-hand side of your window. + +Visit the [Tableau documentation](https://help.tableau.com/current/pro/desktop/en-us/gettingstarted_overview.htm) to learn more about how to use Tableau worksheets and dashboards. + +## Things to note + +- All metrics use the "SUM" aggregation type, and this can't be altered. The dbt Semantic Layer controls the aggregation type and it is intentionally fixed. Keep in mind that the underlying aggregation in the dbt Semantic Layer might not be "SUM" (even though "SUM" is Tableau's default). +- Tableau surfaces all metrics and dimensions from the dbt Semantic Layer on the left-hand side. Note, that not all metrics and dimensions can be combined with one another. You will receive an error message if a particular dimension cannot be sliced with a metric (or vice versa). + - To display available metrics and dimensions, dbt Semantic Layer returns metadata for a fake table with the dimensions and metrics as 'columns' on this table. Because of this, you can't actually query this table for previews or extracts. + - Since this is treated as a table, dbt Semantic Layer can't dynamically change what is available. This means we display _all_ available metrics and dimensions even if a particular metric and dimension combination isn't available. + +- Certain Table calculations like "Totals" and "Percent Of" may not be accurate when using metrics aggregated in a non-additive way (such as count distinct) + +## Unsupported functionality + +The following Tableau features aren't supported at this time, however, the dbt Semantic Layer may support some of this functionality in a future release: + +- Updating the data source page +- Using "Extract" mode to view yur data +- Unioning Tables +- Writing Custom SQL +- Table Extensions +- Cross Database Joins +- All functions in Analysis --> Create Calculated Field +- Filtering on a Date Part time dimension for a Cumulative metric type +- Changing your date dimension to use "Week Number" diff --git a/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md b/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md index 34c0e813725..801227924dd 100644 --- a/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md +++ b/website/docs/guides/best-practices/how-we-build-our-metrics/semantic-layer-2-setup.md @@ -13,7 +13,7 @@ git clone git@github.com:dbt-labs/jaffle-sl-template.git cd path/to/project ``` -Next before we start writing code, we'll need to install the MetricFlow CLI as an extension of a dbt adapter from PyPI. The MetricFlow CLI is compatible with Python versions 3.8 through 3.11. +Next, before you start writing code, you need to install MetricFlow as an extension of a dbt adapter from PyPI (dbt Core users only). The MetricFlow is compatible with Python versions 3.8 through 3.11. We'll use pip to install MetricFlow and our dbt adapter: @@ -33,7 +33,7 @@ Lastly, to get to the pre-Semantic Layer starting state, checkout the `start-her git checkout start-here ``` -For more information you can [look at the docs](/docs/build/metricflow-cli) or checkout a [Quickstart](https://docs.getdbt.com/quickstarts) to get more familiar with setting up a dbt project. +For more information, refer to the [MetricFlow commands](/docs/build/metricflow-commands) or a [quickstart](/quickstarts) to get more familiar with setting up a dbt project. ## Basic commands diff --git a/website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md b/website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md new file mode 100644 index 00000000000..ba1660a8d82 --- /dev/null +++ b/website/docs/guides/best-practices/how-we-mesh/mesh-1-intro.md @@ -0,0 +1,39 @@ +--- +title: "Intro to dbt Mesh" +description: Getting started with dbt Mesh patterns +hoverSnippet: Learn how to get started with dbt Mesh +--- + +## What is dbt Mesh? + +Organizations of all sizes rely upon dbt to manage their data transformations, from small startups to large enterprises. At scale, it can be challenging to coordinate all the organizational and technical requirements demanded by your stakeholders within the scope of a single dbt project. To date, there also hasn't been a first-class way to effectively manage the dependencies, governance, and workflows between multiple dbt projects. + +Regardless of your organization's size and complexity, dbt should empower data teams to work independently and collaboratively; sharing data, code, and best practices without sacrificing security or autonomy. dbt Mesh provides the tooling for teams to finally achieve this. + +dbt Mesh is not a single product: it is a pattern enabled by a convergence of several features in dbt: + +- **[Cross-project references](/docs/collaborate/govern/project-dependencies#how-to-use-ref)** - this is the foundational feature that enables the multi-project deployments. `{{ ref() }}`s now work across dbt Cloud projects on Enterprise plans. +- **[dbt Explorer](/docs/collaborate/explore-projects)** - dbt Cloud's metadata-powered documentation platform, complete with full, cross-project lineage. +- **Governance** - dbt's new governance features allow you to manage access to your dbt models both within and across projects. + - **[Groups](/docs/collaborate/govern/model-access#groups)** - groups allow you to assign models to subsets within a project. + - **[Access](/docs/collaborate/govern/model-access#access-modifiers)** - access configs allow you to control who can reference models. +- **[Model Versions](/docs/collaborate/govern/model-versions)** - when coordinating across projects and teams, we recommend treating your data models as stable APIs. Model versioning is the mechanism to allow graceful adoption and deprecation of models as they evolve. +- **[Model Contracts](/docs/collaborate/govern/model-contracts)** - data contracts set explicit expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. + +## Who is dbt Mesh for? + +The multi-project architecture helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. If you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring this paradigm: + +- The **number of models** in your project is degrading performance and slowing down development. +- Teams have developed **separate workflows** and need to decouple development from each other. +- **Security and governance** requirements are increasing and would benefit from increased isolation. + +dbt Cloud is designed to coordinate the features above and simplify the complexity to solve for these problems. + +If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can _incrementally_ adopt the features in this guide as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow. + +## Learning goals + +- Understand the **purpose and tradeoffs** of building a multi-project architecture. +- Develop an intuition for various **dbt Mesh patterns** and how to design a multi-project architecture for your organization. +- Establish recommended steps to **incrementally adopt** these patterns in your dbt implementation. diff --git a/website/docs/guides/best-practices/how-we-mesh/mesh-2-structures.md b/website/docs/guides/best-practices/how-we-mesh/mesh-2-structures.md new file mode 100644 index 00000000000..937515954af --- /dev/null +++ b/website/docs/guides/best-practices/how-we-mesh/mesh-2-structures.md @@ -0,0 +1,52 @@ +--- +title: Deciding how to structure your dbt Mesh +description: Getting started with dbt Mesh patterns +hoverSnippet: Learn how to get started with dbt Mesh +--- +## Exploring mesh patterns + +When adopting a multi-project architecture, where do you draw the lines between projects? + +How should you organize data workflows in a world where instead of having a single dbt DAG, you have multiple projects speaking to each other, each comprised of their own DAG? + +Adopting the dbt Mesh pattern is not a one-size-fits-all process. In fact, it's the opposite! It's about customizing your project structure to fit _your_ team and _your_ data. Now you can mold your organizational knowledge graph to your organizational people graph, bringing people and data closer together rather than compromising one for the other. + +While there is not a single best way to implement this pattern, there are some common decision points that will be helpful for you to consider. + +At a high level, you’ll need to decide: + +- Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project? +- How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project? + +## Define your project interfaces by splitting your DAG + +The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns. + +### Vertical splits + +Vertical splits separate out layers of transformation in DAG order. Let's look at some examples. + +- **Splitting up staging and mart layers** to create a more tightly-controlled, shared set of components that other projects build on but can't edit. +- **Isolating earlier models for security and governance requirements** to separate out and mask PII data so that downstream consumers can't access it is a common use case for a vertical split. +- **Protecting complex or expensive data** to isolate large or complex models that are expensive to run so that they are safe from accidental selection, independently deployable, and easier to debug when they have issues. + +### Horizontal splits + +Horizontal splits separate your DAG based on source or domain. These splits are often based around the shape and size of the data and how it's used. Let's consider some possibilities for horizontal splitting. + +- **Team consumption patterns.** For example, splitting out the marketing team's data flow into a separate project. +- **Data from different sources.** For example, clickstream event data and transactional ecommerce data may need to be modeled independently of each other. +- **Team workflows.** For example, if two embedded groups operate at different paces, you may want to split the projects up so they can move independently. + +### Combining these strategies + +- **These are not either/or techniques**. You should consider both types of splits, and combine them in any way that makes sense for your organization. +- **Pick one type of split and focus on that first**. If you have a hub-and-spoke team topology for example, handle breaking out the central platform project before you split the remainder into domains. Then if you need to break those domains up horizontally you can focus on that after the fact. +- **DRY applies to underlying data, not just code.** Regardless of your strategy, you should not be sourcing the same rows and columns into multiple nodes. When working within a mesh pattern it becomes increasingly important that we don't duplicate logic or data. + +## Determine your git strategy + +A multi-project architecture can exist in a single repo (monorepo) or as multiple projects, with each one being in their own repository (multi-repo). + +- If you're a **smaller team** looking primarily to speed up and simplify development, a **monorepo** is likely the right choice, but can become unwieldy as the number of projects, models and contributors grow. +- If you’re a **larger team with multiple groups**, and need to decouple projects for security and enablement of different development styles and rhythms, a **multi-repo setup** is your best bet. diff --git a/website/docs/guides/best-practices/how-we-mesh/mesh-3-implementation.md b/website/docs/guides/best-practices/how-we-mesh/mesh-3-implementation.md new file mode 100644 index 00000000000..cfbbc7a1f28 --- /dev/null +++ b/website/docs/guides/best-practices/how-we-mesh/mesh-3-implementation.md @@ -0,0 +1,130 @@ +--- +title: "Implementing your mesh plan" +description: Getting started with dbt Mesh patterns +hoverSnippet: Learn how to get started with dbt Mesh +--- + +As mentioned before, the key decision in migrating to a multi-project architecture is understanding how your project is already being grouped, built, and deployed. We can use this information to inform our decision to split our project apart. + +- **Examine your jobs** - which sets of models are most often built together? +- **Look at your lineage graph** - how are models connected? +- **Look at your selectors** defined in `selectors.yml` - how do people already define resource groups? +- **Talk to teams** about what sort of separation naturally exists right now. + - Are there various domains people are focused on? + - Are there various sizes, shapes, and sources of data that get handled separately (such as click event data)? + - Are there people focused on separate levels of transformation, such as landing and staging data or building marts? + +## Add groups and access + +Once you have a sense of some initial groupings, you can first implement **group and access permissions** within a single project. + +- First you can create a [group](/docs/build/groups) to define the owner of a set of models. + +```yml +# in models/__groups.yml + +groups: + - name: marketing + owner: + - name: Ben Jaffleck + email: ben.jaffleck@jaffleshop.com +``` + +- Then, we can add models to that group using the `group:` key in the model's YAML entry. + +```yml +# in models/marketing/__models.yml + +models: + - name: fct_marketing_model + group: marketing + - name: stg_marketing_model + group: marketing +``` + +- Once you've added models to the group, you can **add [access](/docs/collaborate/govern/model-access) settings to the models** based on their connections between groups, *opting for the most private access that will maintain current functionality*. This means that any model that has *only* relationships to other models in the same group should be `private` , and any model that has cross-group relationships, or is a terminal node in the group DAG should be `protected` so that other parts of the DAG can continue to reference it. + +```yml +# in models/marketing/__models.yml + +models: + - name: fct_marketing_model + group: marketing + access: protected + - name: stg_marketing_model + group: marketing + access: private +``` + +- **Validate these groups by incrementally migrating your jobs** to execute these groups specifically via selection syntax. We would recommend doing this in parallel to your production jobs until you’re sure about them. This will help you feel out if you’ve drawn the lines in the right place. +- If you find yourself **consistently making changes across multiple groups** when you update logic, that’s a sign that **you may want to rethink your groups**. + +## Split your projects + +1. **Move your grouped models into a subfolder**. This will include any model in the selected group, it's associated YAML entry, as well as its parent or child resources as appropriate depending on where this group sits in your DAG. + 1. Note that just like in your dbt project, circular refereneces are not allowed! Project B cannot have parents and children in Project A, for example. +2. **Create a new `dbt_project.yml` file** in the subdirectory. +3. **Copy any macros** used by the resources you moved. +4. **Create a new `packages.yml` file** in your subdirectory with the packages that are used by the resources you moved. +5. **Update `{{ ref }}` functions** — For any model that has a cross-project dependency (this may be in the files you moved, or in the files that remain in your project): + 1. Update the `{{ ref() }}` function to have two arguments, where the first is the name of the source project and the second is the name of the model: e.g. `{{ ref('jaffle_shop', 'my_upstream_model') }}` + 2. Update the upstream, cross-project parents’ `access` configs to `public` , ensuring any project can safely `{{ ref() }}` those models. + 3. We *highly* recommend adding a [model contract](/docs/collaborate/govern/model-contracts) to the upstream models to ensure the data shape is consistent and reliable for your downstream consumers. +6. **Create a `dependencies.yml` file** ([docs](/docs/collaborate/govern/project-dependencies)) for the downstream project, declaring the upstream project as a dependency. + +```yml + +# in dependencies.yml +projects: + - name: jaffle_shop +``` + +### Best practices + +- When you’ve **confirmed the right groups**, it's time to split your projects. + - **Do *one* group at a time**! + - **Do *not* refactor as you migrate**, however tempting that may be. Focus on getting 1-to-1 parity and log any issues you find in doing the migration for later. Once you’ve fully migrated the project then you can start optimizing it for its new life as part of your mesh. +- Start by splitting your project within the same repository for full git tracking and easy reversion if you need to start from scratch. + + +## Connecting existing projects + +Some organizations may already be coordinating across multiple dbt projects. Most often this is via: + +1. Installing parent projects as dbt packages +2. Using `{{ source() }}` functions to read the outputs of a parent project as inputs to a child project. + +This has a few drawbacks: + +1. If using packages, each project has to include *all* resources from *all* projects in its manifest, slowing down dbt and the development cycle. +2. If using sources, there are breakages in the lineage, as there's no real connection between the parent and child projects. + +The migration steps here are much simpler than splitting up a monolith! + +1. If using the `package` method: + 1. In the parent project: + 1. mark all models being referenced downstream as `public` and add a model contract. + 2. In the child project: + 1. Remove the package entry from `packages.yml` + 2. Add the upstream project to your `dependencies.yml` + 3. Update the `{{ ref() }}` functions to models from the upstream project to include the project name argument. +1. If using `source` method: + 1. In the parent project: + 1. mark all models being imported downstream as `public` and add a model contract. + 2. In the child project: + 1. Add the upstream project to your `dependencies.yml` + 2. Replace the `{{ source() }}` functions with cross project `{{ ref() }}` functions. + 3. Remove the unnecessary `source` definitions. + +## Additional Resources +### Our example projects + +We've provided a set of example projects you can use to explore the topics covered here. We've split our [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) project into 3 separate projects in a multi-repo dbt Mesh. Note that you'll need to leverage dbt Cloud to use multi-project architecture, as cross-project references are powered via dbt Cloud's APIs. + +- **[Platform](https://github.com/dbt-labs/jaffle-shop-mesh-platform)** - containing our centralized staging models. +- **[Marketing](https://github.com/dbt-labs/jaffle-shop-mesh-marketing)** - containing our marketing marts. +- **[Finance](https://github.com/dbt-labs/jaffle-shop-mesh-finance)** - containing our finance marts. + +### dbt-meshify + +We recommend using the `dbt-meshify` [command line tool](
the database | All [supported versions](/docs/dbt-versions/core) | -| [seed](/reference/commands/seed) | Loads CSV files into the database | All [supported versions](/docs/dbt-versions/core) | -| [show](/reference/commands/show) | Preview table rows post-transformation | All [supported versions](/docs/dbt-versions/core) | -| [snapshot](/reference/commands/snapshot) | Executes "snapshot" jobs defined in a project | All [supported versions](/docs/dbt-versions/core) | -| [source](/reference/commands/source) | Provides tools for working with source data (including validating that
sources are "fresh") | All [supported versions](/docs/dbt-versions/core) | -| [test](/reference/commands/test) | Executes tests defined in a project | All [supported versions](/docs/dbt-versions/core) | -| [init](/reference/commands/init) | Initializes a new dbt project (CLI only) | All [supported versions](/docs/dbt-versions/core) | - +
dbt Core | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | +| [compile](/reference/commands/compile) | Compiles (but does not run) the models in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [debug](/reference/commands/debug) | Debugs dbt connections and projects | dbt Core | All [supported versions](/docs/dbt-versions/core) | +| [deps](/reference/commands/deps) | Downloads dependencies for a project | All | All [supported versions](/docs/dbt-versions/core) | +| [docs](/reference/commands/cmd-docs) | Generates documentation for a project | All | All [supported versions](/docs/dbt-versions/core) | +| help | Displays help information for any command | dbt Core
dbt Cloud CLI | All [supported versions](/docs/dbt-versions/core) | +| [list](/reference/commands/list) | Lists resources defined in a dbt project | All | All [supported versions](/docs/dbt-versions/core) | +| [parse](/reference/commands/parse) | Parses a project and writes detailed timing info | All | All [supported versions](/docs/dbt-versions/core) | +| reattach | Reattaches to the most recent invocation to retrieve logs and artifacts. | dbt Cloud CLI | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | +| [retry](/reference/commands/retry) | Retry the last run `dbt` command from the point of failure | All | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | +| [run](/reference/commands/run) | Runs the models in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [run-operation](/reference/commands/run-operation) | Invoke a macro, including running arbitrary maintenance SQL against the database | All | All [supported versions](/docs/dbt-versions/core) | +| [seed](/reference/commands/seed) | Loads CSV files into the database | All | All [supported versions](/docs/dbt-versions/core) | +| [show](/reference/commands/show) | Preview table rows post-transformation | All | All [supported versions](/docs/dbt-versions/core) | +| [snapshot](/reference/commands/snapshot) | Executes "snapshot" jobs defined in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [source](/reference/commands/source) | Provides tools for working with source data (including validating that sources are "fresh") | All | All [supported versions](/docs/dbt-versions/core) | +| [test](/reference/commands/test) | Executes tests defined in a project | All | All [supported versions](/docs/dbt-versions/core) | +| [init](/reference/commands/init) | Initializes a new dbt project | dbt Core | All [supported versions](/docs/dbt-versions/core) |
Install with dbt Core "
icon="materialize"/>
Install with dbt Core "
icon="oracle"/>
diff --git a/website/snippets/_adapters-verified.md b/website/snippets/_adapters-verified.md
index 7caf099b7d1..3cc1e800448 100644
--- a/website/snippets/_adapters-verified.md
+++ b/website/snippets/_adapters-verified.md
@@ -2,61 +2,60 @@
diff --git a/website/snippets/_cloud-cli-flag.md b/website/snippets/_cloud-cli-flag.md new file mode 100644 index 00000000000..523591a438c --- /dev/null +++ b/website/snippets/_cloud-cli-flag.md @@ -0,0 +1,5 @@ +:::info Public preview functionality + +The dbt Cloud CLI is currently in [public preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Share feedback or request features you'd like to see on the [dbt community Slack](https://getdbt.slack.com/archives/C05M77P54FL). + +::: diff --git a/website/snippets/_cloud-environments-info.md b/website/snippets/_cloud-environments-info.md index 5388379dc34..b0c8e86a2b5 100644 --- a/website/snippets/_cloud-environments-info.md +++ b/website/snippets/_cloud-environments-info.md @@ -3,17 +3,17 @@ In dbt Cloud, there are two types of environments: - Deployment environment — Determines the settings used when jobs created within that environment are executed. -- Development environment — Determines the settings used in the dbt Cloud IDE for that particular dbt Cloud project. +- Development environment — Determines the settings used in the dbt Cloud IDE or dbt Cloud CLI, for that particular project. Each dbt Cloud project can only have a single development environment but can have any number of deployment environments. | | Development Environments | Deployment Environments | | --- | --- | --- | -| Determines settings for | dbt Cloud IDE | dbt Cloud Job runs | +| Determines settings for | dbt Cloud IDE or dbt Cloud CLI | dbt Cloud Job runs | | How many can I have in my project? | 1 | Any number | :::note -For users familiar with development on the CLI, each environment is roughly analogous to an entry in your `profiles.yml` file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments [here](/docs/core/dbt-core-environments). +For users familiar with development on dbt Core, each environment is roughly analogous to an entry in your `profiles.yml` file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments [here](/docs/core/dbt-core-environments). ::: ## Common environment settings @@ -38,7 +38,7 @@ Both development and deployment environments have a section called **General Set By default, all environments will use the default branch in your repository (usually the `main` branch) when accessing your dbt code. This is overridable within each dbt Cloud Environment using the **Default to a custom branch** option. This setting have will have slightly different behavior depending on the environment type: -- **Development**: determines which branch in the dbt Cloud IDE developers create branches from and open PRs against +- **Development**: determines which branch in the dbt Cloud IDE or dbt Cloud CLI developers create branches from and open PRs against. - **Deployment:** determines the branch is cloned during job executions for each environment. For more info, check out this [FAQ page on this topic](/faqs/Environments/custom-branch-settings)! @@ -59,7 +59,7 @@ Something to note, Extended Attributes doesn't mask secret values. We recommend
-If you're developing in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [orchestrating job runs](/docs/deploy/deployments), Extended Attributes parses through the provided YAML and extracts the `profiles.yml` attributes. For each individual attribute: +If you're developing in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), or [orchestrating job runs](/docs/deploy/deployments), Extended Attributes parses through the provided YAML and extracts the `profiles.yml` attributes. For each individual attribute: - If the attribute exists in another source (such as your project settings), it will replace its value (like environment-level values) in the profile. It also overrides any custom environment variables. diff --git a/website/snippets/_enterprise-permissions-table.md b/website/snippets/_enterprise-permissions-table.md index c89835c8437..3eb313e0f5b 100644 --- a/website/snippets/_enterprise-permissions-table.md +++ b/website/snippets/_enterprise-permissions-table.md @@ -7,7 +7,7 @@ Key: Permissions: * Account-level permissions — Permissions related to management of the dbt Cloud account. For example, billing and account settings. -* Project-level permissions — Permissions related to the projects in dbt Cloud. For example, repos and access to the IDE. +* Project-level permissions — Permissions related to the projects in dbt Cloud. For example, repos and access to the IDE or dbt Cloud CLI. ### Account roles Account roles enable you to manage the dbt Cloud account and manage the account settings (for example, generating service tokens, inviting users, configuring SSO). They also provide project-level permissions. The **Account Admin** role is the highest level of access you can assign. @@ -38,7 +38,7 @@ Account roles enable you to manage the dbt Cloud account and manage the account | Credentials | W | | W | | R | | Custom env. variables | W | | W | | R | | dbt adapters | W | | W | | R | -| Develop (IDE) | W | | W | | | +| Develop (IDE or dbt Cloud CLI) | W | | W | | | | Environments | W | | W | | R | | Jobs | W | | W | | R | | Metadata | R | | R | | R | @@ -52,7 +52,7 @@ Account roles enable you to manage the dbt Cloud account and manage the account ### Project role permissions -The project roles enable you to work within the projects in various capacities. They primarily provide access to project-level permissions such as repos and the IDE, but may also provide some account-level permissions. +The project roles enable you to work within the projects in various capacities. They primarily provide access to project-level permissions such as repos and the IDE or dbt Cloud CLI, but may also provide some account-level permissions. #### Account permissions for project roles @@ -78,7 +78,7 @@ The project roles enable you to work within the projects in various capacities. | Credentials | W | W | W | W | R | W | | | | R | R | | | Custom env. variables | W | W | W | W | W | W | R | | | R | W | | | dbt adapters | W | W | W | W | R | W | | | | R | R | | -| Develop (IDE) | W | W | | W | | | | | | | | | +| Develop (IDE or dbt Cloud CLI) | W | W | | W | | | | | | | | | | Environments | W | R | R | R | R | W | R | | | R | R | | | Jobs | W | R | R | W | R | W | R | | | R | R | | | Metadata | R | R | R | R | R | R | R | R | | R | R | | diff --git a/website/snippets/_new-sl-setup.md b/website/snippets/_new-sl-setup.md index b802db9c5ae..ad248bc3ca9 100644 --- a/website/snippets/_new-sl-setup.md +++ b/website/snippets/_new-sl-setup.md @@ -1,13 +1,13 @@ You can set up the dbt Semantic Layer in dbt Cloud at the environment and project level. Before you begin: -- You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America. +- You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment. Single-tenant coming soon. - You must be part of the Owner group, and have the correct [license](/docs/cloud/manage-access/seats-and-users) and [permissions](/docs/cloud/manage-access/self-service-permissions) to configure the Semantic Layer: * Enterprise plan — Developer license with Account Admin permissions. Or Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. * Team plan — Owner with a Developer license. - You must have a successful run in your new environment. :::tip -If you're using the legacy Semantic Layer, we **highly** recommend you [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to dbt v1.6 or higher to use the new dbt Semantic Layer. Refer to the dedicated [migration guide](/guides/migration/sl-migration) for more info. +If you're using the legacy Semantic Layer, dbt Labs strongly recommends that you [upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to dbt version 1.6 or newer to use the latest dbt Semantic Layer. Refer to the dedicated [migration guide](/guides/migration/sl-migration) for details. ::: 1. In dbt Cloud, create a new [deployment environment](/docs/deploy/deploy-environments#create-a-deployment-environment) or use an existing environment on dbt 1.6 or higher. diff --git a/website/snippets/_sl-connect-and-query-api.md b/website/snippets/_sl-connect-and-query-api.md new file mode 100644 index 00000000000..429f41c3bf6 --- /dev/null +++ b/website/snippets/_sl-connect-and-query-api.md @@ -0,0 +1,10 @@ +You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer. + +You must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment. Single-tenant coming soon. + +- To learn how to use the JDBC or GraphQL API and what tools you can query it with, refer to [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview). + + * To authenticate, you need to [generate a service token](/docs/dbt-cloud-apis/service-tokens) with Semantic Layer Only and Metadata Only permissions. + * Refer to the [SQL query syntax](/docs/dbt-cloud-apis/sl-jdbc#querying-the-api-for-metric-metadata) to query metrics using the API. + +- To learn more about the sophisticated integrations that connect to the dbt Semantic Layer, refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more info. diff --git a/website/snippets/_sl-create-semanticmodel.md b/website/snippets/_sl-create-semanticmodel.md index bc4276efcb6..6e0376ab10b 100644 --- a/website/snippets/_sl-create-semanticmodel.md +++ b/website/snippets/_sl-create-semanticmodel.md @@ -1,4 +1,4 @@ -The following steps will walk you through setting up semantic models, which you can do with the dbt Cloud IDE or the CLI. Semantic models consist of [entities](/docs/build/entities), [dimensions](/docs/build/dimensions), and [measures](/docs/build/measures). +The following steps describe how to set up semantic models. Semantic models consist of [entities](/docs/build/entities), [dimensions](/docs/build/dimensions), and [measures](/docs/build/measures). We highly recommend you read the overview of what a [semantic model](/docs/build/semantic-models) is before getting started. If you're working in the [Jaffle shop example](https://github.com/dbt-labs/jaffle-sl-template), delete the `orders.yml` config or delete the .yml extension so it's ignored during parsing. **We'll be rebuilding it step by step in this example.** diff --git a/website/snippets/_sl-define-metrics.md b/website/snippets/_sl-define-metrics.md index 29af3f5b7c3..3734e819c1b 100644 --- a/website/snippets/_sl-define-metrics.md +++ b/website/snippets/_sl-define-metrics.md @@ -1,4 +1,4 @@ -Now that you've created your first semantic model, it's time to define your first metric! You can define metrics with the dbt Cloud IDE or CLI. +Now that you've created your first semantic model, it's time to define your first metric! You can define metrics with the dbt Cloud IDE or command line. MetricFlow supports different metric types like [simple](/docs/build/simple), [ratio](/docs/build/ratio), [cumulative](/docs/build/cumulative), and [derived](/docs/build/derived). It's recommended that you read the [metrics overview docs](/docs/build/metrics-overview) before getting started. diff --git a/website/snippets/_sl-install-metricflow.md b/website/snippets/_sl-install-metricflow.md deleted file mode 100644 index 73e60d34e85..00000000000 --- a/website/snippets/_sl-install-metricflow.md +++ /dev/null @@ -1,8 +0,0 @@ -Install the [MetricFlow CLI](/docs/build/metricflow-cli) as an extension of a dbt adapter from PyPI. The MetricFlow CLI is compatible with Python versions 3.8, 3.9, 3.10 and 3.11 - -Use pip install `metricflow` and your [dbt adapter](/docs/supported-data-platforms): - -- Create or activate your virtual environment. `python -m venv venv` or `source your-venv/bin/activate` -- Run `pip install "dbt-metricflow[your_adapter_name]"` - - You must specify `[your_adapter_name]`. - - For example, run `pip install "dbt-metricflow[snowflake]"` if you use a Snowflake adapter. diff --git a/website/snippets/_sl-partner-links.md b/website/snippets/_sl-partner-links.md index fa8146c6418..59e5b54b7c0 100644 --- a/website/snippets/_sl-partner-links.md +++ b/website/snippets/_sl-partner-links.md @@ -2,26 +2,61 @@ The following tools integrate with the dbt Semantic Layer:
-Before you connect to these tools, you'll need to first [set up the dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl) and [generate a service token](/docs/dbt-cloud-apis/service-tokens) to create **Semantic Layer Only** and **Metadata Only** permissions. - +Before you connect to these tools, you'll need to first [set up the dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl) and [generate a service token](/docs/dbt-cloud-apis/service-tokens) to create **Semantic Layer Only** and **Metadata Only** permissions. diff --git a/website/snippets/_sl-plan-info.md b/website/snippets/_sl-plan-info.md index 5fba18de6bb..083ab2209bc 100644 --- a/website/snippets/_sl-plan-info.md +++ b/website/snippets/_sl-plan-info.md @@ -1,2 +1,2 @@ -To define and query metrics with the {props.product}, you must be on a {props.plan} multi-tenant plan, {props.instance} (Additional region support coming soon).
The re-released dbt Semantic Layer is available on dbt v1.6 or higher. dbt Core users can use the MetricFlow CLI to define metrics in their local project, but won't be able to dynamically query them with integrated tools.
+To define and query metrics with the {props.product}, you must be on a {props.plan} multi-tenant plan .
diff --git a/website/snippets/_sl-run-prod-job.md b/website/snippets/_sl-run-prod-job.md new file mode 100644 index 00000000000..a637b0b431e --- /dev/null +++ b/website/snippets/_sl-run-prod-job.md @@ -0,0 +1,7 @@ +Once you’ve defined metrics in your dbt project, you can perform a job run in your deployment environment in dbt Cloud to materialize your metrics. The deployment environment is only supported for the dbt Semantic Layer currently. + +1. Select **Deploy** from the top navigation bar. +2. Select **Jobs** to rerun the job with the most recent code in the deployment environment. +3. Your metric should appear as a red node in the dbt Cloud IDE and dbt directed acyclic graphs (DAG). + +
-- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the revamped dbt Semantic Layer. - + - To access the API and query metrics in downstream tools, you must have a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account. dbt Core or Developer accounts can define metrics but won't be able to dynamically query them.
+- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the latest dbt Semantic Layer.