Skip to content

Commit

Permalink
[do not merge] main-sl-cloud-cli (#4191)
Browse files Browse the repository at this point in the history
<p><a
href="https://www.notion.so/dbtlabs/Prod-Docs-Beta-Cloud-CLI-Oct-9th-a389341b7c1f45e2b7c162e4de591d90?d=2b3208f1ff8c49698fde7b3946b86e91">Docs
project</a></p>

this branch and PR addresses docs changes for the cloud-cli beta and SL
GA launch

Closes #4041 #4014 #4040
  • Loading branch information
mirnawong1 authored Oct 17, 2023
2 parents c915480 + c721a7f commit 8ee4a80
Show file tree
Hide file tree
Showing 111 changed files with 2,427 additions and 764 deletions.
4 changes: 4 additions & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,10 @@ exports.versionedPages = [
{
"page": "reference/resource-configs/grants",
"firstVersion": "1.2",
},
{
"page": "docs/build/saved-queries",
"firstVersion": "1.7",
}
]

Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/about-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ pagination_prev: null

dbt compiles and runs your analytics code against your data platform, enabling you and your team to collaborate on a single source of truth for metrics, insights, and business definitions. There are two options for deploying dbt:

**dbt Cloud** runs dbt Core in a hosted (single or multi-tenant) environment with a browser-based interface. The intuitive UI will aid you in setting up the various components. dbt Cloud comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring & alerting, and an integrated developer environment (IDE).
**dbt Cloud** runs dbt Core in a hosted (single or multi-tenant) environment with a browser-based interface. The intuitive user interface aids you in setting up the various components. dbt Cloud comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring, and alerting. It also offers an integrated development environment (IDE) and allows you to develop and run dbt commands from your local command line (CLI) or code editor.

**dbt Core** is an open-source command line tool that can be installed locally in your environment, and communication with databases is facilitated through adapters.

Expand All @@ -21,7 +21,7 @@ To begin configuring dbt now, select the option that is right for you.

<Card
title="dbt Cloud setup"
body="Learn how to connect to a data platform, integrate with secure authentication methods, configure a sync with a git repo, and how to use the IDE."
body="Learn how to connect to a data platform, integrate with secure authentication methods, configure a sync with a git repo, how to use the IDE, and how to install the dbt Cloud CLI."
link="/docs/cloud/about-cloud-setup"
icon="dbt-bit"/>

Expand Down
50 changes: 32 additions & 18 deletions website/docs/docs/build/about-metricflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,36 +8,52 @@ pagination_next: "docs/build/join-logic"
pagination_prev: null
---

This guide introduces MetricFlow's fundamental ideas for new users. MetricFlow, which powers the dbt Semantic Layer, helps you define and manage the logic for your company's metrics. It's an opinionated set of abstractions and helps data consumers retrieve metric datasets from a data platform quickly and efficiently.
This guide introduces MetricFlow's fundamental ideas for people new to this feature. MetricFlow, which powers the dbt Semantic Layer, helps you define and manage the logic for your company's metrics. It's an opinionated set of abstractions and helps data consumers retrieve metric datasets from a data platform quickly and efficiently.

:::info
MetricFlow handles SQL query construction and defines the specification for dbt semantic models and metrics. It allows you to define metrics in your dbt project and query them with [MetricFlow commands](/docs/build/metricflow-commands) whether in dbt Cloud or dbt Core.

MetricFlow is a new way to define metrics and one of the key components of the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl). It handles SQL query construction and defines the specification for dbt semantic models and metrics.
Before you start, consider the following guidelines:

MetricFlow is currently available on dbt v1.6 or higher for all users. dbt Core users can use the MetricFlow CLI to define metrics in their local dbt Core project. However, to experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.
- Define metrics in YAML and query them using these [new metric specifications](https://github.com/dbt-labs/dbt-core/discussions/7456).
- You must be on [dbt version](/docs/dbt-versions/upgrade-core-in-cloud) 1.6 or higher to use MetricFlow.
- Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (dbt Core only), or Redshift.
- Discover insights and query your metrics using the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and its diverse range of [available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations). You must have a dbt Cloud account on the [Team or Enterprise plan](https://www.getdbt.com/pricing/).

:::
## MetricFlow

There are a few key principles:
MetricFlow is a SQL query generation tool designed to streamline metric creation across different data dimensions for diverse business needs.
- It operates through YAML files, where a semantic graph links language to data. This graph comprises [semantic models](/docs/build/semantic-models) (data entry points) and [metrics](/docs/build/metrics-overview) (functions for creating quantitative indicators).
- MetricFlow is a [BSL package](https://github.com/dbt-labs/metricflow) with code source available, and compatible with dbt version 1.6 and higher. Data practitioners and enthusiasts are highly encouraged to contribute.
- As a part of the dbt Semantic Layer, MetricFlow empowers organizations to define metrics using YAML abstractions.
- To query metric dimensions, dimension values, and validate configurations, use [MetricFlow commands](/docs/build/metricflow-commands).

- **Flexible, but complete** &mdash; Ability to create any metric on any data model by defining logic in flexible abstractions.
- **Don't Repeat Yourself (DRY)** &mdash; Avoid repetition by allowing metric definitions to be enabled whenever possible.
- **Simple with progressive complexity** &mdash; Make MetricFlow approachable by relying on known concepts and structures in data modeling.
- **Performant and efficient** &mdash; Allow for performance optimizations in centralized data engineering while still enabling distributed definition and ownership of logic.

## MetricFlow
**Note** &mdash; MetricFlow doesn't support dbt [builtin functions or packages](/reference/dbt-jinja-functions/builtins) at this time, however, support is planned for the future.

MetricFlow abides by these principles:

- MetricFlow is a SQL query generation engine that helps you create metrics by constructing appropriate queries for different granularities and dimensions that are useful for various business applications.
- **Flexibility with completeness**: Define metric logic using flexible abstractions on any data model.
- **DRY (Don't Repeat Yourself)**: Minimize redundancy by enabling metric definitions whenever possible.
- **Simplicity with gradual complexity:** Approach MetricFlow using familiar data modeling concepts.
- **Performance and efficiency**: Optimize performance while supporting centralized data engineering and distributed logic ownership.

- It uses YAML files to define a semantic graph, which maps language to data. This graph consists of [semantic models](/docs/build/semantic-models), which serve as data entry points, and [metrics](/docs/build/metrics-overview), which are functions used to create new quantitative indicators.
<!--MetricFlow is a SQL query generation engine that helps you create metrics by constructing appropriate queries for different granularities and dimensions that are useful for various business applications.
- MetricFlow is a [BSL package](https://github.com/dbt-labs/metricflow) (code is source available) and available on dbt versions 1.6 and higher. Data practitioners and enthusiasts are highly encouraged to contribute.
- It uses YAML files to define a semantic graph, which maps language to data. This graph consists of , which serve as data entry points, and [, which are functions used to create new quantitative indicators.
- MetricFlow is a () and available on dbt versions 1.6 and higher.
- MetricFlow, as a part of the dbt Semantic Layer, allows organizations to define company metrics logic through YAML abstractions, as described in the following sections.
- You can install MetricFlow using PyPI as an extension of your [dbt adapter](/docs/supported-data-platforms) in the CLI. To install the adapter, run `pip install "dbt-metricflow[your_adapter_name]"` and add the adapter name at the end of the command. For example, for a Snowflake adapter run `pip install "dbt-metricflow[snowflake]"`.
- To query metrics dimensions, dimension values, and validate your configurations; install MetricFlow in
- To query metrics dimensions, dimension values, and validate your configurations; install the [MetricFlow CLI](/docs/build/metricflow-cli).
MetricFlow has the following principles:
- **Flexible, but complete** &mdash; Ability to create any metric on any data model by defining logic in flexible abstractions.
- **Don't Repeat Yourself (DRY)** &mdash; Avoid repetition by allowing metric definitions to be enabled whenever possible.
- **Simple with progressive complexity** &mdash; Make MetricFlow approachable by relying on known concepts and structures in data modeling.
- **Performant and efficient** &mdash; Allow for performance optimizations in centralized data engineering while still enabling distributed definition and ownership of logic.
-->

### Semantic graph

Expand Down Expand Up @@ -114,8 +130,6 @@ group by 1, 2
</TabItem>
<TabItem value="metricflow" label="Calculate with MetricFlow">

> Introducing MetricFlow, a key component of the dbt Semantic Layer 🤩 - simplifying data collaboration and governance.
In the following three example tabs, use MetricFlow to define a semantic model that uses order_total as a metric and a sample schema to create consistent and accurate results &mdash; eliminating confusion, code duplication, and streamlining your workflow.

<Tabs>
Expand Down
31 changes: 12 additions & 19 deletions website/docs/docs/build/build-metrics-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,24 @@ pagination_next: "docs/build/sl-getting-started"
pagination_prev: null
---

Use MetricFlow in dbt to centrally define your metrics. As a key component of the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), MetricFlow is responsible for SQL query construction and defining specifications for dbt semantic models and metrics.
Use MetricFlow in dbt to centrally define your metrics. As a key component of the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), MetricFlow is responsible for SQL query construction and defining specifications for dbt semantic models and metrics. It uses familiar constructs like semantic models and metrics to avoid duplicative coding, optimize your development workflow, ensure data governance for company metrics, and guarantee consistency for data consumers.

Use familiar constructs like semantic models and metrics to avoid duplicative coding, optimize your development workflow, ensure data governance for company metrics, and guarantee consistency for data consumers.

:::info
MetricFlow is currently available on dbt v1.6 or higher and allows users to define metrics in their dbt project whether in dbt Cloud or dbt Core. dbt Core users can use the MetricFlow CLI to define metrics in their local dbt Core project. However, to experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.

:::

Before you start, consider the following guidelines:

- Define metrics in YAML and query them using these [new metric specifications](https://github.com/dbt-labs/dbt-core/discussions/7456).
- You must be on dbt v1.6 or higher to use MetricFlow. [Upgrade your dbt version](/docs/dbt-versions/upgrade-core-in-cloud) to get started.
- Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (CLI only), or Redshift. (dbt Cloud Postgres support coming soon)
- Unlock insights and query your metrics using the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and its diverse range of [available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations).
MetricFlow allows you to:
- Intuitively define metrics in your dbt project
- Develop from your preferred environment, whether that's the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), or [dbt Core](/docs/core/installation)
- Use [MetricFlow commands](/docs/build/metricflow-commands) to query and test those metrics in your development environment
- Harness the true magic of the universal dbt Semantic Layer and dynamically query these metrics in downstream tools (Available for dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) accounts only).


<div className="grid--3-col">

<Card
title="Get started with the dbt Semantic Layer"
body="Use this guide to build and define metrics with MetricFlow, set up the dbt Semantic Layer, and query them using downstream tools."
link="/docs/build/sl-getting-started"
icon="dbt-bit"/>

<Card
title="About MetricFlow"
body="Understand MetricFlow's core concepts, key principles, and how to use this powerful tool."
Expand All @@ -52,12 +51,6 @@ Before you start, consider the following guidelines:
link="/docs/use-dbt-semantic-layer/dbt-sl"
icon="dbt-bit"/>

<Card
title="Get started with the dbt Semantic Layer"
body="Use this guide to build and define metrics, set up the dbt Semantic Layer, and query them using the Semantic Layer API"
link="/docs/use-dbt-semantic-layer/quickstart-sl"
icon="dbt-bit"/>

<Card
title="Available integrations"
body="Discover the diverse range of partners that seamlessly integrate with the powerful dbt Semantic Layer, allowing you to query and unlock valuable insights from your data ecosystem."
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/derived-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ You can query any granularity and offset window combination. The following examp
alias: bookings_7_days_ago
```

When you run the query `mf query --metrics d7_booking_change --group-by metric_time__month` for the metric, here's how it's calculated:
When you run the query `dbt sl query --metrics d7_booking_change --group-by metric_time__month` for the metric, here's how it's calculated. For dbt Core, you can use the `mf query` prefix.

1. We retrieve the raw, unaggregated dataset with the specified measures and dimensions at the smallest level of detail, which is currently 'day'.
2. Then, we perform an offset join on the daily dataset, followed by performing a date trunc and aggregation to the requested granularity.
Expand Down
25 changes: 20 additions & 5 deletions website/docs/docs/build/dimensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,10 @@ semantic_model:

Dimensions have 2 types. This section further explains the definitions and provides examples.

1. [Categorical](#categorical)
1. [Time](#time)
- [Dimensions types](#dimensions-types)
- [Categorical](#categorical)
- [Time](#time)
- [SCD Type II](#scd-type-ii)

### Categorical

Expand All @@ -108,9 +110,14 @@ You can use multiple time groups in separate metrics. For example, the `users_cr


```bash
mf query --metrics users_created,users_deleted --dimensions metric_time --order metric_time
# dbt Cloud users
dbt sl query --metrics users_created,users_deleted --dimensions metric_time --order metric_time
# dbt Core users
mf query --metrics users_created,users_deleted --dimensions metric_time --order metric_time
```


You can set `is_partition` for time or categorical dimensions to define specific time spans. Additionally, use the `type_params` section to set `time_granularity` to adjust aggregation detail (like daily, weekly, and so on):

<Tabs>
Expand All @@ -121,9 +128,13 @@ Use `is_partition: True` to show that a dimension exists over a specific time wi

You can also use `is_partition` for [categorical](#categorical) dimensions as well.

MetricFlow enables metric aggregation during query time. For example, you can aggregate the `messages_per_month` measure. If you originally had a `time_granularity` for the time dimensions `metric_time`, you can specify a yearly granularity for aggregation in your CLI query:
MetricFlow enables metric aggregation during query time. For example, you can aggregate the `messages_per_month` measure. If you originally had a `time_granularity` for the time dimensions `metric_time`, you can specify a yearly granularity for aggregation in your query:

```bash
# dbt Cloud users
dbt sl query --metrics messages_per_month --dimensions metric_time --order metric_time --time-granularity year
# dbt Core users
mf query --metrics messages_per_month --dimensions metric_time --order metric_time --time-granularity year
```

Expand Down Expand Up @@ -344,7 +355,11 @@ In the sales tier example, if sales_person_id 456 is Tier 2 from 2022-03-08 onwa

The following command or code represents how to return the count of transactions generated by each sales tier per month:

```
```bash
# dbt Cloud users
dbt sl query --metrics transactions --dimensions metric_time__month,sales_person__tier --order metric_time__month --order sales_person__tier
# dbt Core users
mf query --metrics transactions --dimensions metric_time__month,sales_person__tier --order metric_time__month --order sales_person__tier
```
Expand Down
9 changes: 9 additions & 0 deletions website/docs/docs/build/measures.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,15 @@ metrics:

We can query the semi-additive metrics using the following syntax:

For dbt Cloud:

```bash
dbt sl query --metrics mrr_by_end_of_month --dimensions metric_time__month --order metric_time__month
dbt sl query --metrics mrr_by_end_of_month --dimensions metric_time__week --order metric_time__week
```

For dbt Core:

```bash
mf query --metrics mrr_by_end_of_month --dimensions metric_time__month --order metric_time__month
mf query --metrics mrr_by_end_of_month --dimensions metric_time__week --order metric_time__week
Expand Down
Loading

0 comments on commit 8ee4a80

Please sign in to comment.