Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[do not merge] dimensions/measures/entities support meta tag #6633

Open
wants to merge 10 commits into
base: current
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions website/docs/docs/build/dimensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa
| `description` | A clear description of the dimension. | Optional | String |
| `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String |
| `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String |
| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | {<dictionary>} |
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

Refer to the following for the complete specification for dimensions:

Expand All @@ -37,6 +38,8 @@ dimensions:

Refer to the following example to see how dimensions are used in a semantic model:

<VersionBlock firstVersion="1.9">

```yaml
semantic_models:
- name: transactions
Expand All @@ -59,13 +62,49 @@ semantic_models:
type_params:
time_granularity: day
label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data
meta:
data_owner: "Finance team"
expr: ts
- name: is_bulk
type: categorical
expr: case when quantity > 10 then true else false end
- name: type
type: categorical
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
semantic_models:
- name: transactions
description: A record for every transaction that takes place. Carts are considered multiple transactions for each SKU.
model: {{ ref('fact_transactions') }}
defaults:
agg_time_dimension: order_date
# --- entities ---
entities:
- name: transaction
type: primary
...
# --- measures ---
measures:
...
# --- dimensions ---
dimensions:
- name: order_date
type: time
type_params:
time_granularity: day
label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data
expr: ts
- name: is_bulk
type: categorical
expr: case when quantity > 10 then true else false end
- name: type
type: categorical
```
</VersionBlock>

Dimensions are bound to the primary entity of the semantic model they are defined in. For example the dimension `type` is defined in a model that has `transaction` as a primary entity. `type` is scoped to the `transaction` entity, and to reference this dimension you would use the fully qualified dimension name i.e `transaction__type`.

Expand Down Expand Up @@ -101,12 +140,27 @@ This section further explains the dimension definitions, along with examples. Di

Categorical dimensions are used to group metrics by different attributes, features, or characteristics such as product type. They can refer to existing columns in your dbt model or be calculated using a SQL expression with the `expr` parameter. An example of a categorical dimension is `is_bulk_transaction`, which is a group created by applying a case statement to the underlying column `quantity`. This allows users to group or filter the data based on bulk transactions.

<VersionBlock firstVersion="1.9">

```yaml
dimensions:
- name: is_bulk_transaction
type: categorical
expr: case when quantity > 10 then true else false end
meta:
usage: "Filter to identify bulk transactions, like where quantity > 10."
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
dimensions:
- name: is_bulk_transaction
type: categorical
expr: case when quantity > 10 then true else false end
```
</VersionBlock>

## Time

Expand All @@ -130,12 +184,16 @@ You can set `is_partition` for time to define specific time spans. Additionally,

Use `is_partition: True` to show that a dimension exists over a specific time window. For example, a date-partitioned dimensional table. When you query metrics from different tables, the dbt Semantic Layer uses this parameter to ensure that the correct dimensional values are joined to measures.

<VersionBlock firstVersion="1.9">

```yaml
dimensions:
- name: created_at
type: time
label: "Date of creation"
expr: ts_created # ts_created is the underlying column name from the table
meta:
notes: "Only valid for orders from 2022 onward"
is_partition: True
type_params:
time_granularity: day
Expand All @@ -156,6 +214,37 @@ measures:
expr: 1
agg: sum
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
dimensions:
- name: created_at
type: time
label: "Date of creation"
expr: ts_created # ts_created is the underlying column name from the table
is_partition: True
type_params:
time_granularity: day
- name: deleted_at
type: time
label: "Date of deletion"
expr: ts_deleted # ts_deleted is the underlying column name from the table
is_partition: True
type_params:
time_granularity: day

measures:
- name: users_deleted
expr: 1
agg: sum
agg_time_dimension: deleted_at
- name: users_created
expr: 1
agg: sum
```
</VersionBlock>

</TabItem>

Expand Down
69 changes: 63 additions & 6 deletions website/docs/docs/build/entities.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,17 +95,67 @@ Natural keys are columns or combinations of columns in a table that uniquely ide

The following is the complete spec for entities:

<VersionBlock firstVersion="1.9">

```yaml
semantic_models:
- name: semantic_model_name
..rest of the semantic model config
entities:
- name: entity_name ## Required
type: Primary, natural, foreign, or unique ## Required
description: A description of the field or role the entity takes in this table ## Optional
expr: The field that denotes that entity (transaction_id). ## Optional
Defaults to name if unspecified.
[config](/reference/resource-properties/config): Specify configurations for entity. ## Optional
[meta](/reference/resource-configs/meta): {<dictionary>} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
semantic_models:
- name: semantic_model_name
..rest of the semantic model config
entities:
- name: entity_name ## Required
type: Primary, or natural, or foreign, or unique ## Required
description: A description of the field or role the entity takes in this table ## Optional
expr: The field that denotes that entity (transaction_id). ## Optional
Defaults to name if unspecified.
```
</VersionBlock>

Here's an example of how to define entities in a semantic model:

<VersionBlock firstVersion="1.9">

```yaml
entities:
- name: transaction ## Required
type: Primary or natural or foreign or unique ## Required
- name: transaction
type: primary
expr: id_transaction
- name: order
type: foreign
expr: id_order
- name: user
type: foreign
expr: substring(id_order from 2)
entities:
- name: transaction
type:
description: A description of the field or role the entity takes in this table ## Optional
expr: The field that denotes that entity (transaction_id). ## Optional
expr: The field that denotes that entity (transaction_id).
Defaults to name if unspecified.
[config](/reference/resource-properties/config):
[meta](/reference/resource-configs/meta):
data_owner: "Finance team"
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

Here's an example of how to define entities in a semantic model:

```yaml
entities:
- name: transaction
Expand All @@ -117,11 +167,18 @@ entities:
- name: user
type: foreign
expr: substring(id_order from 2)
entities:
- name: transaction
type:
description: A description of the field or role the entity takes in this table ## Optional
expr: The field that denotes that entity (transaction_id).
Defaults to name if unspecified.
```
</VersionBlock>

## Combine columns with a key

If a table doesn't have any key (like a primary key), use _surrogate combination_ to form a key that will help you identify a record by combining two columns. This applies to any [entity type](/docs/build/entities#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a _surrogate key_. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator.
If a table doesn't have any key (like a primary key), use _surrogate combination_ to form a key that will help you identify a record by combining two columns. This applies to any [entity type](/docs/build/entities#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a _surrogate key_. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator.

```yaml

Expand Down
132 changes: 124 additions & 8 deletions website/docs/docs/build/measures.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,41 @@ import MeasuresParameters from '/snippets/_sl-measures-parameters.md';

An example of the complete YAML measures spec is below. The actual configuration of your measures will depend on the aggregation you're using.

<VersionBlock firstVersion="1.9">

```yaml
semantic_models:
- name: semantic_model_name
..rest of the semantic model config
measures:
- name: The name of the measure
description: 'same as always' ## Optional
agg: the aggregation type.
expr: the field
agg_params: 'specific aggregation properties such as a percentile' ## Optional
agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional
non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional
[config](/reference/resource-properties/config): Use the config property to specify configurations for your measure. ## Optional
[meta](/reference/resource-configs/meta): {<dictionary>} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional
```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
measures:
- name: The name of the measure
description: 'same as always' ## Optional
agg: the aggregation type.
expr: the field
agg_params: 'specific aggregation properties such as a percentile' ## Optional
agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional
non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional
semantic_models:
- name: semantic_model_name
..rest of the semantic model config
measures:
- name: The name of the measure
description: 'same as always' ## Optional
agg: the aggregation type.
expr: the field
agg_params: 'specific aggregation properties such as a percentile' ## Optional
agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional
non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional
```
</VersionBlock>

### Name

Expand Down Expand Up @@ -96,6 +121,96 @@ If you use the `dayofweek` function in the `expr` parameter with the legacy Snow

### Model with different aggregations

<VersionBlock firstVersion="1.9">

```yaml
semantic_models:
- name: transactions
description: A record of every transaction that takes place. Carts are considered multiple transactions for each SKU.
model: ref('schema.transactions')
defaults:
agg_time_dimension: transaction_date

# --- entities ---
entities:
- name: transaction_id
type: primary
- name: customer_id
type: foreign
- name: store_id
type: foreign
- name: product_id
type: foreign

# --- measures ---
measures:
- name: transaction_amount_usd
description: Total USD value of transactions
expr: transaction_amount_usd
agg: sum
config:
meta:
used_in_reporting: true
- name: transaction_amount_usd_avg
description: Average USD value of transactions
expr: transaction_amount_usd
agg: average
- name: transaction_amount_usd_max
description: Maximum USD value of transactions
expr: transaction_amount_usd
agg: max
- name: transaction_amount_usd_min
description: Minimum USD value of transactions
expr: transaction_amount_usd
agg: min
- name: quick_buy_transactions
description: The total transactions bought as quick buy
expr: quick_buy_flag
agg: sum_boolean
- name: distinct_transactions_count
description: Distinct count of transactions
expr: transaction_id
agg: count_distinct
- name: transaction_amount_avg
description: The average value of transactions
expr: transaction_amount_usd
agg: average
- name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition
description: The total USD value of valid transactions only
expr: CASE WHEN is_valid = True then transaction_amount_usd else 0 end
agg: sum
- name: transactions
description: The average value of transactions.
expr: transaction_amount_usd
agg: average
- name: p99_transaction_value
description: The 99th percentile transaction value
expr: transaction_amount_usd
agg: percentile
agg_params:
percentile: .99
use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile.
- name: median_transaction_value
description: The median transaction value
expr: transaction_amount_usd
agg: median

# --- dimensions ---
dimensions:
- name: transaction_date
type: time
expr: date_trunc('day', ts) # expr refers to underlying column ts
type_params:
time_granularity: day
- name: is_bulk_transaction
type: categorical
expr: case when quantity > 10 then true else false end

```
</VersionBlock>

<VersionBlock lastVersion="1.8">

```yaml
semantic_models:
- name: transactions
Expand Down Expand Up @@ -177,6 +292,7 @@ semantic_models:
expr: case when quantity > 10 then true else false end

```
</VersionBlock>

### Non-additive dimensions

Expand Down
Loading
Loading