Abstract model - aka Contracts - defined in a .yaml properties file, and a model configuration to tell dbt that it "implements" that abstract model/contract

[fabrice-etanchaud]

Hi all !

It's quite usual that business dbt models are first defined with business people before being implemented by analytics engineers.
It's would be great to have a way to define "abstract" models - or models "contracts", not related to a given sql or python implementation.
Then we could use a new configuration - maybe config(implements=my_abstract_model) ? - to tell dbt that a given model implements a model contract.

Would you find this helpful ?

We could use yaml anchors, but this wouldn't show up in downstream documentation or semantic layer.

[jtcohen6]

@fabrice-etanchaud Yes! We've been hearing this more and more. Have you been following our conversation over in #6079? :)

Copying some internal Slack messages I sent a few weeks ago:

The team defining the data model for the "business-ready" tables could define the column spec via yaml files in dbt. Then, another team is responsible for writing the SQL to produce a table matching that defined spec.

It's possible today to create .yml specs for a model that does not yet exist. dbt will show a warning ("hey I couldn't find a model by this name, ignoring the config").
Then when the model is created (.sql file) by that name, that spec would take effect, including both:

• tests on that model
• column names/types/constraints, where we can leverage Snowflake/BigQuery support for create table <name> <column_spec> as <query>, and if there's a mismatch between <column_spec> and <query> SF/BQ will raise an error

My questions are all about the user stories / workflows here:

• I don't know the "abstract model designer" persona as well as I know the SQL writer persona. Is this person open to writing yaml? Or using a separate tool (perhaps GUI) that writes yaml and opens PRs on the back-end?
• Do we need a mechanism to "lock down" permissions on who can touch which code? Do these definitions need to live in separate codebases/repositories, or could they just live in separate file paths within the same repo?

[fabrice-etanchaud]

Bonsoir Jérémy !

I don't know the "abstract model designer" persona as well as I know the SQL writer persona. Is this person open to writing yaml? Or using a separate tool (perhaps GUI) that writes yaml and opens PRs on the back-end?

I would say both but usually, formalizing the data model is up to the engineer working with the business team on the enterprise data model.

Do we need a mechanism to "lock down" permissions on who can touch which code? Do these definitions need to live in separate codebases/repositories, or could they just live in separate file paths within the same repo?

No extra features on permissions I would say, but it would be great to have in a same package any mixture of abstraction and implementation. We could even envision referencing a pure abstract package - modeling an enterprise area - from a package implementing the related business concepts.

[fabrice-etanchaud]

There is a lot of buz around "Data Contracts". It seems to me that dbt has - as usual :-) - already the solution to that problem : A contract could be formalized with dbt as a yaml model properties file, containing not only the description of the model's attributes - name, type, null/not_null, accepted values - but also and more importantly the semantic rules, expressed as dbt tests.

generic tests would formalize attribute, or group of attributes semantics

singular tests would formalize row level semantics or model global computation rules (a kind of class invariant but for data sets)

But there is something missing for now : the possibility to express singular tests on abstract models. What does dbt do with generic tests referencing unimplemented models ? I will give it a try. It would help to be able to list the singular tests modeling the model semantics inside the yaml file, to keep things together.

What are your feelings ?