Let's get rid of the source call!

[randypitcherii]

I've wondered for a long time why sources are so special that they need their own jinja function to reference them.

Instead of something like:

select * from {{source('my_src', 'table_1')}} 

Why not just:

select * from {{ref('my_src.table_1')}} 

Or even:

select * from {{ref('my_src', 'table_1')}} 

Note: I know we have a 2 arg ref already for handling abiguous model names across packages, but maybe we can move that to a 3rd positional argument and make room for source refs to thrive!

Pros:

• ergonomics! Less complex to learn or think about.

Cons:

• probably tons of work for something that's not really that big of a deal

What do you think? Am I just being a baby?

[gshank]

It would probably be possible, but yeah, a fair amount of work. This brings up something else that I've been wondering about, which is that we put "nodes" into different dictionaries in the manifest, nodes, sources, metrics, exposures, disabled. Maybe we could just put everything into one "nodes" dictionary?

[jtcohen6]

For sure — I remember talking about this one last year, in the leadup to v1.0 (didn't end up getting in): #3557

Lots of things to like about a single centralized dict... especially if it makes it easier for us to (someday) split up and index the manifest, rather than expecting it all to live in One Big File :)

[ChenyuLInx]

I tried to do something similar in the external node hackathon, it worked(halfway). We probably want to refactor the package part(I believe that's how source with different table works now) and how we decide unique node in order to make future changes easier

[Fleid]

I'm wondering if we shouldn't wait until we unpack "multi-project deployments (1.5+)" to touch the ergonomics of interacting with external resources. But then, absolutely yes.

[joellabes]

I'm tentatively negative on it, but might be wrong. Here's what I'm thinking:

source being its own thing means that it is 100% entirely clear and unambiguous when you're doing the wrong thing. Sources are different to models that dbt makes, and you get to treat them differently.

If I instead made my staging model like this:

-- model_1.sql, materialized into the analytics schema
select * from {{ ref('raw.source_table') }}

It might feel natural to then go and do this in an intermediate model/mart:

select * from {{ ref('analytics.model_1') }}

Admittedly, this will fail because you don't have analytics.model_1 defined as a source. But maybe then you think you have to go and re-define it as a source?

The reason Im going down this line of thought: when I made my first snapshots, I tried to define them as sources because I didn't know that I could just ref them. It felt wrong, but I couldn't work out what to do instead.

This feels somewhat similar.

[enrico-spadaTR]

IMHO we need the source and DBT should more mature feature on them.

Ideally, in the future DBT deploys a feature that handles source nodes differently.
Specifically, why does ref nodes waits for their dependency upstream to complete but this feature is missing in source?
When DBT will further mature, the source should include a dependency to the landing table it refresh (e.g., do not execute it until its landing table is refreshed).