feat: transforms aggregated completion "progress" events into a new fact

[pomegranited]

Extends https://github.com/openedx/aspects-dbt/ to add a new fact_aggregated_completions table to the reporting schema.
This new table is the same as fact_completion but also includes the results.completion field as a completed boolean.

Part of: openedx/openedx-aspects#222

[pomegranited]

Hi @bmtcril @Ian2012 I wrote this dbt package to go along with the completion aggregator xAPI events added by open-craft/openedx-completion-aggregator#205, but I'm having trouble with the target schema: it's using reporting_xapi instead of reporting for some reason I can't see. See PR description for a screenshot.

I also tried:

• re-running tutor dev do init -l aspects to see if re-running this grant statement would fix the issue. It didn't.
• hardcoding database: "reporting" instead of using the DBT_PROFILE_TARGET_DATABASE env var.

Can you see where I've gone wrong?

[bmtcril]

I'm seeing the same issue you are and I think this is your problem. Since Clickhouse doesn't have a concept of database, only schema when database and schema are both defined dbt_clickhouse seems to treat database as a prefix. I think if you delete this line it should work as expected.

FWIW I do also see this successfully being created in a new reporting_xapi schema and otherwise it seems to work!

[Ian2012]

Aspects v1.0.2 solved an issue in which the environment variable for the reporting schema was not correctly set.

[pomegranited]

Since Clickhouse doesn't have a concept of database, only schema when database and schema are both defined dbt_clickhouse seems to treat database as a prefix.

That's exactly what happened! I ended up deleting the schema="xapi" instead of removing database="reporting" here, because the rest of the fact_* tables are all under the reporting database, cf 8859f6a.

[pomegranited]

@bmtcril FYI I also had to qualify this Aspects get_engine macro with the aspects. prefix.
Will note this in a "troubleshooting" docs PR, because this was difficult to debug :)

[bmtcril]

Oh nice, good find 👍

[Ian2012]

For this to "extend" Aspects, you need to add a packages.yml file with:

packages:
  - git: "https://github.com/openedx/aspects-dbt.git"
    revision: vX.X.X

I'm not sure how to keep it in sync for every release of aspects-dbt.

[bmtcril]

You can use semantic versioning, fwiw:

packages:
  - git: "https://github.com/openedx/aspects-dbt.git"
    version: [">=0.7.0", "<0.8.0"]

It's difficult to know what's a "breaking change" in dbt land, we should probably write up criteria.

[pomegranited]

You can use semantic versioning, fwiw:

Nope, dbt doesn't like that, raises a Validator error (docs)

I'm not sure how to keep it in sync for every release of aspects-dbt.

Maybe there's a tutor templated way? But I think it can be manual for now.

[Ian2012]

We could inject it, but @bmtcril wanted to avoid introducing breaking changes