Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 98 additions & 8 deletions docs/openedx_learning/decisions/0002-competency-criteria-model.rst
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Decision

This new database table will have the following columns:
1. ``taxonomy_ptr_id``: Primary key and one-to-one foreign key to ``oel_tagging_taxonomy.id``.
2. ``taxonomy_overrides_org``: Boolean, defaults to ``false``. Used only while computing which single ``CompetencyRuleProfile`` to assign to a ``CompetencyCriterion`` (Decision 4). If, for a criterion's context, both an organization-scoped profile row and a taxonomy-scoped profile row exist as candidates, this field decides which one gets assigned: ``false`` (default) assigns the organization-scoped row; ``true`` assigns this taxonomy's own row instead, so it cannot be overridden by an organization. Once assigned, the criterion stores that one profile's id and this field plays no further part. Not exercised in this phase; see the MVP note in Decision 4.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions:

  • the taxonomy here decides if its own competency rule profile should be overridden by the org course profile if present or not. Why is this decided by the taxonomy?
  • "Not exercised in this phase" - what does that mean for MVP implementation? Create but not use this field?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm drafting some clarifications to this in the ADR now, but to answer your questions:

  1. Why the taxonomy decides the override: a single taxonomy can compete against many organization-scoped profiles, so the tiebreak needs one consistent answer that applies to all of them. Putting it on the taxonomy avoids setting (and keeping in sync) the same flag on every org-scoped profile row. Added as a new bullet under Decision 1's "Relationship to other concepts."

  2. "Not exercised in this phase" means: the column is created and defaults to false, but no code reads it yet, since organization-scoped profiles aren't created in this phase (see the MVP note in Decision 4), so the conflict it resolves can't occur yet. Reworded that sentence in the column description to say this directly.


Lifecycle rules for this parent/child pair:

Expand Down Expand Up @@ -113,19 +114,27 @@ Decision

Relationship to other concepts:

- Can be scoped by taxonomy, course, and/or organization.
- Each row is scoped by at most one of taxonomy, course, or organization (or by none, for the system default). See Decision 4 for how a criterion is assigned a profile when rows in more than one of these scopes could apply to it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: if taxonomy + course + organization are all none, this is system default. Does uniqueness apply to the system default as well, i.e. there can only be one system default rule profile?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there should only be 1 system default rule profile.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question:
Trying to understand how this works in practice. Let's say for example we have a course, and this course has various assignments. Some of these assignments demonstrate the "Python" skill, some of these demonstrate the "JS" skill. So we make sure the correct competency taxonomy that contains different skills is linked (is this linking done by the course author?)
Then the author can use AND/OR statements to apply the competency rule profile to different assignments.
Since there is a course-wide or taxonomy-wide rule profile, initially the taxonomy decides that all assignments need to have gte 75% of success (why is this specified for the taxonomy, as it applies to assignments in a course?) or the author can set a taxonomy rule profile for the course stating that all assignments need to meet 75% of success, and he can override this per assignment if desired.
But unless the author manually edits assignment rules, they can't set e.g. 75% for Python assignments, 50% for JS assignments, and 65% for take-home assignments in Python?
So... how is having one rule-profile per course or even per taxonomy helpful? Why don't they differentiate by skill or assignment type or something else?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively: why have this complicated construct if it can create hurdles (like not distinguishing between skill and assignment types)?
Why not like this?

  • rule_type and rule_payload can be set per assignment if desired, like before, overriding any course-wide or org-wide values or values that are set up higher in the criteria tree (similar to how problem grading works now)
  • The author can define a CompetencyRule that he assigns to one or more assignments/tag applications (criterion), or to a skill tag via CompetencyCriteriaGroup / to a tree node, or to the whole course or multiple courses. That relationship can be reflected via another table
  • Even CompetencyRules that are applied to a whole course can be marked to apply only to specific skills or assignment types
  • No more fixed inflexible rule profiles - the author just creates a rule and assigns it to whatever he wants, and being able to apply it to the whole course or even org allows him to be efficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is mute (see #636 (review)).

- At most one profile row may exist per distinct scope value (unique constraint, Decision 5).
- The system default is the single profile row where all three scope fields are null. It is seeded, not created/edited/deleted through the profile API.
- Is referenced by ``CompetencyCriterion``, which may override its type/payload.
- Never hard-deleted; retirement is archive-only (Decision 7).

This new database table will have the following columns:

1. ``id``: unique primary key
2. ``organization_id``: The ``organization_id`` of the organization that this competency rule profile is scoped to. Null if it is not scoped to a specific organization.
3. ``course_id``: The ``course_id`` of the course that this competency rule profile is scoped to. Null if it is not scoped to a specific course.
4. ``competency_taxonomy_id``: The ``CompetencyTaxonomy.taxonomy_ptr_id`` of the competency taxonomy that this competency rule profile is scoped to.
4. ``competency_taxonomy_id``: The ``CompetencyTaxonomy.taxonomy_ptr_id`` of the competency taxonomy that this competency rule profile is scoped to. Null if it is not scoped to a specific taxonomy.
5. ``rule_type``: “View”, “Grade”, “MasteryLevel” (Only “Grade” will be supported for now)
6. ``rule_payload``: JSON payload keyed by ``rule_type`` to avoid freeform strings. It is structured JSON (not arbitrary freeform data): each ``rule_type`` defines the allowed payload shape and required keys, and validation enforces this contract. JSON is used instead of fixed columns like ``op``, ``value``, and ``scale`` so that future rule types (for example, ``MasteryLevel`` thresholds or plugin-defined evaluators such as CEL-based rules) can add their own fields without repeated schema migrations or many nullable columns. Examples:

1. ``Grade``: ``{"op": "gte", "value": 75, "scale": "percent"}``
1. ``Grade``: ``{"op": "gte", "value": 0.75, "scale": "percent"}``. Allowed ``op`` values: ``gte``, ``lte``, ``eq``. ``value`` must be a fraction between 0.0 and 1.0 inclusive, matching the platform's existing fractional grade representation, not a 0-100 scale.
7. ``archived``: Boolean, defaults to false. Set instead of deleting a profile that is no longer wanted. Archived profiles are hidden from authoring and new associations but remain queryable, so existing ``CompetencyCriterion`` rows and learner status history stay resolvable.

Editing a profile may change ``rule_type``/``rule_payload`` only; scope fields are immutable after creation, to avoid silently re-scoping criteria that already resolved to this profile.

MVP note: only taxonomy-scoped and system-default profiles are created; every profile row created in this phase has ``course_id`` and ``organization_id`` null. Enabling course- or organization-scoped profiles later needs no schema change here -- the columns, the uniqueness constraint, and the re-assignment behavior (Decision 4) already support them.

4. ``CompetencyCriterion`` concept (``CompetencyCriteria`` database table)

Expand All @@ -142,11 +151,80 @@ Decision
1. ``id``: unique primary key
2. ``competency_criteria_group_id``: Foreign key to ``CompetencyCriteriaGroup.id``.
3. ``oel_tagging_objecttag_id``: Tag/Object Association id
4. ``competency_rule_profile_id``: Nullable FK to the ``CompetencyRuleProfile`` applied to this criterion. If null, evaluate using fallback lookup order: taxonomy-scoped profile, then course-scoped profile, then organization-scoped profile, then system default.
4. ``competency_rule_profile_id``: Nullable FK to the ``CompetencyRuleProfile`` applied to this criterion.
5. ``rule_type_override``: Nullable enumerated rule type: “View”, “Grade”, “MasteryLevel” (Only “Grade” will be supported for now). When set, this overrides the ``rule_type`` in the associated ``CompetencyRuleProfile`` for this criterion.
6. ``rule_payload_override``: Nullable JSON payload keyed by ``rule_type`` to avoid freeform strings. When set, this overrides the ``rule_payload`` in the associated ``CompetencyRuleProfile`` for this criterion. The same typed/validated payload contract as ``rule_payload`` applies. Examples:

1. ``Grade``: ``{"op": "gte", "value": 75, "scale": "percent"}``
1. ``Grade``: ``{"op": "gte", "value": 0.75, "scale": "percent"}``. Allowed ``op`` values: ``gte``, ``lte``, ``eq``. ``value`` must be a fraction between 0.0 and 1.0 inclusive, matching the platform's existing fractional grade representation, not a 0-100 scale.

Exactly one of the following holds for a given criterion, never both, never neither:

(a) ``competency_rule_profile_id`` is set and both override fields are null, or
(b) ``competency_rule_profile_id`` is null and both override fields are set.

``competency_rule_profile_id`` is not assigned once and left alone. The same assignment computation -- using whatever scope the relevant authoring screen operates in, resolved per the table below -- is re-run at each of these points, and each one writes a new value:

1. Creation: the authoring screen that creates the criterion assigns it using the scope that screen itself operates in (for example, adding Competency Criteria to the Course Outline page would have it supply its own course id; the Competency Management page will supply its own taxonomy id). This is independent of ``CompetencyCriteriaGroup.course_id``, which scopes evaluation windowing (Decision 2), not rule assignment; the two may coincidentally match but neither determines the other.
2. A more specific profile is created later that would now apply to an existing criterion: that criterion is reassigned to it. Treated as an edit for the in-use warning (ADR 0003 Decision 4).
3. A user sets a per-criterion override (for example, editing a ``CompetencyCriteriaGroup``'s default rule, which cascades to every ``CompetencyCriterion`` under that group): ``competency_rule_profile_id`` is set to null and the override fields are set instead.
4. A user's override is changed to match what the computation in (1)/(2) would already produce for this criterion: the criterion is reassigned back to that profile and the override fields are cleared, rather than keeping a redundant override in place.

(A future authoring action to revert a criterion to the next-closest scope, once course/organization scoping exists, would be a fifth trigger using the same computation; out of scope for this phase.)

In no case is the FK re-resolved dynamically at evaluation time -- only these explicit write events change it.

Which profile a criterion is assigned, by which profile rows exist for its context:

.. list-table::
:header-rows: 1

* - Course profile exists?
- Org profile exists?
- Taxonomy profile exists?
- Assigned
- Notes
* - Yes
- (any)
- (any)
- Course
- Always wins outright, no exceptions.
* - No
- Yes
- Yes
- See below
- The only contested case.
* - No
- Yes
- No
- Organization
- Nothing else to compete with.
* - No
- No
- Yes
- Taxonomy
- Nothing else to compete with.
* - No
- No
- No
- System default
- Nothing else exists.

The contested case (no course profile; both an organization-scoped row and a taxonomy-scoped row exist) is resolved by ``CompetencyTaxonomy.taxonomy_overrides_org`` (Decision 1):

.. list-table::
:header-rows: 1

* - ``taxonomy_overrides_org``
- Example
- Assigned
* - ``false`` (default)
- A general skill taxonomy ("Communication") used by multiple departments, each wanting its own threshold
- Organization
* - ``true``
- A taxonomy adopted from a third-party standard body ("Nursing"), which should not be locally weakened
- Taxonomy

MVP note: only taxonomy-scoped and system-default profiles are created, so every ``CompetencyCriterion`` created in this phase is assigned one of those two -- the only authoring screens that exist today (Competency Management, and eventually Libraries) supply a competency tag but no course or organization. Organization scoping in particular has no defined source yet: no authoring screen has been decided to expose an organization context, so there is nothing to assign an organization-scoped profile from until that is decided. Course- and organization-scoped profiles, and therefore both tables above, are dormant until those scope levels are built; no schema change is needed to enable them then.

5. Indexes for common lookups

Expand All @@ -158,7 +236,7 @@ Decision
6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)``
7. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)``
8. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)``
9. ``CompetencyRuleProfile(competency_taxonomy_id, course_id, organization_id)``
9. ``CompetencyRuleProfile(competency_taxonomy_id, course_id, organization_id)`` (unique -- at most one profile per distinct scope combination)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are we ensuring unique combinations if there are nullable fields?

10. ``CompetencyMasteryStatuses(status)`` (unique)

6. Learner progress status concepts (``StudentCompetency*Status`` database tables)
Expand Down Expand Up @@ -216,8 +294,9 @@ Decision

- If no learner status rows exist for a competency definition, hard delete is allowed and cascades through competency metadata tables.
- Once any related learner status exists in ``StudentCompetencyStatus``, ``StudentCompetencyCriteriaGroupStatus``, or ``StudentCompetencyCriteriaStatus``, deletion of associated competency definition rows is blocked.
- This delete protection applies to ``oel_tagging_taxonomy``, ``CompetencyTaxonomy``, ``oel_tagging_tag``, ``oel_tagging_objecttag``, ``CompetencyCriteriaGroup``, ``CompetencyCriteria``, and ``CompetencyRuleProfile``.
- This delete protection applies to ``oel_tagging_taxonomy``, ``CompetencyTaxonomy``, ``oel_tagging_tag``, ``oel_tagging_objecttag``, ``CompetencyCriteriaGroup``, and ``CompetencyCriteria``.
- Once any related learner status exists, retiring definitions may be archive-only (hidden from authoring and new associations), not hard delete.
- Exception: ``CompetencyRuleProfile`` is never hard-deleted, even before any learner status exists, because it is shared across many ``CompetencyCriterion`` rows and reused going forward rather than tied to one. Retirement is always archive-only, via its ``archived`` column (Decision 3).

.. image:: images/CompetencyCriteriaModel.png
:alt: Competency Criteria Model
Expand Down Expand Up @@ -245,7 +324,7 @@ Content objects:

2. ``CompetencyRuleProfile``:

- Course-scoped default: ``Grade >= 75%`` for this competency taxonomy
- Taxonomy-scoped default: ``Grade >= 75%`` for this competency taxonomy

3. ``CompetencyCriteriaGroup``:

Expand Down Expand Up @@ -314,3 +393,14 @@ Rejected Alternatives
2. Cons

1. Increases table count and join complexity as new rule types are added

5. Require a strict, always-cascading scope on ``CompetencyRuleProfile`` (organization always set; taxonomy only settable alongside organization; course only settable alongside both), instead of letting each row be scoped by at most one of the three independently.

1. Pros

1. At most one profile can ever apply to a given criterion by construction, without a separate uniqueness constraint or tiebreak field.
2. Cons

1. Does not fit taxonomies that span multiple organizations: a single taxonomy-wide default would need one duplicate profile per associated organization.
2. Requires reconciling profiles whenever an organization is added to or removed from a taxonomy.
3. Organization and taxonomy are not naturally nested (a taxonomy can belong to many organizations and vice versa), so forcing one to always contain the other does not reflect the actual relationship between them.
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ For the initial implementation, versioning and traceability of competency achiev

- If a user edits competency criteria definitions or competency object/tag associations after related learner status exists, Studio must display an explicit warning that student statuses have already been set, and these changes will be applied going forward, so existing learner statuses will not be retroactively updated.
- Applying these changes requires explicit user confirmation.
- A ``CompetencyRuleProfile`` is "in use" if any ``CompetencyCriterion`` assigned to it (``competency_rule_profile_id``) has an associated ``StudentCompetencyCriteriaStatus`` row. Editing an in-use profile's ``rule_type``/``rule_payload`` requires the same warning and confirmation.
- The same warning applies when creating a more specific profile causes existing criteria to be reassigned to it, and when an authoring action switches a criterion between a profile assignment and per-criterion overrides (ADR 0002 Decision 4).

5. Learner status models/tables are append-only history and do not use ``django-simple-history``:

Expand Down