docs: ADR 0005 competency mastery storage, without history

docs/openedx_learning/decisions/0002-competency-criteria-model.rst

@@ -135,7 +135,7 @@ Decision
    7. ``rule_payload``: JSON payload keyed by ``rule_type`` to avoid freeform strings. It is structured JSON (not arbitrary freeform data): each ``rule_type`` defines the allowed payload shape and required keys, and validation enforces this contract. JSON is used instead of fixed columns like ``op``, ``value``, and ``scale`` so that future rule types (for example, ``MasteryLevel`` thresholds or plugin-defined evaluators such as CEL-based rules) can add their own fields without repeated schema migrations or many nullable columns. Examples:
 
       1. ``Grade``: ``{"op": "gte", "value": 0.75, "scale": "percent"}``. Allowed ``op`` values: ``gte``, ``lte``, ``eq``. ``value`` must be a fraction between 0.0 and 1.0 inclusive, matching the platform's existing fractional grade representation, not a 0-100 scale.
-   8. ``archived``: Boolean, defaults to false. Set instead of deleting a profile that is no longer wanted. Archived profiles are hidden from authoring and new associations but remain queryable, so existing ``CompetencyCriterion`` rows and learner status history stay resolvable.
+   8. ``archived``: Boolean, defaults to false. Set instead of deleting a profile that is no longer wanted. Archived profiles are hidden from authoring and new associations but remain queryable, so existing ``CompetencyCriterion`` rows and learner status rows stay resolvable.
 
    A check constraint requires that at most one of ``organization_id``, ``course_id``, and ``competency_taxonomy_id`` is non-null on any row, matching the scoping rule above.
 
@@ -240,9 +240,9 @@ Decision
    3. ``oel_tagging_objecttag(object_id)``
    4. ``CompetencyCriteria(oel_tagging_objecttag_id)``
    5. ``CompetencyCriteria(competency_criteria_group_id)``
-   6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)``
-   7. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)``
-   8. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)``
+   6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)`` (unique)
+   7. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)`` (unique)
+   8. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)`` (unique)
    9. ``CompetencyRuleProfile(scope_code)`` (unique -- at most one profile per distinct scope value; a plain unique constraint on the three raw nullable scope columns would not enforce this, since SQL never treats two ``NULL`` values as equal and this project's MySQL backend does not support the conditional/partial unique indexes that would otherwise route around that; see the ``scope_code`` column in Decision 3)
    10. ``CompetencyMasteryStatuses(status)`` (unique)
 
@@ -268,6 +268,7 @@ Decision
 
       1. ``id``: unique primary key
       2. ``status``: unique status value (seeded values: “Demonstrated”, “AttemptedNotDemonstrated”, and “PartiallyAttempted”)
+      3. ``rank``: unique small integer ordering the statuses from lowest to highest: “AttemptedNotDemonstrated”, “PartiallyAttempted”, “Demonstrated”. ``id`` does not carry that order and must not be used for it. This column is what lets the monotone merge in ADR 0004 Decision 2 compare the stored status against the newly computed one in a single ``UPDATE``, rather than reading the row into Python and writing it back under a lock the design does not otherwise need.
 
       Notes:
 
@@ -279,23 +280,26 @@ Decision
       2. ``competency_criteria_id``: Foreign key to ``CompetencyCriterion.id``
       3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table
       4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id``
-      5. ``created``: The timestamp at which the student's criterion status was set.
+      5. ``created``: The timestamp at which this row was first written.
+      6. ``modified``: The timestamp at which the student's criterion status last changed. Rows are updated in place (ADR 0003 Decision 5), so ``created`` alone cannot date the current status. Set when the stored status actually changes, not on every write: under the monotone merge (ADR 0004 Decision 2) a redelivered grade event commonly rewrites the same value, and dating that as a change would make this column mean "last written" instead. Django's ``auto_now`` implements the latter, and does not fire at all for ``queryset.update()`` or ``bulk_update()``.
 
    3. Add a new database table for ``StudentCompetencyCriteriaGroupStatus`` with these columns:
 
       1. ``id``: unique primary key
       2. ``competency_criteria_group_id``: Foreign key to ``CompetencyCriteriaGroup.id``
       3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table
       4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id``
-      5. ``created``: The timestamp at which the student's criteria-group status was set.
+      5. ``created``: The timestamp at which this row was first written.
+      6. ``modified``: The timestamp at which the student's criteria-group status last changed.
 
    4. Add a new database table for ``StudentCompetencyStatus`` with these columns:
 
       1. ``id``: unique primary key
       2. ``oel_tagging_tag_id``: Foreign key pointing to Tag id
       3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table
       4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id``. This table should have a constraint to only allow status values of “Demonstrated” and “PartiallyAttempted” since it represents overall competency demonstration state, not in-progress states.
-      5. ``created``: The timestamp at which the student's competency status was set.
+      5. ``created``: The timestamp at which this row was first written.
+      6. ``modified``: The timestamp at which the student's competency status last changed.
 
 7. Delete protection boundaries
 
@@ -422,3 +426,23 @@ Rejected Alternatives
 
       1. Silently does not work on this project's tested and production database backend. Django compiles a conditional ``UniqueConstraint`` to a partial index, which MySQL does not support; Django raises only a non-fatal system-check warning (``models.W036``) and skips creating the constraint, leaving the uniqueness rule completely unenforced at the database level.
       2. The gap would surface only as a data-integrity incident under concurrent writes, not as a test or migration failure, since SQLite (used for quick local test runs) does support partial indexes and would mask the problem in that environment.
+
+Changelog
+---------
+
+2026-07-27:
+
+* Made the learner status indexes unique, so there is one row per learner and node. This is what
+  the in-place, monotone status updates in :ref:`openedx-learning-adr-0004` read, lock, and update.
+
+2026-08-11:
+
+* Gave each learner status table a ``modified`` timestamp and narrowed ``created`` to the row's
+  first write. In-place updates mean ``created`` no longer dates the current status, and adding the
+  column after the tables have rows would be a migration on the largest table in the schema
+  (:ref:`openedx-learning-adr-0005`).
+* Corrected the ``CompetencyRuleProfile.archived`` note to say archived profiles keep learner status
+  *rows* resolvable. It said "history", which :ref:`openedx-learning-adr-0005` does not provide.
+* Gave ``CompetencyMasteryStatuses`` a ``rank`` column. Without a stored ordering, the monotone merge
+  in :ref:`openedx-learning-adr-0004`, Decision 2 cannot be a single statement, and ``id`` does not
+  supply one: the seeded values are listed highest-first.

docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst

@@ -13,7 +13,7 @@ Typically, institutions and instructional designers do not change the mastery re
 
 Currently, Open edX always displays the latest edited version of content in the Studio UI and always shows the latest published version of content in the LMS UI, despite having more robust version tracking on the backend (Publishable Entities).
 
-Authoring data (criteria definitions) and runtime learner data (status) have different governance needs. The former is long-lived and typically non-PII, while the latter is user-specific, can be large (learners x criteria/competencies x time), and may require stricter retention and access controls. These differing lifecycles can make deep coupling of authoring and runtime data harder to manage at scale. Performance is also a consideration as computing or resolving versioned criteria for large courses could add overhead in Studio authoring screens or LMS views.
+Authoring data (criteria definitions) and runtime learner data (status) have different governance needs. The former is long-lived and typically non-PII, while the latter is user-specific, can be large (learners x criteria/competencies), and may require stricter retention and access controls. These differing lifecycles can make deep coupling of authoring and runtime data harder to manage at scale. Performance is also a consideration as computing or resolving versioned criteria for large courses could add overhead in Studio authoring screens or LMS views.
 
 Decision
 --------
@@ -44,12 +44,20 @@ For the initial implementation, versioning and traceability of competency achiev
    - A ``CompetencyRuleProfile`` is "in use" if any ``CompetencyCriterion`` assigned to it (``competency_rule_profile_id``) has an associated ``StudentCompetencyCriteriaStatus`` row. Editing an in-use profile's ``rule_type``/``rule_payload`` requires the same warning and confirmation.
    - The same warning applies when creating a more specific profile causes existing criteria to be reassigned to it, and when an authoring action switches a criterion between a profile assignment and per-criterion overrides (ADR 0002 Decision 4).
 
-5. Learner status models/tables are append-only history and do not use ``django-simple-history``:
+5. Learner status models/tables are updated in-place:
 
-   - For ``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and ``StudentCompetencyStatus``, each status change is stored as a new row with ``created`` as the write timestamp.
-   - Existing learner status rows are not updated in place.
-   - Current status is determined by the most recent row for a given learner + target entity (ordered by ``created``, with ``id`` as a tie-breaker).
-   - Older rows represent the learner status history and remain available for audit/tracing.
+   - For ``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and ``StudentCompetencyStatus``,
+     each status change updates the responsible row.
+   - Automatic status updates only ever increase a status, as relied on by
+     :ref:`openedx-learning-adr-0004`. A downward adjustment (for example ``Demonstrated`` to
+     ``PartiallyAttempted``) is never applied by a grade change or by a competency criteria rule
+     change.
+   - Direct edits by staff, through Django admin or as a deliberate instructor correction, are
+     exempt: they may set a status to any value, including a lower one, and the ancestors above the
+     edited node are recomputed to match.
+   - How learner status history is retained is not decided here. See
+     :ref:`openedx-learning-adr-0005`, "Out of Scope", for what the current status rows can and
+     cannot account for, and for the constraints on whatever decides it.
 
 
 Rejected Alternatives
@@ -85,3 +93,21 @@ Rejected Alternatives
     - Cons:
         - Requires custom tooling to reconstruct past versions
         - Does not align with existing publishable versioning patterns
+
+Changelog
+---------
+
+2026-07-27:
+
+* Reworked Decision 5 for :ref:`openedx-learning-adr-0004`: learner status rows are now updated in
+  place, and automatic updates only ever increase a status, with direct staff edits exempt.
+  Previously append-only, with current status resolved as the most recent row. How status history is
+  retained is left undecided.
+
+2026-08-11:
+
+* Pointed Decision 5's undecided history question at :ref:`openedx-learning-adr-0005`, which owns
+  the deferral.
+* Dropped the "x time" factor from the Context's sizing of learner status data. It described the
+  append-only model that Decision 5 replaced; there is now one row per learner and node
+  (:ref:`openedx-learning-adr-0005`).

docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst

@@ -0,0 +1,130 @@
+.. _openedx-learning-adr-0004:
+
+4. How should learner competency mastery be recorded concurrently and at scale?
+================================================================================
+
+Status
+------
+Proposed.
+
+Context
+-------
+A learner's mastery of one competency is stored at three levels of the competency criteria tree:
+the leaf criterion that was graded, each criteria group above it, and the competency itself. There
+is one row per learner and node, updated in place (:ref:`openedx-learning-adr-0002`,
+:ref:`openedx-learning-adr-0003`). Each row holds one of three values, lowest to highest:
+``AttemptedNotDemonstrated``, ``PartiallyAttempted``, ``Demonstrated``.
+So one grade change updates the leaf and then every row above
+it, for a very large number of learners, in bursts. This ADR decides how those writes stay correct
+when two of them for the same learner overlap.
+
+What triggers a change is a subsection grade, or any other learning instrument tied to a
+competency by a competency criterion, such as a course grade or a rubric criterion. The rows above
+the leaf are stored rather than recomputed on read because they also drive notifications, badges,
+and certificate issuing, so the roll-up has to happen when the grade does either way.
+
+Decision
+--------
+
+1. **The platform's grading task calls one openedx-core function, in the same atomic transaction as the
+   grade write.** Subsection grading already happens in a celery task; that task writes the grade
+   and then calls this function, which updates the leaf and walks up. Writing up the tree stops
+   where :ref:`openedx-learning-adr-0002`, Decision 6 says it stops.
+
+2. **Automatic updates only move a status up: each write stores the higher of the stored value and
+   the newly computed one.** This makes the recorder safe
+   against celery delivering the same work twice or out of order. Applying one grade event twice
+   lands on the same value as applying it once.
+
+3. **Before recomputing a group, lock that group's row.**
+   If two children advance at the same moment, each recomputation could read the other child as not yet
+   advanced. Both would then compute the same too-low value. Locking the group first makes the two writers take turns, so the
+   second one reads the first's committed children. Handling deadlocks is needed: see Unresolved 1.
+
+4. **A direct staff edit is the exception to all of the above.** An instructor or admin correcting
+   a status by hand may set any value, including a lower one, and the rows above the edited one are
+   recomputed and overwritten rather than merged, including any an earlier staff edit set by hand.
+   So a staff edit or a Django admin change is the only thing that can lower
+   a status, and a later grade change can raise what an edit lowered but can never re-lower what an
+   edit raised.
+
+Unresolved
+----------
+
+1. How to avoid deadlocks on competency group node locks that a) involve a grade change locking one group and
+   b) involve a grade change locking multiple groups, which happens when one subsection manifests as multiple leafs
+   in the same tree.
+2. How notifications, badges, and certificates learn that a row moved.
+
+Assumptions
+-----------
+
+1. Connections run at ``READ COMMITTED`` isolation. Decision 3 depends on it: the read taken after
+   the lock has to see current data rather than a snapshot from earlier in the transaction. MySQL's
+   own default is ``REPEATABLE READ``, but Django overrides it to ``READ COMMITTED`` on every
+   connection and edx-platform leaves that alone. A deployment that changes the setting breaks
+   decision 3 with no error and no failing test.
+
+Rejected Alternatives
+---------------------
+
+1. Prevent concurrent writes with a coarser lock, either deployment-wide or per-learner.
+
+    - Pros:
+        - Correctness comes from a single lock rather than from the merge argument in Decision 2,
+          so it is simpler to reason about.
+        - A per-learner lock (for example a database advisory lock keyed on a hash of the user id)
+          still lets different learners record in parallel, and gives the same per-learner
+          serialization the chosen design relies on.
+    - Cons:
+        - A single deployment-wide lock serializes recording across every learner, giving up the
+          throughput the design needs under bursty grading.
+        - A per-learner lock still serializes a single learner's independent competencies against
+          each other even when they never contend.
+        - Either lock adds lock-lifecycle machinery (acquisition, release, and handling a holder
+          that dies) across a very large key space.
+        - The chosen design needs no lock beyond the row locks the database already takes for the
+          statements it issues: the merge (Decision 2) makes each single-row write safe, and the
+          parent row lock (Decision 3) serializes only writers that actually contend for the same
+          parent row of the same learner, so different learners, and different competencies of one
+          learner, still record in parallel.
+
+2. Recompute derived levels on read instead of materializing them.
+
+    - Pros:
+        - Eliminates the derived group and competency status rows and the roll-up writes entirely,
+          leaving nothing to keep consistent on write.
+    - Cons:
+        - Moves the full bottom-up tree evaluation onto the hot read path, the opposite of what
+          dashboards and other read surfaces need (a direct indexed lookup).
+        - Settled against in :ref:`openedx-learning-adr-0002`.
+
+3. Send an event to openedx-core and update competency statuses in a separate celery task.
+
+    - Pros:
+        - Decouples the mastery update from the grade write, so grade recording does not depend on
+          competency code being installed or fast.
+    - Cons:
+        - Without a shared transaction, a failure or a lost event leaves the grade and its mastery
+          rows permanently out of sync (data drift), with no way to roll them back together.
+        - Writing the statuses in the same transaction as the grade (Decision 1) instead makes the
+          grade and its mastery consequences commit or fail as a unit.
+
+4. Commit the leaf, then re-read the leaves before rolling up, with no lock.
+
+    - Pros:
+        - Correct, and lock-free. Every writer commits its leaf before reading, so whichever writer
+          reads last sees every leaf already committed and computes the true value; the merge in
+          Decision 2 keeps it.
+    - Cons:
+        - The leaf has to commit before the roll-up reads, so the roll-up cannot share the grade's
+          transaction, which reopens the partial-failure window Decision 1 exists to close.
+
+5. Detect conflicts optimistically: a version column plus a unique constraint, and the losing write
+   retries.
+
+    - Pros:
+        - Contention costs a retry rather than a wait, so no writer ever blocks.
+    - Cons:
+        - Every writer needs conflict handling and a retry loop, and repeated contention on one
+          parent multiplies retries. The row lock in Decision 3 reaches the same result by waiting.