Competency Status Storage ADR

[jesperhodge]

UPDATE: ADRs in this PR have been completely rewritten as of July 17, 2026.

Strong involvement from Claude (AI).

I haven't fully reviewed changes to diagrams and ADRs 2 and 3 yet, made with Claude, need my own review still.

Summary

These are some significant changes to the data model and approach. They will warrant a good amount of discussion.
A main concern is that the leaf node table and leaf node history table could possibly contain billions of rows, at the scale of the PersistentSubsectionGrade table of edx-platform. The current solution accepts this, but is this acceptable for OpenEdx? If not, there is also a rejected alternative to not store leaf nodes and just compute them at read time, but that would make it harder to keep subsection statuses frozen so that later competency criteria changes don't change the status.

ADR 4: How learner competency mastery is recorded
correctly under concurrent, out-of-order grade-change events without
paying a per-event serialization cost at scale.

ADR 5: How competency criteria statuses for learners are stored at scale, given the tables
can be massive (billions of rows).

[openedx-webhooks]

Thanks for the pull request, @jesperhodge!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[mgwozdz-unicon]

Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.

I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.

Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.

Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.

[jesperhodge]

Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.

I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.

Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.

Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.

@mgwozdz-unicon I'll improve the description in the ADR.

Here's why this solution is not good:

It fights the batching that the performance depends on. The chosen design's throughput comes from batching across learners: one bulk read of current statuses, evaluate the whole batch in memory, one bulk write — a few round-trips regardless of how many learners/events are in the batch. A per-learner lock is naturally per-learner (in the earlier design, per-event): acquire lock → read that learner → evaluate → write → release, one learner at a time. That reintroduces per-learner (or per-event) transaction/commit overhead plus lock acquire/release churn, which is exactly the overhead batching was collapsing. Under bursty grading across many learners, that per-unit overhead dominates.

Lock-lifecycle machinery would be multiplied across millions of keys. One deployment-wide lock has exactly one lifecycle to run: acquisition, timeout, stale-lock recovery on a crashed worker. A per-learner lock needs a per-learner lock table (or keyed advisory locks) and that same lifecycle replicated across potentially millions of learner keys, held and waited on concurrently, plus a connection+worker tied up per contended lock.

[jesperhodge]

@ormsbee @mgwozdz-unicon @thelmick-unicon two open questions:

• Do we need the history table to keep records forever, or can we auto-truncate it after a certain period?
• If we only store advancing states in the history table, does that conflict with any edge cases? For example, if a learner grade gets corrected by an instructor, or a django admin corrects mistaken entries - does the correction need to be traceable in the history table?

[mgwozdz-unicon]

• Do we need the history table to keep records forever, or can we auto-truncate it after a certain period?

I think it would be safe to auto-truncate as long as the instance owner is able to control what the auto-truncation period is.

• If we only store advancing states in the history table, does that conflict with any edge cases? For example, if a learner grade gets corrected by an instructor, or a django admin corrects mistaken entries - does the correction need to be traceable in the history table?

I think we would want corrections and edits traceable in the history table as well.

[jesperhodge]

I think it would be safe to auto-truncate as long as the instance owner is able to control what the auto-truncation period is.

@mgwozdz-unicon I'll defer to you on this, since I don't really know what the pieces are that need to be read off of the history table. -> So I'll add auto-truncation, but what should the mechanism for instance owners be - an environment variable? And what should the default setting for this be?

I think we would want corrections and edits traceable in the history table as well.

@mgwozdz-unicon how about we make it so there are two routes for this?

1. we record only advances, no repeat attempts or evaluations, and we don't allow downwards corrections that result from criteria tree changes. This keeps the history table small
2. Only in the case where the assignment is directly changed by a course instructor or admin we record this change in the django history (even if it's downward).

[jesperhodge]

@ormsbee I addressed your feedback; only the two open questions discussed with Mary above are not handled yet.

[thelmick-unicon]

@jesperhodge

Do we need the history table to keep records forever, or can we auto-truncate it after a certain period?

I agree with Mary, this is going to be variable by institution, depending on when they will want to truncate the history. As for a default, my first thought is 2 years, but I think that's open to discussion. This might be a good question for Jason.

If we only store advancing states in the history table, does that conflict with any edge cases? For example, if a learner grade gets corrected by an instructor, or a django admin corrects mistaken entries - does the correction need to be traceable in the history table?

Again, I agree with Mary that manual changes should be captured and traceable in the history.

how about we make it so there are two routes for this?
we record only advances, no repeat attempts or evaluations, and we don't allow downwards corrections that result from criteria tree changes. This keeps the history table small
Only in the case where the assignment is directly changed by a course instructor or admin we record this change in the django history (even if it's downward).

Feedback we received from institutions was that they explicitly wanted to capture all repeat attempts and evaluations so both the instructor and student can see all attempts to evaluate if progress is being made or to review feedback from previous attempts.

[mgwozdz-unicon]

I think it would be safe to auto-truncate as long as the instance owner is able to control what the auto-truncation period is.

@mgwozdz-unicon I'll defer to you on this, since I don't really know what the pieces are that need to be read off of the history table. -> So I'll add auto-truncation, but what should the mechanism for instance owners be - an environment variable? And what should the default setting for this be?

Yes, I think an environment variable sounds good. Claude is recommending a Django setting as a precedent in the codebase. The 2 years Tammie suggested sounds good to me, though that's probably an implementation detail that can get worked out later. I put a question out to Jason about it.

I think we would want corrections and edits traceable in the history table as well.

@mgwozdz-unicon how about we make it so there are two routes for this?

1. we record only advances, no repeat attempts or evaluations, and we don't allow downwards corrections that result from criteria tree changes. This keeps the history table small
2. Only in the case where the assignment is directly changed by a course instructor or admin we record this change in the django history (even if it's downward).

I agree with Tammie that we definitely want to keep data for repeat attempts/evaluations. In general, we probably do want to allow recording downwards corrections whether the correction was authored by a person or by software since I'm pretty sure I recall that D2L Brightspace supports displaying a log of all grade changes to an assignment over time whether it was changed by a system or a specific person. That way we can support parity with what institutions already expect.

I think the main consolation here is that ADR 3 says "If a user edits competency criteria definitions or competency object/tag associations after related learner status exists, Studio must display an explicit warning that student statuses have already been set, and these changes will be applied going forward, so existing learner statuses will not be retroactively updated." So at least they won't get retroactively updated by the software in that case.

[jesperhodge]

@mgwozdz-unicon @thelmick-unicon The platform stores attempts and their scoring already, right? Why can't we just pull the scores and history from that table without saving the skill status updates for attempts? I would like to avoid adding any of this to the history table.
Unless there's anything that the platform does not store already?

Followup: if you can specify exactly what data we want from the audit trail, maybe we don't need to store history at all? Instead just pull everything from the existing data on attempted subsections and such.

[mgwozdz-unicon]

@mgwozdz-unicon @thelmick-unicon The platform stores attempts and their scoring already, right? Why can't we just pull the scores and history from that table without saving the skill status updates for attempts? I would like to avoid adding any of this to the history table. Unless there's anything that the platform does not store already?

Followup: if you can specify exactly what data we want from the audit trail, maybe we don't need to store history at all? Instead just pull everything from the existing data on attempted subsections and such.

@jesperhodge The platform's attempt/submission data doesn't carry any notion of competency or criteria, that mapping only exists here. So pulling from it doesn't remove the storage need, it just relocates it: we'd still need to record which CompetencyCriteria (and which version of it) applied when a status was evaluated. Without that, this collapses into rejected alternative 1 in this same ADR (transient computation), which we already ruled out because retroactive criteria edits would make recomputed history wrong, there's no frozen record of what was true at the time. Separately, the ADR already frames this as covering "any of the associated measurement instruments," not just gradeable subsections, so keeping competency status/history in one place avoids re-deriving this pattern per instrument as new measurement types get added.

[jesperhodge]

I've worked with Claude to come up with some sort of suggestion on how to handle this. Here is a summary:
"
Two different things are being kept in one table, and they want opposite things. One is a record of mastery: this learner reached Demonstrated on this criterion at this time. It stays small, and it should be kept forever, because it is effectively a transcript. The other is a record of activity: every attempt, every score, the feedback an instructor wrote. It grows without limit, and it is the kind of data that ought to expire. That mismatch, not the size, is the real problem. The request for full attempt history is also not scope creep. ADR 0005 says the history tables exist to understand "why a learner did or didn't achieve mastery," then records only advances, so a learner who tried five times and never passed leaves no trace, and the design cannot answer its own question. The recommendation is to keep the mastery record in ADR 0005, and to add a few counters to the row it already writes: attempts, and best score. Those reveal whether a learner is making progress, and they cost nothing, because the row is written anyway. The attempt log itself belongs in a separate ADR, where it can have its own retention policy, and where the harder question about how to store the record of activity. Note that written instructor feedback lives in ORA anyway, which is not currently owned by openedx-core.
"

[jesperhodge]

Okay, some new perspectives.
I investigated two directions with Claude:
A) ways to compact, aggregate, and offload the history to reduce size / migration complications;
B) my doubts on whether we need the history at all - Claude's evaluation is we do not need it. (More promising avenue.)

More details:
A) in short, there are a couple alternatives here. 1. We can use analytics tools like aspects / clickhouse to completely avoid MySQL; 2. we can compress rows older than 1 month into (user_id, course_id, month) rows that use compression algorithms, which saves a huge amount of space; 3. we can do the same thing as in 2 but offload everything older than a month to S3 as well.

B)
I have my doubts we need the history table for leaf criteria statuses at all, because it mainly reflects data that is already stored elsewhere. I had Claude analyze how we could avoid the history table, and it's opinion is also that the history table is not needed. Here is a summary of its analysis:

"We do not need the huge attempt-history table, and dropping it removes the whole scaling problem. The audit requirement bundled together three different things: every problem attempt a learner makes, every recalculation of a subsection score, and the much smaller record of when a learner's competency status actually changed. edx-platform already stores the first one and always has, so duplicating it into our competency tables would be paying for tens of billions of rows twice. The genuine concern behind the request is not volume but traceability: when we say a learner earned a competency, we need to know what the criteria looked like at that moment, since criteria can be edited later. That is a question about authored definitions, not learner activity, and definitions are tiny by comparison. So the fix is to keep our small advance-only history and have each recorded advance point at the criteria configuration that produced it. Importantly that is a pointer, not a copy: the learner row gains a single column, while the configuration it names is stored once and shared by every learner evaluated against it, so that side grows with authoring edits rather than with learners. At the leaf level we add almost nothing, because ADR 0003 already versions criteria definitions, so the pointer is simply the version number of something we are keeping anyway. This is the same trick edx-platform already uses to freeze what content made up a graded subsection. Instructor reviews get their own separate table, because their volume is bounded by how much staff actually write, not by learner activity. The cost is that some audit questions become slow, awkward, cross-database queries, which we have agreed is acceptable. In exchange we avoid partitioning, sharding, archiving to S3, and multi-terabyte schema migrations entirely."

@mgwozdz-unicon @ormsbee thoughts?

[kdmccormick]

try to write in plainer english and cut right to the point.

here is my understanding of points 1-3. let me know if you disagree.

1. Writes to a node's status will only ever advance its value. Roughly, status := max(stored status, newly computed status). As a result, out-of-order delivery and re-delivery are harmless.
2. When a child's value advances, its parent will be recomputed in the same transaction. Parent nodes may have conjunctive mastery rules like "demonstrated only when all children are demonstrated". Imagine that two child nodes share a parent, and each child advanced in mastery at the same time. If each child tried to recompute the parent's mastery at the same time, then they might use the other child's outdated mastery value. To protect against this, a node that is advanced in mastery should take a row-level lock and update just its immediate parent in the same transaction. Two updates that touch the same ancestors will essentially take turns turns up the path to the root, in a consistent order, so concurrent updates cannot deadlock.
3. Subsection grade changes will trigger the mastery computation. openedx-platform
   computes subsection grades in an async celery task (recalculate_subsection_grade_v3) triggered by a score-change signal. After that task writes the subsection grade, it will call a public openedx-core function within the same transaction to recompute mastery as described in 1 and 2. As other kinds competency criteria are defined (e.g. completion), additional triggers will be added.

i know it can be very challenging to distill text like this--it took me 20 minutes write this comment :P but the practice of doing it will force you to refine your ideas and it will help dave and I review more quickly.

[jesperhodge]

Thanks so much for doing that! Sounds good, I'll take another pass reviewing / improving the language. This indeed looks much better!

It's great to have an example of how this text can be improved. Now that I've looked at it, if you ever see something like this where text is not clear enough or needs to be more understandable / more in plain English - in the future, you can also just leave a comment for me to rewrite it. No need to dig around for you to decipher what I mean or find suggestions yourself - if it's not clear enough, I can rewrite so it becomes clear, even before you review the content.

[kdmccormick]

in this case, the exercise of writing it helped me come up to speed on the content of the ADR. but in the future i'll just ask, thanks!

[jesperhodge]

Pulled ADR 4 out into #713 .

[mgwozdz-unicon]

@jesperhodge Thank you for updating this PR based off of the consensus I'm seeing in Slack that we don't need to store learner competency status history for MVP.

After contemplating this a bit, I'm getting the impression that we may not need ADR 5 at all anymore given that the decision was to not store history for student competency statuses. I think it would be cleaner to just update Decision 5 of ADR 3 to say that history won't be stored for the learner status tables for MVP and just add some Rejected Alternatives covering everything that has happened in this PR.

Following are some suggestions for what updates to make, but you don't need to consider them set in stone if you feel there are some other points that are important to include that I have missed or if you'd rather state something differently:

Suggested Decision 5 replacement (0003-competency-criteria-versioning.rst)

5. Do not store history for learner competency status tables, and update rows in place.

   For ``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and
   ``StudentCompetencyStatus``, each row is updated in place when a learner's status
   changes. There is no history of prior status values beyond the ``modified`` timestamp,
   and no separate history table.

   Open edX has no concept of gradeable subsection attempts. This means that an attempt will actually be defined at the level of an individual problem, which can result in tens of billions of rows for a large Open edX instance. That scale creates real operational burden (schema migrations, backups, truncation and retention policy). Therefore, we did additional market research and found that storing history for learner competency status will not be required by the initial pilot partners for the MVP of the CBE implementation, and it would be safe to add later if needed.

Suggested Rejected Alternatives additions (append after existing item 4)

5. Keep the learner status tables append-only, storing every status change as a new row
   (the design this replaces).

    - Pros:
        - Every write is an insert rather than a read-modify-write, so there is no current
          row to keep consistent.
        - Preserves a full audit trail of every status change.
    - Cons:
        - No MVP requirement calls for this history.
        - Grows the leaf table by a further multiplier of problem attempts per learner per
          leaf, reaching tens of billions of rows for a large instance.
        - A read must resolve the latest row for a learner and node rather than reading one
          in-place row, which is more expensive and more complex.

6. Compute leaves transiently, never store them.

    - Pros:
        - Eliminates the largest table, since leaf demonstration would be computed on
          demand from the leaf's rule and the learner's grade.
    - Cons:
        - A recomputed leaf reflects the rule as it stands now, not the rule in force when
          the learner was graded, which contradicts Decision 4 above and can silently
          lower a status.

7. Store child evaluations on the parent group row instead of a leaf table.

    - Pros:
        - Avoids the largest table entirely.
    - Cons:
        - A leaf write becomes a read-modify-write of a column shared with every sibling.
        - No unique index or foreign key behind a status packed into a per-group array.
        - Couples a leaf's frozen mastery to the current shape of the criteria tree.
        - Removes the ability to individually track competency status progress by learning object (e.g. subsection)

8. Put the leaf table behind its own database alias/router, a separate physical database,
   or native partitioning/sharding, from the start.

    - Pros:
        - Physically isolates or splits the largest table from the start.
    - Cons:
        - A second alias gives up the atomicity these writes need with the grade write.
        - Imposes real operational cost on every deployment with nothing measured to
          justify it, and remains available later if a specific need is proven.

9. Serve heavy leaf-table reads from a read replica.

    - Pros:
        - Keeps dashboard and reporting reads off the primary.
    - Cons:
        - Premature: no measurement shows the primary struggling, and these are point
          lookups on a composite index, not the wide, expensive reads that drive
          ``StudentModule`` load in edx-platform.
        - Adding it later is a per-query choice, not a schema decision.

10. Give the leaf table a custom unsigned 64-bit primary key.

    - Cons:
        - ``BigAutoField``'s range is already far out of reach for this table.
        - Unsigned integers do not exist in PostgreSQL, and the custom field type carries
          ongoing maintenance cost for no real benefit at this scale.

11. Drop the database-level foreign key constraint on the learner column.

    - Cons:
        - This repo's convention is a real foreign key to ``settings.AUTH_USER_MODEL``.
          Reports of contention on the user row exist elsewhere in the ecosystem, but are
          not understood well enough here to design around.

[mgwozdz-unicon]

Thank you for getting those changes in!

[ormsbee]

Approved, but it looks like it will need manual rebase before it can be merged.