RFC: Store Hyperion Consistency Checks as Exercise Reviews (multi-location)

[FelixTJDietrich]

Status: Draft
Depends on: #10967 (versioning)
Related: #10969 (inline reviews) · #11339 (consistency check UX)

Summary

Treat AI-found consistency issues as first-class review comments so they persist across reloads, drive unresolved counts, and support instructor actions.

Motivation (Why)

• Make AI findings durable and actionable (approve/dismiss/resolve/unresolve).
• Show unresolved counts in file tree and repository switcher.
• Correlate runs with Langfuse/OTel for debugging and product insights.

Proposal (Approach)

• Each Hyperion run creates one ExerciseReview for the active ExerciseVersion with review_type = AI_CONSISTENCY.
• Each AI “consistency issue” becomes an ExerciseReviewComment on that review.
• Instructor actions update comment status; unresolved counts derive from status.
• Support issues that span multiple places (statement + files) via a single JSONB locations array (no extra tables).

Schema delta (builds on #10969)

erDiagram
    Exercise ||--o{ ExerciseVersion : "has versions"
    ExerciseVersion ||--o{ ExerciseReview : "reviewed in"
    ExerciseReview ||--o{ ExerciseReviewComment : "contains"
    User ||--o{ ExerciseReview : "authors"

    Exercise {
        long id PK
    }

    ExerciseVersion {
        long id PK
        long exercise_id FK
        text problem_statement
        text template_commit_sha
        text solution_commit_sha
        text tests_commit_sha
        timestamp created_at
    }

    ExerciseReview {
        long id PK
        long exercise_version_id FK
        long author_id FK "user who triggered the consistency check"
        enum review_type "MANUAL | AI_CONSISTENCY"
        jsonb ai_metadata "model, prompt_version, trace_id (nullable)"
        timestamp submitted_at "for AI: set at creation"
    }

    ExerciseReviewComment {
        long id PK
        long review_id FK
        enum status "OPEN | APPROVED | DISMISSED | RESOLVED"
        jsonb locations "[] of {component_type, path?, start_line?, end_line?, start_offset?, end_offset?}"
        enum consistency_issue_severity "LOW | MEDIUM | HIGH (nullable)"
        enum consistency_issue_category "<CONSISTENCY ISSUE TYPES> (nullable)"
        text body
    }

    User {
        long id PK
    }

Loading

locations example

[
  {"component_type":"PROBLEM_STATEMENT","start_line":10,"end_line":12},
  {"component_type":"REPOSITORY_TESTS","path":"test/FooTest.java","start_line":5,"end_line":9}
]

Counting & Telemetry

• Unresolved counts: status IN (OPEN, APPROVED); exclude DISMISSED, RESOLVED.
• Tracing: store model/prompt/trace in ExerciseReview.ai_metadata for Langfuse/OTel correlation.

Feedback welcome.

[wasnertobias]

ExerciseReviewComment needs suggested_fix and probably an extension of suggested_patch as well!

Edit: Ok, suggested_fix could also be simply saved in the attribute text instead.