Skip to content

feat: preserve question dates and disclose CLI execution provenance #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
100yenadmin wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: preserve question dates and disclose CLI execution provenance #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c0f0d6d
feat: preserve benchmark dates and CLI provenance
Jul 21, 2026
5d07dc1
fix: fail closed on incomplete checkpoint copies
Jul 21, 2026
c9fde57
feat: fold hermes-lcm provider bridge overlay into harness PR
Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions src/judges/cli.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import type { LanguageModel } from "ai"
import type { Judge, JudgeConfig, JudgeInput, JudgeResult } from "../types/judge"
import type { ProviderPrompts } from "../types/prompts"
import { buildJudgePrompt, parseJudgeResponse, getJudgePrompt } from "./base"
import { logger } from "../utils/logger"
import { CliCallError, cliComplete, cliLlmModelId, type CliCallTelemetry } from "../utils/cli-llm"

export class CliJudge implements Judge {
name = "cli"

async initialize(_config: JudgeConfig): Promise<void> {
logger.info(`Initialized CLI judge (${cliLlmModelId("judge")})`)
}

async evaluate(input: JudgeInput): Promise<JudgeResult> {
let execution: CliCallTelemetry | undefined
try {
const text = await cliComplete(buildJudgePrompt(input), {
role: "judge",
retry: false,
onTelemetry: (telemetry) => {
execution = telemetry
},
})
return { ...parseJudgeResponse(text), execution }
} catch (error) {
if (execution && !(error instanceof CliCallError)) {
throw new CliCallError(error instanceof Error ? error.message : String(error), execution)
}
throw error
}
}

getPromptForQuestionType(questionType: string, providerPrompts?: ProviderPrompts): string {
return getJudgePrompt(questionType, providerPrompts)
}

getModel(): LanguageModel {
throw new Error("CliJudge.getModel() is unavailable for CLI execution")
}
}
48 changes: 32 additions & 16 deletions src/orchestrator/checkpoint.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ import {
mkdirSync,
rmSync,
readdirSync,
cpSync,
copyFileSync,
renameSync,
unlinkSync,
} from "fs"
import { join } from "path"
import { basename, join } from "path"
import type {
RunCheckpoint,
QuestionCheckpoint,
Expand Down Expand Up @@ -356,6 +356,8 @@ export class CheckpointManager {
benchmark: source.benchmark,
judge: overrides?.judge || source.judge,
answeringModel: overrides?.answeringModel || source.answeringModel,
answererProvenance: overrides?.answeringModel ? undefined : source.answererProvenance,
judgeProvenance: overrides?.judge ? undefined : source.judgeProvenance,
Comment on lines +359 to +360

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve provenance when copied phases are retained

When the server creates an advanced copy it always passes answeringModel || sourceCheckpoint.answeringModel and a required judgeModel, so these truthy checks clear both provenance fields even when the copied run starts from evaluate or report and the corresponding completed answer/evaluation phases are kept. That produces reports with retained CLI answers/judgments but no actual execution identity, which regresses the provenance this change is trying to preserve; only clear provenance when that phase is actually reset or the model really changes.

Useful? React with 👍 / 👎.

createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
limit: source.limit,
Expand All @@ -365,26 +367,40 @@ export class CheckpointManager {
questions: newQuestions,
}

// Create directories
// Create a fresh destination. A copy must never merge with a prior run.
const newRunPath = this.getRunPath(newRunId)
const newResultsDir = this.getResultsDir(newRunId)
if (!existsSync(newRunPath)) {
mkdirSync(newRunPath, { recursive: true })
}
if (!existsSync(newResultsDir)) {
mkdirSync(newResultsDir, { recursive: true })
if (existsSync(newRunPath)) {
throw new Error(`Destination checkpoint already exists: ${newRunId}`)
}
mkdirSync(newResultsDir, { recursive: true })

// Copy results directory if we're keeping search results (fromPhase is after search)
const sourceResultsDir = this.getResultsDir(sourceRunId)
if (existsSync(sourceResultsDir) && fromIndex > PHASE_ORDER.indexOf("search")) {
// Copy search results files
try {
cpSync(sourceResultsDir, newResultsDir, { recursive: true })
const preserveSearchResults = fromIndex > PHASE_ORDER.indexOf("search")
try {
if (preserveSearchResults) {
for (const question of Object.values(newQuestions)) {
const search = question.phases.search
if (search.status !== "completed") continue
const sourceFile = search.resultFile
if (!sourceFile || !existsSync(sourceFile)) {
throw new Error(
`completed search result is missing for ${question.questionId}: ${sourceFile || "no path"}`
)
}
const destinationFile = join(newResultsDir, basename(sourceFile))
copyFileSync(sourceFile, destinationFile)
if (!existsSync(destinationFile)) {
throw new Error(`completed search result was not copied for ${question.questionId}`)
}
search.resultFile = destinationFile
}
logger.info(`Copied results from ${sourceRunId} to ${newRunId}`)
} catch (e) {
logger.warn(`Failed to copy results: ${e}`)
}
} catch (error) {
rmSync(newRunPath, { recursive: true, force: true })
throw new Error(
`Failed to copy checkpoint ${sourceRunId} to ${newRunId}: ${error instanceof Error ? error.message : String(error)}`
)
}

this.save(newCheckpoint)
Expand Down
49 changes: 41 additions & 8 deletions src/orchestrator/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
import type { ProviderName } from "../types/provider"
import type { BenchmarkName } from "../types/benchmark"
import type { JudgeName } from "../types/judge"
import type { RunCheckpoint, SamplingConfig } from "../types/checkpoint"
import type { LlmExecutionProvenance, RunCheckpoint, SamplingConfig } from "../types/checkpoint"
import type { ConcurrencyConfig } from "../types/concurrency"
import { createProvider } from "../providers"
import { createBenchmark } from "../benchmarks"
import { createJudge } from "../judges"
import { CliJudge } from "../judges/cli"
import { CheckpointManager } from "./checkpoint"
import { getProviderConfig, getJudgeConfig } from "../utils/config"
import { resolveModel } from "../utils/models"
Expand All @@ -16,6 +17,8 @@ import { runSearchPhase } from "./phases/search"
import { runAnswerPhase } from "./phases/answer"
import { runEvaluatePhase } from "./phases/evaluate"
import { generateReport, saveReport, printReport } from "./phases/report"
import { questionCheckpointMetadata, syncQuestionCheckpointMetadata } from "./question-metadata"
import { cliLlmBackend, cliLlmProvenance } from "../utils/cli-llm"

export interface OrchestratorOptions {
provider: ProviderName
Expand Down Expand Up @@ -79,6 +82,21 @@ function selectQuestionsBySampling(
return allQuestions.map((q) => q.questionId)
}

function resolveExecutionProvenance(
configuredModel: string,
role: "answerer" | "judge"
): LlmExecutionProvenance {
const cli = cliLlmProvenance(role)
if (cli) return { ...cli, configuredModel, tokenizerModel: "gpt-4o" }

@evaos-code-review-bot evaos-code-review-bot Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Hardcoded tokenizerModel=gpt-4o for every CLI backend

resolveExecutionProvenance returns { ...cli, configuredModel, tokenizerModel: "gpt-4o" } for any CLI backend (line 90), including claude/non-OpenAI models. Token counts in the answer phase are still computed via countTokens(prompt, modelConfig) where modelConfig is getModelConfig(DEFAULT_ANSWERING_MODEL) (gpt-4o) on the CLI path (answer.ts:120-121), so the value is internally consistent today, but the provenance field asserts the tokenizer model is gpt-4o regardless of the actual backend. Either derive tokenizerModel from the resolved modelConfig actually used for counting, or document that gpt-4o tokenization is applied to all CLI backends by design, so downstream readers don't infer Anthropic-native token counts.

Category: Auth

Why this matters: Provenance fields are meant to disclose exactly how a run was executed; an unconditional tokenizer label can mislead analysis comparing token economics across codex vs claude backends.

const resolved = resolveModel(configuredModel)
return {
transport: "ai-sdk",
model: resolved.id,
modelExplicit: true,
configuredModel,
}
}

export class Orchestrator {
private checkpointManager: CheckpointManager

Expand Down Expand Up @@ -252,16 +270,29 @@ export class Orchestrator {

for (const q of questionsToInit) {
const containerTag = `${q.questionId}-${checkpoint.dataSourceRunId}`
this.checkpointManager.initQuestion(checkpoint, q.questionId, containerTag, {
question: q.question,
groundTruth: q.groundTruth,
questionType: q.questionType,
})
this.checkpointManager.initQuestion(
checkpoint,
q.questionId,
containerTag,
questionCheckpointMetadata(q)
)
}

this.checkpointManager.updateStatus(checkpoint, "running")
}

const metadataUpdates = syncQuestionCheckpointMetadata(checkpoint, allQuestions)
if (metadataUpdates > 0) {
logger.info(`Backfilled questionDate for ${metadataUpdates} checkpoint questions`)
}
if (phases.includes("answer")) {
checkpoint.answererProvenance = resolveExecutionProvenance(answeringModel, "answerer")
}
if (phases.includes("evaluate")) {
checkpoint.judgeProvenance = resolveExecutionProvenance(judgeModel, "judge")
Comment on lines +288 to +292

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve stored provenance when nothing reruns

On a normal resume the default phase list includes answer and evaluate, so this overwrites the checkpoint provenance before those phases discover that all questions are already completed. For AI SDK runs there is no per-call ledger to reconcile, so simply regenerating a report with a different -m/-j value or CLI environment can record the current labels as the actual answerer/judge even though the retained answers and judgments were produced earlier; only refresh these fields when the phase is actually reset or has pending work.

Useful? React with 👍 / 👎.

}
this.checkpointManager.save(checkpoint)

const provider = createProvider(providerName)
await provider.initialize(getProviderConfig(providerName))

Expand Down Expand Up @@ -300,8 +331,10 @@ export class Orchestrator {
}

if (phases.includes("evaluate")) {
const judge = createJudge(judgeName)
const judgeConfig = getJudgeConfig(judgeName)
const judge = cliLlmBackend() ? new CliJudge() : createJudge(judgeName)
const judgeConfig = cliLlmBackend()
? { apiKey: "", model: judgeModel }
: getJudgeConfig(judgeName)
judgeConfig.model = judgeModel
await judge.initialize(judgeConfig)
await runEvaluatePhase(
Expand Down
91 changes: 78 additions & 13 deletions src/orchestrator/phases/answer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,16 @@ import { buildContextString } from "../../types/prompts"
import { ConcurrentExecutor } from "../concurrent"
import { resolveConcurrency } from "../../types/concurrency"
import { countTokens } from "../../utils/tokens"
import {
cliCallTelemetryFromError,
cliCallsFromPhase,
cliComplete,
cliLlmBackend,
cliLlmModelId,
reconcileCliProvenanceIdentity,
summarizeCliLedger,
type CliCallTelemetry,
} from "../../utils/cli-llm"

type LanguageModel =
| ReturnType<typeof createOpenAI>
Expand Down Expand Up @@ -46,7 +56,7 @@ function getAnsweringModel(modelAlias: string): {
}
}

function buildAnswerPrompt(
export function buildAnswerPrompt(
question: string,
context: unknown[],
questionDate?: string,
Expand Down Expand Up @@ -79,6 +89,18 @@ export async function runAnswerPhase(
? questions.filter((q) => questionIds.includes(q.questionId))
: questions

const missingSearchResults = targetQuestions.filter((question) => {
const checkpointQuestion = checkpoint.questions[question.questionId]
if (!checkpointQuestion || checkpointQuestion.phases.answer.status === "completed") return false
const search = checkpointQuestion.phases.search
return search.status === "completed" && (!search.resultFile || !existsSync(search.resultFile))
})
if (missingSearchResults.length > 0) {
throw new Error(
`Cannot answer because a completed search result file is missing for: ${missingSearchResults.map((question) => question.questionId).join(", ")}`
)
}

const pendingQuestions = targetQuestions.filter((q) => {
const status = checkpointManager.getPhaseStatus(checkpoint, q.questionId, "answer")
const searchStatus = checkpointManager.getPhaseStatus(checkpoint, q.questionId, "search")
Expand All @@ -89,15 +111,19 @@ export async function runAnswerPhase(
})

if (pendingQuestions.length === 0) {
updateAnswerCliLedger(checkpoint, checkpointManager)
logger.info("No questions pending answering")
return
}

const { client, modelConfig } = getAnsweringModel(checkpoint.answeringModel)
const useCli = cliLlmBackend() !== null
const { client, modelConfig } = useCli
? { client: null, modelConfig: getModelConfig(DEFAULT_ANSWERING_MODEL) }
: getAnsweringModel(checkpoint.answeringModel)
const concurrency = resolveConcurrency("answer", checkpoint.concurrency, provider?.concurrency)

logger.info(
`Generating answers for ${pendingQuestions.length} questions using ${modelConfig.displayName} (concurrency: ${concurrency})...`
`Generating answers for ${pendingQuestions.length} questions using ${useCli ? cliLlmModelId("answerer") : modelConfig.displayName} (concurrency: ${concurrency})...`
)

await ConcurrentExecutor.execute(
Expand All @@ -107,6 +133,10 @@ export async function runAnswerPhase(
"answer",
async ({ item: question, index, total }) => {
const resultFile = checkpoint.questions[question.questionId].phases.search.resultFile!
const priorLlmCalls = cliCallsFromPhase(
checkpoint.questions[question.questionId].phases.answer
)
let llmCall: CliCallTelemetry | undefined

const startTime = Date.now()
checkpointManager.updatePhase(checkpoint, question.questionId, "answer", {
Expand All @@ -129,25 +159,36 @@ export async function runAnswerPhase(
// custom prompt functions that transform context (e.g. Zep's XML-like tags).
const contextTokens = Math.max(0, promptTokens - basePromptTokens)

const params: Record<string, unknown> = {
model: client(modelConfig.id),
prompt,
maxTokens: modelConfig.defaultMaxTokens,
}

if (modelConfig.supportsTemperature) {
params.temperature = modelConfig.defaultTemperature
let text: string
if (useCli) {
text = await cliComplete(prompt, {
role: "answerer",
retry: false,
Comment on lines +164 to +166

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor configured model for CLI answerer

When HERMES_MB_LLM_CLI is set, the normal -m/--answering-model value is still stored in checkpoint.answeringModel, but this CLI path never passes it to cliComplete. Unless a separate HERMES_MB_CODEX_MODEL/HERMES_MB_CLAUDE_MODEL env var is also set, the benchmark runs the subscription CLI's default unpinned model while the run arguments show the requested model; pass the checkpoint model as the CLI model override or reject the unsupported combination.

Useful? React with 👍 / 👎.

onTelemetry: (telemetry) => {
llmCall = telemetry
},
})
} else {
const params: Record<string, unknown> = {
model: client!(modelConfig.id),
prompt,
maxTokens: modelConfig.defaultMaxTokens,
}
if (modelConfig.supportsTemperature) {
params.temperature = modelConfig.defaultTemperature
}
text = (await generateText(params as Parameters<typeof generateText>[0])).text
}

const { text } = await generateText(params as Parameters<typeof generateText>[0])

const durationMs = Date.now() - startTime
checkpointManager.updatePhase(checkpoint, question.questionId, "answer", {
status: "completed",
hypothesis: text.trim(),
promptTokens,
basePromptTokens,
contextTokens,
llmCall,
llmCalls: llmCall ? [...priorLlmCalls, llmCall] : priorLlmCalls,

@evaos-code-review-bot evaos-code-review-bot Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Failed retries inflate the per-question llmCalls ledger without distinguishing completed vs failed attempts

On the success path the phase records llmCalls: llmCall ? [...priorLlmCalls, llmCall] : priorLlmCalls (line 191). When the CLI path is used with retry: false this is a single call, but cliComplete still aggregates usage across attempts and the appended telemetry represents the whole call. The concern is narrower but real on the failure path (line 209): llmCall ||= cliCallTelemetryFromError(e) only attaches telemetry when the error is a CliCallError. A cliComplete call that fails on the FIRST attempt with retry:false throws a CliCallError (good), but any non-CLI failure (e.g. prompt-build throw before cliComplete is awaited, or an upstream error) leaves llmCall undefined, so llmCalls is set to priorLlmCalls and the failed attempt is invisible to the ledger. summarizeCliLedger/callLedgerComplete (cli-llm.ts:130) then report the ledger as complete for a phase that actually failed mid-call. Confirm whether failed-but-started attempts should be recorded as a distinct status telemetry entry so callLedgerComplete cannot read 'true' over a silent dropped call.

Category: Runtime correctness

Why this matters: The CLI ledger/provenance is the disclosure mechanism this PR adds; a 'complete' ledger over a dropped attempt misrepresents which model actually produced (or failed to produce) an answer.

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Drop stale CLI calls after non-CLI answer retry

If a question has a failed CLI answer attempt and the run is resumed with HERMES_MB_LLM_CLI unset, the successful AI SDK answer leaves llmCall undefined, so this carries the old failed CLI ledger into the completed phase. The report reconciles provenance from any stored CLI call and will attribute that AI SDK answer to the previous CLI identity (and mark usage incomplete); clear old CLI calls when a non-CLI retry succeeds, or only preserve them when appending a new CLI call.

Useful? React with 👍 / 👎.

completedAt: new Date().toISOString(),
durationMs,
})
Expand All @@ -159,10 +200,13 @@ export async function runAnswerPhase(
)
return { questionId: question.questionId, durationMs }
} catch (e) {
llmCall ||= cliCallTelemetryFromError(e)
const error = e instanceof Error ? e.message : String(e)
checkpointManager.updatePhase(checkpoint, question.questionId, "answer", {
status: "failed",
error,
llmCall,
llmCalls: llmCall ? [...priorLlmCalls, llmCall] : priorLlmCalls,
completedAt: new Date().toISOString(),
durationMs: Date.now() - startTime,
})
Expand All @@ -173,5 +217,26 @@ export async function runAnswerPhase(
}
)

updateAnswerCliLedger(checkpoint, checkpointManager)

logger.success("Answer phase complete")
}

function updateAnswerCliLedger(
checkpoint: RunCheckpoint,
checkpointManager: CheckpointManager
): void {
if (!checkpoint.answererProvenance) return
const phases = Object.values(checkpoint.questions).map((question) => question.phases.answer)
const ledger = summarizeCliLedger(phases)
if (ledger.callCount === 0) return
Object.assign(checkpoint.answererProvenance, {
callCount: ledger.callCount,
retryCount: ledger.retryCount,
executionIdentityCount: ledger.executionIdentityCount,
mixedExecutionIdentity: ledger.mixedExecutionIdentity,
callLedgerComplete: ledger.callLedgerComplete,
})
reconcileCliProvenanceIdentity(checkpoint.answererProvenance, ledger.calls)
checkpointManager.save(checkpoint)
}
Loading