Skip to content

Commit 6dd52c6

Browse files
committed
fix: pass questionDate into question checkpoint so answer prompts get the real date
The benchmark loaders (e.g. LongMemEval) put question_date into question.metadata.questionDate, and the answer phase reads checkpoint.questions[id].questionDate to fill the {{questionDate}} / 'Question Date:' slot in every answer prompt. But the orchestrator's initQuestion call never copied questionDate from the loaded question into the checkpoint, so the value was always undefined and every answer prompt rendered 'Question Date: Not specified'. Temporal-reasoning questions ('what did I buy 10 days ago?') are unanswerable without the question date, so temporal category scores were understated for every provider measured on the harness.
1 parent 118209a commit 6dd52c6

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

src/orchestrator/index.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,7 @@ export class Orchestrator {
242242
question: q.question,
243243
groundTruth: q.groundTruth,
244244
questionType: q.questionType,
245+
questionDate: q.metadata?.questionDate as string | undefined,
245246
})
246247
}
247248

0 commit comments

Comments
 (0)