While working with ConvFinQA I found two conversation turns in the dev set whose gold execution answers don't match the question being asked. In both cases the discrepancy is traceable to the turn-collapse/decomposition step, and the annotation fields in the same record contain the evidence for the correct answer.
1. Double_AOS/2004/page_11.pdf, turn 0
- Question: "as of december 31, 2004, what percentage of the total of aggregate contractual obligations was due to long-term debt?"
- Current
turn_program[0] = "275.1", exe_ans_list[0] = 275.1 — this is the bare numerator, not a percentage.
- The same record's
original_program_0 = divide(275.1, 521.3) and step_list_0 = ["Ask for number 275.1", "Ask for number 521.3", "divide(275.1, 521.3)"]. When the two "n/a" ask-for-number turns were collapsed (dialogue_break_ori → dialogue_break), turn 0's program was aligned to the intermediate 275.1 step instead of the final divide.
- Correct answer:
divide(275.1, 521.3) ≈ 0.5277 (52.77%). This is also consistent with the parallel turn 1 ("and what percentage of it was due to purchase obligations?" → divide(177.3, 521.3)).
2. Double_VLO/2018/page_25.pdf, turn 0
- Question: "what was the total number of shares purchased in november for the q4 ended 12/31/18?"
- Current
turn_program[0] = divide(3655945, 7673266), exe_ans_list[0] = 0.47645 — a ratio, but the question asks for a count.
- The table value for November total shares purchased is 3,655,945, which is the answer to the question as phrased. (The
divide(...) ratio would answer a "what percentage of total" question, not this one.)
Both reproduce in train.json / dev.json from the current data.zip. Happy to provide a small patch/errata file if useful.
While working with ConvFinQA I found two conversation turns in the dev set whose gold execution answers don't match the question being asked. In both cases the discrepancy is traceable to the turn-collapse/decomposition step, and the
annotationfields in the same record contain the evidence for the correct answer.1.
Double_AOS/2004/page_11.pdf, turn 0turn_program[0]="275.1",exe_ans_list[0]=275.1— this is the bare numerator, not a percentage.original_program_0=divide(275.1, 521.3)andstep_list_0=["Ask for number 275.1", "Ask for number 521.3", "divide(275.1, 521.3)"]. When the two"n/a"ask-for-number turns were collapsed (dialogue_break_ori→dialogue_break), turn 0's program was aligned to the intermediate275.1step instead of the finaldivide.divide(275.1, 521.3)≈ 0.5277 (52.77%). This is also consistent with the parallel turn 1 ("and what percentage of it was due to purchase obligations?" →divide(177.3, 521.3)).2.
Double_VLO/2018/page_25.pdf, turn 0turn_program[0]=divide(3655945, 7673266),exe_ans_list[0]=0.47645— a ratio, but the question asks for a count.divide(...)ratio would answer a "what percentage of total" question, not this one.)Both reproduce in
train.json/dev.jsonfrom the currentdata.zip. Happy to provide a small patch/errata file if useful.