Skip to content

Two mislabeled gold answers in dev set (AOS/2004, VLO/2018) #5

Description

@ashalliants

While working with ConvFinQA I found two conversation turns in the dev set whose gold execution answers don't match the question being asked. In both cases the discrepancy is traceable to the turn-collapse/decomposition step, and the annotation fields in the same record contain the evidence for the correct answer.

1. Double_AOS/2004/page_11.pdf, turn 0

  • Question: "as of december 31, 2004, what percentage of the total of aggregate contractual obligations was due to long-term debt?"
  • Current turn_program[0] = "275.1", exe_ans_list[0] = 275.1 — this is the bare numerator, not a percentage.
  • The same record's original_program_0 = divide(275.1, 521.3) and step_list_0 = ["Ask for number 275.1", "Ask for number 521.3", "divide(275.1, 521.3)"]. When the two "n/a" ask-for-number turns were collapsed (dialogue_break_oridialogue_break), turn 0's program was aligned to the intermediate 275.1 step instead of the final divide.
  • Correct answer: divide(275.1, 521.3)0.5277 (52.77%). This is also consistent with the parallel turn 1 ("and what percentage of it was due to purchase obligations?"divide(177.3, 521.3)).

2. Double_VLO/2018/page_25.pdf, turn 0

  • Question: "what was the total number of shares purchased in november for the q4 ended 12/31/18?"
  • Current turn_program[0] = divide(3655945, 7673266), exe_ans_list[0] = 0.47645 — a ratio, but the question asks for a count.
  • The table value for November total shares purchased is 3,655,945, which is the answer to the question as phrased. (The divide(...) ratio would answer a "what percentage of total" question, not this one.)

Both reproduce in train.json / dev.json from the current data.zip. Happy to provide a small patch/errata file if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions