Skip to content

REPORT.md

Latest commit

 

History

History
131 lines (66 loc) · 6.94 KB

REPORT.md

File metadata and controls

131 lines (66 loc) · 6.94 KB
Raw

ConvFinQA Report

Before executing the program, please update your Anthropic API key in the .env file:

ANTHROPIC_API_KEY=your_api_key

Method

The given dataset is a conversational financial Q&A dataset containing numerical reasoning questions across multiple dialogue turns.

A traditional RAG system is not well suited for this problem because it treats each question as an independent query and does not retain prior conversational context.

Therefore, I considered two alternative approaches:

  1. Building an agent-based system with LangGraph

  2. Using an LLM with built-in tools for reasoning and arithmetic operations

The LangGraph's agent-based solution was considered but rejected for this prototype as it require predefined condition flow which doesnt suit this problem as the tools used here are dynamic. It also became complicated because the conversations do not follow a fixed pattern. Therefore, I chose the LLM + Tools approach for solving ConvFinQA.

Tech Stack Used

  1. Programming Language: Python

  2. LLM Model: Anthropic Claude Haiku 4.5 (claude-haiku-4-5-20251001)

Execution Flow

Setup:

  1. Define the arithmetic tools required for numerical reasoning: Add, Subtract, Multiply and Divide.

  2. JSON schema is defined for each tool containing the tool name, description, input parameters, data types and required fields.

  3. execute_tools() function is defined which performs the corresponding arithmetic operation whenever a tool is called.

Runtime:

  1. The user provides a documentID and runs the command:

     uv run main chat <record_id>

  2. The program reads the JSON dataset and retrieves the matching record. It then extracts the pretext, posttext and table data.

  3. The table data is reformatted into a clearer structure so the LLM can easily understand row headers, column headers, numerical values and their relationships.

  4. A prompt is created containing reasoning instructions, response rules and the extracted pretext, table and posttext content.

  5. Since the model can call tools multiple times, tool usage is restricted to a maximum of 10 steps for a single question. This acts as a guardrail to prevent infinite loops and excessive token cost.

  6. The Anthropic Claude Haiku model is executed with the following parameters:

    a. Tools enabled

    b. Maximum output tokens: 1000

    c. Temperature: 0 (for grounded responses as this is a financial dataset)

    d. Message history (The user query and conversation history are passed to the model)

    e. System prompt

  7. When the model determines that a tool is required, it stops generation, selects the appropriate tool, performs the arithmetic operation and receives the result.

  8. The tool output is appended to conversation history so the model can continue reasoning and use other tools if required.

  9. Once the final answer is computed, the response is displayed to the user.

Evaluation:

For evaluation, I selected 200 random sample documents.

The outputs were compared using the following rules:

  1. If the expected result was a string (Yes/No), the model response was directly compared.

  2. If the expected result was an integer or numeric value, both values were converted to float and compared.

  3. If the expected result was a decimal value, comparison was performed using a tolerance of 0.01.

Turn-wise Accuracy:

Turn 1: 75.00% (150/200)

Turn 2: 72.00% (144/200)

Turn 3: 72.19% (109/151)

Turn 4: 72.28% (73/101)

Turn 5: 53.70% (29/54)

Turn 6: 63.64% (7/11)

Turn 7: 100.00% (2/2)

Turn 8: 0.00% (0/1)

Turns 9 and 10 had no available samples.

Error Analysis

Evaluation was performed on 200 sample records and the following failure patterns were observed.

  1. For percentage related questions, the model sometimes automatically converted ratio values into percentages. For example, instead of returning 0.141, it returned 14.1%. This behavior is beneficial in some cases, as it suggests the model has been pretrained to present outputs in a human friendly percentage format. However, during evaluation, this caused two issues:

    a. Mismatch with Ground Truth Labels:

    When ratios were automatically converted into percentages, the output no longer matched the executed_answers field in the dataset, which was used for direct comparison. This reduced the measured accuracy even though the answer was mathematically equivalent from a human perspective.

    b. Error in Multi-turn Conversations:

    A more critical issue was that converted percentage outputs were sometimes reused in later conversational turns. If a later question depended on the earlier ratio value, using the converted percentage instead could lead to incorrect calculations.

    This issue was mitigated by adding stricter prompt instructions requiring the model to preserve the original numeric format unless explicitly asked to convert it.

  2. Several conversational turns require the result from a previous step as an input for the next calculation. If the model predicts an incorrect value in an earlier turn, then subsequent turns that depend on that result are also likely to be incorrect. This effect was observed during evaluation, where accuracy gradually decreased in later turns as errors from previous calculations passed through the conversation history.

Future Work

  1. This tool uses a simple prompt, I am currently working on creating a complex prompt to define edge cases, gaurdrails and safe response.

  2. Structured JSON Response: Currently, the model is prompted to return outputs in either numeric format or Yes/No responses. This can be further improved by updating the prompt to require a structured JSON response format. Using structured JSON outputs would make response parsing more reliable, simplify automated evaluation and reduce ambiguity in model-generated answers.

  3. Multi document analysis: At present, the model is designed for single-document analysis. This can be extended to support multi-document analysis by providing multiple documents as input and enabling comparison based questions across those documents. Such an enhancement would allow the system to perform cross-document reasoning, comparative financial analysis and more advanced conversational queries involving multiple sources.

  4. Enhanced Error analysis: At present, the evaluation is based on 200 random samples and the accuracy for each conversational turn is reported as turn-wise accuracy. This analysis can be further improved by identifying which turn failures were independent errors and which occurred due to incorrect results carried forward from previous turns.

Additionally, the dataset includes turn-level program information that can be used to verify whether the model selected the appropriate tools for each step. If the input value was incorrect due to an earlier mistake, but the model still chose the correct tool sequence for the current turn, then that turn may be considered partially correct from a reasoning perspective.

Incorporating these deeper diagnostics would provide a more accurate understanding of model performance beyond simple answer matching.