Skip to content

About

MultiDRECS: A Multi-Dimensional Requirement-Aligned Automated Evaluation Framework for Code Summarization

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

MultiDRECS

This repository is used to provide supplementary materials related to "MultiDPECS: A Multi-Dimensional Requirement-Aligned Automated Evaluation Framework for Code Summarization", which has been accepted by Internetware 2026.

Project Structure

  • /Benchmark
    • ConstructBench.py: Specific code for constructing the DiscrimEval test set
    • SampleData.jsonl: Raw data parsed from the project (346 samples), each containing id, code, reference summary, and context information
    • CodeSumBench.jsonl: The constructed DiscrimEval test set, including original fields and 9 summary variants per sample
  • CalculateReward.py: MultiDRECS implementation (Classify Agent, Consistency Agent, Relevance Agent)
  • LLM_eval.py: LLM-as-a-Judge evaluation (Reference-Based & Reference-Free modes)
  • test_rm.py: Evaluates CodeSumBench using MultiDPECS
  • test_llm.py: Evaluates CodeSumBench using LLM-as-a-Judge

Quickstart

Construct DiscrimEval Dataset

cd ./Benchmark
python ConstructBench.py -i ./SampleData.jsonl -o ./CodeSumBench.jsonl

Evaluate using our MultiDRECS

cd ..
python test_rm.py -d ./Benchmark/CodeSumBench.jsonl -m [model_name]

Evaluate using the LLM-as-a-Judge method

python test_llm.py -d ./Benchmark/CodeSumBench.jsonl -m [model_name] -r [rb/rf]

About

MultiDRECS: A Multi-Dimensional Requirement-Aligned Automated Evaluation Framework for Code Summarization

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages