This repository is used to provide supplementary materials related to "MultiDPECS: A Multi-Dimensional Requirement-Aligned Automated Evaluation Framework for Code Summarization", which has been accepted by Internetware 2026.
- /Benchmark
- ConstructBench.py: Specific code for constructing the DiscrimEval test set
- SampleData.jsonl: Raw data parsed from the project (346 samples), each containing
id,code,reference summary, andcontext information - CodeSumBench.jsonl: The constructed DiscrimEval test set, including original fields and 9 summary variants per sample
- CalculateReward.py: MultiDRECS implementation (Classify Agent, Consistency Agent, Relevance Agent)
- LLM_eval.py: LLM-as-a-Judge evaluation (Reference-Based & Reference-Free modes)
- test_rm.py: Evaluates CodeSumBench using MultiDPECS
- test_llm.py: Evaluates CodeSumBench using LLM-as-a-Judge
cd ./Benchmark
python ConstructBench.py -i ./SampleData.jsonl -o ./CodeSumBench.jsonlcd ..
python test_rm.py -d ./Benchmark/CodeSumBench.jsonl -m [model_name]python test_llm.py -d ./Benchmark/CodeSumBench.jsonl -m [model_name] -r [rb/rf]