Incomplete instructions regarding 'Add New Bug Benchmark'

[jose]

Hi @nus-apr,

Although this page/document provides some initial instructions on how to add a new bug benchmark, it does not provide instructions for the second and third points listed.

• Benchmark Image: a Dockerfile describing how to construct the benchmark container
• Benchmark metadata file: A Json file containing an array of objects with the following features

(The last sentence ends weirdly). It would be great if you could provide more info.

[Marti2203]

Hi @jose ,
We appreciate your interest in Cerberus! I have pushed commit a111e52 to the dev branch with extra information in the aforementioned document, your feedback is highly appreciated

[jose]

Thanks @Marti2203 for your prompt reply.

I believe there is a typo in that commit. Where it's written

Create a new file in app/core/drivers/benchmarks/ with the Benchmark name (i.e. NewBenchmark.py) that contains the following code:

should be

Create a new file in app/drivers/benchmarks/ with the Benchmark name (i.e. NewBenchmark.py) that contains the following code:

right?

Another question that's not yet clear to me is, how does one create the meta-data.json file? Manually?

[Marti2203]

Hi,
Thank you for catching that incorrect path. I will fix it in a commit now. To create the meta-data.json file, you can construct it manually or make a script that generates it. For benchmarks, which are more uniform in structure and metadata, I have used a script (examples are ITSP and Refactory benchmarks) .