Experiments
As a registered user of the platform who has uploaded a system which conforms the specification (API) of one of the benchmarks allows the user to benchmark the system.
The benchmarking of a system is done via the Benchmarks menu where at first the benchmark is selected to be used for the experiment. The drop down menu displays all possible benchmarks.
Having selected the benchmark, the system to be benchmarked is selected. Only systems uploaded by the user and fitting the API of the chosen benchmark are displayed. Then the benchmark experiment is configured by setting the benchmark specific parameters. These might vary amongst the different benchmarks due to their different nature. Parameters can be e.g. numeric values, string values, dates or even nominal with pre-defined values that can be selected by a drop-down box. Some of the values might also have restrictions.
When the experiment is configured it can be submitted via the Submit button. This button is inactive as long as the configuration is not completed. After successful submission a page with the submission details is presented.
Please note that refreshing this page always creates a new experiment. Therefore, please avoid refreshing commands.
Note that the created experiment has got an ID that is helpful to identify the experiment later on.
The status of the currently running experiment and the experiments waiting in a queue can be accessed by clicking on Experiments > Experiment Status.
Any user can explore published results of experiments. Additionally, registered users can explore unpublished results for their systems, e.g., results in a challenge that has not published its complete result list. Challenge owners are allowed to see the results of all experiments that belong to their challenge allowing them to see and announce the best performing system even if the results haven't been published.
In order to see the results the user selects the Experiment Results page of the Experiments menu item
The results page displays the details of finished experiments the user is allowed to explore.
One can select various experiments in order to compare the results of these experiments or only select one experiment to explore its result. The results are displayed after pressing the Show Details button. One additional option is to see all results of a specific challenge task. This is done via the selection of the challenges task name at the bottom of the challenge page.
The results are displayed next to each other for easy comparison with the parameter- (KPI-) names displayed on the left side of the table. If the experiment caused and logged an error, it is displayed as well. The error will be highlighted.
