…in evaluator

…valuator

…valuator

Co-authored-by: Teucher, Roman <[email protected]>

* adding generic measure class
* adding perplexity as well as cross entropy loss computations
* starting unit tests

done:
- integrated new aggregated measures into evaluator
- updated and expanded evaluator tests
- extracted throughput measurement into class
- fixed missing keys in measure implementations
- code clean up
todo:
- integration into trainer
- parameterization f eval losses/metrics
- testing the loss implementations (in particular perplexity)

…and_metrics_in_evaluator

* add aggregative measure factories to constructor of Evaluator
* extract big train method into smaller ones
* adapt test_evaluator to new params of changed Evaluator class
* gym: remove loss functions and metrics for evaluation from evaluation method

…ixtures

WIP: The current version does not yet work.

… configs

…xity impl and test

* add throughput aggregator factory callable; needed because otherwise it was intantiated in the class itself and was not mockable
* fix perplexity computation: now tracking losses, summing over them and then computing torch.exp(loss_sum/#samples)
*

Perplexity now gets computed correctly for each sequence in a batch and summed up afterwards.

Also: Small fix in perplexity tests.

… "mean".

Otherwise (with reduction="sum"), this would drastically impact training loss.
Instead, for accumulating and logging the training loss, the added losses now get divided by number of batches instead of the added batch sizes.

Also, small refactoring of corresponding tests.

…_losses_and_metrics_in_evaluator

# Conflicts:
#	config_files/config_example_hf_meditron_7B_instruction.yaml
#	config_files/config_example_mem_map_dataset.yaml
#	config_files/config_lorem_ipsum.yaml
#	src/modalities/__main__.py
#	src/modalities/config/config.py
#	src/modalities/config/lookup_types.py
#	src/modalities/evaluator.py
#	src/modalities/resolver_register.py
#	src/modalities/trainer.py
#	tests/conftest.py

…size

…ltiple_losses_and_metrics_in_evaluator

…egate().

…d of Evaluator.

… compute() to aggregate().

…n Trainer instead of iterable decorator.

This code should be better readable than using start_throughput_measurement().

…ltiple_losses_and_metrics_in_evaluator

…ltiple_losses_and_metrics_in_evaluator