[MNT] Unit testing revamp part 2: classification

[MatthewMiddlehurst]

Follows on from #1479, part of AEP 05.

• Removes the current test_all_classiifers and deep learning classifier tests.
• Replaces them with a yielf function in the new testing framework.
• Reworks the test generator to create test instances while collecting tests. This allows us to avoid creating multiple instances for tests which only need one or a class.
• Reworks testing data to contain both a train/test set and data/labels for each key.
• Updates the rng generation and results for Arsenal and related estimators, which failed the deterministic test on the new version.

pytest -k 'ProximityTree' -vv run:

Before:

collected 47930 items / 47890 deselected / 40 selected                                                                                                                                                                                 

aeon/classification/tests/test_all_classifiers.py::TestAllClassifiers::test_does_not_override_final_methods[ProximityTree] PASSED                                                                                                [  2%]
aeon/classification/tests/test_all_classifiers.py::TestAllClassifiers::test_classifier_against_expected_results[ProximityTree] PASSED                                                                                            [  5%] 
aeon/classification/tests/test_all_classifiers.py::TestAllClassifiers::test_contracted_classifier[ProximityTree] PASSED                                                                                                          [  7%] 
aeon/classification/tests/test_all_classifiers.py::TestAllClassifiers::test_classifier_tags_consistent[ProximityTree] PASSED                                                                                                     [ 10%] 
aeon/classification/tests/test_all_classifiers.py::TestAllClassifiers::test_classifier_train_estimate[ProximityTree] PASSED                                                                                                      [ 12%] 
aeon/classification/tests/test_all_classifiers.py::TestAllClassifiers::test_classifier_output[ProximityTree-ClassifierFitPredict] PASSED                                                                                         [ 15%]
aeon/testing/test_softdeps.py::test_est_construct_without_modulenotfound[ProximityTree] PASSED                                                                                                                                   [ 17%] 
aeon/testing/test_softdeps.py::test_est_fit_without_modulenotfound[ProximityTree] PASSED                                                                                                                                         [ 20%]
aeon/classification/distance_based/_proximity_tree.py::aeon.classification.distance_based._proximity_tree.ProximityTree PASSED                                                                                                   [ 22%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[ProximityTree-ClassifierFitPredict-get_fitted_params] PASSED                                                                     [ 25%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[ProximityTree-ClassifierFitPredict-predict] PASSED                                                                                           [ 27%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_raises_not_fitted_error[ProximityTree-ClassifierFitPredict-get_fitted_params] PASSED                                                                                [ 30%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_multiprocessing_idempotent[ProximityTree-ClassifierFitPredict-predict] SKIPPED (hangs on mac and unix remote tests)                                                 [ 32%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_raises_not_fitted_error[ProximityTree-ClassifierFitPredict-predict] PASSED                                                                                          [ 35%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[ProximityTree-ClassifierFitPredict-predict_proba] PASSED                                                                         [ 37%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_multiprocessing_idempotent[ProximityTree-ClassifierFitPredict-predict_proba] SKIPPED (hangs on mac and unix remote tests)                                           [ 40%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[ProximityTree-ClassifierFitPredict-predict] PASSED                                                                               [ 42%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_dl_constructor_initializes_deeply[ProximityTree] PASSED                                                                                                             [ 45%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_raises_not_fitted_error[ProximityTree-ClassifierFitPredict-predict_proba] PASSED                                                                                    [ 47%] 
aeon/testing/test_all_estimators.py::TestAllEstimators::test_fit_deterministic[ProximityTree-ClassifierFitPredict-predict_proba] PASSED                                                                                          [ 50%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_fit_deterministic[ProximityTree-ClassifierFitPredict-predict] PASSED                                                                                                [ 52%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[ProximityTree-ClassifierFitPredict-predict_proba] PASSED                                                                                     [ 55%]
aeon/testing/test_all_estimators.py::TestAllEstimators::test_fit_updates_state[ProximityTree-ClassifierFitPredict] PASSED                                                                                                        [ 57%]
aeon/testing/test_all_estimators.py::TestAllObjects::test_valid_estimator_tags[ProximityTree] PASSED                                                                                                                             [ 60%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_has_common_interface[ProximityTree] PASSED                                                                                                                             [ 62%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_estimator_tags[ProximityTree] PASSED                                                                                                                                   [ 65%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_set_params_sklearn[ProximityTree] PASSED                                                                                                                               [ 67%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_clone[ProximityTree] PASSED                                                                                                                                            [ 70%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_get_params[ProximityTree] PASSED                                                                                                                                       [ 72%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_create_test_instance[ProximityTree] PASSED                                                                                                                             [ 75%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_valid_estimator_class_tags[ProximityTree] PASSED                                                                                                                       [ 77%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_create_test_instances_and_names[ProximityTree] PASSED                                                                                                                  [ 80%]
aeon/testing/test_all_estimators.py::TestAllObjects::test_no_cross_test_side_effects_part2[ProximityTree] PASSED                                                                                                                 [ 82%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_set_params[ProximityTree] PASSED                                                                                                                                       [ 85%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_no_between_test_case_side_effects[ProximityTree-ClassifierFitPredict-True] PASSED                                                                                      [ 87%]
aeon/testing/test_all_estimators.py::TestAllObjects::test_no_between_test_case_side_effects[ProximityTree-ClassifierFitPredict-42] PASSED                                                                                        [ 90%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_inheritance[ProximityTree] PASSED                                                                                                                                      [ 92%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_no_cross_test_side_effects_part1[ProximityTree] PASSED                                                                                                                 [ 95%]
aeon/testing/test_all_estimators.py::TestAllObjects::test_repr[ProximityTree] PASSED                                                                                                                                             [ 97%] 
aeon/testing/test_all_estimators.py::TestAllObjects::test_constructor[ProximityTree] PASSED                                                                                                                                      [100%] 

After:

collected 46365 items / 46339 deselected / 26 selected                                                                                                                                                                                 

aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_constructor(estimator_class=ProximityTree)] PASSED                                                                                                          [  3%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[test_classifier_output(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy3D)] PASSED                                                   [  7%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_raises_not_fitted_error(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy2D)] PASSED                                            [ 11%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[test_persistence_via_pickle(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy2D)] PASSED                                              [ 15%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_create_test_instance(estimator_class=ProximityTree)] PASSED                                                                                                 [ 19%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_get_params(estimator=ProximityTree())] PASSED                                                                                                               [ 23%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_non_state_changing_method(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy2D)] PASSED                                          [ 26%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_set_params_sklearn(estimator_class=ProximityTree)] PASSED                                                                                                   [ 30%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_fit_updates_state(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy2D)] PASSED                                                  [ 34%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_create_test_instances_and_names(estimator_class=ProximityTree)] PASSED                                                                                      [ 38%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_valid_estimator_class_tags(estimator_class=ProximityTree)] PASSED                                                                                           [ 42%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_estimator_tags(estimator_class=ProximityTree)] PASSED                                                                                                       [ 46%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[test_classifier_tags_consistent(estimator_class=ProximityTree)] PASSED                                                                                            [ 50%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_has_common_interface(estimator_class=ProximityTree)] PASSED                                                                                                 [ 53%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_valid_estimator_tags(estimator=ProximityTree())] PASSED                                                                                                     [ 57%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_repr(estimator=ProximityTree())] PASSED                                                                                                                     [ 61%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_inheritance(estimator_class=ProximityTree)] PASSED                                                                                                          [ 65%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_clone(estimator=ProximityTree())] PASSED                                                                                                                    [ 69%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[test_does_not_override_final_methods(estimator_class=ProximityTree)] PASSED                                                                                       [ 73%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_set_params(estimator=ProximityTree())] PASSED                                                                                                               [ 76%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[test_classifier_against_expected_results(estimator_class=ProximityTree)] PASSED                                                                                   [ 80%]
aeon/testing/tests/test_all_estimators.py::test_all_estimators[test_classifier_output(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy2D)] PASSED                                                   [ 84%] 
aeon/testing/tests/test_all_estimators.py::test_all_estimators[check_fit_deterministic(estimator=ProximityTree(),datatype=EqualLengthUnivariate-Classification-numpy2D)] PASSED                                                  [ 88%]
aeon/testing/tests/test_softdeps.py::test_est_fit_without_modulenotfound[ProximityTree] PASSED                                                                                                                                   [ 92%] 
aeon/testing/tests/test_softdeps.py::test_est_construct_without_modulenotfound[ProximityTree] PASSED                                                                                                                             [ 96%]
aeon/classification/distance_based/_proximity_tree.py::aeon.classification.distance_based._proximity_tree.ProximityTree PASSED                                                                                                   [100%]

[aeon-actions-bot]

Thank you for contributing to aeon

I did not find any labels to add based on the title. Please add the [ENH], [MNT], [BUG], [DOC], [REF], [DEP] and/or [GOV] tags to your pull requests titles. For now you can add the labels manually.
I have added the following labels to this PR based on the changes made: [ $\color{#BCAE15}{\textsf{classification}}$, $\color{#7E0206}{\textsf{regression}}$, $\color{#2C2F20}{\textsf{testing}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

• Run pre-commit checks for all files
• Run all pytest tests and configurations
• Run all notebook example tests
• Run numba-disabled codecov tests
• Stop automatic pre-commit fixes (always disabled for drafts)

[TonyBagnall]

I think I need to sit down and work through it all, but I think its all good. Only question was about setting "classifier" in test_all_estimators

[TonyBagnall]

LGTM, but there is a conflict with main

[hadifawaz1999]

there was a new test i added recently for saving/loading in classification/deep_learning, might put it in yield as well ?

[MatthewMiddlehurst]

Can leave that for next PR IMO. A lot going on here.

[hadifawaz1999]

we're working on the same testing functions here
how should we proceed with this, i can wait for your PR to get in and then adjust on the new placed test

[MatthewMiddlehurst]

I don't mind which one goes in first, this has other PRs waiting on it however.

[TonyBagnall]

lets get this one in first

[hadifawaz1999]

lets get this one in first

i dont mind for sure, can put my changes on the new placed test functions after this