What happened:

The current implementation of Hard Example Mining in Incremental Learning is incorrect, causing the PCB-AoI/IncrementalLearning to fail to run.

What you expected to happen:

Fix this bug to make the hard example mining interface functional in Ianvs. #122 will also require this feature.

How to reproduce it (as minimally and precisely as possible):

1. Build an environment in which PCB-AoI is runnable.
2. Fix path errors in yamls.

You need to check the following files:
examples/pcb-aoi/incremental_learning_bench/fault detection/benchmarkingjob.yaml
examples/pcb-aoi/incremental_learning_bench/fault detection/testalgorithms/fpn/fpn_algorithm.yaml
examples/pcb-aoi/incremental_learning_bench/fault detection/testenv/testenv.yaml.

The main issue is that the fault detection folder needs to be added to the path.

For example: The following line needs to be modified to ./examples/pcb-aoi/incremental_learning_bench/fault detection/testenv/testenv.yaml

You may also need to fix paths for workspace, train_url, test_url, initial_model_url according to your fold structure

4. Run PCB-AoI/IncrementalLearning by

ianvs -f examples/pcb-aoi/incremental_learning_bench/fault detection/benchmarkingjob.yaml

Then you will see errors like below:

Traceback (most recent call last):
  File "./ianvs/core/testcasecontroller/testcase/testcase.py", line 74, in run
    res, system_metric_info = paradigm.run()
  File "./ianvs/core/testcasecontroller/algorithm/paradigm/incremental_learning/incremental_learning.py", line 91, in run
    r)
  File "./ianvs/core/testcasecontroller/algorithm/paradigm/incremental_learning/incremental_learning.py", line 132, in _inference
    job = self.build_paradigm_job(ParadigmType.INCREMENTAL_LEARNING.value)
  File "./ianvs/core/testcasecontroller/algorithm/paradigm/base.py", line 100, in build_paradigm_job
    ModuleType.HARD_EXAMPLE_MINING.value))
  File "/home/XXX/miniconda3/envs/ianvs-py36/lib/python3.6/site-packages/sedna-0.4.1-py3.6.egg/sedna/core/incremental_learning/incremental_learning.py", line 75, in __init__
    hem = hard_example_mining.get("method", "IBT")
AttributeError: 'IBTFilter' object has no attribute 'get'

Anything else we need to know?:

The reason for this error is that Ianvs passed incorrect parameters for hard_example_mining when calling Sedna's IncrementalLearning interface.

Ianvs's IncrementalLearning is implemented by introducing Sedna's IncrementalLearning, as shown below.

Ianvs passes an instance of the user-defined hard_example_mining module (PCB-AoI/hard_example_mining.py is an example) into the hard_example_mining keyword argument of Sedna's IncrementalLearning.

However, hard_example_mining in Sedna is defined as a dictionary with method and param fields, and the correct parameters for instantiating IncrementalLearning are as bellow.

il_model = IncrementalLearning(
            estimator=Estimator,
            hard_example_mining={
                "method": "IBT",
                "param": {
                    "threshold_img": 0.9
                }
            }
        )

Inside Sedna's IncrementalLearning, an instance of the hard_example_mining algorithm will be created using the ClassFactory.get_cls() method based on this dictionary. See details in Sedna's Source Code

        self.hard_example_mining_algorithm = None
        if not hard_example_mining:
            hard_example_mining = self.get_hem_algorithm_from_config()
        if hard_example_mining:
            hem = hard_example_mining.get("method", "IBT")
            hem_parameters = hard_example_mining.get("param", {})
            self.hard_example_mining_algorithm = ClassFactory.get_cls(
                ClassType.HEM, hem
            )(**hem_parameters)

According to the current parameter passing method of Ianvs, the hard_example_mining in the above code will not be None, thus triggering hem = hard_example_mining.get("method", "IBT").

However, we are passing a module that lacks the get method, which is typical of a dict. Additionally, the example does not provide any further definitions, leading to an AttributeError.