Refactor/parallel jobs

[Riul1999]

Summary

Refactored the jobs in order to execute them in parallel, using multiprocessing Process.
In order to do so the Job Queue was refacored, decoupling the run method from the DashAI core, now it can run in another process.

Type of change

• Back end new feature.
• Refactoring.

Changes

• Refactored run method from ModelJob, now it receive all the dependencies required to be execute and it not depends on the Container.
• Added method get_args to BaseJob and ModelJob to get the required arguments of the job to be executed (run).
• Added method store_results to BaseJob and ModelJob to persist the results of the job, storing the target data in the DB.
• Added methods deliver_job, start_job, finish_job and terminate__job to register the state of the job and give feedback to the user.
• Now it's not necessary to pass the db connection directly to the Job, so the job POST endpoint was modified to not use the db.
• The job_queue_loop was updated to use the new API of Jobs, a Pipe is used to send/receive the results of the run method.
• Added a new wait_point in the job_queue_loop function to wait until the child process sends data or an exception occurs. If an exception occurs, the execution status is set to error and the next job is executed.
• Refactored DummyJob in test_simple_job_queue to use the new Job API.

How to Test

Run pytest .\tests\back\api\test_jobs.py.

[notion-workspace]

Fix Queue Test Bug

[notion-workspace]

Parallel Jobs