DLCS Dependent Workflow

[tomcrane]

What does this change?

With the new Delivery Channels work, we rely on the DLCS named query manifest to tell us the exact sizes it has made for thumbnails.
But this means we can't build a IIIF Manifest until the DLCS batches have finished processing.

This PR changes the way Workflow Jobs are processed. If a workflow job has options (see RunnerOptions) that are dependent on the DLCS and ALSO has the RunnerOptions.RegisterImages option, the WorkflowProcessor starts the register images operation, marks the job as having running ingests (the IngestJobStarted DateTime? field), removes the RunnerOptions.RegisterImages option from the job, and keeps the job waiting to be picked up.

The logic for picking a job up now requires that it can't have running DLCS ingests - that its IngestJobStarted field is null.

A separate process polls the DLCS to see whether IngestJobStarted can be set to null.

We could go further - it's still possible to manually sync from the dashboard, which the workflow processor won't know about because there's no workflow job to be updated.

Problems:

WorkflowProcessor is now doing three related but distinct things:

• processing jobs from the workflow jobs table
• polling SQS for new jobs to add to the table (typically from Goobi)
• checking the DLCS to see if jobs with a non-null IngestJobStarted field

While this PR can be released to see how this behaves, we should probably split these tasks into three independent services.

How to test

tbc

How can we measure success?

tbc

Have we considered potential risks?

Risks are that the workflow processor is doing too much work.

I also want to at least attempt to calculate the expected thumbnails from w, delivery channel policies and see whether they match, and report on this. If we find that we are 100% accurate in our predictions, we can toggle this behaviour off for the DDS specifically.