Detecting failures

[donaldgray]

We'll need to confirm whether there is ongoing but it seems like there may be an issue were task failure isn't detected correctly.

I initially thought this could be related to running out of resources (due to memory issue resolved in #71). When using the default django-q broker the max_retries seemed to be ignored, items were continuously picked up and aborted. The endpoint to check status wasn't returning an error.

#68 added support for SQS broker, it also looks like max_retries is ignored in this case. I tested using deadletter queues via SQS to detect failures but in this instance we need to investigate how we can determine which message has failed. The tasks are pickled and signed so we'd need to investigate whether it is possible or not. Does the API / worker may need to listen to the DLQ and mark the task as failed? Is there a standard pattern for this?

This may be releated to how the API has been implemented and something has been overlooked.