feat: baseline spadev0 integration

[jcace]

Closes #24

Adds an integration to the Retrieval bot codebase to enable pulling CIDs from Spade and adding them to the task queue

[jcace]

@xinaxu , do you think we should have an entirely separate Module here? ex task.Spade? Reason I ask is I don't see the metadata retrieve_type used anywhere (maybe I'm not looking in the right place though).. is is OK to use that metadata for control flow?

[xinaxu]

Reason it is not used is because it is currently assuming there is only one type of retrieve_type and I'm expecting, as part of this PR or next PR, you will fork the logic, i.e., if retrieve_type is spade, do spade, otherwise, do regular.

Also, metadata will be part of the result payload, so you will be able to query the result collection with certain filtering on metadata.

[xinaxu]

Meanwhile, please double check the current retrieval logic to make sure that it will not panic or crash on unexpected metadata/retrieve_type. You should be fine just by looking at your code

[xinaxu]

Finally, have you checked the logic here. This is where retrieval worker polls the task queue for tasks. If we end up filling the task but no-one is pulling from it, the task queue will get full and no new tasks will be added. I think you're fine by looking at your logic but want to make sure that you're aware.

RetrievalBot/pkg/task/task_worker.go

Line 98 in 342f350

if strings.HasPrefix(t.acceptedCountries, "!") {

[jcace]

Meanwhile, please double check the current retrieval logic to make sure that it will not panic or crash on unexpected metadata/retrieve_type. You should be fine just by looking at your code

Yup, confirmed - the metadata retrieve_type is not actually read anywhere in the codebase, so should not cause any issues.

Finally, have you checked the logic here. This is where retrieval worker polls the task queue for tasks. If we end up filling the task but no-one is pulling from it, the task queue will get full and no new tasks will be added. I think you're fine by looking at your logic but want to make sure that you're aware.

should we add some kind of back-pressure check to decide to not run the integration if the queue is too full?

[ribasushi]

This is a very dcent first pass! The method is solid, left a few comments regarding implementation errors ( nothing architecturally bad though )

[xinaxu]

Please take a look at the comments before merging

[xinaxu]

Reason it is not used is because it is currently assuming there is only one type of retrieve_type and I'm expecting, as part of this PR or next PR, you will fork the logic, i.e., if retrieve_type is spade, do spade, otherwise, do regular.

Also, metadata will be part of the result payload, so you will be able to query the result collection with certain filtering on metadata.

[xinaxu]

Meanwhile, please double check the current retrieval logic to make sure that it will not panic or crash on unexpected metadata/retrieve_type. You should be fine just by looking at your code

[xinaxu]

Finally, have you checked the logic here. This is where retrieval worker polls the task queue for tasks. If we end up filling the task but no-one is pulling from it, the task queue will get full and no new tasks will be added. I think you're fine by looking at your logic but want to make sure that you're aware.

RetrievalBot/pkg/task/task_worker.go

Line 98 in 342f350

if strings.HasPrefix(t.acceptedCountries, "!") {