Patch Consumer to support dd-trace-js traceability

[rbonestell]

Background

It seems that due to the implementation style of promise chaining in the Consumer, there is functionally no way for the dd-trace-js library to persist context and continue the same trace context from which the message was enqueued.

To be candid, I'm no NodeJS expert and don't thoroughly understand the root cause here. Please refer to the explanation from a maintainer of dd-trace-js, found here: DataDog/dd-trace-js#1680 (comment)

Resolving this functional conflict with dd-trace-js will allow users of sqs-consumer to support distributed tracing across services which communicate using sqs-consumer and sqs-producer.

Objectives

1. Refactor the complexity of promise chaining in the Consumer.poll() function in order to support implicit distributed traceability over SQS with the dd-trace-js library.

[nicholasgriffintn]

Hey, thanks for posting the request.

We don't use DataDog within the BBC as far as I'm aware, and we'd need some sort of agreement with them to use their services (legal stuff I'm afraid).

I could probably look personally, but I also don't really have a working knowledge with the service to really understand how it works.

It might be helpful if we could get an example app setup that showed the library in use with sqs-consumer (we have a repo here: https://github.com/bbc/sqs-consumer-starter/tree/main/examples/ts-esm), then maybe I could look personally or someone from the community could take a look and post a PR.

[rbonestell]

Thanks for the quick response, @nicholasgriffintn! To be honest, I was hoping that someone would find the solution more obvious than I do regarding async_hooks and the async sqs-consumer callback pattern.

I'll create a fork and add a new example using dd-trace-js in the /examples folder. You won't need any Datadog accounts or anything, I'll simply make a small demo that starts a trace and logs the info, enqueues a message, then dequeues the message and and log out the new trace ID that should match the original, demonstrating that the metadata of the message was not properly handled by dd-trace-js (as per the explanation of the maintainer in the previously linked transaction).

I appreciate your willingness to help! Cheers!

[rbonestell]

Following up here again, I haven't had the capacity to dedicate to a repro example yet but very much intend on doing that to help progress resolving this issue!