nameko-opentelemetry

OpenTelemetry instrumentation for Nameko microservices.

Usage

Inside your codebase, before the service is started, apply the instrumentation:

# Instrument nameko
from nameko_opentelemetry import NamekoInstrumentor
NamekoInstrumentor().instrument(
    # optionally pass any customised entrypoint adapters (see below for details)
    entrypoint_adapters={
        "my.custom.EntrypointType": "my.custom.EntrypointAdapter"
    },
    # turn on logging of request headers and context data
    send_headers=True,
    # turn on logging of request payloads and entrypoint arguments
    send_request_payloads=True,
    # turn on logging of response payloads and entrypoint results
    send_response_payloads=True,
    # change the default length at which headers and payloads are truncated
    max_truncate_length=1000,
)

# You will also want to instrument any other libraries you're using, e.g.
from opentelemetry.instrumentation.requests import RequestsInstrumentor
RequestsInstrumentor().instrument()

# All entrypoints, and all built-in dependency providers and clients are now instrumented.
# To use the instrumentation somehow, attach a span processor, e.g.
from opentelemetry import trace
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
trace.get_tracer_provider().add_span_processor(
    SimpleSpanProcessor(ConsoleSpanExporter())
)

Approach

This library uses monkey-patching, as many other third-party libraries do when adding instrumentation. This has the advantage of keeping the main library "clean" of instrumentation code, but at the expense of some brittleness. It's probably fine for now.

Applied Instrumentation

Server spans are generated for every entrypoint execution, with helpful default span names and attributes. To customise these for a third-party entrypoint, you can define a new EntrypointAdapter. See below.

Client spans are generated by all the built-in dependency providers and standalone clients, again with helpful defaults for span names and attributes. To instrument third-party dependency providers or clients, you may need to create standalone instrumentation -- or rely on pre-existing instrumentation of an underlying library (e.g. requests, sqlalchemy, etc.)

Entrypoint Adapters

This instrumentation generates a default span name and attributes for every entrypoint that executes. It assumes that the context_data for the call includes any propagation headers from whichever transport was used. If you want to customise the span name or attributes, or if the entrypoint does not populate the worker's context_data with headers from the transport, you can implement an entrypoint adapter. As an example, parts of the adapter for the built-in @http entrypoint:

class HttpEntrypointAdapter(EntrypointAdapter):
    """ Implemented according to https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md
    """

    def get_metadata(self):
        # http headers are not automatically added to the worker's context data,
        # so we get them from the `request` argument
        return self.worker_ctx.args[0].headers

    def get_span_name(self):
        # http entrypoints use the url path as a span name, as per
        # https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md
        return self.worker_ctx.entrypoint.url

    def get_attributes(self):
        """ Arbitrary additional attributes to include
        """
        default_attributes = super(). get_attributes()
        default_attributes.update({
            "appropriately.namespaced.key": "value"
        })

    def get_result_attributes(self, result):
        """ Return attributes derived from the `result` returned by the entrypoint method.
        """

    def get_exception_attributes(self, exc_info):
        """ Return attributes derived from any exception thrown by the entrypoint method.
        """