tracing-actix-web

Structured diagnostics for actix-web applications.

Note

This is a fork of tracing-actix-web library updated specifically with the needs of Blockscout rust microservices in mind.

Original decription

tracing-actix-web provides TracingLogger, a middleware to collect telemetry data from applications built on top of the actix-web framework.

tracing-actix-web was initially developed for the telemetry chapter of Zero to Production In Rust, a hands-on introduction to backend development using the Rust programming language.

Getting started

How to install

Add tracing-actix-web to your dependencies:

[dependencies]
# ...
tracing-actix-web = "0.7"
tracing = "0.1"
actix-web = "4"

tracing-actix-web exposes these feature flags:

• opentelemetry_0_13: attach OpenTelemetry's context to the root span using opentelemetry 0.13;
• opentelemetry_0_14: same as above but using opentelemetry 0.14;
• opentelemetry_0_15: same as above but using opentelemetry 0.15;
• opentelemetry_0_16: same as above but using opentelemetry 0.16;
• opentelemetry_0_17: same as above but using opentelemetry 0.17;
• opentelemetry_0_18: same as above but using opentelemetry 0.18;
• opentelemetry_0_19: same as above but using opentelemetry 0.19;
• opentelemetry_0_20: same as above but using opentelemetry 0.20;
• opentelemetry_0_21: same as above but using opentelemetry 0.21;
• emit_event_on_error: emit a tracing event when request processing fails with an error (enabled by default).
• uuid_v7: use the UUID v7 implementation inside RequestId instead of UUID v4 (disabled by default).

Quickstart

use actix_web::{App, web, HttpServer};
use tracing_actix_web::TracingLogger;

fn main() {
    // Init your `tracing` subscriber here!

    let server = HttpServer::new(|| {
        App::new()
            // Mount `TracingLogger` as a middleware
            .wrap(TracingLogger::default())
            .service( /*  */ )
    });
}

Check out the examples on GitHub to get a taste of how TracingLogger can be used to observe and monitor your application.

From zero to hero: a crash course in observability

tracing: who art thou?

TracingLogger is built on top of tracing, a modern instrumentation framework with a vibrant ecosystem.

tracing-actix-web's documentation provides a crash course in how to use tracing to instrument an actix-web application.
If you want to learn more check out "Are we observable yet?" - it provides an in-depth introduction to the crate and the problems it solves within the bigger picture of observability.

The root span

tracing::Span is the key abstraction in tracing: it represents a unit of work in your system.
A tracing::Span has a beginning and an end. It can include one or more child spans to represent sub-unit of works within a larger task.

When your application receives a request, TracingLogger creates a new span - we call it the root span.
All the spans created while processing the request will be children of the root span.

tracing empowers us to attach structured properties to a span as a collection of key-value pairs.
Those properties can then be queried in a variety of tools (e.g. ElasticSearch, Honeycomb, DataDog) to understand what is happening in your system.

Customisation via RootSpanBuilder

Troubleshooting becomes much easier when the root span has a rich context - e.g. you can understand most of what happened when processing the request just by looking at the properties attached to the corresponding root span.

You might have heard of this technique as the canonical log line pattern, popularised by Stripe. It is more recently discussed in terms of high-cardinality events by Honeycomb and other vendors in the observability space.

TracingLogger gives you a chance to use the very same pattern: you can customise the properties attached to the root span in order to capture the context relevant to your specific domain.

[TracingLogger::default] is equivalent to:

use tracing_actix_web::{TracingLogger, DefaultRootSpanBuilder};

// Two ways to initialise TracingLogger with the default root span builder
let default = TracingLogger::default();
let another_way = TracingLogger::<DefaultRootSpanBuilder>::new();

We are delegating the construction of the root span to DefaultRootSpanBuilder.
DefaultRootSpanBuilder captures, out of the box, several dimensions that are usually relevant when looking at an HTTP API: method, version, route, etc. - check out its documentation for an extensive list.

You can customise the root span by providing your own implementation of the RootSpanBuilder trait.
Let's imagine, for example, that our system cares about a client identifier embedded inside an authorization header. We could add a client_id property to the root span using a custom builder, DomainRootSpanBuilder:

use actix_web::body::MessageBody;
use actix_web::dev::{ServiceResponse, ServiceRequest};
use actix_web::Error;
use tracing_actix_web::{TracingLogger, DefaultRootSpanBuilder, RootSpanBuilder};
use tracing::Span;

pub struct DomainRootSpanBuilder;

impl RootSpanBuilder for DomainRootSpanBuilder {
    fn on_request_start(request: &ServiceRequest) -> Span {
        let client_id: &str = todo!("Somehow extract it from the authorization header");
        tracing::info_span!("Request", client_id)
    }

    fn on_request_end<B: MessageBody>(_span: Span, _outcome: &Result<ServiceResponse<B>, Error>) {}
}

let custom_middleware = TracingLogger::<DomainRootSpanBuilder>::new();

There is an issue, though: client_id is the only property we are capturing.
With DomainRootSpanBuilder, as it is, we do not get any of that useful HTTP-related information provided by DefaultRootSpanBuilder.

We can do better!

use actix_web::body::MessageBody;
use actix_web::dev::{ServiceResponse, ServiceRequest};
use actix_web::Error;
use tracing_actix_web::{TracingLogger, DefaultRootSpanBuilder, RootSpanBuilder};
use tracing::Span;

pub struct DomainRootSpanBuilder;

impl RootSpanBuilder for DomainRootSpanBuilder {
    fn on_request_start(request: &ServiceRequest) -> Span {
        let client_id: &str = todo!("Somehow extract it from the authorization header");
        tracing_actix_web::root_span!(request, client_id)
    }

    fn on_request_end<B: MessageBody>(span: Span, outcome: &Result<ServiceResponse<B>, Error>) {
        DefaultRootSpanBuilder::on_request_end(span, outcome);
    }
}

let custom_middleware = TracingLogger::<DomainRootSpanBuilder>::new();

root_span! is a macro provided by tracing-actix-web: it creates a new span by combining all the HTTP properties tracked by DefaultRootSpanBuilder with the custom ones you specify when calling it (e.g. client_id in our example).

We need to use a macro because tracing requires all the properties attached to a span to be declared upfront, when the span is created.
You cannot add new ones afterwards. This makes it extremely fast, but it pushes us to reach for macros when we need some level of composition.

root_span! exposes more or less the same knob you can find on tracing's span! macro. You can, for example, customise the span level:

use actix_web::body::MessageBody;
use actix_web::dev::{ServiceResponse, ServiceRequest};
use actix_web::Error;
use tracing_actix_web::{TracingLogger, DefaultRootSpanBuilder, RootSpanBuilder, Level};
use tracing::Span;

pub struct CustomLevelRootSpanBuilder;

impl RootSpanBuilder for CustomLevelRootSpanBuilder {
    fn on_request_start(request: &ServiceRequest) -> Span {
        let level = if request.path() == "/health_check" {
            Level::DEBUG
        } else {
            Level::INFO
        };
        tracing_actix_web::root_span!(level = level, request)
    }

    fn on_request_end<B: MessageBody>(span: Span, outcome: &Result<ServiceResponse<B>, Error>) {
        DefaultRootSpanBuilder::on_request_end(span, outcome);
    }
}

let custom_middleware = TracingLogger::<CustomLevelRootSpanBuilder>::new();

The RootSpan extractor

It often happens that not all information about a task is known upfront, encoded in the incoming request.
You can use the RootSpan extractor to grab the root span in your handlers and attach more information to your root span as it becomes available:

use actix_web::body::MessageBody;
use actix_web::dev::{ServiceResponse, ServiceRequest};
use actix_web::{Error, HttpResponse};
use tracing_actix_web::{RootSpan, DefaultRootSpanBuilder, RootSpanBuilder};
use tracing::Span;
use actix_web::get;
use tracing_actix_web::RequestId;
use uuid::Uuid;

#[get("/")]
async fn handler(root_span: RootSpan) -> HttpResponse {
    let application_id: &str = todo!("Some domain logic");
    // Record the property value against the root span
    root_span.record("application_id", &application_id);

    // [...]
    # todo!()
}

pub struct DomainRootSpanBuilder;

impl RootSpanBuilder for DomainRootSpanBuilder {
    fn on_request_start(request: &ServiceRequest) -> Span {
        let client_id: &str = todo!("Somehow extract it from the authorization header");
        // All fields you want to capture must be declared upfront.
        // If you don't know the value (yet), use tracing's `Empty`
        tracing_actix_web::root_span!(
            request,
            client_id, application_id = tracing::field::Empty
        )
    }

    fn on_request_end<B: MessageBody>(span: Span, response: &Result<ServiceResponse<B>, Error>) {
        DefaultRootSpanBuilder::on_request_end(span, response);
    }
}

Unique identifiers

Request Id

tracing-actix-web generates a unique identifier for each incoming request, the request id.

You can extract the request id using the RequestId extractor:

use actix_web::get;
use tracing_actix_web::RequestId;
use uuid::Uuid;

#[get("/")]
async fn index(request_id: RequestId) -> String {
  format!("{}", request_id)
}

The request id is meant to identify all operations related to a particular request within the boundary of your API.
If you need to trace a request across multiple services (e.g. in a microservice architecture), you want to look at the trace_id field - see the next section on OpenTelemetry for more details.

Optionally, using the uuid_v7 feature flag will allow RequestId to use UUID v7 instead of the currently used UUID v4.

However, the uuid crate requires a compile time flag uuid_unstable to be passed in RUSTFLAGS="--cfg uuid_unstable" in order to compile. You can read more about it here.

Trace Id

To fulfill a request you often have to perform additional I/O operations - e.g. calls to other REST or gRPC APIs, database queries, etc.
Distributed tracing is the standard approach to trace a single request across the entirety of your stack.

tracing-actix-web provides support for distributed tracing by supporting the OpenTelemetry standard.
tracing-actix-web follows OpenTelemetry's semantic convention for field names.
Furthermore, it provides an opentelemetry_0_17 feature flag to automatically performs trace propagation: it tries to extract the OpenTelemetry context out of the headers of incoming requests and, when it finds one, it sets it as the remote context for the current root span. The context is then propagated to your downstream dependencies if your HTTP or gRPC clients are OpenTelemetry-aware - e.g. using reqwest-middleware and reqwest-tracing if you are using reqwest as your HTTP client.
You can then find all logs for the same request across all the services it touched by looking for the trace_id, automatically logged by tracing-actix-web.

If you add tracing-opentelemetry::OpenTelemetryLayer in your tracing::Subscriber you will be able to export the root span (and all its children) as OpenTelemetry spans.

Check out the relevant example in the GitHub repository for reference.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in tracing-actix-web by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.