OpenTelemetry Testbed for NodeJS

This project is an OpenTelemetry for NodeJS. It is designed to be a simple application that provides all the necessary endpoint which when invoked exercise all the core OpenTelemetry instrumentation.

Installation

N/A

Usage

To use the application, run devbox run demo. This will start the application and the associated services - the Jaeger backend, and, the OpenTelemetry collector.

The Jaeger collector is accessible at http://localhost:16686/ and the application at http://localhost:3000/.

• Invoke /captive to see outbound-requests being instrumented. This endpoint simply makes a HTTP request to an external endpoint.
• Invoke /goboom to see an exception being instrumeted. This endpoint does nothing more than throw an internal-server-error response (500).
• Invoke /logme to see logs being instrumented. This endpoint does nothing more than log message and return all-ok response (200).

Invoking these endpoints will lead to traces being generated and you will be able to see them in the Jaeger UI.

Using Elasticsearch

If you would like to send traces from your

https://github.com/elastic/observability-docs/blob/b8f4f2327ed3bbfe3965c9cffa974e268a853616/docs/en/serverless/apm/collect-application-data/open-telemetry/otel-direct.asciidoc#send-data-from-an-upstream-opentelemetry-collector

  otlp/elastic:
    # Elastic APM server https endpoint (without the "https://" prefix)
    endpoint: "${env:ELASTIC_APM_SERVER_ENDPOINT}"
    tls:
        # Must be enabled if using self-signed certificates
		insecure: false
    headers:
      # Elastic APM Server secret token (base64 decoded)
      Authorization: "Bearer ${env:ELASTIC_APM_SECRET_TOKEN}"

Table of Contents

• Architecture
• Developing
• Contributing
• License

Architecture

The app is a NestJS application deployed on AWS Lambda and fronted by CloudFront. All SSL certificates are managed automatically via Certificate Manager (using DNS validation).

To manage the lock and unlock schedules, EventBridge Scheduler is used which in turn invokes the Lambda. All credentials are stored in Secrets Manager.

The application uses X-Ray for tracing and Cloudwatch for logging.

The application is designed to be stateless and does not have any sort of persistence—this includes all ephemeral persistence, e.g. caches.

Developing

The app is built with Typescript 5.3 using NestJS and requires Node 20 to run.

Notes:

Even after installing the OTEL libraries, you must explicitly install these packages

• The https://www.npmjs.com/package/@opentelemetry/id-generator-aws-xray package so that the IDs are compliant.
• The `@opentelemetry/propagator-aws-xray library which I'm unusre about.

These libraries do not need to be installed @opentelemetry/instrumentation-aws-sdk as these are bundled.

On AWS, the OTEL must generate Ids that are understandable by XRay and that's why the AWSXRayIdGenerator is used.

A trace_id consists of three numbers separated by hyphens. For example, 1-58406520-a006649127e371903a2de979. For trace IDs created by an X-Ray SDK, or by AWS services integrated with X-Ray, a trace ID includes:

Trace ID Format

• The version number, for instance, 1.
• The time of the original request, in Unix epoch time, in 8 hexadecimal digits. For example, 10:00AM December 2nd, 2016 PST in epoch time is 1480615200 seconds, or 58406520 in hexadecimal.
• A 96-bit identifier for the trace, globally unique, in 24 hexadecimal digits.

Note Trace IDs created via OpenTelemetry have a different format based on the W3C Trace Context specification. A W3C trace ID must be formatted in the X-Ray trace ID format when sending to X-Ray. For example, a W3C trace ID 4efaaf4d1e8720b39541901950019ee5 should be formatted as 1-4efaaf4d-1e8720b39541901950019ee5 when sending to X-Ray. While X-Ray trace IDs include the original request timestamp in Unix epoch time, this is not required or validated.

https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html#API_PutTraceSegments_RequestSyntax

By exporting all the traces to the console, we can see that the ids generated by the AWSXRayIdGenerator don't comply with this format. With little to no literature around this, I assume that this is to do with the fact the there is an internal spec that denotes what the trace ids must be like.

The

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awsxrayexporter/internal/translator/segment.go#L556-L595

---

[Minor] When running Jaeger, you need to set up SSL certificates or you're going to need to switch to insecure communication between the collector and Jaeger.

[Major] Most extensions in the Otel Collector are still in development or in beta. This includes simple extensions like the health check which are currently marked as in beta. Since so many extensions are currently in beta, a production rollout of the Otel Collector seems scary as you don't know what will go south in production.

Contributing

If you have suggestions for how this app could be improved, or want to report a bug, open an issue - we'd love all and any contributions.

License

Apache License 2.0 © 2024 Mridang Agarwalla