google-cloud-functions

Instrumenting a Node.js Google Cloud Run Function with OpenTelemetry

This example demonstrates how to instrument an serverless Google Cloud Run function written in Node.js using OpenTelemetry, and then export the data to Splunk Observability Cloud. We'll use Node.js v20 for this example, but the steps for other Node.js versions are similar.

Prerequisites

The following tools are required to deploy Node.js Google Cloud Run functions:

• An Google Cloud Platform account with permissions to create and execute Google Cloud Run functions
• An OpenTelemetry collector that's accessible to the Google Cloud Run function
• gCloud CLI

Splunk Distribution of the OpenTelemetry Collector

For this example, we deployed the Splunk Distribution of the OpenTelemetry Collector onto a virtual machine in GCP using Gateway mode, and ensured it's accessible to our Google Cloud Run function.

We configured it with the SPLUNK_HEC_TOKEN and SPLUNK_HEC_URL environment variables, so that it exports logs to our Splunk Cloud instance.

Please refer to Install the Collector using packages and deployment tools for collector installation instructions.

Application Overview

If you just want to build and deploy the example, feel free to skip this section.

The application used for this example is a simple Hello World application.

We added the @splunk/otel package with npm, which we can see in the package.json file:

{
  "dependencies": {
    "@google-cloud/functions-framework": "^3.0.0",
    "@splunk/otel": "^3.0.0",
    "pino": "^9.5.0"
  }
}

Build and Deploy

Open a terminal and navigate to the following directory:

splunk-opentelemetry-examples/instrumentation/nodejs/google-cloud-functions

Initialize the gCloud CLI

If you haven't already done so, install and initialize the gcloud CLI.

Build and Deploy the Google Cloud Run Function

Use the following command to deploy the Google Cloud Run function, substituting the region that's best for you. To allow OpenTelemetry to send trace data to Splunk Observability Cloud, we also need to set the OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, and NODE_OPTIONS environment variables as part of the gcloud deploy command:

gcloud functions deploy node-js-gcloud-function-example \
    --gen2 \
    --region=us-central1 \
    --runtime=nodejs20 \
    --source=./src \
    --entry-point=helloHttp \
    --trigger-http \
    --set-env-vars OTEL_SERVICE_NAME=node-js-gcloud-function-example,OTEL_EXPORTER_OTLP_ENDPOINT=http://<Collector IP>:4318,OTEL_RESOURCE_ATTRIBUTES=deployment.environment=test,NODE_OPTIONS='--require @splunk/otel/instrument'

Answer "y" to the following question when asked:

Allow unauthenticated invocations of new function [node-js-gcloud-function-example]? (y/N)? 

If the function is created successfully, it should provide you with a URL such as the following:

https://us-central1-gcp-<account name>.cloudfunctions.net/node-js-gcloud-function-example

Test the Google Cloud Run Function

Take the URL provided by the gcloud CLI above and enter it into your browser. It should return:

Hello World! 

View Traces in Splunk Observability Cloud

After a minute or so, you should start to see traces for the serverless function appearing in Splunk Observability Cloud:

Add Trace Context to Logs

Logs generated by a Google Cloud Run function get sent to Google Cloud Logging. Various methods exist for streaming logs into Splunk platform from Google Cloud Logging, as described in Stream logs from Google Cloud to Splunk.

Once the logs are in Splunk platform, they can be made available to Splunk Observability Cloud using Log Observer Connect.

Here's an example log entry, which includes the trace_id and span_id:

{
    hostname: localhost, 
    level: 30, 
    msg: helloHttp handler was invoked, 
    pid: 1, 
    service.environment: test, 
    service.name: node-js-gcloud-function-example, 
    span_id: c3fd896135dfedf5, 
    time: 1741039055180, 
    trace_flags: 01, 
    trace_id: f3adbec68648bcc4172e9b7098887959
}

We can see that the log entry includes a trace_id and span_id, which allows us to correlate logs with traces.