[SVLS-7018] Support profiling in Azure Functions #40

kathiehuang · 2025-10-01T18:40:22Z

What does this PR do?

Implements functionality in Azure Functions to handle proxy payloads - for now, specifically profile payloads
Starts a background task that runs a proxy flusher
Adds a profiling endpoint that receives profile proxy payloads and sends them through an mpsc channel where the proxy flusher receives it and forwards it to Datadog, handling retries and exponential backoffs
Attaches additional Azure Function-specific tags to the profile
Works in Python, Node.js, and Java, but the Java and .NET Continuous Profilers in serverless are not currently supported by the Profiling team
Added datadog-fips to Cargo.toml to use its reqwest client builder method

Motivation

Profiling is not yet supported in Azure Functions.

Additional Notes

Tested across all hosting plans. This works for Python and Node.js.

For Node.js, profiling only has prebuilt binaries for Linux arm64/x64, Alpine Linux x64, macOS arm64/x64, and Windows x64. The default for Azure Apps in Terraform is Windows x32, so we had to go to the Configuration tab in the Azure portal and change Platform to 64 Bit to get profiling to work.
- This is documented in the Node.js profiler troubleshooting page
This worked in Java as well in testing, but the Java Continuous Profiler in serverless is not currently supported by the Profiling team.
The .NET Continuous Profiler in serverless is also not currently supported by the Profiling team - could not test because the .NET tracer shuts down the profiler if it detects it's in an Azure Function.

Describe how to test/QA your changes

To test this, follow the Serverless Compatibility Layer instructions to deploy function apps across runtimes and hosting plans with the built binary. Add DD_PROFILING_ENABLED=true in Azure app settings. Generate load on the function app and check Datadog to see profiles show up, and click into the profile > Runtime Info to see the added tags!

Profile tags:

Ticket for handling profile payloads: SVLS-7982

Ticket for adding tags to profiles: SVLS-7983

FRSLES-759

Follow up task: Talk with APM Serverless about changes in their tracer code that will need to be made. SVLS-7841

…ushed in proxy flusher

…andle forwarding and retries in background task

kathiehuang · 2025-10-27T16:59:33Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

crates/datadog-trace-agent/Cargo.toml

chatgpt-codex-connector · 2025-10-27T17:06:41Z

crates/datadog-trace-agent/src/mini_agent.rs

+        // Extract headers and body
+        let (parts, body) = request.into_parts();
+
+        let body_bytes = match body.collect().await {
+            Ok(collected) => collected.to_bytes(),
+            Err(e) => {
+                return log_and_create_http_response(
+                    &format!("Error reading profiling request body: {e}"),
+                    StatusCode::BAD_REQUEST,
+                );
+            }
+        };
+
+        // Create proxy request
+        let proxy_request = ProxyRequest {
+            headers: parts.headers,
+            body: body_bytes,
+            target_url: config.profiling_intake.url.to_string(),
+        };


Forward profiling query string to Datadog

The profiling proxy handler builds the outgoing request URL from the static profiling_intake endpoint and ignores parts.uri.query(). Profiling clients pass service, environment, and other metadata via the query string when posting to /profiling/v1/input; dropping these parameters means the flusher forwards payloads without required metadata, causing ingestion failures or misattribution. Append the incoming query parameters to ProxyRequest.target_url before sending to the proxy flusher.

Useful? React with 👍 / 👎.

The lambda extension doesn't preserve query parameters: https://github.com/DataDog/datadog-lambda-extension/blob/main/bottlecap/src/traces/proxy_flusher.rs#L119

kathiehuang · 2025-10-27T19:10:12Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-10-27T19:14:39Z

crates/datadog-trace-agent/src/mini_agent.rs

+    /// Handles incoming proxy requests for profiling - can be abstracted into a generic proxy handler for other proxy requests in the future
+    async fn profiling_proxy_handler(
+        config: Arc<config::Config>,
+        request: hyper_migration::HttpRequest,
+        proxy_tx: Sender<ProxyRequest>,
+    ) -> http::Result<hyper_migration::HttpResponse> {
+        debug!("Trace Agent | Received profiling request");
+
+        // Extract headers and body
+        let (parts, body) = request.into_parts();
+
+        let body_bytes = match body.collect().await {
+            Ok(collected) => collected.to_bytes(),
+            Err(e) => {
+                return log_and_create_http_response(
+                    &format!("Error reading profiling request body: {e}"),
+                    StatusCode::BAD_REQUEST,
+                );


Enforce profiling payload size before buffering

The new profiling proxy endpoint reads the entire request body into memory (body.collect().await) without validating the Content-Length. The trace and stats handlers call verify_request_content_length to reject payloads larger than Config::max_request_content_length before allocation, but the profiling handler omits this guard. A client can therefore POST an arbitrarily large profiling payload and the mini agent will buffer it in RAM before forwarding, allowing a single request to exhaust memory and crash the function. Consider applying the same content length check as the other endpoints before collecting the body.

Useful? React with 👍 / 👎.

The lambda extension also doesn't validate the content-length, probably because profiling payloads are typically much larger and they're coming from the Datadog profiler

duncanpharvey

Looks good! Just a few minor comments.

Let's make sure to test this in all hosting plans and operating systems prior to releasing. I've seen issues in the past with TLS certificates. See DataDog/libdatadog#523

crates/dogstatsd/src/datadog.rs

crates/datadog-trace-agent/src/config.rs

crates/datadog-trace-agent/src/http_utils.rs

kathiehuang · 2025-11-10T20:13:48Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

crates/datadog-trace-agent/src/mini_agent.rs

kathiehuang changed the title ~~[SVLS-7018] Support profiling in Azure Functions~~ WIP [SVLS-7018] Support profiling in Azure Functions Oct 1, 2025

kathiehuang added 2 commits October 1, 2025 14:42

Handle proxy payload

3be23ab

Send proxy payload through mpsc channel to then get aggregated and fl…

d9ea13b

…ushed in proxy flusher

kathiehuang force-pushed the kathie.huang/receive-profile-payload branch from c0abc20 to d9ea13b Compare October 15, 2025 18:58

kathiehuang added 2 commits October 16, 2025 17:11

Remove aggregator logic, just send payload through channel and then h…

601c893

…andle forwarding and retries in background task

cleanup

516bc94

kathiehuang changed the title ~~WIP [SVLS-7018] Support profiling in Azure Functions~~ [SVLS-7018] Support profiling in Azure Functions Oct 16, 2025

kathiehuang changed the title ~~[SVLS-7018] Support profiling in Azure Functions~~ [SVLS-7673] Support profiling in Azure Functions Oct 16, 2025

kathiehuang added 9 commits October 16, 2025 17:29

Resolve merge conflicts with main

435363c

Fix error handling

9502707

Fix reqwest dependency

a381612

Fix license

0e45726

nit: formatting

420e805

Renaming for clarity

4698d0f

Minor refactoring

3ec3d73

fix

e63d8ac

Reduce request timeout

638acb3

kathiehuang marked this pull request as ready for review October 27, 2025 16:58

kathiehuang marked this pull request as draft October 27, 2025 16:59

chatgpt-codex-connector bot reviewed Oct 27, 2025

View reviewed changes

Add rustls-tls

d529656

chatgpt-codex-connector bot reviewed Oct 27, 2025

View reviewed changes

kathiehuang marked this pull request as ready for review October 27, 2025 19:20

kathiehuang changed the title ~~[SVLS-7673] Support profiling in Azure Functions~~ [SVLS-7018] Support profiling in Azure Functions Oct 27, 2025

kathiehuang requested review from a team and duncanpharvey October 27, 2025 19:47

Merge branch 'main' into kathie.huang/receive-profile-payload

574b371

duncanpharvey approved these changes Nov 5, 2025

View reviewed changes

crates/dogstatsd/src/datadog.rs Outdated Show resolved Hide resolved

crates/datadog-trace-agent/src/config.rs Show resolved Hide resolved

crates/datadog-trace-agent/src/http_utils.rs Show resolved Hide resolved

kathiehuang added 7 commits November 5, 2025 15:51

Revert to original dogstatsd build_client function

6feb3b1

Add additional tags

a8a5a6b

Formatting

5a8984a

Avoid panicking

6a64f40

Add aas tags

4d2ddc2

Fix aas tags

6c8a8db

Fix additional tags in profiles

0045627

chatgpt-codex-connector bot reviewed Nov 10, 2025

View reviewed changes

crates/datadog-trace-agent/src/mini_agent.rs Show resolved Hide resolved

kathiehuang requested a review from duncanpharvey November 10, 2025 20:19

Remove unnecessary debug logs

8522ac0

kathiehuang marked this pull request as draft November 12, 2025 15:58

Refactor

c774b3b

[SVLS-7018] Support profiling in Azure Functions #40

Are you sure you want to change the base?

[SVLS-7018] Support profiling in Azure Functions #40

Uh oh!

Conversation

kathiehuang commented Oct 1, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Describe how to test/QA your changes

Uh oh!

kathiehuang commented Oct 27, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang commented Oct 27, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

duncanpharvey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kathiehuang commented Nov 10, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kathiehuang commented Oct 1, 2025 •

edited by atlassian bot

Loading

kathiehuang Oct 27, 2025 •

edited

Loading