Skip to content

Fix/native tracing #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Daedaelius merged 2 commits into from
Dec 30, 2025
Merged

Fix/native tracing #87

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f347485
propagate Langfuse credentials for native tracing when doing automate…
Daedaelius Dec 27, 2025
947e59c
Cleanup and fixes
Daedaelius Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions agent-server/nodejs/src/api-server.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,26 +593,33 @@ class APIServer {
// 1. Try tier-specific endpoint (e.g., main_model.endpoint)
// 2. Fall back to top-level endpoint (e.g., model.endpoint)
// 3. Fall back to LITELLM_ENDPOINT env var (for litellm provider)
const getEndpoint = (tierConfig) => {
const explicitEndpoint = tierConfig?.endpoint || requestBody.model.endpoint;
// @param {Object} resolvedConfig - Config after extractModelTierConfig (has defaults applied)
// @param {Object} rawTierConfig - Original tier config from request (may be undefined)
const getEndpoint = (resolvedConfig, rawTierConfig) => {
const explicitEndpoint = rawTierConfig?.endpoint || requestBody.model.endpoint;
if (explicitEndpoint) return explicitEndpoint;
// Use env var default for litellm provider
if (tierConfig?.provider === 'litellm') return defaultLiteLLMEndpoint;
// Use env var default for litellm provider (check resolved config for provider)
if (resolvedConfig?.provider === 'litellm') return defaultLiteLLMEndpoint;
return undefined;
};

// Extract tier configs first so getEndpoint can use resolved provider
const mainConfig = this.extractModelTierConfig('main', requestBody.model.main_model, defaults);
const miniConfig = this.extractModelTierConfig('mini', requestBody.model.mini_model, defaults);
const nanoConfig = this.extractModelTierConfig('nano', requestBody.model.nano_model, defaults);

return {
main_model: {
...this.extractModelTierConfig('main', requestBody.model.main_model, defaults),
endpoint: getEndpoint(requestBody.model.main_model)
...mainConfig,
endpoint: getEndpoint(mainConfig, requestBody.model.main_model)
},
mini_model: {
...this.extractModelTierConfig('mini', requestBody.model.mini_model, defaults),
endpoint: getEndpoint(requestBody.model.mini_model)
...miniConfig,
endpoint: getEndpoint(miniConfig, requestBody.model.mini_model)
},
nano_model: {
...this.extractModelTierConfig('nano', requestBody.model.nano_model, defaults),
endpoint: getEndpoint(requestBody.model.nano_model)
...nanoConfig,
endpoint: getEndpoint(nanoConfig, requestBody.model.nano_model)
}
};
}
Expand Down
17 changes: 12 additions & 5 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,18 @@ RUN git remote add upstream https://github.com/BrowserOperator/browser-operator-
RUN git fetch upstream
RUN git checkout upstream/main

# Force automated mode (matches parent Dockerfile.devtools approach - rock-solid)
RUN sed -i 's/AUTOMATED_MODE: false/AUTOMATED_MODE: true/' \
front_end/panels/ai_chat/core/BuildConfig.ts || true

# Build Browser Operator version with current changes
# Copy local LLM changes on top of the upstream code
# This allows iterative development without breaking BUILD.gn compatibility
COPY front_end/panels/ai_chat/LLM /workspace/devtools/devtools-frontend/front_end/panels/ai_chat/LLM

# AUTOMATED_MODE: Default true for API mode (Type 2/3). Override with --build-arg AUTOMATED_MODE=false for Type 1.
ARG AUTOMATED_MODE=true
RUN if [ "$AUTOMATED_MODE" = "true" ]; then \
sed -i 's/AUTOMATED_MODE: false/AUTOMATED_MODE: true/' \
front_end/panels/ai_chat/core/BuildConfig.ts; \
fi

# Build Browser Operator version with local changes
RUN npm run build

# ==============================================================================
Expand Down
4 changes: 4 additions & 0 deletions docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

This directory contains Docker configuration files for building and running the Browser Operator DevTools Frontend with integrated Agent Server in a containerized environment.

> **Deployment Types:** This directory supports **Type 1** (DevTools only) and **Type 2** (DevTools + Agent Server).
> For **Type 3** (full dockerized browser), see the [web-agent](https://github.com/BrowserOperator/web-agent) repository.
> Quick reference: [CLAUDE.md](../CLAUDE.md)

## Overview

The Docker setup uses a multi-stage build process:
Expand Down
32 changes: 30 additions & 2 deletions front_end/panels/ai_chat/evaluation/remote/EvaluationAgent.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { ToolRegistry, ConfigurableAgentTool } from '../../agent_framework/Confi
import { AgentService } from '../../core/AgentService.js';
import { AIChatPanel } from '../../ui/AIChatPanel.js';
import { createLogger } from '../../core/Logger.js';
import { createTracingProvider, withTracingContext, isTracingEnabled, getTracingConfig } from '../../tracing/TracingConfig.js';
import { createTracingProvider, withTracingContext, isTracingEnabled, getTracingConfig, setTracingConfig, refreshTracingProvider } from '../../tracing/TracingConfig.js';
import { AgentDescriptorRegistry, type AgentDescriptor } from '../../core/AgentDescriptorRegistry.js';
import '../../core/BaseOrchestratorAgent.js';
import type { TracingProvider, TracingContext } from '../../tracing/TracingProvider.js';
Expand Down Expand Up @@ -433,9 +433,37 @@ export class EvaluationAgent {
hasTracing: !!params.tracing,
tracingKeys: Object.keys(requestTracing),
sessionId: requestTracing.session_id,
traceId: requestTracing.trace_id
traceId: requestTracing.trace_id,
hasLangfuseCredentials: !!(requestTracing.langfuse_endpoint && requestTracing.langfuse_public_key && requestTracing.langfuse_secret_key)
});

// Auto-configure Langfuse tracing from request if credentials provided and not already enabled
if (requestTracing.langfuse_endpoint &&
requestTracing.langfuse_public_key &&
requestTracing.langfuse_secret_key &&
!isTracingEnabled()) {
logger.info('Auto-configuring DevTools Langfuse tracing from request', {
endpoint: requestTracing.langfuse_endpoint,
hasPublicKey: true,
hasSecretKey: true
});

setTracingConfig({
provider: 'langfuse',
endpoint: requestTracing.langfuse_endpoint,
publicKey: requestTracing.langfuse_public_key,
secretKey: requestTracing.langfuse_secret_key
});

// Refresh the tracing provider to pick up new configuration
await refreshTracingProvider();

// Update this instance's tracing provider
this.tracingProvider = createTracingProvider();

logger.info('DevTools Langfuse tracing configured successfully from request');
}
Comment on lines +440 to +465
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if there are any validation or allowlist mechanisms for Langfuse endpoints
rg -n "langfuse" --type ts -C 3 | head -100

Repository: BrowserOperator/browser-operator-core

Length of output: 9141

🏁 Script executed:

#!/bin/bash
# Check where requestTracing comes from and any validation before this point
rg -n "requestTracing" --type ts -B 5 -A 2 | head -150

Repository: BrowserOperator/browser-operator-core

Length of output: 7338

🏁 Script executed:

#!/bin/bash
# Check the context of the EvaluationAgent file and function signature
rg -n "setTracingConfig|refreshTracingProvider|createTracingProvider" --type ts -B 2 -A 5

Repository: BrowserOperator/browser-operator-core

Length of output: 33776

🏁 Script executed:

#!/bin/bash
# Check the evaluate method signature and params type definition
rg -n "evaluate.*params" --type ts -B 5 -A 10 front_end/panels/ai_chat/evaluation/remote/EvaluationAgent.ts | head -80

Repository: BrowserOperator/browser-operator-core

Length of output: 63

🏁 Script executed:

#!/bin/bash
# Look for any interface/type definition of params containing tracing
rg -n "interface.*Params|type.*Params" --type ts -A 15 front_end/panels/ai_chat/evaluation/remote/ | head -100

Repository: BrowserOperator/browser-operator-core

Length of output: 3111

🏁 Script executed:

#!/bin/bash
# Search for any validation or security-related comments in the file
rg -n "security|validate|allowlist|trusted|endpoint" --type ts front_end/panels/ai_chat/evaluation/remote/EvaluationAgent.ts

Repository: BrowserOperator/browser-operator-core

Length of output: 1093

🏁 Script executed:

#!/bin/bash
# Get the complete EvaluationParams interface including the tracing field
sed -n '87,130p' front_end/panels/ai_chat/evaluation/remote/EvaluationProtocol.ts

Repository: BrowserOperator/browser-operator-core

Length of output: 1281

🏁 Script executed:

#!/bin/bash
# Look for the tracing field definition in EvaluationParams
rg -n "tracing" --type ts front_end/panels/ai_chat/evaluation/remote/EvaluationProtocol.ts -A 5

Repository: BrowserOperator/browser-operator-core

Length of output: 460

🏁 Script executed:

#!/bin/bash
# Check if there's any server-side validation of the request before it reaches EvaluationAgent
rg -n "BrowserAgentServer\|executeRequest" --type ts -A 10 | head -100

Repository: BrowserOperator/browser-operator-core

Length of output: 63

🏁 Script executed:

#!/bin/bash
# Check for any documentation or comments about the security model
rg -n "trusted|evaluation.*server|eval.*framework" --type ts front_end/panels/ai_chat/evaluation/remote/ -B 2 -A 2 | head -50

Repository: BrowserOperator/browser-operator-core

Length of output: 2570

🏁 Script executed:

#!/bin/bash
# Check if there are any configuration files or server-side setup for this remote evaluation feature
find front_end/panels/ai_chat/evaluation/remote -type f -name "*.ts" -o -name "*.md" | xargs ls -la

Repository: BrowserOperator/browser-operator-core

Length of output: 307

Clarify security model and consider validating Langfuse endpoint against an allowlist.

This code auto-configures Langfuse tracing using credentials from the incoming evaluation request, which is intentionally designed for remote evaluation frameworks. However, without endpoint validation, an untrusted or compromised evaluation server could redirect tracing data to a malicious Langfuse instance or cause connections to arbitrary endpoints.

Add documentation clarifying that this feature assumes a trusted evaluation server, or implement endpoint validation (e.g., allowlist of permitted Langfuse hosts).

🤖 Prompt for AI Agents 
In front_end/panels/ai_chat/evaluation/remote/EvaluationAgent.ts around lines
440 to 465, the code auto-configures Langfuse tracing from request-provided
credentials without validating the endpoint; update this so untrusted endpoints
cannot be used by either (preferred) validating the
requestTracing.langfuse_endpoint against a configured allowlist (hostnames or
base URLs) before calling setTracingConfig/refreshTracingProvider and reject or
log and skip configuration if not matched, or (alternative) add clear inline
docs and runtime assertion that this flow only runs when the evaluation server
is trusted; also ensure you sanitize/log only non-sensitive metadata (do not log
secretKey) when emitting audit logs and return a clear error or telemetry event
when an invalid endpoint is supplied.


// Create a trace for this evaluation - use tracing from request if available
const traceId = requestTracing.trace_id || `eval-${params.evaluationId}-${Date.now()}`;
const sessionId = requestTracing.session_id || `eval-session-${Date.now()}`;
Expand Down
5 changes: 5 additions & 0 deletions front_end/panels/ai_chat/evaluation/remote/EvaluationProtocol.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@ export interface EvaluationParams {
category?: string;
tags?: string[];
trace_name?: string;
// Langfuse credentials for auto-configuration
// When provided, DevTools will auto-configure Langfuse tracing if not already enabled
langfuse_endpoint?: string;
langfuse_public_key?: string;
langfuse_secret_key?: string;
};
}

Expand Down