Skip to content

feat: add verbose request tracing and structured API error responses.#2007

Draft
zhang-minchao wants to merge 10 commits into
xLLM-AI:mainfrom
zhang-minchao:feat/verbose_error_trace_id
Draft

feat: add verbose request tracing and structured API error responses.#2007
zhang-minchao wants to merge 10 commits into
xLLM-AI:mainfrom
zhang-minchao:feat/verbose_error_trace_id

Conversation

@zhang-minchao

@zhang-minchao zhang-minchao commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR improves API request observability and makes HTTP error responses consistent across OpenAI-compatible and Anthropic-compatible endpoints.

Request tracing

  • Add an opt-in asynchronous rotating-file logger for verbose request traces.
  • Resolve one x-request-id for each request, preferring the client-provided x-request-id or x-ms-client-request-id header and generating an ID when neither is present.
  • Reuse the resolved ID across API logs, verbose trace events, and engine request parameters, and return it in the HTTP response header for end-to-end correlation.
  • Record request lifecycle events (request_received, request_completed, and request_error) with the HTTP status code.
  • For early request failures, record the endpoint, status, reason, and up to 8 KiB of the raw request body in the verbose trace log.
  • Keep the serving hot path lightweight: tracing costs one relaxed atomic load when disabled; when enabled, entries are assembled in a thread-local buffer and submitted to a non-blocking spdlog async queue. If the queue is full, the oldest queued entry is dropped rather than blocking request processing.

Verbose tracing is disabled by default and can be configured with:

Option Default Description
--enable_verbose_trace_log false Enable verbose request-trace logging.
--verbose_trace_log_path log/verbose_trace.log Output log path.
--verbose_trace_log_max_size_mb 1024 Maximum size of each rotated file.
--verbose_trace_log_max_files 100000 Maximum number of rotated files retained.

Structured API errors

  • Centralize internal StatusCode to HTTP status and API error-type mapping.
  • Return structured JSON errors with the appropriate HTTP status instead of relying on brpc::Controller::SetFailed, which can replace the custom response with a generic HTTP 500.
  • Preserve the expected protocol-specific envelopes:
    • OpenAI-compatible endpoints return {"error":{"message":...,"type":...,"code":...,"param":null}}.
    • Anthropic-compatible endpoints return {"type":"error","error":{"type":...,"message":...}}.
  • Emit terminal SSE error events for failures after a streaming response has already committed HTTP 200.
  • Apply the shared error path to JSON parsing, unsupported endpoints, model/service lookup, profiling, and control APIs.

Build fixes

  • Fix stopping_checker_test static-library link dependencies.
  • Remove the unnecessary prefix_cache -> request dependency that introduced circular static-archive link ordering after rebasing onto the latest main.

Related Issues

N/A

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Validation

  • GitHub Actions format-check passed.
  • GitHub Actions build jobs are still in progress.
  • CUDA build and tests.
  • NPU build and tests.
  • MLU build and tests.

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

  • Verbose tracing is disabled by default. When enabled, failed-request traces can contain up to 8 KiB of raw request content; deployments should protect the log path and choose retention settings appropriate for potentially sensitive data.
  • Please focus on protocol compatibility of the OpenAI/Anthropic error envelopes, StatusCode to HTTP mappings, streaming terminal-error behavior, and the non-blocking logging path.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@zhang-minchao zhang-minchao changed the title Feat/verbose error trace feat: add verbose request tracing and structured API error responses. Jul 22, 2026
@zhang-minchao
zhang-minchao marked this pull request as draft July 22, 2026 13:30
@zhang-minchao
zhang-minchao force-pushed the feat/verbose_error_trace_id branch from 673dae3 to 81ef010 Compare July 22, 2026 13:33
…onses.

- VerboseTraceLogger: spdlog async rotating logger; thread-local buffer
  eliminates per-call heap allocation. XLLM_VERBOSE_TRACE() << ... macro
  is zero-cost when disabled (one relaxed atomic load).
- Resolve a single x_request_id per request in Call::init (client header
  or server-generated), pass directly to engine for end-to-end correlation.
- Replace scattered ctrl->SetFailed + LOG(ERROR) with fail_http_request:
  structured JSON error body, proper HTTP status codes, x-request-id in
  logs, and verbose trace of failed context with raw request_body.
- New gflags: enable_verbose_trace_log, verbose_trace_log_path (default
  log/verbose_trace.log), verbose_trace_log_max_size_mb,
  verbose_trace_log_max_files.
- AnthropicCall::write_and_finish: add missing request_completed trace
  event so every request_received has a matching completion in the log.
- ModelsHttp: ProtoMessageToJson failure now returns a structured error
  response instead of an empty HTTP 200.
- Call::init: reuse get_header_x_request_id() instead of duplicating the
  header extraction logic inline.
- request_completed: http=200
- request_failed_context: http=<code> (from status_code_to_http_status)
- request_error: already had http=<code>, unchanged
…-id in response header.

- Return const std::string& from Call accessors and mark them const.
- Rename get_x_request_time() to x_request_time() for consistency with x_request_id().
- Use std::string_view for message/x_request_id params in api_error functions.
- Add override and = default to ~AnthropicCall().
- Echo x-request-id in HTTP response header for end-to-end tracing.
…ency.

- Add :block and :prefix_cache as explicit DEPS for stopping_checker_test
  to resolve undefined reference to create_prefix_cache caused by
  request ↔ prefix_cache circular static-archive link ordering.
- Update text_generation_service_impl.cpp to use renamed accessors.
- Remove spurious :request dep from prefix_cache to break circular
  static-archive link ordering (prefix_cache ↔ request ↔ block).
@zhang-minchao
zhang-minchao force-pushed the feat/verbose_error_trace_id branch from 81ef010 to 38d7488 Compare July 23, 2026 06:15
- preserve the configured path for single-rank serving
- derive independent rotating paths for every multi-rank process
- add path-resolution coverage and document per-rank retention
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant