bug/feat(llmobs): expose input_documents/output_documents on span processor

[MrinalJain17]

Description

The span processor (register_processor / enable(span_processor=...)) exposes input and output (messages/values) for mutation, but not documents. Embedding and retrieval span documents pass through to Datadog without any way for users to redact or modify them.

This adds input_documents and output_documents fields to LLMObsSpan, following the same populate-before/write-back-after pattern used by input and output.

Usage

def redact_processor(span: LLMObsSpan) -> LLMObsSpan:
    for doc in span.input_documents:
        doc["text"] = ""
    for doc in span.output_documents:
        doc["text"] = ""
    return span

LLMObs.register_processor(redact_processor)

Testing

Added two tests in TestLLMIOProcessing:

• test_document_processor_redact — mutate document text in place
• test_document_processor_remove — clear documents by setting to []

Risks

None. The change is backward-compatible — input_documents and output_documents default to [], so existing processors that do not reference them are unaffected.

Additional Notes

Related issues: #11179

The fix in #13479 was only partial. For enterprise use-cases, being able to redact sensitive information is critical.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13d3236b57

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[MrinalJain17]

Hello @ZStriker19 @Yun-Kim @gnufede

Would appreciate if you can take a look at this.

The issue can lead to (sensitive) document content being leaked to Datadog LLMobs silently, without the ability for users to redact.