Distributed tracing with OpenTelemetry

[greatest0fallt1me]

Description

• Export to OTLP; propagate context through jobs and RPC clients.

Requirements and context

• Must be secure, tested, and documented for production operation.
• Should be efficient, observable, and easy to review in PRs.
• Scope is Predictify backend / off-chain services only (no Soroban contract changes, no frontend UI in this issue).

Suggested execution

• Fork the repository and create a branch:
  • git checkout -b feature/otel-tracing
• Implement changes:
  1. Service / module: implement in src/telemetry/otel.rs and related packages as needed.
  2. Tests: add or extend trace propagation tests (unit + integration where applicable).
  3. Documentation: update or add docs/backend/OBSERVABILITY.md (architecture notes, OpenAPI, or runbooks).
  4. Comments & types: document public APIs, config knobs, and failure modes clearly.

Primary touchpoints: src/telemetry/otel.rs

Test and commit

• Run the project test command (e.g. pytest, cargo test, or npm test — follow repo conventions).
• Cover edge cases listed in the description; add regression tests for any bug found.
• In the PR, include summarized test output and security / ops notes (authn/z, data handling, rate limits).

Example commit message

feat(obs): opentelemetry tracing

Guidelines

• Target ≥ 95% coverage on new or materially changed modules (per language/tooling configured in CI).
• Documentation must allow a new engineer to operate and verify the feature locally.
• Timeframe: 96 hours from assignment.