Pre-filing checklist
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Is your feature request related to a problem?
The https://www.stefdata.net/ is a new alternative to OTLP that demonstrates substantially better compression than we have for OTAP using the Arrow IPC encoding. While we see Arrow IPC encoding as a good option for fast/good compression,
- Users may be willing to trade compute for compression; STEF's compression results.
- Without an implementation in Rust, we do not feel we know the relative CPU cost. In our opinion, the Golang implementation of OTAP is not applicable because of the overhead of interacting with OTLP message objects (pdata).
Proposed Solution
There is a Go implementation we can use for reference: https://github.com/splunk/stef/tree/main/go/pdata/metrics
Alternatives Considered
As a requirement, we expect to follow the pattern used for ingesting OTLP bytes: we expect to interpret STEF and construct OTAP records directly in memory (receiver), then to interpret OTAP records and construct STEF bytes directly.
Additional Context
No response
Pre-filing checklist
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Is your feature request related to a problem?
The https://www.stefdata.net/ is a new alternative to OTLP that demonstrates substantially better compression than we have for OTAP using the Arrow IPC encoding. While we see Arrow IPC encoding as a good option for fast/good compression,
Proposed Solution
There is a Go implementation we can use for reference: https://github.com/splunk/stef/tree/main/go/pdata/metrics
Alternatives Considered
As a requirement, we expect to follow the pattern used for ingesting OTLP bytes: we expect to interpret STEF and construct OTAP records directly in memory (receiver), then to interpret OTAP records and construct STEF bytes directly.
Additional Context
No response