Pre-filing checklist
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Description
The num_items() for OTLP data costs at least one memory allocation. It shouldn't need to, since there is a class of iterator we could have that intentionally skips unrecognized fields, instead of pushing them onto a stack.
The batch processor has a bunch of uses of num_items that show up in profiles, and it doesn't need to use this method for its bookkeeping. When the num_items result is used only for metrics, it should be an opt-in metric.
I suggest that nodes should not calculate simple throughput metrics themselves, this should be performed in channel metrics for all nodes (opt-in).
I will file 3 separate child issues.
Additional Context
No response
Pre-filing checklist
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Description
The
num_items()for OTLP data costs at least one memory allocation. It shouldn't need to, since there is a class of iterator we could have that intentionally skips unrecognized fields, instead of pushing them onto a stack.The batch processor has a bunch of uses of num_items that show up in profiles, and it doesn't need to use this method for its bookkeeping. When the num_items result is used only for metrics, it should be an opt-in metric.
I suggest that nodes should not calculate simple throughput metrics themselves, this should be performed in channel metrics for all nodes (opt-in).
I will file 3 separate child issues.
Additional Context
No response