Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lua as a processor for Traces and Metrics #9534

Open
mp3monster opened this issue Oct 28, 2024 · 0 comments
Open

Lua as a processor for Traces and Metrics #9534

mp3monster opened this issue Oct 28, 2024 · 0 comments

Comments

@mp3monster
Copy link

It would be very powerful if Lua could be used as a Processor not just for Logs, but also Traces and Metrics. This would allow users to address issues such as implementing sampling on traces (currently an option with https://opentelemetry.io/docs/concepts/sampling/)

Describe alternatives you've considered
Ability to convert a Trace or Metric to a log and back again—this would allow all of Fluent Bit's features to be leveraged.

Additional context
Unable to apply sampling or filtering of traces to create metrics for traces with outlier thresholds (e.g a span execution time greater than X) and to sample traces to reduce the traffic volumes being sent onwards. While processing multiple events if we can sample the the latency for events that aren't showing anomalies - we can start to analyze performance levels over time without sharing every event.

We would need the downstream handling to also deal with null records.

The Processor with Lua could then look like:
service:
log_level: debug
http_server: on
flush: 1

pipeline:
inputs:
- name: opentelemetry
listen: 0.0.0.0
port: 4317
processors:
traces:
- name: lua
call: modify
code: |
function modify(tag, timestamp, record)
sampleThreshold = 0.7
if (math.random() > sampleThreshold)
record = null
-- we're assuming that returning a null record would effectively be dropped
return 1, timestamp, record
end

outputs:
- name: stdout
match: "*"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant