HTTP header and body extraction

[NimrodAvni78]

We wanted to open this issue to discuss the design of technical implementation of header and body extraction for http

Requirements:

• attach http request and or response headers to spans as attributes
  • needs to be opt-in
  • some glob pattern to specify which headers to add to spans
  • needs to follow otel semantic conventions
    • http.(request|response).header.<key>
  • for some reason semantic convention says the value of the attribute should be a is a list of values instead of a single one, where a header can appear multiple times, don’t think thats a real case?
• attach http request and or response body to a span
  • customer should limit the size of the payload to extract
  • i’m pretty sure there are no otel semantic conventions for this
    • we can invent our own and try to push them to otel, http.(request|response).body.payload
  • we can also populate the http.(request|response).body.size attributes
  • needs to be opt-in
  • maybe we should concider adding sampling to this, maybe we dont want every span to have the payload since it might add significant cost, or is normal span sampling good enough
  • obfuscation, we must allow users to remove sensitive parts from their request / response payloads, and replace them with some filler (***), we might want to have multiple strategies for obfuscating payloads

    • json obfuscation: we can detect the payload is json relying on the content-type header:
      for example, lets look at this example json

      {"root":{"username": "myusername", "age": 51, "password": "mypassword"}}

      we can have multiple strategies of how to obfuscate jsons

      • json_values: obfuscate all json values

        {"root":{"username": "***", "age": "***", "password": "***"}}

      • json_path: obfuscate all json values matching a specific jsonPath

        • jsonPath: $.root.password

        {"root":{"name": "myusername", "age": 51, "password": "***"}}

      • json_value_pattern: obfuscate all json values matching some glob pattern

        • pattern: my*

      {"root":{"name": "***", "age": 51, "password": "***"}}

      • note, this might be too cpu intensive to be worth implementing

    • Plain Text obfuscation: for plain text payloads we can just remove the entire payload and replace it with *** using the plaintext obfuscation

this is an example of how a client can configure this:

payload_extraction:
  http:
    max_payload_size: 8192 # maximum size of http packtes to send
    truncate: true # if http request size is bugger the max_payload_size, should we truncate it to fit or ignore the payload extraction for this request
    mode: # maybe a better name
      # add validation for repating values, should we warn or error?
      - "request_headers"
      - "response_headers"
      - "request_body"
      - "response_body"
      - "headers"
      - "body"
      - "all"
    headers:
      pattern: '!(*Authorization*)'
      request:
        # do we want this to be override or merged?
        pattern: 'x-my-custom-req-header'
      response:
        # do we want this to be override or merged?
        pattern: 'x-my-custom-res-header'
      # obfuscation only runs on body
    body:
      obfuscation:
        - type: "json_path"
          json_path: ".root.password"
        - type: "json_value_pattern"
          json_value_pattern: "PK-*"
        - type: "json_values"
        - type: "plaintext"

implementation:

• this can depend on the large_buffers feature that was added, to send the full http payload from kernel space, and do the header and value parsing in userspace

open questions:

• what is the difference between obfuscation of headers and just remove them them, do we need both?

what else it gives us:

• besides giving additional data on spans, it lets us infer a lot of http based protocols in the future, stuff like graphql, elastic and so on

[NimrodAvni78]

@mmat11 @mariomac @grcevski @rafaelroquetto @marctc would love to hear your opinions on this

[rafaelroquetto]

Hey @NimrodAvni78, I think that's a pretty cool idea. In the past months I've been wondering myself about the pros and cons of userspace parsing, as not only does it provides users with richer data, but it gives us a lot more flexibility by allowing us to do work in the userspace. The downside, as you are aware, is that it probably comes with a huge overhead and performance penalty. In a different project prior to Grafana, we used to do something analogous, but the overhead of dispatching eBPF events to userspace for post-processing became a real CPU and memory problem (and by extension, a cost issue). Obviously, that was a different project so I am comparing a apples with oranges, but this serves as a north. Closer to home, we've already experienced signifcant CPU spikes which culminated with this PR - which is just the tip of the iceberg (there's more to be done on that front, we just didn't have the time to approach it).

As a matter of fact, a similar issue still happens to one of our ring buffers: the logger. This ring buffer is probably the one with the largest traffic, and the kind of data flowing through (strings) it is somewhat analogous to what we would be working with when parsing HTTP headers and friends. As a result, when OTEL_EBPF_BPF_DEBUG=1 is set, it's not uncommon to see the ringbuffer overflow and log events being dropped because the userspace simply can't keep up with it (and why often times we end up digging directly into /sys/kernel/debug/tracing/trace_pipe).

So, IMHO, I think we should probably work on benchmarking the current large buffer implementation as a pre-requisite to this, in order to understand how far we can stretch it and consequently, whether shipping things to userspace as you are proposing is actually feasible. I am happy to help in that front, and I actually have some things in mind on how we can go about it.

Happy to elaborate more, either here, or during the SIG meeting.

[NimrodAvni78]

yeah @rafaelroquetto would love to hear more, what exactly you want to benchmark and what you have in mind about it

[rafaelroquetto]

The usual culprits are good starting points, namely:

• CPU usage
• memory consumption/allocation
• event throughput

[mmat11]

Hi @rafaelroquetto, I added support for large buffers in the HTTP path and collected some benchmark data using pprof. The test setup was a VM running a simple server, with a client that sent parallel requests configured as follows:

concurrency = flag.Int("c", 100, "number of concurrent workers")
qps         = flag.Int("qps", 200, "global max requests per second")
timeout     = flag.Duration("timeout", 3*time.Second, "http client timeout")

OBI config:

log_config: true
open_port: "8080"
log_level: WARN
profile_port: 6060

discovery:
  skip_go_specific_tracers: true

ebpf:
  buffer_sizes:
    http: 0 // 1024, 2048, 4096, 8192

trace_printer: "text"

Bench script:

#!/usr/bin/env bash
SIZE=$1

curl -s -o "heap.${SIZE}.before.pprof" "http://localhost:6060/debug/pprof/heap"
curl -s -o "cpu.${SIZE}.pprof" "http://localhost:6060/debug/pprof/profile?seconds=60"
curl -s -o "heap.${SIZE}.after.pprof" "http://localhost:6060/debug/pprof/heap"
curl -s -o "allocs.${SIZE}.pprof" "http://localhost:6060/debug/pprof/allocs"

NOTE: the 8192 sized test is currently gated to 4096 by the HTTP handling ebpf code (whereas the TCP code works fine). I haven't investigated this further, 4096 bytes are already a lot for now.

From what we could observe, there isn't much difference cpu-wise, there's a difference in the allocations, which is expected.
I attach here all the .pprof files in case you want to have a look.

pprofs.zip

This is the branch I used for testing: #631

[grcevski]

I wonder if we used to have much bigger problem with CPU overhead before @rafaelroquetto changed the use of the ringbuffer to casts instead of binary parsing? I'm happy to take the win, I'll try to reproduce on my end to check what it might look like.

[rafaelroquetto]

Thanks again for submitting those. I've done some profiling on my end as well from that branch, and I don't see anything unexpected. The bottleneck is the number of allocations we have to make, that puts pressure both on the CPU and on the heap - but that's just the nature of using the large buffers. This may be something we want to revisit in the future.

Just for quick reference, this is what I saw both in terms of CPU and memory when torturing OBI:

CPU

Mem

Used 8K buffers as a baseline.

IMHO we should go ahead with this, but be mindful.

We can look into optimising those allocations at a later time.