[Feedback] Profiles premature optimization

[bogdandrutu]

There is a premature optimization in the profiles proto https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto#L53 and the whole ProfilesDictionary idea. The premature optimization is not that it exists this dictionary, is that it is a global dictionary instead of a resource level dictionary. The benefit is very minimal compare with the huge downside that merging multiple requests coming from multiple resources is impossible (or close to impossible).

[aalexand]

The motivation stems from here and it's supporting host-level profiles where resources correspond to a container. Per-container profiles will have duplication such as kernel function names, system libraries function names. Or even application symbol names if multiple replicas of the app run on the host.

cc @open-telemetry/profiling-maintainers

[felixge]

+1 to what @aalexand said. We're aware of the tradeoffs, but merging profiles from different requests is a less important use case to us than supporting container level resource profiles.

We expect that merging only needs to be implemented once in the collector, and the complexity is acceptable there.

Let us know if this makes sense or if we should continue the discussion.

[bogdandrutu]

The motivation stems from #628 (comment) and it's supporting host-level profiles where resources correspond to a container. Per-container profiles will have duplication such as kernel function names, system libraries function names. Or even application symbol names if multiple replicas of the app run on the host.

I understand, but I do believe the tradeoff does not have a good ROI, mostly because in a case (assume 200 containers on a host) the win of doing manual encryption vs relying on some compression mechanism snappy/zstd/etc. does not make sense to me.

[felixge]

I understand, but I do believe the tradeoff does not have a good ROI, mostly because in a case (assume 200 containers on a host) the win of doing manual encryption vs relying on some encryption mechanism snappy/zstd/etc. does not make sense to me.

I assume you mean compression rather than encryption?

As far as potential efficiency arguments are concerned, could you elaborate a bit more? Wouldn't the case of having 200 containers on a host greatly benefit from a global directory rather than a resource level one? I'd imagine that many/most of these containers would be using the same languages and runtimes with shared symbols for stdlib and common libraries?

We're definitely happy to take your feedback into account, but right now it's a bit too vague to be actionable.

[tigrannajaryan]

I think what @bogdandrutu is saying is that zstd and other general purpose compression algos are really good at eliminating this duplication and even with 200-fold repetition of dictionaries after the compression on the wire you will likely see a fairly small total size difference. I think that's fair and it would be useful to see benchmarks that show this (or disprove it).

However, while that is true about wire sizes, it is not relevant for in-memory sizes. After you uncompress the payload in-memory you do end up with 200 duplicate dictionaries, with potentially a lot more memory usage.

It is also possible that much larger uncompressed input to zsts/snappy/etc cause an increase in compute even if the compressed output size ends up being similar.

I would really love to see some benchmarking done that demonstrate these aspects (wire size, in-mem usage, compress+serde compute) when changes are made to the proto for the purpose of optimization.

[bogdandrutu]

Wouldn't the case of having 200 containers on a host greatly benefit from a global directory rather than a resource level one?

How do you get to the point of producing this "global directory". You will most likely pay more CPU to construct this out of multiple incoming requests than a highly optimized compression algorithm that have lots of small engineers behind.

I do feel that this is a hypothetical scenario, with real good understanding on how do we get to really benefit of this. So I stand by my words that this is a premature optimization, and most likely we will never be able to merge these maps across multiple sources (or very rarely) and does not justify the complications when it comes to usability.

I assume you mean compression rather than encryption?

Thank you for fixing my words on this.

[aalexand]

How do you get to the point of producing this "global directory". You will most likely pay more CPU to construct this out of multiple incoming requests than a highly optimized compression algorithm that have lots of small engineers behind.

In-memory data is not compressed though. If a given machine runs 100 containers that are replicas of the same program and each container profile references 10000 functions with each function name being 100 bytes on average, the current approach seems like a reasonable way to store 1 MB of function names in memory instead of 100 MB. I wouldn't say it's hypothetical.

most likely we will never be able to merge these maps across multiple sources (or very rarely)

Can you clarify? It's not very clear what inability to merge this is about.

[felixge]

@bogdandrutu can you outline the specific use case you have in mind? Premature optimization can be very subjective term when different stakeholders get together.

I think @aalexand and my perspective is biased by working on profilers and tools that analyze/visualize profiling data. I suspect you're biased towards collector use cases? Or something different?

Maybe you can join our next meeting on Thursday for a high bandwidth discussion?

The profiling SIG has spend a lot of time discussing the tradeoffs of leaving compression to general purpose compressor algorithms versus leveraging lookup tables and other domain specific knowledge about data distributions. The general conclusion is that neither of the extreme approaches are good. You can't leave it all to gzip and friends, nor can you do it all in the wire format. There is obviously a lot of room to argue the details when it comes to balancing the tradeoffs, but I just want to reassure you that we're not ignorant of them and had a lot of discussions about this already.

[bogdandrutu]

In-memory data is not compressed though. If a given machine runs 100 containers that are replicas of the same program and each container profile references 10000 functions with each function name being 100 bytes on average, the current approach seems like a reasonable way to store 1 MB of function names in memory instead of 100 MB. I wouldn't say it's hypothetical.

Maybe my knowledge about containers is wrong, but I do expect that you cannot share this map across different containers (unless you do SHM, etc.), so my question still stands, how do you get into 1 linux process with all these data so that you benefit out of this? And more than how you get, how much you pay to get to build this map?

Can you clarify? It's not very clear what inability to merge this is about.

Can you please start and explain to me a basic setup where you have on a VM with 100 containers and how do you get with all the data in 1 place so that you can build this map.

[aalexand]

A profiler that uses something like Linux perf in system-wide mode profiles all processes and containers at once. https://github.com/open-telemetry/opentelemetry-ebpf-profiler is not per cgroup either I think. So the per cgroup data would get compacted early.

Another example is processing the profiling data on the backend - e.g. offline symbolization. Decoding a profile that has duplicated per cgroup strings will have the in-memory duplication I mentioned earlier.

[felixge]

@bogdandrutu as mentioned by @aalexand, a major use case is the ebpf profiler. We're happy to elaborate further on this. But I'd like to reiterate our requests for learning more about your use cases as well. How do you intend to build on the profiling signal? What profiling data source (profiler) are you planning to tap into?

[bogdandrutu]

@felixge I am coming from usability perspective, these data are very hard to use by a backend or by any transformation service. Wearing my TC hat, I need to ensure that we are making the right tradeoffs of usability vs some "hypothetical" performance win.