http.route for client & server both as opposed to only server?

[dpk83]

What did you expect to see?

http.route is only present for server. I expect a similar tag for client also to capture the request route.

Additional context.

Add any other context about the problem here. If you followed an existing documentation, please share the link to it.

[dpk83]

@reyang Could you please help adding [area:semantic-conventions] [semconv:HTTP] labels

[reyang]

@reyang Could you please help adding [area:semantic-conventions] [semconv:HTTP] labels

Done.

[lmolkova]

HTTP route is not available on the client side and calculating it in the instrumentation is not possible without knowing the pattern. The best we can do here is an opt-in mechanism where it's populated manually or matched using provided regex.
In any case, this attribute can only be added at optional requirement level and it's unlikely instrumentation libraries would support it.

[dpk83]

In any case, this attribute can only be added at optional requirement level and it's unlikely instrumentation libraries would support it.

This attribute as optional is perfectly fine but this is important to be in otel semantics as it’s an important attributes.

[lmolkova]

@dpk83 can you share some scenarios of how it can be used and why it's important?

[dpk83]

@lmolkova It's important piece of information to be able to monitor the health of APIs. A lot of services has dependencies on lot of external services and the reliability of individual requests is as important as the overall health of the target service. Most of our systems care about the monitoring health by request and the request route is important piece of information to be able to create dashboards and setup alerting on per request basis.

How a user sets is can differ but the convention of what the name of the dimension to use and what the values should be is critical for standardization. As this is pretty core requirement for outgoing request metrics, standardizing it is as important as it is for incoming request metric.

[MrAlias]

@dpk83 why not just add the route to your instrumentation?

Adding this to the semantic conventions and enforcing all users to have their clients include the route is unreasonable. Many users do not want the high cardinality route tag of a client to pollute their observability systems.

[dpk83]

@dpk83 why not just add the route to your instrumentation?

Adding this to the semantic conventions and enforcing all users to have their clients include the route is unreasonable. Many users do not want the high cardinality route tag of a client to pollute their observability systems.

It's not forcing as we are talking about this attribute as optional here. What you are saying here applies to a lot of recommended and optional attributes which are part of the semantic convention.

I was actually also going to question making the http.scheme as required, as we have 1000s of services running in our org and not one of them utilizes this today in metrics.

[MrAlias]

I was actually also going to question making the http.scheme as required, as we have 1000s of services running in our org and not one of them utilizes this today in metrics.

The http.scheme has two possible values, the client side http.route has infinite. That comparison seems to indicate the point others and I are trying to make is being missed.

Suggesting OTel sponsored instrumentation should include this attribute on the client side means all users will have a cardinality explosion on the distinct telemetry they send to their observability systems. It is not something we should do.

[lmolkova]

@dpk83

Regarding route, I still don't understand how solution would look like. Route is not readily available on the client. Client have http.url, i.e. https://foobar.com/v1/api/customers/foo/orders/xyz/123?p=42. Route would be v1/api/customers/{name}/orders/{id}/{sub-id}
Without customization (e.g. span processors with regex) or fully manual instrumentation, it's not possible to populate route. However, when it's populated manually, there is no limitation on adding http.route to client spans. It can be added, and custom queries or dashboard can be built on top of it.
Expecting vendors to build any experience based on route on client spans would be unrealistic since it can only be added manually and won't be part of instrumentation libraries.

Information about http.scheme is available on both server and client spans. On server spans it's available as a separate attribute, on clients it's available as part of http.url. This allows us to avoid duplication and keep instrumentations efficient. Clients usually only know URL, servers only know URL components. It's relatively easy to add span processor that would extract components from http.url and populate them in addition to URL.

[dpk83]

The http.scheme has two possible values, the client side http.route has infinite. That comparison seems to indicate the point others and I are trying to make is being missed.

Suggesting OTel sponsored instrumentation should include this attribute on the client side means all users will have a cardinality explosion on the distinct telemetry they send to their observability systems. It is not something we should do.

How's that different than http.route for incoming requests? Why do we have it for incoming request then?

[dpk83]

In fact I am fine if it's not added to otel semantic convention as long as that stand doesn't change in future. i.e. we can go ahead and standardize this additional tag for our services as long as there is an assurance that in future otel semantic convention won't add it and thus we won't have to change all services to adapt to it.

[lmolkova]

@dpk83 for incoming requests it calculated by the HTTP framework. E.g. ASP.NET Core calculates it (knowing all the routes in the server app) and instrumentation takes readily available data.

HTTP spec explicitly says that route can only be populated when available.

[1]: 'http.route' MUST NOT be populated when this is not supported by the HTTP server framework as the route attribute should have low-cardinality and the URI path can NOT substitute it.

If a pure http-listener kind of server app is used that does not support routing, it's not available on server and not populated.

Any of these requirements do not prevent user apps from adding any attributes anywhere manually.

WRT stability, this document is experimental and might change.

[crsantos]

I understand what @dpk83 said about:

the request route is important piece of information to be able to create dashboards and setup alerting on per request basis

And I have exactly the same issue with dashboards: aggregation.

@lmolkova is right, this should not be created on clients.

But I would like to validate if injecting http.route on the collector with a processor is the correct way.
And how can we do it? Since I only know how to replace_match a specific field to mask parameters 🤔