Cannot ignore a particular request

[CatalinMustata]

I've just migrated our system to remote strategies and while the strategies seem to be picked-up (they were const before, now they are probabilistic), I can't seem to get "unnamed" requests to be ignored (they appear as "http_request").

The server is a NodeJS / Hapi one using the jaeger-client-node. This seems to have a strange limitation in that sampler.param is required even if sampler.type is remote. I've set this to 1.

The Collector is deployed in K8S, and the sampling strategies file looks like this:

{
  "service_strategies": [
    {
      "service": "Gateway",
      "type": "probabilistic",
      "param": 1.0,
      "operation_strategies": [
        {
          "operation": "http_request",
          "type": "probabilistic",
          "param": 0.0
        }
      ]
    }
  ],
  "default_strategy": {
    "type": "probabilistic",
    "param": 1.0
  }
}

The idea is that everything should be probabilistic with param 1.0, except http_request which should be 0.0 (ignored).

However, I did receive this span:

{
    "data": [
        {
            "traceID": "d439103202ee204a",
            "spans": [
                {
                    "traceID": "d439103202ee204a",
                    "spanID": "d439103202ee204a",
                    "flags": 1,
                    "operationName": "http_request",
                    "references": [],
                    "startTime": 1637081523292000,
                    "duration": 1945000,
                    "tags": [
                        {
                            "key": "sampler.type",
                            "type": "string",
                            "value": "probabilistic"
                        },
                        {
                            "key": "sampler.param",
                            "type": "float64",
                            "value": 1
                        },
// removed to keep this short

For some reason, this span is not ignored and reports sampler.param to be 1.

[yurishkuro]

strange limitation in that sampler.param is required even if sampler.type is remote

I think the value might be used as default probability when the client cannot reach the server to retrieve the actual sampling strategies

[CatalinMustata]

Hmmm, that would make some sense, yeah. It would've been nice if that was documented though.

[yurishkuro]

For some reason, this span is not ignored and reports sampler.param to be 1.

Are you sure that the remote strategy was actually retrieved by the client? You need to validate that other operation names are getting the expected non-default probability before considering that specifically "http_request" is not working (i.e. maybe everything is not working / not communicating)

[CatalinMustata]

My initial assumption was that the client picked it up, since the sampling type changed from const to probabilistic, but seeing your previous comment made me think it might have just defaulted to that.

The collector does have logs this one below, so it does make me thing communication is happening:

{"level":"debug","ts":1637081202.5178516,"caller":"static/strategy_store.go:89","msg":"sampling strategy not found, using default","service":"Gateway"}

I'll try to set logging to info on the collector and set different values to all of the entries in the strategies file, so I can identify which one is being picked up.

[CatalinMustata]

You were right in that the strategies were not picked up. There was a typo in my yaml file which lead to the collector using its own default strategy. Now it seems to apply the params I set for the main requests, although I did need to restart the client pods after I changed the values back to 1.0 - it seems like the clients do not query the collector on each request, so I'm assuming this happens from time to time.

An interesting note was that the sampling param seems to affect attached logs as well - not sure if this is a bug or not, but when running with a value less than 1.0, I noticed that some log entries were dropped from the trace.

You were also right in that the sampling.param from the client sets a default in case no valid strategy can be retrieved. Now, comes the interesting part - I've set that default to 0.9 in the Gateway service and this seems to be used for that pesky "http_request". That is very strange considering the service is setup to use 1.0 for all operations except "http_request".

EDIT: Scratch all of that, I've let it sit for an hour and when I looked at the previously recorded traces (just after restarting the Gateway service) I noticed all of them were recorded with a param of 0.9. I've done a new set of requests and these now come with 1, making me think the client did not pick up the strategy instantly. What is interesting is that when using the remote strategy, some child spans are completely ignored (and some logs are still missing).