Support Remote Entities in New Relic Backend

[jaloren]

In the past few months, newrelic has a brand new go SDK for generating infrastructure metrics. See here. The current metrics in the backend are rolled by hand and don't conform to the newer version of the newrelic infrastructure metrics.

If this functionality is something you'd be willing to accept, I'd be willing to submit the PR for it.

[tiedotguy]

cc @Kav91 - thoughts?

[Kav91]

@jaloren @tiedotguy the New Relic Infrastructure Go SDK essentially just produces a JSON payload which the agent then reads and processes. The implementation I've added within GoStatsD obviously meets the exact same spec, which is just a very simple structure.

Adding the full SDK is over kill imo, we would be adding weight with many features that would be unused, which then spits out the exact same JSON structure but with more moving parts. We would not see any value/functionality add, and may in fact see a performance overhead, since it has its own way of storing metric sets, before flushing them out which overlaps with GoStatsD's own internals/concepts.

The other thing to note, is without the SDK it remains very simple to manipulate the payload to support sending to the insights platform directly as well which this already does. Whereas the Go SDK is specifically for via the NR Infrastructure Agent.

edit: also the payload being sent complies with the latest V3 SDK (v3 sdk sends json schema/protocol v2 and now v3), this already sends protocol version 2, and protocol version 3 is basically the same "but it improves remote entities uniqueness by adding "local address replacement on entity names" at agent level." so we could just change "v2", to "v3" in the payload sent and thats it.

[tiedotguy]

@Kav91 - thanks for the quick input!

@jaloren - is there a specific issue of (in)compatibility you've encountered? Personally I would prefer to keep dependencies low, but that's my general philosophy.

[jaloren]

@tiedotguy There is. It looks like the current newrelic backend supports the v2 schema. I am looking for support for the v3 schema, which handles multiple entities. See the following for details:

• https://github.com/newrelic/infra-integrations-sdk/blob/master/docs/v2tov3.md
• https://github.com/newrelic/infra-integrations-sdk/blob/master/docs/tutorial_multiple_entities.md

In my particular case, I want to have multiple statsd clients send their metrics to a single gostatsd server and then have that server generate newrelic integration where each client's metrics are a different entity. Additionally, licensing and metric retention for infrastructure metrics is very different then just pushing this into insights, not to mention that we can add important labeling through the agent.

I fully understand the desire to avoid adding a big dep just to get a json doc out of it. I'll just create a small program that uses gostatsd as a library, which will achieve what I want.

[tiedotguy]

I admit - I completely lack knowledge of Newrelic and the implications for this, so I'll have to defer back to @Kav91 again.

A note on consuming gostatsd as a library - it's certainly doable (https://github.com/sensu/sensu-go consumes it for example), but you'll want to be sure to pin the commit/version. It's semver, but library usage is largely a secondary consideration, which is why the major number gets bumped for almost every change.

If you make such a program and it's open source, if you could leave a link that would be great. It would be neat to have a list of examples in the README.md or something.

[Kav91]

@jaloren support for multiple entities exist in the v2 schema, the second link you have actually shows the redis integration supporting multiple entities using v2, and I can confirm this as I've written a lot of integrations at NR. (I think you are getting confused between the SDK versions, and the schema protocol versions.)

As I previously linked before the v3 protocol only improves remote entities uniqueness by adding "local address replacement on entity names" at agent level. Basically to avoid naming collisions.

But too rewind back, the problem statement/issue has changed now. It's not about using the latest SDK or protocol version, but remote entity support.

So I can definitely see the merit in having unique entities, but with StatsD we have to shuffle our thinking on how we address this. Basically if GoStatsD is running say on an EC2, and the NR EC2 integration is setup, any infrastructure events that go through that agent/host, gets decorated with the metadata like ec2tags for that very same entity being what is running GoStatsD, which you've referred to as labels already. So having remote entity support will associate the incoming metrics to the entity they are coming from instead.

Now with Redis for example, since we are defining the config with each remote entity ahead of time, its easy to have those entities distinguished. With GoStatsD we would need to think about having a tag we reserve, eg. hostname, and then group entities by that tag/host, and assign the metrics accordingly to each entity before sending over. We need a way to tell GoStatsD hey this came from this remote entity and should be grouped accordingly.

I think this is easily achievable with v2 protocol, but we can total just change the value to v3 protocol anyway, and just do some grouping before flushing. Ultimately don't think loading up the entire SDK for grouping is needed, and also prefer keeping dependencies low when possible.

[tiedotguy]

For that there's the Hostname field on all the metric types, you can see how it's consumed in the Datadog backend (which also has that as part of its schema). There's a few ways it's populated, but ultimately it's the source, and it may be empty (ie, for non-host based timer aggregation), so that needs to be handled as well.

[jaloren]

@Kav91 you are right. I was definitely guilty of engaging in an XY problem. I don't care about the SDK, I care that the data structure is emitted in a particular way. To that end, I'd just like to make sure we are on the same page.

This is currently what gostatsd is generated for the newrelic backend.

{
  "name": "com.newrelic.gostatsd",
  "protocol_version": "2",
  "integration_version": "2.1.0",
  "data": [
    {
      "metrics": [
        {
          "event_type": "GoStatsD",
          "integration_version": "2.1.0",
          "interval": 10,
          "name": "vault.vault3.norvax.test.runtime.malloc_count",
          "timestamp": 1563025140,
          "type": "gauge",
          "value": 345400
        },
        {
          "event_type": "GoStatsD",
          "integration_version": "2.1.0",
          "interval": 10,
          "name": "vault.vault2.norvax.test.runtime.malloc_count",
          "timestamp": 1563025140,
          "type": "gauge",
          "value": 345405
        }
      ]
    }
  ]
}

And here's roughly what I would expect it to look like. its important not only that the entity correspond to the remote addr but also the event type is configurable as well. For example, if i have consul agent and vault servers generating statsd metrics, those should be different event types.

A nice to have would be dynamically creating string attributes based on event type or entity. For example, it would nice to specify what environment the metrics came from.

{
  "name": "com.newrelic.gostatsd",
  "protocol_version": "2",
  "integration_version": "2.1.0",
  "data": [
    {
      "entity": {
        "name": "vault2.test",
        "type": "vault"
      },
      "metrics": [
        {
          "event_type": "hashicorp_vault",
          "interval": 10,
          "timestamp": 1563025140,
          "type": "gauge",
          "runtime.malloc_count": 345400
        }
      ]
    },
    {
      "entity": {
        "name": "vault3.test",
        "type": "vault"
      },
      "metrics": [
        {
          "event_type": "hashicorp_vault",
          "interval": 10,
          "timestamp": 1563025140,
          "type": "gauge",
          "runtime.malloc_count": 345400
        }
      ]
    }
  ]
}

[tiedotguy]

One thing I want to avoid is having gostatsd parse tags out of metric names in an unstructured format.

For example, I believe there's a statsd client that doesn't use the dogstatsd line protocol, but rather something like metric[foo=value;bar=value]:10|ms - I would consider adding that, but something like metric.<foo>.<bar> I wouldn't. There's a few reasons for this, but the top two are non-technical - 1. tags are the way of things, services should start using them, and 2. it puts ownership of the list on to the gostatsd administrator, and I don't believe that should be something they manage.

Perhaps one day I'll make a proxy which can parse and rewrite metrics like that (someone probably already has), but that's ok because the maintenance of any proxy belongs to the service emitting metrics.

So with that view in mind, if the metric being emitted can be emitted with tags, then the Newrelic backend could use a magic prefix to route tags to being part of the entity, rather than being part of the metric. So instead of vault.vault2.norvax.test.runtime.malloc_count, it could be runtime.malloc_count with NR_type=vault, NR_name=vault2.test, NR_event_type=hashicorp_vault. I don't know the implications of changing the event_type, but I assume any implementor would test that :)

My ideal scenario however is the metric has a host tag, the host goes in to the name field of the entity, and on the backend Newrelic is able to correlate that name with other entity metadata provided by something else, such as the Newrelic infrastructure agent (this is basically the way it's done with Datadog host tags). I don't know if Newrelic supports such correlation though.