Support prometheus

[mmpei]

#161
add 'http_' as prefix of metrics name. for there is a restriction in prometheus that metrics name should start with letter.
and you can set backend to 'promethues' to active prometheus metrics.

metrics:
  backend: prometheus
  m3:
    service: kraken-agent
    host_port: 10740
  statsd:
    host_port: 0.0.0.0:10740
    prefix: kraken-agent
  prometheus:
    listen_address: 0.0.0.0:10740

[codygibb]

This is going to break all of our existing monitoring. Is there a way to make this backwards compatible?

[evelynl94]

This is going to break all of our existing monitoring. Is there a way to make this backwards compatible?

+1, maybe its better to wrap promethues metrics to add a prefix.

[evelynl94]

also http_ usually implies its related to http endpoints...you should just use kraken_

[mmpei]

This is going to break all of our existing monitoring. Is there a way to make this backwards compatible?

+1, maybe its better to wrap promethues metrics to add a prefix.

it is difficult to do. i have considered that but it will break the abstract architecture if only add prefix to promethues. i will try.

[mmpei]

also http_ usually implies its related to http endpoints...you should just use kraken_

you are right. maybe kraken_http_ to identify that it's from kraken and related to http.

[yiranwang52]

I actually think this change is fine...let's discuss a little more about this

[jlegrone]

The yaml config keys are camel cased in the tally package:

https://github.com/uber-go/tally/blob/f266f90e9c4d5894364039a324a05d061f2f34e2/prometheus/config.go#L33-L58

Perhaps we should prefer camel case here as well so that it's not a breaking change if we want to swap PrometheusConfig out with the tally prometheus.Configuration type in the future.