cloudwatch-sender

Retrieves metrics from Cloudwatch and sends them to InfluxDB/Graphite.

What is this?

Allows you to send EC2, SQS and custom metrics from Cloudwatch to InfluxDB or Graphite. EC2 metrics are gathered via EC2 tags instead of EC2 Instance IDs, making the tool far more dynamic.

Installation

Add this line to your Gemfile:

gem "cloudwatch-sender"

Then execute:

bundle

Or install it yourself:

gem install cloudwatch-sender

Usage

Command Line

cloudwatch-sender send_metrics /path/to/config.yaml --region=eu-west-1 --access-key-id=$AWS_ACCESS_KEY --secret-access-key= $AWS_SECRET_KEY

If you would like to stream metrics to your endpoint at a set interval, use continuous:

cloudwatch-sender continuous /path/to/config.yaml --region=eu-west-1 $INTERVAL --access-key-id=$AWS_ACCESS_KEY --secret-access-key= $AWS_SECRET_KEY

Note - the default $INTERVAL is 60 seconds.

cloudwatch-sender continuous /path/to/config.yaml --region=eu-west-1 $INTERVAL --provider=iam

Programmatically

require "cloudwatch/sender/cli"

loop do
  Cloudwatch::Sender::CLI.new.send_metrics(config_path = './configs/default.yml', { 'region' => 'eu-west-1', 'provider' => 'iam'})
  sleep 120
end

require "cloudwatch/sender/cli"

Cloudwatch::Sender::CLI.new.continuous(config_path = './configs/default.yml', 60, { 'region' => 'eu-west-1', 'provider' => 'iam'})

Configs

The gem is powered via a YAML config file, see example.yaml for an example.

Note: take into account how often metrics update for each AWS product:

• EC2 - every 60 seconds.
• SQS - every 5 minutes.
• Custom - every 60 seconds.

How it works

The gem extracts metrics for a given set of EC2 instances based on an EC2 tag key/value. For example:

ec2_tag_key:   ProjectName
ec2_tag_value: bbc_news

As seen in the example above a tag relates to a single project. Thus if that key/value is called, it returns all the instances attached to that project - which is what the gem then uses to gather metrics on each instance.

The gem collects EC2 metrics for the previous 3 minutes and SQS metrics for the previous 20 minutes. Thus running the gem every 60 seconds for EC2 and every 5 minutes for SQS will provide sufficient data. This allows for the gem to remain unaffected by network/database issues.

Why make this?

We found the existing tools heavily rely upon AWS variables which often become out-of-date, e.g. EC2 Instance ID. This becomes a problem when a component is being regularly re-built and deployed. Thus we made the decision to base cloudwatch-sender on a broder set of criteria, such as EC2 tags.

TODO

See open issues.

License

Standard MIT License, see included license file.

Authors

• David Blooman
• Charlie Revett

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am "Add some feature"
4. Push to the branch: git push origin my-new-feature
5. Create a new Pull Request.

Please feel free to raise an issue if you find a bug or have a feature request.