Create an alarm on the Kinesis stream for GetRecords.IteratorAge

[alexwlchan]

Earlier this week, the Lambda got stuck and stopped processing records from Kinesis into Elasticsearch. The Lambda was timing out; we fixed it by increasing the Lambda timeout – but we only noticed because we were looking at the reporting cluster and noticed the lack of search information.

We could be alerted to this in future by creating an alarm on the GetRecords.IteratorAge metric, and sending a Slack alert when it goes above 0 – this means records aren't being processed off the stream. The graph above shows how this metric spiked when we had the outage.