Add ability to configure lookup credentials for multiple AWS accounts

[knightsg]

The detail lookups configuration for this plugin only provides the option to enter credentials for a single AWS account. I am currently collecting logs from kinesis streams in multiple accounts but the detail lookup only works for one of the accounts, which means I can only filter/stream on entity types for a single AWS account.

It would be useful to have this feature as maintaining multiple AWS accounts is now a common use case.

[radykal-com]

Hello,

I agree that would be nice to have lookups for multiple AWS accounts.

As far I can see, this plugin should be developed taking into account that it's very common to run Graylog inside AWS infrastructure and also that is very common to have more than one AWS account you would like to collect logs from, without forget that there are multiple ways to implement the authentication process (secret keys, cross accounts, IAM roles...)

Right now, the lookup processor is configured at same level that the generic settings for the plugin.
Maybe, to implement this feature the lookup processor should be refactored to some kind of multi-instance service with it's own configuration page, allowing CRUD operations of processor instances (maybe a new configuration page for the processor, accessible from the plugin configuration page?)

[hybby]

👍 from me.

It'd be great if this plugin could tell that the running host was an EC2 instance and if so, provide an option to use its IAM Role instead of hardcoded credentials.

Just noticed @radykal-com's comment on #57 which addresses my use-case:

This functionality is already implemented. If you don't specify the keys in the input config, it will try to authenticate using the roles assigned to the instance profile.

[knightsg]

We have a single cluster of graylog servers ingesting logs from multiple AWS accounts, so if we leave the key fields blank and it uses an IAM role, I would imagine it's still only going to use the role for the account where the graylog server is and only look up instance details from that account - is that correct? In which case, it doesn't solve the original issue, which is looking up details for logs from other AWS accounts.

[radykal-com]

Exactly, the plugin doesn't support right now to run instances lookup in multiple accounts.