Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement better key naming fn #8

Open
lbradstreet opened this issue Jan 13, 2017 · 0 comments
Open

Implement better key naming fn #8

lbradstreet opened this issue Jan 13, 2017 · 0 comments

Comments

@lbradstreet
Copy link
Member

We need the key naming fn to randomize the key prefixes to ensure proper s3 sharding. This could be done by generating a timestamp and reversing the end of the timestamp to ensure randomness.

The input plugin would then need to be modified to make this more pleasing to use, and ensure order is maintained.

This would break prefixing, as it depends on keys sharing a prefix. This could be worked around by listing all keys and filtering from onyx, rather than having amazon's servers pre filter. If we go this route we should have a metric for the time it takes to filter the keys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant