Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement better cogs accounting for worker time usage #233

Open
markstory opened this issue Mar 7, 2025 · 1 comment
Open

Implement better cogs accounting for worker time usage #233

markstory opened this issue Mar 7, 2025 · 1 comment

Comments

@markstory
Copy link
Member

To give better visibility into the costs of taskbroker/workers we should record usage in finer detail.

Workers

Workers should already be measuring task execution time. We should use sentry.usageaccountant to record time spent on tasks per app_feature. We'll need a way to map between namespaces and app_features. Perhaps that could be metadata that is included in the TaskNamespaces.

Brokers

Brokers could periodically emit task counts for each namespace they are handling. Again we'd need a map between namespaces and app_features. The https://github.com/getsentry/rust-usage-accountant library can be used to record and publish usage results.

@markstory
Copy link
Member Author

Recommended labeling for brokers & workers:

  • app_feature: the same as the corresponding celery task
  • system: task_broker
  • subsystem: broker / worker.

As brokers ingest messages from Kafka we could generate counts of activations by app_feature and emit those as kafka messages using usage-accountant. This implies that there is a configuration map of namespace : app_feature within broker configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant