periodic audit

[satra]

there are many things we will likely need to monitor, report, etc.,. it would be good to start building an internal monitoring dashboard.

[yarikoptic]

yeah, let's keep https://github.com/dandi/dandi-api-webshots/ separate but may be incorporate its "stats" into it.
Another related item to include: #253, and possibly dandi/dandisets#39

[waxlamp]

@satra can you give me some examples of the sort of stuff we need to capture?

[satra]

here are a few. but we should create an exhaustive wishlist.

• delta between dandiset size and blob size (since people can upload data without associating with dandiset)
  • staleness of blobs, who it belongs to, and blob specific requests (in case people are misusing the archive)
• number of API keys being used to upload data
• access stats
  • geolocations
  • bytes downloaded/uploaded
  • posts
• status of services (although this should be a separate heartbeat monitoring system)
  • how many times did each service raise errors

[waxlamp]

@satra, just to confirm, is this the topic you mentioned in today's standup? We've got some ideas on how to begin building this feature

[satra]

@waxlamp - yes, the first bullet above is more of the computation on the archive db type of audit that is specific to dandi, the rest i think are more of the automation pieces that would aggregate streams of info from other sources.

[mvandenburgh]

Closing in favor of #653, #654, #655.