Refine #5315 and create subtasks

[Gman0909]

Context

We want to create a work plan for delivering group association for usage and costs monitoring, and ensure we have a good understanding of the necessary subtasks.

If there is App Engineering capacity, we can start bringing in some sub-tasks into the same sprint.

Task list

• Write design document
• Get feedback 1
• Jenny conducts a spike into Prometheus
• Refine design doc
• Get feedback 2
• Use design doc to add subtasks to [Platform Initiative] Associate users with groups for usage monitoring #5315

Design

https://hackmd.io/@jnywong/rkH_8h5nyg

Preliminary work

• Use annotations instead of labels for usernames on pods #5804
• Use kube_pod_annotations for usernames instead of kube_pod_labels in Grafana dashboards #5806
• Separate pod and user diagnostics jupyterhub/grafana-dashboards#146

Definition of Done

• The issue's sub-tasks are complete

[yuvipanda]

Me and @jnywong have caught up a little about this, and she's working on a design document for me to review soon.

[jnywong]

Document sent to Yuvi for review.

[jnywong]

Updated the top card. I will conduct a spike exploring Prometheus to refine the design doc more before getting feedback again.

[jnywong]

I've added sub-tasks to #5315.

Marking this as closed :)