[Step 2/4] Queue Time Histgram

[yangw-dev]

Description

Add two main class for queue time histogram lambda:

• TimeIntervalGenerator: generate list of time intervals based on the latest timestamp from source table and target table
  • generate a single data from source table if no timestamp found in target table
  • throw error if the interval is greater than maxi-interval threshold
• WorkerPoolProcessor:
  • input the interval list from TimeIntervalGenerator, and process histogram data in parrallel
  • It process each interval by calling the QueueTimeProcessor.process(), and in the end write all result to destination

Others

• added unit tests for all utilization methods
• clickhouse_client is not thread-safe, make sure each thread has its own client in thread pool

Next steps

step 3

• Add HistogramGenerator
• Add runner_label logic in HistogramGenerator
• Fix the issue with GithubConfigRetriever
• Change the target db to misc. instead of fortesting
  step 4
• setup AWS resources
• setup deployment

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
torchci	⬜️ Ignored (Inspect)	Visit Preview	Apr 3, 2025 8:17am

[yangw-dev]

just side talk, I since this file gets bigger, and i can see we can put some methods in another common file
I will re-org this file once things are functionally working, right now I'm focusing on the functionality rather than the org of files

[jeanschmidt]

LGTM :)