Research on analytics infrastructure

[harshita-srivastava-yral]

Background

• Investigate as big query is not good enough
• We need to cater to 4 UCs:

1. Dashboard and most tracked metrics are visible
2. Custom querying powerful queuing on actual day unlike GA where there is leakage of data. Eg 1024 sign up will end up showing 1000 ad blocks etc make it more difficult…We were ok till now for trading with these number n accuracy
3. Developer workflows - metrics tracing and logging..there are lot of overlaps
4. PNs require reading into the actual data and listen to specific events…!

• Our ingestion pipeline should be good for getting a sense of this
• If we choose right tooling we can get all of it

Solution proposed

• Vector.dev by Datadog has unified ingestion sources
• It also has Kafka type que based mechanism

Actionables suggested

• Primarily @komal-sai-yral to setup infrastructure for this and do research on vector.dev which is rust based platform
• @vishnu-shankar-yral to work on tasks at hand and push Komal for the relevant use cases catering via Vecto.dev

[harshita-srivastava-yral]

• Directly use vector dev and try to align on the same

[komal-sai-yral]

Task 1

• is there wasm for vector.dev ?
• test fly log shipper for other sources

Task 2

• check if fly managed grafana is accessible and can we add custom sources - https://chat.google.com/room/AAAA4jH22SI/KQXdy5TzvzI/KQXdy5TzvzI?cls=10

Task 3
sinks for logs

• GCP cloud monitoring
• S3
  • test with storj

Task 4

• . use quickwit - https://github.com/quickwit-oss/quickwit

• else Evaluate

  • Kibana
  • Bigquery connect to S3 omni and looker studio
  • databricks

• Setup fly logs -> vector -> quickwit pipeline

Task 5

• canister logs

Task 6

• migrate events to vector

[harshita-srivastava-yral]

@vishnu-shankar-yral to deep dive into quickwit - https://github.com/quickwit-oss/quickwit

[harshita-srivastava-yral]

• DAU unique across all domains is the metric we would want to trigger it daily on "Events-alerts" space
• Airflow job should run it periodic and push numbers

[siyara-m-yral]

• ⁠Komal - Figured out vector isnt wasm compatible. Will start testing out the fly log shipper version

[siyara-m-yral]

• Will be setting up quickwit today
• Should be done around Monday