Skip to content

Architect a rolling Count-Min Sketch filter to aggressively cache high-frequency fractional asset reads #462

Description

@Damola-Sodiq

Description

Integrate a Count-Min Sketch algorithm in the Node.js backend to probabilistically track hot fractional assets and aggressively cache them in memory.

Architecture & Context

Standard LRU caches consume excessive memory when dealing with thousands of long-tail assets. A Count-Min Sketch provides a mathematically bounded, low-memory footprint way to identify the top 1% of queried tokens and cache their metadata heavily.

Technical Requirements

  • Implement the Count-Min Sketch data structure strictly in TypeScript.
  • Hook the estimator into the Fastify request lifecycle to score asset IDs on read.
  • Evict cold assets and promote hot assets into a V8 in-memory map dynamically.

Acceptance Criteria

  • The filter accurately identifies trending assets with a known error bound.
  • Memory consumption for the tracker stays under 15MB.
  • Database read load is empirically reduced by > 50% during traffic spikes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions