[Feature] Add Full Iterator Pattern to BanyanDB's Query Pipeline

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

BanyanDB's query pipeline currently utilizes the iterator pattern for sorting, aggregating, and limiting data. However, in the initial stage of the pipeline—the raw data retrieval—all data in the segments are loaded into memory. This approach can lead to excessive memory usage, especially for heavy aggregation queries, such as retrieving the top 10 items ordered by a tag over a large time range (e.g., "last month").

We propose extending the iterator pattern to the initial raw data retrieval step to address this issue. By doing so, we can significantly reduce memory consumption by streaming data from segments on-demand rather than loading all segment data into memory at once.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct