### Is there an existing issue for this? - [ ] I have searched the existing issues ### New feature In order to handle large data sets, we have need for a system that allows for the data set to be processed in chunks, so that memory usage is limited. ### Unclear documentation _No response_