Parametric UMAP saves multiple copies of full input data

[bnelsj]

Parametric UMAP stores multiple copies of the full input data, but these are unnecessary for transforming new data points. By deleting self._raw_data and self._knn_search_index._raw_data from my Parametric UMAP model object, I was able to reduce the size of the saved model from 90 GB to 300 MB (the input data is a distance matrix with 80K locations). This might not work for models that require additional training, but perhaps should be an option when model size is an issue?

[timsainb]

Sounds like a good idea.

…

On Thu, May 2, 2024 at 5:37 PM Brad Nelson ***@***.***> wrote: Parametric UMAP stores multiple copies of the full input data, but these are unnecessary for transforming new data points. By deleting self._raw_data and self._knn_search_index._raw_data from my Parametric UMAP model object, I was able to reduce the size of the saved model from 90 GB to 300 MB (the input data is a distance matrix with 80K locations). This might not work for models that require additional training, but perhaps should be an option when model size is an issue? — Reply to this email directly, view it on GitHub <#1118>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYKBWB42UCIRPIVNCSIJLZAKWZJAVCNFSM6AAAAABHEPKJE6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TMNJUGUYDQNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Tim Sainburg <https://timsainburg.com/> Postdoctoral Fellow Harvard Medical School 814.574.7780, ***@***.***

[bartbroere]

I'm having the same issue, where I want the trained model to be as small as possible (the inference machine does not have as much memory as the training machine). I'll link a PR where I added a parameter to remove the raw data to the save method.