npzstringreader / npzreader throw std::runtime_error("Error inflating stream") when reading NPZ entries written by numpy.savez_compressed() once the compressed entry exceeds roughly 2 MB (i.e. spans more than one of npy_inflate's 1 MiB input chunks). Small entries read fine. The root cause seems to be the inner loop condition in npy_inflate in src/zip.cpp, which drops unconsumed input.
Repro
Generate a small and a large archive with numpy:
import numpy as np
rng = np.random.default_rng(0)
np.savez_compressed("small.npz", rng.standard_normal(219_040).astype("float32")) # ~800 KB, reads OK
np.savez_compressed("large.npz", rng.standard_normal(876_160).astype("float32")) # ~3.2 MB, fails
Read them back with libnpy:
#include <npy/npy.h>
#include <fstream>
#include <iostream>
int main() {
std::ifstream input("large.npz", std::ios::binary);
std::string bytes{std::istreambuf_iterator<char>(input), {}};
npy::npzstringreader reader(std::move(bytes));
auto tensor = reader.read<npy::tensor<float>>(reader.keys().front()); // throws
std::cout << tensor.size() << '\n';
}
small.npz reads correctly butlarge.npz throws Error inflating stream.
I'm happy to open a PR with a fix. Just say the word.
npzstringreader/npzreaderthrowstd::runtime_error("Error inflating stream")when reading NPZ entries written bynumpy.savez_compressed()once the compressed entry exceeds roughly 2 MB (i.e. spans more than one of npy_inflate's 1 MiB input chunks). Small entries read fine. The root cause seems to be the inner loop condition innpy_inflateinsrc/zip.cpp, which drops unconsumed input.Repro
Generate a small and a large archive with numpy:
Read them back with libnpy:
small.npzreads correctly butlarge.npzthrowsError inflating stream.I'm happy to open a PR with a fix. Just say the word.