Skip to content

npy_inflate fails with "Error inflating stream" on deflate streams larger than ~2 MB #46

Description

@olokobayusuf

npzstringreader / npzreader throw std::runtime_error("Error inflating stream") when reading NPZ entries written by numpy.savez_compressed() once the compressed entry exceeds roughly 2 MB (i.e. spans more than one of npy_inflate's 1 MiB input chunks). Small entries read fine. The root cause seems to be the inner loop condition in npy_inflate in src/zip.cpp, which drops unconsumed input.

Repro

Generate a small and a large archive with numpy:

import numpy as np

rng = np.random.default_rng(0)
np.savez_compressed("small.npz", rng.standard_normal(219_040).astype("float32"))  # ~800 KB, reads OK
np.savez_compressed("large.npz", rng.standard_normal(876_160).astype("float32"))  # ~3.2 MB, fails

Read them back with libnpy:

#include <npy/npy.h>
#include <fstream>
#include <iostream>

int main() {
    std::ifstream input("large.npz", std::ios::binary);
    std::string bytes{std::istreambuf_iterator<char>(input), {}};
    npy::npzstringreader reader(std::move(bytes));
    auto tensor = reader.read<npy::tensor<float>>(reader.keys().front());  // throws
    std::cout << tensor.size() << '\n';
}

small.npz reads correctly butlarge.npz throws Error inflating stream.


I'm happy to open a PR with a fix. Just say the word.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions