Possibility to compress/decompress in smaller segments instead of all at once?

[TomArrow]

Hi,

this looks like a pretty cool library, but is there any way to compress/decompress smaller segments? I have a use case where I might wanna continuously keep adding data byte by byte until everything's all finished up, and the same for reading.

Cheers

[SSBMTonberry]

Hello, @TomArrow :)
No, there is no proper way to do that directly, but you could use a std::vector<uint8_t> where you append the bytes for each chunk, and then compress it when all the bytes are added. The most obvious reason why this is not possible, and would be tricky to handle properly, is that for LZMA to properly compress anything, a dictionary must be built, and how well this can be done is based on the context/content of what you want to compress. This doesn't mean it would be impossible to implement, but I'll be honest with you and say that looking into a way of compressing things in chunks is not something I'll implement any time soon.

[TomArrow]

Ah, I see. I actually managed to do it myself in the end. I ended up using the classic LZMA SDK API with multithreading and an std::condition_variable. So I start the encoding on a separate thread and then in the Read() callback I just do the waiting until I can deliver as much as the LZMA SDK is asking for before I return.

Here's the code if you're curious: https://github.com/TomArrow/JK2DemoCutter/blob/master/shared/files_lzma.hpp

Obviously not terribly elegant but I think as a proof of concept it works and one could probably do something similar without multithreading by modifying the API.

[SSBMTonberry]

Great job!
As long as it does what you want to do, it doesn't need to be the most elegant code in the world. It can be refactored later! 😄

I'm planning to at least update the backend to use some of the latest code from the LZMA-SDK. If the changelog is correct this should include cross-platform multithreading 🙂
As for the scope of this project: I intend to keep it up to date, but I don't think there will be a lot of extra features, as I have a lot of other projects I also work with, so I need to sort out my priorities 😛

Could be I add support for data chunks eventually. But I can't promise anything.