Added ZX0 compression

[nestornillo]

This patch adds ZX0 compression/decompression to CPCTelera

[nestornillo]

Einar Saukas, the author of ZX7, has recently released ZX0 https://github.com/einar-saukas/ZX0
It compresses more than ZX7, and is faster uncompressing. My own test with CPCTelera:
Input file: 952 bytes
ZX7b: 704 bytes 33543 us
ZX0: 672 bytes 29251 us
This patch tries to implement ZX0 in the same way as ZX7b
I've used the standard version of ZX0, which does not uncompress backwards, so the inputs are the start of destination and the start of compressed data.
It works fine for me but there are some things to be done:

-of course more testing

-In ZX0 Makefile, Einar Saukas source was using using owcc compiler with CFLAGS:
CFLAGS = -ox -ob -ol+ -onatx -oh -zp8 -fp6 -g0 -Ot -oe -ot -Wall -xc -s -finline-functions -finline-intrinsics -finline-math -floop-optimize -frerun-optimizer -fno-stack-check -march=i386 -mtune=i686
Using gcc it works for me with "CFLAGS = -O3 -fsigned-char -Wall -DUNIX -pedantic" (as used in ZX7b makefile), and also works for me with no CFLAGS at all. But I am a complete noob on this subject.

-I've used this parameters for cpct_pack: (edit:changed this. see coment below)
-czx7b | --compressor-zx7b - Use ZX7B compressor (Default).
-czx0 | --compressor-zx0 - Use ZX0 compressor.
Would it be better to use -c7b/-c0? -z0/-z7b? -zx0/-zx7b?

-documentation, formatting

-time measurements

[nestornillo]

I've also added ZX0b, the backwards version of ZX0, which is similar in speed and compression ratio to normal ZX0 (depends on data).
There is only one little problem with ZX0 algorithm, the compression method is very slow on big files. In my old PC, a file about 15KB is almost instantly compressed, but a test with a 65KB file took about 5 minutes to compress. I don't think this is a big problem for Amstrad programming. The author knows about it and zx0 has the option to use quick compression (with lower compression rate, just for testing), but I don't think its needed here.

[nestornillo]

czx7b | --compressor-zx7b - Use ZX7B compressor (Default).
-czx0 | --compressor-zx0 - Use ZX0 compressor.

I've changed the parameters for cpct_pack for a more rational:

-c | --compressor [method] - Sets the compression method. Possible values: zx7b zx0 zx0b (Default: zx7b)

[nestornillo]

Regarding ZX0 compression time, I've done some testing using cpct_pack with text data as input. (my PC is a Core 2 Duo 3GHz running linux)

[nestornillo]

I've updated ZX0 compressor to the latest version, released some hours ago. It fixes a bug in the "prefix" parameter, which is not used by cpct_pack but it's a very interesting ZX0 feature. Tests ok with this update

About cpct_pack script, the output of zx7b compressor command was redirected to NULL, so I also redirected to NULL the output of ZX0 command. However, ZX0 has a nice onscreen output (with progress bar and delta info). Was ZX7 redirected to NULL just for the ugly output?

[lronaldo]

@nestornillo Thank you very much for all this great and deep work. This is trully a lot of interesting and well done work. I need some spare time to sit and review everything before merging it to development branch. Sorry for not being able to respond to your work faster. Some general comments on CPCtelera approach:

• The general idea is that everything included into CPCtelera should be thouroughly docummented and comented.
• Development model is slow by nature: slow introduction of high-quality developed and documented stuff
• Always need a focus on learning: anyone should be able to read and understand what the code does. Therefore, comments on code are highly relevant

More or less, I see in your work that you are already on a similar path to these general guidelines, which is truly great and appreciable. Your analyses and comparison graphs are really valuable and should have a place into the documentation. I think it is really interesting to add a file with general documentation on how CPCtelera compressors work, and these comparisons to be able to pick one of them.

In these sense, comments in the ZX0 code need some improvement. Just by adding the standard [ms] count on each instruction will be okay to follow a general standard. Deeper comments on how the algorithm works would be highly appreciated, but I could be too much work to do. It is not necessary for inclusion.

With respect to documentation on ZX0 and ZX7B, it would be interesting to add, in the time and size measures, a table of tests on time required to decompress some standard test sets (time required for a 1K, 2K, 4K, 8K sprite sets and a 16K screen, in microseconds). That is highly valuable information for the user to pick up one version or the other.

I agre with your change in the cpct_pack interface for using different algorithms. That's much more convenient.

With respect to your question on redirection to NULL, in order just not to pollute the terminal, and give the user more relevant and curated information. If the new version gives that information, that's really nice. It can be used as long as it results appropriate.

Another thing that we are already missing is new examples of use of compressors. We need some very simple and direct examples of projects using ZX7b and ZX0. A simple project that many people would use, for instance, is directly decompressing an image into video memory. That could be an interesting example to show how compressors work. Another one could be having 2 compressed sets of sprites and code that decompresses and shows them, using a key to change from one set to the other. These two examples using both compressors could give an interesting idea on how to use them.

One last thing about the inclusion of ZX0 and ZX0B: as I understand, the decruncher has different versions, similar to ZX7b. For that, it is interesting to name them in their function label names in a coherent way. Actually, zx7b_decrunch has _s version, normal version and f0 f1 f2 versions, as they are labelled by ZX7B. I don't know if these are standard names also for ZX0, but there is the need to ensure that the user knows these functions, what they do, and why they are different. So, we need a common way to tag them for the user to pick the most appropriate. Now, the tags mean _s (small), _fX (fast X), and no tag for standard version. We can maintain this convention and explain it to the user in the "compression.txt" documentation.

Again, thanks a million for your work. As soon as I have enough time to do a proper review, I will help merging it to the development branch of CPCtelera :)

[nestornillo]

ZX0 has 6 decrunchers: Standard (69 bytes), Turbo(128 bytes 20% faster) and Mega(408 bytes 25% faster), each of them with normal and backwards version.
ZX7 "offficial" release (https://spectrumcomputing.co.uk/entry/27996/ZX-Spectrum/ZX7) also had 6 decompressors:
Standard (69 bytes), Turbo (88 bytes 25% faster) and Mega (244 bytes 30% faster). Also normal and backwards versions.
Later Antonio Villena released his own version of the backwards ZX7, and named his decrunching routines as:
Slow (64 bytes), Medium (67 bytes), Fast0 (156 bytes), Fast1 (177 bytes) and Fast2 (191 bytes). Cpctelera uses this Slow version.
For naming convention, I think we can assume that "Official Standard"="Villena's Medium". As I am using the Standard versions of ZX0, I think I should remove the _s tag.

As I have learned a lot using CPCTelera, I fully understand your comments on CPCtelera approach and the improvements that are needed in this patch. So I will continue working on it

[nestornillo]

Ok, I found what I was doing wrong, now I can correctly compress files using compression.mk

[nestornillo]

I added a table and chart to file compression.txt. I've done the tests with my own set of sprites, as I can't find any standard test set. ¿do you know where can I get some standard sets for testing?