Too many Gif files, too big files

[thomas694]

In the following post I want to share something for "advanced" users.

Disclaimer: This is not an official recommendation by the makers of TumblThree nor a request to do anything described here in. We do not take responsibility for any damages to or losses of your files.

Do you also have dozens of Gif files in your downloaded blogs and ever wondered why a little bit blinking or a movement of few seconds needs several MB storage?
If it is your own content, you need them in original state or anything similar, you can stop reading any further.

Gif files have been invented a very long time ago (you remember 1987?) and weren't meant to be used for displaying an anime or short movie clip and the like. Therefore the format and compression (is no longer) isn't efficient enough.
But the decades made it a very widespread file format, nearly every image editor and player supports it. And it's well-known and used by the masses.

Are there any alternatives, at least or especially if you want to efficiently store Gif content/message away?
At least two, I think.

WebP

This format is already used by Tumblr, if you view a blog in the browser. Many Gif files are "shown" as Gifv (see their image link) and if your browser supports it, the content which is downloaded by and displayed in the browser is actually a WebP file (some kind of movie format, VP8 to be exact).

• all features of Gif files (e.g. loops, animation, metadata, transparency) are supported
• supports a lossless mode, saving in average 10%
• supports different lossy compression levels, depends on the actual content, but sometimes saves even 80%
• it's a "new" technique, several years old, but not yet widespread
• no Windows built-in support, only some image viewers or the web browsers
• criticisms: for some converted graphic art Gifs it shall have problems with crispness and color*

You can download and play around with the Google WebP Tools.
There are a few settings, but not too much.

Converting Gif to lossless WebP:
gif2webp -mt picture.gif -o picture.webp
Uses the default compression/speed factor of 75. Even if 100 is used, most of the times the gain is comparable low for the extra time spent.

Converting Gif to "mixed" WebP (for each frame either lossless or lossy is chosen):
gif2webp -mt -mixed picture.gif -o picture.webp (default setting 75)

Quite a bit smaller the file ...gif.mixed75.webp. But how is the quality of the converted content?

You can also find here a summary of the advantages and disadvantages of WebP.
From there is also the dancing banana demo image which they use to demonstrate the space savings.

* If you convert a graphic art Gif like this to WebP and open it in some image viewers (e.g. Windows Photo Viewer), you can see slightly changed colors. But if you open the same images in a photo editor (e.g. GIMP) or a web browser, you can see that the colors are totally ok. I believe that at least some of the complaints of internet users are due to the fact of using an incapable program. Interestingly, but also sadly, the current Windows image viewer "Photos" does not support WebP images by default after so many years.
Which settings is Tumblr using? Are the complaining users right or wrong considering the above mentioned fact?

Opened in wrong image viewer:

Maybe TumblThree should get an option to download the Tumblr converted Gif images (WebP images).

WebM

It's a video container format. As compression formats VP8 as well as the successors VP9 and AV1 are supported. The format also supports transparency.

• compression levels of VP9 are better as VP8 at same visual quality
• AV1 would be even a bit better, but currently takes too much cpu power resp. time to encode
• AV1 encoded files made problems in a couple of movie players
• supports transparency
• supported by Windows built-in media/movie player, wide range of other movie players and web browsers
• no auto-looping by just opening the file, only by option in movie players

As I don't care much about loops when storing Gif files and as most movie players can do an auto-loop on playback, I want to look a bit deeper on that format.

It took quite a while to get all the information about the required tools and especially their settings. How can the conversion result of different encoder settings be objectively measured? As it will be an irrevocable process, at least after the original Gifs have been deleted and the originating blogs are no longer available, you want to be sure that everything was correct and they are stored in the "best" possible quality.
As outlined above AV1 isn't an option yet.

The already known FFmpeg serves quite well for this purpose. The newest FFmpeg can be downloaded from their download page. The file ffmpeg-n5.0-latest-win64-gpl-shared-5.0.zip (download link) contains the current stable version and all en-/decoders.

The VP9 encoder settings are described in details here.

Eventually we need a command line like this:

ffmpeg -i image.gif -c:v libvpx-vp9 -lag-in-frames 0 -b:v 0 -crf 25 -pass 1 -an -f null NUL && ^
ffmpeg -i image.gif -c:v libvpx-vp9 -lag-in-frames 0 -b:v 0 -crf 25 -pass 2 -an image.gif.webm

As different kind of Gifs need to be converted, it's not possible to specify a general bit rate. Anyway two-pass mode is the encoders preferred encoding mode.
Gifs don't contain audio, so no audio processing with option -an. Some Gifs don't convert properly and frames are missing in the output, so the option lag-in-frames is necessary, which solves this problem. But what value to choose for the parameter crf?
Note: For batch processing in parallel, you either need to set your command prompt to different folders or use the option -passlogfile to specify different prefixes for the two-pass log file.

The lossless conversion was actually bigger than the original Gif, lossless and CRF value 0 are nearly the same. The recommendations for the CRF value range from 15 to 35. But even the "lowest" value recommendation of crf 15 already brought impressive space savings of ~85%.
Is there a possibility to do some test conversions and compare the videos with some tool?

I found out that in the meantime FFmpeg also included a quality measuring library (VMAF).
Therefore a trained model file vmaf_v0.6.1.json is needed. For convenience it can be put in the FFmpeg bin folder. Actually it is for movies and higher resolutions than normal Gifs are, so you maybe have to put the numbers from the original gif file and the converted files in relation to get something meaningful. Originally values above 90 are considered "good" and one of the streaming providers targets 93.

Measuring the quality of a video file:

ffmpeg.exe -i converted.gif.webm -i original.gif -lavfi libvmaf="model_path=vmaf_v0.6.1.json" -f null -

Comparing the original Gif to itself gives the best VMAF score of 99.x%.
The higher the CRF value (more compression, less quality) gets, the more the score value drops.
Sometimes already a lossless converted Gif to it's original gave a "bad" value (much less than 99.x%). But if using this value as base and setting it in relation to the other conversions, it gives the usual numbers starting in the 90s all the way down. A CRF value of around 25 resulted in a VMAF score of about 93.

Final statistics

6360 Gifs of all sizes and kind (average size 2.9 MB) have been converted.
Space usage:

Gif   17.88 GB
WebP   4.31 GB
WebM   1.66 GB

That is around 75% less space for WebP and astonishing 90% for WebM.
Maybe one could split the Gif set at a file size of e.g. 500KB to convert the smaller ones (high chance it's a real graphic art) to WebP and the bigger ones to WebM (most likely converted video clips).