Error during processing of HEIC input files #2930

robskrob · 2020-03-22T20:54:14Z

Environment

Tesseract Version: 4.1.1
Platform: macOS Catalina 10.15

Current Behavior:

Whenever I execute $ tesseract images/IMG_3958.HEIC output/grocery_bill I get this error:

$ tesseract images/IMG_3958.HEIC output/grocery_bill
Tesseract Open Source OCR Engine v4.1.1 with Leptonica
Error during processing.

Expected Behavior:

I would expect tesseract to output the text from the grocery bill into the output file output/grocery_bill.

Is there something wrong with processing HEIC images? Also, is there a location where I can tail the logs to see if I can get a richer description of the error?

Here's more information about the tesseract program that I installed with Homebrew:

$ tesseract -v
tesseract 4.1.1
 leptonica-1.79.0
  libgif 5.2.1 : libjpeg 9d : libpng 1.6.37 : libtiff 4.1.0 : zlib 1.2.11 : libwebp 1.1.0 : libopenjp2 2.3.1
 Found AVX2
 Found AVX
 Found FMA
 Found SSE

Also attached please find the image I had tesseract process.
IMG_3958.HEIC.zip

The text was updated successfully, but these errors were encountered:

stweil · 2020-03-22T21:08:44Z

You closed that, so what was the solution?

robskrob · 2020-03-22T21:21:18Z

@stweil I apologize for closing without an explanation.

The problem is the image is not a JPG. It is a HEIC. When take a photo with my iphone, upload it to google drive and then download this photo onto my machine, the image is saved as a HEIC file, which to be honest is a new file extension to me.

So the "solution" for me was to convert the file on my machine from HEIC to JPG. Once I did this tesseract had no problem processing the image.

I suppose then my original issue is still a bug -- except there's nothing wrong with the JPG file. It's the HEIC file:

$ tesseract images/IMG_3958.HEIC output/grocery_bill
Tesseract Open Source OCR Engine v4.1.1 with Leptonica
Error during processing.

Attached please find the HEIC file

IMG_3958.HEIC.zip

stweil · 2020-03-22T21:27:46Z

Thanks. So the error message should be improved and report that the input file could not be read. That is not specific for macOS, therefore I changed the title.

robskrob · 2020-03-22T21:30:10Z

Yeah I wanted to tail the logs of my tesseract process so I could potentially learn more about what was going on. I definitely think the error message could be improved. And yes, there's something about the input file that appears to not be readable. And yes, changing the title of the issue makes sense.

stweil · 2020-03-22T21:37:16Z

See the Wikipedia article on HEIC or HEIF. Maybe Leptonica can be extended to support that new format (so Tesseract would support it, too). There exists a C library libheif which might be used. @DanBloomberg, would that be interesting?

stweil · 2020-03-22T21:57:44Z

It looks like the format is covered by patents. Debian provides libheif nevertheless, and GIMP supports it. I don't know whether its use in Leptonica and Tesseract would be a problem.

2021-12-04: libheif uses the GNU Lesser General Public License which should be compatible with Leptonica. See https://github.com/strukturag/libheif/.

DanBloomberg · 2020-03-22T22:54:55Z

I've asked an expert on coding about this format, and will report here when I hear back.

For leptonica to support it is a high bar, because support is a serious commitment and invariably a lot of work. That work will include passing very intensive fuzzing, both internally at Google and externally by oss-fuzz on github.

amitdo · 2020-12-28T00:38:46Z

Maybe Leptonica/Tesseract can support gdk-pixbuf, which will bring indirect support for heic and avif.

DanBloomberg · 2020-12-28T01:52:13Z

Each new format supported demands a big cost in development and support effort.

As mentioned on leptonica#546, jpeg-xl seems more interesting: both easier to support and having compatibility with jpeg.

amitdo · 2020-12-28T02:53:56Z

Dan. I understand your view.

The thing is, these two formats (HEIC and AVIF) are becoming quite popular. HEIC is the default file format for photos in iOS. Recently, Firefox followed Chrome's lead, and it now supports AVIF by default (previously users had to enable it manually).

So, since you do not want to support these formats in your software. maybe we, Tesseract devs, should discuss whether we want to support them ourselves.

If we will decide to support them. we will convert them to Leptonica's pix.

To be honest, this was mainly directed toward @stweil, hoping that he would want this enough to implement it... :-)

DanBloomberg · 2020-12-28T07:17:30Z

Supporting AVIF and HEIC within tesseract is certainly an option. As I said, my experience with the older I/O libraries is that it's a lot of work. webp has been easier, because there are not a lot of options and the implementation is more "modern", with the basic encoder and decoder going between memory and not file streams (or, worse, with tiff, unix file descriptors). With the fuzzers that have been made for leptonica, both internally in Google and now externally with the oss-fuzz project, maintenance work has increased considerably.

But overall, maintenance on the I/O libraries has been a significant time-sink. I'm glad to have been able to relieve Ray Smith and tesseract of that burden.

I can't promise anything about jpeg-xl, but it does seem to be something I should look into.

amitdo · 2020-12-28T14:40:39Z

I'm glad to have been able to relieve Ray Smith and tesseract of that burden.

Last time we saw Ray here was 2 years ago. I don't know if he plans to return to contribute to this project.

amitdo · 2022-06-13T00:33:36Z

So, since you do not want to support these formats in your software. maybe we, Tesseract devs, should discuss whether we want to support them ourselves.

Answering myself: We should not do it.

amitdo · 2022-06-13T00:38:32Z

Still, a proper error message for unsupported image formats is desired.

DanBloomberg · 2022-06-13T04:33:10Z

pixReadStream() emits the message:
Unknown format: no pix returned
if the format is not supported for reading.

I believe that tesseract should not rely on leptonica error messages -- you might not even emit them by default.
Instead, tesseract should have its own error message if the image file can't be read to a pix.

zdenop · 2022-06-13T18:58:45Z

The funny part is that OCR process of this image fails here:

tesseract/src/api/baseapi.cpp

Lines 974 to 976 in b5878c2

    
           if (lines.empty()) { 
        
             return false; 
        
           }

Because when the file format is unknown (IFF_UNKNOWN), tesseract API(???) expects it is a file list:

tesseract/src/api/baseapi.cpp

Lines 1201 to 1210 in b5878c2

    
           // Maybe we have a filelist 
        
           if (r != 0 || format == IFF_UNKNOWN) { 
        
             std::string s; 
        
             if (data != nullptr) { 
        
               s = buf.c_str(); 
        
             } else { 
        
               std::ifstream t(filename); 
        
               std::string u((std::istreambuf_iterator<char>(t)), std::istreambuf_iterator<char>()); 
        
               s = u.c_str(); 
        
             }

I think we should remove this "guess what is the input" out of OCR API...

vid-bin · 2023-10-24T15:17:11Z

Is there any update on this? My entire photo library is heic (including screenshots). I’ve been trying to get OCR working with heic on fscrawler and Nextcloud with no success.

OCR works on my Mac because Apple has support for it in their system but I want to get this working on my NAS.

the only solution would be to convert my library to jpeg which I do not want to do.

amitdo · 2023-10-24T16:18:02Z

We don't plan to support the HEIC format.

stweil · 2023-10-24T16:19:06Z

No, there isn't any update, neither for Leptonica nor for Tesseract. The only solution is currently to convert from HEIC to JPEG, so Tesseract can process the JPEG file.

vid-bin · 2023-10-24T17:37:32Z

Does tesseract support AVIF or jpeg-xl then? I don’t want to convert to jpeg because I’ll lose HDR on my photos. The storage space is also significantly higher when converting my images.

Ideally the applications indexing the files would convert to jpeg on the fly for tesseract and then delete the temporary file when done but the ones I’m trying to use do not do this.

DanBloomberg · 2023-10-24T17:56:34Z

We've looked a couple of times at jpeg-xl. There is a very high bar to cross before deciding to support a new format with leptonica, and the reasons for this have been described several times. AVIF doesn't meet it. Initially, jpeg-xl was an interesting possibility because it was supported by Google, it is a significantly more efficient encoder than jpeg, and it is designed to have very good compatibility with the jpeg library. All these things meant that there was some likelihood of widespread adoption within a few years, essentially as a replacement for jpeg, and the work of supporting it in leptonica would be less than that for a completely new format.

Just a reminder: part of the work to support any new format is to build and run fuzzers for months, and to harden it not to crash or hang for any possible input.

As of 2022, however, it was evident that Google was not supporting it.
For details why it is no longer of interest, see:
DanBloomberg/leptonica#692 (comment)

amitdo · 2023-10-25T05:55:42Z

https://tesseract-ocr.github.io/tessdoc/InputFormats

DanBloomberg · 2023-10-25T06:26:15Z

For the tesseract input formats page, you might clarify that reading animated webp (a-webp) is not supported -- only writing.

I lost interest in fully supporting a-webp when I found that Google had so little interest in supporting their own format (which is far superior in compression to animated gif) that even with several billion gmail accounts, they didn't let you view an a-webp attachment! Only the first frame is displayed. Nevertheless, a-webp is supported in browsers; see the Google webp faq: https://developers.google.com/speed/webp/faq

amitdo · 2023-10-25T08:53:10Z

Hi Dan,

Thanks for the info. Is the description on that page in the 'Animated GIF' section also correct for animated WebP?

DanBloomberg · 2023-10-25T17:47:41Z

No, leptonica can not read a webp anim file. It does not return the first image.

Here are the error messages:

Error in pixReadMemWebP: WebP decode failed
Error in pixReadStream: webp: no pix returned

robskrob closed this as completed Mar 22, 2020

stweil added the question label Mar 22, 2020

robskrob reopened this Mar 22, 2020

robskrob changed the title ~~Error during processing of JPG -- macOS Catalina 10.15~~ Error during processing of HEIC -- macOS Catalina 10.15 Mar 22, 2020

stweil changed the title ~~Error during processing of HEIC -- macOS Catalina 10.15~~ Error during processing of HEIC input files Mar 22, 2020

stweil mentioned this issue Oct 29, 2020

Support for av1-avif DanBloomberg/leptonica#546

Closed

amitdo added the leptonica label Mar 21, 2021

dadoonet mentioned this issue Oct 29, 2023

Add support for HEIC files (convert to JPEG) dadoonet/fscrawler#1745

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error during processing of HEIC input files #2930

Error during processing of HEIC input files #2930

robskrob commented Mar 22, 2020 •

edited

Loading

stweil commented Mar 22, 2020

robskrob commented Mar 22, 2020

stweil commented Mar 22, 2020 •

edited

Loading

robskrob commented Mar 22, 2020 •

edited

Loading

stweil commented Mar 22, 2020 •

edited

Loading

stweil commented Mar 22, 2020 •

edited

Loading

DanBloomberg commented Mar 22, 2020

amitdo commented Dec 28, 2020

DanBloomberg commented Dec 28, 2020

amitdo commented Dec 28, 2020 •

edited

Loading

DanBloomberg commented Dec 28, 2020

amitdo commented Dec 28, 2020

amitdo commented Jun 13, 2022

amitdo commented Jun 13, 2022

DanBloomberg commented Jun 13, 2022 •

edited

Loading

zdenop commented Jun 13, 2022

vid-bin commented Oct 24, 2023

amitdo commented Oct 24, 2023

stweil commented Oct 24, 2023

vid-bin commented Oct 24, 2023

DanBloomberg commented Oct 24, 2023

amitdo commented Oct 25, 2023

DanBloomberg commented Oct 25, 2023 •

edited

Loading

amitdo commented Oct 25, 2023

DanBloomberg commented Oct 25, 2023

Error during processing of HEIC input files #2930

Error during processing of HEIC input files #2930

Comments

robskrob commented Mar 22, 2020 • edited Loading

Environment

Current Behavior:

Expected Behavior:

stweil commented Mar 22, 2020

robskrob commented Mar 22, 2020

stweil commented Mar 22, 2020 • edited Loading

robskrob commented Mar 22, 2020 • edited Loading

stweil commented Mar 22, 2020 • edited Loading

stweil commented Mar 22, 2020 • edited Loading

DanBloomberg commented Mar 22, 2020

amitdo commented Dec 28, 2020

DanBloomberg commented Dec 28, 2020

amitdo commented Dec 28, 2020 • edited Loading

DanBloomberg commented Dec 28, 2020

amitdo commented Dec 28, 2020

amitdo commented Jun 13, 2022

amitdo commented Jun 13, 2022

DanBloomberg commented Jun 13, 2022 • edited Loading

zdenop commented Jun 13, 2022

vid-bin commented Oct 24, 2023

amitdo commented Oct 24, 2023

stweil commented Oct 24, 2023

vid-bin commented Oct 24, 2023

DanBloomberg commented Oct 24, 2023

amitdo commented Oct 25, 2023

DanBloomberg commented Oct 25, 2023 • edited Loading

amitdo commented Oct 25, 2023

DanBloomberg commented Oct 25, 2023

robskrob commented Mar 22, 2020 •

edited

Loading

stweil commented Mar 22, 2020 •

edited

Loading

robskrob commented Mar 22, 2020 •

edited

Loading

stweil commented Mar 22, 2020 •

edited

Loading

stweil commented Mar 22, 2020 •

edited

Loading

amitdo commented Dec 28, 2020 •

edited

Loading

DanBloomberg commented Jun 13, 2022 •

edited

Loading

DanBloomberg commented Oct 25, 2023 •

edited

Loading