questions on usage and observations

[ritonlaveur]

Hi all

First of all, great work, after a week of testing docling and quackling I am still impressed by how well it works and how useful it is.

I did have a few questions though:

1. I was wondering if there would be a way of handling in multiprocessing a single document in order for it to be faster?
2. If I sent a PDF document from the frontend directly to my flask backend via an API call, will the "from_streams" conversion work?
3. The processing works really well on research papers and such documents. However, I stumbled across a few documents like reports of earnings where I found a few hiccups (nothing too major). These documents resemble a powerpoint presentation in the sense that they have mostly tables, figures, big text in bold and numbers across the place. The tables were not the cleanest and some of them had two liners on a single column name, and the remaining columns were single liners. I had the same table being repeated across 3 different pages, and it found a different markdown(and json) formatting for each table. 1 table had the second line of the two liners column being propagated as a value under each other column, 1 table had the title of the table being propagated as column names and the column names were put in a row as values, 1 table was correct. Now the model seemed to misinterpret what was a title (which should be the header that it correctly caught). It believed that any text slightly big or in bold was a subtitle-1 (even the main header title). The another thing I found is that since the text is all over the place it did not manage to fully comprehend the text structure in the order it should be in (such as individual components were seen as a single component despite the major horizontal space between them and thus read things line by line, and sometime our of 3 component, 1 would be correctly seen and sometimes not). Since it was in a less structured format, text had big vertical spaces between them and were then seen by the converter as being separate, sometimes this happened for each line being seen as a single entity. I found that many times I had a title (##) with another one under it directly and once again another one. Of course I could do some post processing on this but I was wondering if any of these observations could already be solved by the use of different parameters or if there was any other way which I am not aware of.
4. When storing the chunked data, I was wondering if there were any per section or per page storing possible since with the current node transformation I find myself getting the page header "Performance details" returned many time on its own without the remaining text on the page when I query "What are the performance details of the portfolio in the document", which of course the llm model would find itself having no context to answer (of course prev next augmentation could work but maybe a per section or per page chunking would be nice too).
5. With quackling being consolidated in docling, does it mean that I now have to process tables myself into a sentence/text? Because in quackling there was a method that directly formatted tables in sentences.
6. I have some text chunks that are repeated many times like a similar page header for multi pages, or the same side header for the name of the document and so on. Is there a way appart from post processing to directly remove repeating sections in order to not overload the vector db?
7. Finally, are there plans to make this porject work with more than pdf document (eg. docx, pptx, ...)?

Hope my explanations were clear.

Once again, I find this project really interesting.

Best,

Henry Valeyre

[dolfim-ibm]

Thanks for reaching out, here is some input for your questions. We can then dive deeper on a few (like point 3).

1. We actually have a commented out feature for processing batches of pages. Unfortunately, we found that some the pdf backends are not thread-safe and get into segfaults. It is anyway a topic high on the list.
2. Yes, I think it is very similar to this https://github.com/DS4SD/docling-serve/blob/main/docling_serve/app.py#L71
3. We can follow up in more details, if you want to share some of the docs. But in general, we actually trained on various types of documents. For example using the https://github.com/DS4SD/DocLayNet dataset.
4. Yes, we are actively working on those.
5. These functions are being migrated as well.
6. I think you might be interested in data-prep-kit, there you find also RAG steps for deduplication. Some end-to-end example is about to be merged there.
7. Yes, it is also on the list!

[ritonlaveur]

Hey Michele,

Thanks for your fast reply. It is good to know that most of my observations/questions already are being worked on or are in the pipeline.

For the point 2, this is the function I was referring to, I will test it out and see if it works.

In regards of the document for point 3. I will ask if I am allowed to share the document/part of it, or if it is possible to generate a fake report, and if it is possible I will put it here tomorrow. If I am not allowed to share it in such a public way, would there be another way for me to send it?

For point 6, thanks for sharing this kit, I will deep dive into it.

[aeamaea]

Re: #3 ... I'm also working on a similar problem in terms of processing docs with mainly tables and would be interesting to see this functionality develop in docling! thanks for such an amazing work!