Hallucinations for complex scientific tables

[borisbolliet]

The system seems to hallucinate for academic paper with difficult tables.

Parsing page 40 of https://arxiv.org/pdf/1807.06209, we find that olmocr:

• only reports Table 4
• mixes up loads of signs in table 4 (for asymmetric uncertainties in particular)
• mis-reports some labels (see for instance second line of table)
• completely misses Table 5

It looks promising though (and already much better than llamaparse for instance)!

See attached screenshot of Table 4 from the pdf, and what olmocr came back with.

Is there some hyper-parameters that could be tuned to improve this? Thanks a lot!

Here is the full output:

<p>Table 4. Constraints on 1-parameter extensions to the base-ΛCDM model for combinations of Planck power spectra, Planck lensing, and BAO (equivalent results using the CamSpec likelihood are given in Table A.2). Note that we quote 95% limits here.</p>
<table>
<thead>
<tr>
<th>Parameter(s)</th>
<th>TT+lowE</th>
<th>TT, TE, EE+lowE</th>
<th>TT, TE, EE+lowE+lensing</th>
<th>TT, TE, EE+lowE+lensing+BAO</th>
</tr>
</thead>
<tbody><tr>
<td>Ωχ ..........</td>
<td>−0.056^{−0.044}_{+0.050}</td>
<td>−0.044^{−0.033}_{+0.042}</td>
<td>−0.011^{−0.013}_{+0.012}</td>
<td>0.0007^{−0.0037}_{+0.0037}</td>
</tr>
<tr>
<td>Σₙₙ [eV] ....</td>
<td>&lt; 0.537</td>
<td>&lt; 0.257</td>
<td>&lt; 0.241</td>
<td>&lt; 0.120</td>
</tr>
<tr>
<td>Nₑₑ ........</td>
<td>3.00^{+0.57}_{−0.53}</td>
<td>2.92^{+0.36}_{−0.37}</td>
<td>2.89^{+0.36}_{−0.38}</td>
<td>2.90^{+0.34}_{−0.33}</td>
</tr>
<tr>
<td>Yₑ ..........</td>
<td>0.246^{+0.039}_{−0.031}</td>
<td>0.240^{+0.024}_{−0.023}</td>
<td>0.239^{+0.024}_{−0.025}</td>
<td>0.243^{+0.023}_{−0.024}</td>
</tr>
<tr>
<td>δₙₙ/d ln k ......</td>
<td>−0.004^{−0.013}_{+0.012}</td>
<td>−0.006^{−0.013}_{+0.012}</td>
<td>−0.005^{−0.013}_{+0.012}</td>
<td>−0.004^{−0.013}_{+0.012}</td>
</tr>
<tr>
<td>r₀.₀₂ .......</td>
<td>&lt; 0.102</td>
<td>&lt; 0.107</td>
<td>&lt; 0.101</td>
<td>&lt; 0.106</td>
</tr>
<tr>
<td>w₀ ..........</td>
<td>−1.56^{−0.48}_{+0.44}</td>
<td>−1.58^{−0.48}_{+0.44}</td>
<td>−1.57±0.40</td>
<td>−1.04^{−0.10}_{+0.10}</td>
</tr>
</tbody></table>
<p>Table 5. Constraints on standard cosmological parameters from Planck TT,TE,EE+lowE+lensing when the base-ΛCDM model is extended by varying additional parameters. The constraint on τ is also stable but not shown for brevity; however, we include H₀ (in km s⁻¹ Mpc⁻¹) as a derived parameter (which is very poorly constrained from Planck alone in the ΛCDM+των extension). Here α−₁ is a matter isocurvature amplitude parameter, following PCC15. All limits are 68% in this table. The results assume standard BBN except when varying Yₑ independently (which requires non-standard BBN). Varying Aₐ is not a physical model (see Sect. 6.2).</p>
<p></p>
<p>and adding BAO data tightens (slightly) the nₛ constraint. Using the Planck temperature likelihoods we find</p>
<p>r₀.₀₀₂ &lt; 0.055<br>(95%, TT+lowE+lensing+BK15+BAO),</p>
<p>(43)</p>
<p>with nₛ = 0.9661 ± 0.0040 at 1 σ, or adding polarization</p>
<p>r₀.₀₀₂ &lt; 0.058<br>(95%, TT,TE,EE+lowE+lensing+BK15+BAO),</p>
<p>(44)</p>
<p>with nₛ = 0.9668 ± 0.0037 at 1 σ. However, the small change when adding polarization is not stable to the choice of polarization likelihood; when using the CamSpec TT,TE,EE+lowE likelihood in place of P14k, we find the weaker constraint r₀.₀₀₂ &lt; 0.065 for the same data combination as that used in Eq. (44).</p>
<p>All the combined nₛ−r contours exclude convex potentials at about the 95% confidence (marginally less if we use the CamSpec likelihood, see Fig. 28), which substantially restricts the range of allowed inflation models and disfavours all simple integer power law potentials. More generally, since r depends on the slope of the potential, the smallness of the empirical upper limit on r implies that the inflationary potential must have been nearly flat when modes exited the horizon. The measured nₛ must then be determined largely by the second derivative of the potential, suggesting a hierarchy in the magnitudes of the slow-roll parameters, favouring hilltop-like potentials. For a detailed discussion of the implications for specific inflation models see Planck Collaboration X (2020).</p>
<p>If we allow running of the spectral index in addition to tensor modes, the constraint on r₀.₀₀₂ weakens if we use only the Planck likelihood; a negative running allows nₛ at large scales to shift to higher values, lowering the large-scale scalar amplitude, and hence allowing a larger tensor contribution. Inclusion of the BK15 likelihood significantly reduces the extent of this degeneracy by constraining the tensor amplitude more directly, giving</p>
<p>r₀.₀₀₂ &lt; 0.16,<br>dₙₙ/d ln k = −0.008^{−0.014}_{+0.015},<br>95%, TT,EE+lowE+lensing.</p>
<p>(45a)</p>

And here is the prompt:

Below is the image of one page of a document, as well as some raw textual content that was previously extracted for it. Just return the plain text representation of this document as if you were reading it naturally.
Do not hallucinate.
RAW_TEXT_START
Page dimensions: 595.0x842.0
[52x69]40
[52x97]integer power law potentials. More generally, since
[52x108]the range of allowed inflation models and disfavours all simple
[52x129]at about the 95 % confidence (marginally less if we use the
[52x161]lihood in place of
[52x171]tion likelihood; when using the
[52x573]is a matter isocurvature amplitude parameter, following
[52x583]km s
[52x604]Table 5.
[52x761]Table 4.
[57x430]m
[57x480]d
[57x392]Y
[57x411]w
[57x420]↵
[60x150]065 for the same data combination as that used in Eq. (
[61x490]n
[62x381]P
[62x223]002
[63x410]0
[63x509]................ 0
[63x427],
[63x433]e
[66x498]s
[66x419]1
[66x433]↵
[68x392]............... 0
[68x499]/
[69x382]N
[71x441],
[73x471].............. 0
[75x655]r
[75x696]⌃
[75x687]N
[75x480]d ln
[75x285]<
[81x645]0
[82x430],
[82x654]002
[83x685]↵
[84x225]0
[85x461]/
[89x480]k
[92x441].......... 0
[93x483]2
[95x192]0
[95x314]temperature likelihoods we find
[98x723]Parameter TT
[111x461]...... 0
[111x480]/
[115x219]+
[115x230](95 %, TT,TE,EE
[128x258]0
[133x192]0
[139x561]P
[144x219]+
[147x139]–
[157x371].
[157x452].
[157x499].
[159x471]02224
[159x480]02237
[159x499]02240
[159x509]02237
[159x518]02237
[159x382]02224
[159x452]02236
[174x171]CamSpec
[175x706]
[179x118]), which substantially restricts
[181x665]0
[181x706]0
[182x676].
[183x382]±
[183x461]±
[183x471]±
[183x490]±
[183x499]±
[183x509]±
[183x696]<
[187x706]056
[188x427]
[189x646].
[190x723]+
[190x452]0
[190x471]0
[191x646]56
[192x427]0
[194x411].
[194x452].
[194x499].
[194x371].
[194x392].
[195x655].
[195x696].
[196x382]00022 0
[196x411]00015 0
[196x471]00022 0
[196x499]00015 0
[196x509]00014 0
[197x679]+
[200x662]
[200x703]
[200x709]+
[200x690]0
[201x673]0
[201x679]0
[203x649]0
[203x684].
[204x662]0
[206x643].
[207x703].
[207x230]+
[208x703]050
[212x171]TT,TE,EE
[213x230]lensing
[214x286]+
[220x286]BK15
[220x782]Planck Collaboration: Cosmological parameters
[232x324]constraint. Using
[235x480].
[235x371].
[235x452].
[235x461].
[237x392]1201
[237x411]1193
[237x509]1199
[237x371]1182
[242x430].
[242x441].
[244x382]1171
[244x430]1200
[245x665]
[248x646]
[249x286]BAO),
[249x723]TE
[250x665]0
[251x171]+
[252x655]<
[255x706].
[256x461]±
[256x471]±
[256x509]±
[256x401]±
[256x452]±
[257x665]006
[257x706]044
[257x535]c
[258x646].
[260x696]0
[260x646]58
[261x438]
[261x385]+
[264x499]0
[264x420]0
[264x452]0
[264x461]0
[264x696].
[264x539]2
[265x379]0
[265x433]0
[266x684]
[266x655]107
[267x679]+
[267x385].
[267x427].
[268x161]r
[268x411].
[268x420].
[268x461].
[268x499].
[269x643]
[269x649]+
[269x433]0032
[269x385]0042
[270x668]+
[270x703]
[270x452]0013 1
[270x480]0012 1
[270x490]0012 1
[270x518]0012 1
[270x371]0015 1
[270x392]0012 1
[270x401]0015 1
[271x159]0
[273x679].
[274x690]36
[276x662].
[276x668].
[276x703].
[276x761]⇤
[278x709]033
[279x723]lowE TT
[281x225](44)
[289x594]⌧
[291x161]<
[294x573]. All limits are 68 % in this table. The results assume standard BBN
[305x441].
[305x452].
[305x471].
[305x490].
[307x471]04116
[307x518]04092
[307x420]04087
[308x430]1
[308x172]eracy by constraining the tensor amplitude more directly, giving
[308x182]BK15 likelihood significantly reduces the extent of this degen-
[308x193]and hence allowing a larger tensor contribution. Inclusion of the
[308x293]n
[309x382].
[312x430].
[319x293]must then be determined largely by the second derivative of
[320x536]100
[320x723],
[324x723]TE
[330x371]±
[330x382]±
[330x411]±
[330x452]±
[330x490]±
[330x509]±
[331x116]0
[334x723],
[334x116].
[335x706]
[336x116]002
[338x382]0
[338x461]0
[338x518]0
[338x676]0
[338x723]EE
[339x433]0
[341x687]2
[342x411].
[342x461].
[342x509].
[342x676].
[342x433].
[344x518]00031 67
[344x371]00032 68
[344x382]0012 66
[344x411]00031
[344x441]00044 65
[344x471]00043 66
[346x687].
[348x646].
[350x583]Planck
[354x723]lowE
[354x696].
[356x684]
[357x696]241
[358x102]
[359x643]
[359x649]+
[360x662]
[360x703]
[363x649]0
[363x709]0
[365x673]025
[365x679]024
[366x649].
[366x668].
[366x709].
[367x649]50
[371x102]008
[387x518].
[389x499]36
[389x518]36
[389x392]19
[390x99]0
[391x471].
[392x430].
[393x99].
[393x461]2
[394x430]11
[394x604]+
[395x99]015
[395x106]014
[396x536]H
[396x411]...
[396x382].
[396x401].
[396x452]1
[398x382]0
[400x471]±
[400x509]±
[400x604]lensing when the base-
[402x399]
[402x404]+
[406x404]2
[406x427]0
[407x471]1
[407x518]0
[407x449].
[409x404].
[409x449]67
[409x455]2
[409x761]Planck
[409x427].
[410x438]6
[410x444]8
[410x115]=
[410x117]>
[410x123]9
[411x427]79
[411x471].
[411x490].
[411x518].
[411x392].
[413x480]56 0
[413x499]53 0
[413x509]54 0
[413x518]54 0
[422x104]+
[429x723]TE
[431x223]002
[433x583]CDM
[435x382]0
[439x471].
[439x499].
[439x382].
[441x401]9688
[441x411]9666
[441x441]9582
[441x452]9647
[441x480]9625
[441x490]9647
[441x509]9659
[443x723]EE
[445x224]weakens if we use only the
[451x706]0
[454x676]0
[455x583]+
[457x706]0007
[459x655]<
[459x696]<
[459x723]lowE
[460x536]n
[461x420]±
[461x461]±
[461x490]±
[461x499]±
[464x665]004
[465x646].
[468x411]0
[468x420]0
[468x452]0
[468x471]0
[468x480]0
[468x490]0
[468x382]0
[468x392]0
[471x655].
[471x696].
[472x490].
[472x518].
[472x401].
[472x411].
[473x684]
[473x690]+
[473x427]0056
[473x433]0045
[474x673]
[474x679]+
[474x420]0061 3
[474x480]0048 3
[474x490]0044 3
[474x509]0041 3
[474x703]
[477x668]+
[480x684].
[480x690].
[480x662]0
[480x673].
[480x679].
[481x684]33
[481x709].
[482x649].
[482x709]0037
[483x662].
[483x668].
[484x662]013
[488x604]⇤
[494x761]Planck
[495x604]CDM model is
[504x499].
[504x420].
[507x382]036
[507x392]042
[507x441]037
[507x452]046
[507x471]036
[507x480]049
[507x499]047
[510x401].
[512x430]050
[520x562]6.2
[521x382]±
[521x392]±
[521x411]±
[521x480]±
[521x499]±
[521x509]±
[525x539]10
[525x369]
[525x427]
[529x420]0
[529x471]0
[529x518]0
[529x398]0
[529x594]H
[531x536]A
[532x562]).
[532x427].
[532x433].
[532x404].
[533x461].
[533x471].
[533x490].
[533x509].
[533x518].
[533x382].
[533x420].
[535x411]014
[535x420]014
[535x441]017
[535x461]017
[535x499]015
[539x536])
[545x582]
[549x582]1

RAW_TEXT_END

This was run on the drag and drop of https://olmocr.allenai.org/

[jakep-allenai]

Yeah, I see what you mean, sadly it's still an issue for the models when the data gets very dense as in your table.

We will add this paper to olmocr-bench and eval our progress on it in V2.

If you run locally, you can lower the generation temperature, by default we have it at 0.8, so 0.1 may work better. The risk that you run is that it can cause the model to degenerate into repeated tokens/sentences.

[whisper-bye]

Not only in complex scientific tables, but also in other complex report-type tables, the effect is very poor.