Repository navigation
HtmlRewriter: title is cut at the first & (entity or bare ampersand) #401
Description
HtmlRewriter.rewrite(...).title only keeps the text before the first & in <title>.
from zimscraperlib.html import find_title_in
from zimscraperlib.rewriting.html import HtmlRewriter
from zimscraperlib.rewriting.url_rewriting import ArticleUrlRewriter, HttpUrl
def title(html: str) -> str:
url_rewriter = ArticleUrlRewriter(article_url=HttpUrl("https://www.example.com/page.html"))
return HtmlRewriter(url_rewriter, None, None, None).rewrite(html).title
for html in (
"<html><head><title>Tom & Jerry</title></head></html>",
"<html><head><title>Rock & Roll – My Blog</title></head></html>",
"<html><head><title>L'Illustre Maurin</title></head></html>",
):
print(repr(title(html)), "| find_title_in:", repr(find_title_in(html, "text/html")))<title> |
HtmlRewriter(...).rewrite().title |
find_title_in (same library) |
|---|---|---|
Tom & Jerry |
'Tom' |
'Tom & Jerry' |
Rock & Roll – My Blog |
'Rock' |
'Rock & Roll – My Blog' |
L'Illustre Maurin |
'L' |
"L'Illustre Maurin" |
Tested on main (d1deb81, 5.5.3-dev0), Python 3.14.
Why it matters
warc2zim uses this value as the entry title: Rewriter.rewrite_html() returns HtmlRewriter(...).rewrite(...) and items.py does self.title, self.content = rewriter.rewrite(...). So any page whose title contains &, ', and so on gets a truncated title in the ZIM, which is what title search and suggestions show. French and Italian titles with an escaped apostrophe are an easy way to hit it.
Cause
The parser runs with convert_charrefs=False, so HTMLParser delivers a title in pieces: handle_data("Tom "), handle_entityref("amp"), handle_data(" Jerry"). handle_data stores the first piece and then ignores the rest because self.title is no longer None:
if self.html_rewrite_context == "title" and self.title is None:
self.title = data.strip()Proposed fix
Collect the pieces while inside the first <title> and join them when it closes:
- in
handle_data, appenddatato a list instead of assigning the title; - in
handle_entityref/handle_charref, append the decoded reference (html.unescape(f"&{name};")/html.unescape(f"&#{name};")); - in
handle_endtag("title"), setself.title = "".join(parts).strip()if it isn't set yet.
The rewritten HTML is not affected, only the extracted title. "First <title> wins" stays as it is, so test_extract_title still holds. I tried this as a subclass outside the repo and it gives the same titles as find_title_in for the cases above, with identical content.
I can open a PR with this and tests in tests/rewriting/test_html_rewriting.py if the approach looks right to you.
Disclosure: I found this with the help of an AI assistant (Claude Code). The reproduction above was run locally against main.
Activity
benoit74 commentedon Nov 10, 2026
This is indeed a genuine bug, thank you for the report
Why not using
find_title_inmethod directly to get the proper title? It would cause parsing the document twice (and hence "cost" some performance), but is it that a significant concern?rajanpanth commentedon Nov 10, 2026
Thanks for looking at it so quickly.
find_title_inreturns the right title for all the cases above, so it would work. On the cost, measured locally onmainwith synthetic pages (a title, then N paragraphs with a link, an image and an inline style each):HtmlRewriter.rewrite()find_title_in()The absolute numbers are from a slow laptop (Windows, Python 3.14); the ratio is the useful part: about 15% more time per HTML entry.
It may not be necessary to pay that, though.
rewrite()already builds a BeautifulSoup tree of the whole document inextract_base_href(). Ifrewrite()builds that tree once and reads both the base href and the title from it, the title costs onesoup.find("title")on a tree that already exists (0.02 ms on a 100 KiB page) and nothing is parsed twice.extract_base_href(content)can keep its signature for other callers.One behaviour difference with either BeautifulSoup variant: when the first
<title>is empty and a later one is not (for example<head><title></title></head>followed by an inline SVG that has its own<title>), the current code returns the later title and BeautifulSoup returns"". That is whatdocument.titlegives in a browser, so I think it is fine, but I wanted to flag it.So the options are:
find_title_ininrewrite()and accept the extra parse;rewrite()and use it for both the base href and the title.I would go with 2 and keep the
.strip()the rewriter applies today. Tell me which you prefer and I will open the PR.