Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

extract_text does not work on lxml XHTML element #24

Open
keturn opened this issue Feb 10, 2020 · 1 comment
Open

extract_text does not work on lxml XHTML element #24

keturn opened this issue Feb 10, 2020 · 1 comment

Comments

@keturn
Copy link

keturn commented Feb 10, 2020

I guess the docs do explicitly state lxml.html.HtmlElement, but the lxml docs say

Note that XHTML is best parsed as XML, parsing it with the HTML parser can lead to unexpected results.

so I had been using lxml in XML-mode, and it failed with the not-so-obvious error:

…/python3.7/site-packages/html_text/html_text.py in parse_html(html)
     47     XXX: mostly copy-pasted from parsel.selector.create_root_node
     48     """
---> 49     body = html.strip().replace('\x00', '').encode('utf8') or b'<html/>'
     50     parser = lxml.html.HTMLParser(recover=True, encoding='utf8')
     51     root = lxml.etree.fromstring(body, parser=parser)

AttributeError: 'lxml.etree._Element' object has no attribute 'strip'

Test case:

def test_extract_text_from_xml_tree():
    xhtml = (u'<html xmlns="http://www.w3.org/1999/xhtml"><head/><body>'
             '<p>Hello,   World!</p>'
             '</body></html>')

    text = u'Hello, World!'
    assert extract_text(etree.fromstring(xhtml,parser=etree.XMLParser()),
                                         guess_punct_space=False, guess_layout=False) == text
@keturn keturn mentioned this issue Feb 10, 2020
Open
@lopuhin
Copy link
Contributor

lopuhin commented Feb 10, 2020 •
edited
Loading

@keturn right, good catch - this is something we should fix. In the meantime, you can try calling html_text.etree_to_text directly, that won't fail in parse_html (but may fail later as I didn't check it). EDIT as I see you already tried that in #25.

Also I didn't experience issues with parsing XHTML with HTML parser, at least as far as html-text is concerned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@keturn @lopuhin