Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an option to guess page layout (try to preserve some of the formatting) #11

Merged
merged 54 commits into from
Sep 25, 2018
Merged

Add an option to guess page layout (try to preserve some of the formatting) #11

merged 54 commits into from
Sep 25, 2018

Commits on Aug 24, 2018

  1. add first working approach plus debug code

    @Kebniss
    Kebniss committed Aug 24, 2018
    Configuration menu
    Copy the full SHA
    0ae6d24 View commit details
    Browse the repository at this point in the history

  2. add newline only at the end of selected tags

    @Kebniss
    Kebniss committed Aug 24, 2018
    Configuration menu
    Copy the full SHA
    566dc9b View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2018

  1. fix multiple consecutive newlines

    @Kebniss
    Kebniss committed Aug 27, 2018
    Configuration menu
    Copy the full SHA
    587e9a7 View commit details
    Browse the repository at this point in the history

  2. add guess_space = False option

    @Kebniss
    Kebniss committed Aug 27, 2018
    Configuration menu
    Copy the full SHA
    6c9d27e View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2018

  1. move add space and newline checks to a function

    @Kebniss
    Kebniss committed Aug 28, 2018
    Configuration menu
    Copy the full SHA
    c22f3fa View commit details
    Browse the repository at this point in the history

  2. add tests guess_page_layout

    @Kebniss
    Kebniss committed Aug 28, 2018
    Configuration menu
    Copy the full SHA
    8a78fc5 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2018

  1. remove old test

    @Kebniss
    Kebniss committed Aug 29, 2018
    Configuration menu
    Copy the full SHA
    a783e31 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2018

  1. guess_punct_space = False behavior same as before this PR

    @Kebniss
    Kebniss committed Aug 30, 2018
    Configuration menu
    Copy the full SHA
    cb8dc1c View commit details
    Browse the repository at this point in the history

  2. fix tests

    @Kebniss
    Kebniss committed Aug 30, 2018
    Configuration menu
    Copy the full SHA
    fb599bc View commit details
    Browse the repository at this point in the history

  3. fixed tests

    @Kebniss
    Kebniss committed Aug 30, 2018
    Configuration menu
    Copy the full SHA
    90e37b7 View commit details
    Browse the repository at this point in the history

  4. fix indent and make add_space more readable

    @Kebniss
    Kebniss committed Aug 30, 2018
    Configuration menu
    Copy the full SHA
    ae26d29 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2018

  1. add double newline before and after title, p and h tags

    @Kebniss
    Kebniss committed Aug 31, 2018
    Configuration menu
    Copy the full SHA
    bb33d4b View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2018

  1. by default tail of root node will not be extracted

    @Kebniss
    Kebniss committed Sep 6, 2018
    Configuration menu
    Copy the full SHA
    3069a73 View commit details
    Browse the repository at this point in the history

  2. add test

    @Kebniss
    Kebniss committed Sep 6, 2018
    Configuration menu
    Copy the full SHA
    dd03201 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2018

  1. fix indentation

    @Kebniss
    Kebniss committed Sep 7, 2018
    Configuration menu
    Copy the full SHA
    0f2fb2b View commit details
    Browse the repository at this point in the history

  2. newline tags as set and extendable, add new features comments, delete… 

    … new argument
    @Kebniss
    Kebniss committed Sep 7, 2018
    Configuration menu
    Copy the full SHA
    e8da507 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2018

  1. make html_to_text private, fix its signature

    @Kebniss
    Kebniss committed Sep 8, 2018
    Configuration menu
    Copy the full SHA
    0b9d139 View commit details
    Browse the repository at this point in the history

  2. add new tags to handle

    @Kebniss
    Kebniss committed Sep 8, 2018
    Configuration menu
    Copy the full SHA
    ba7cdc0 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2018

  1. handle more tags

    @Kebniss
    Kebniss committed Sep 10, 2018
    Configuration menu
    Copy the full SHA
    952d895 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2018

  1. remove cleaning of inline tags

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    9dafbf0 View commit details
    Browse the repository at this point in the history

  2. fix bug with multiple newlines

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    b3229d6 View commit details
    Browse the repository at this point in the history

  3. remove newline

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    695b458 View commit details
    Browse the repository at this point in the history

  4. add test html without text

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    03259b9 View commit details
    Browse the repository at this point in the history

  5. fix newline + space bug

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    cba531f View commit details
    Browse the repository at this point in the history

  6. add bad punct test

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    9811349 View commit details
    Browse the repository at this point in the history

  7. add newline

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    d47138c View commit details
    Browse the repository at this point in the history

  8. add tests on real webpages

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    76f9028 View commit details
    Browse the repository at this point in the history

  9. tests to hopefully make codecov happy

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    05c7702 View commit details
    Browse the repository at this point in the history

  10. remove pathlib import

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    4505e24 View commit details
    Browse the repository at this point in the history

  11. fix test

    @Kebniss
    Kebniss committed Sep 11, 2018
    Configuration menu
    Copy the full SHA
    a27e4c8 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2018

  1. remove space

    @Kebniss
    Kebniss committed Sep 12, 2018
    Configuration menu
    Copy the full SHA
    b926c8c View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2018

  1. handle list of selectors

    @Kebniss
    Kebniss committed Sep 19, 2018
    Configuration menu
    Copy the full SHA
    73f49ad View commit details
    Browse the repository at this point in the history

  2. a list of selectors returns a list of texts

    @Kebniss
    Kebniss committed Sep 19, 2018
    Configuration menu
    Copy the full SHA
    15d22e0 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2018

  1. selectors_to_text add to res only if something is extracted

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    8f68b2c View commit details
    Browse the repository at this point in the history

  2. selectors_to_text merge results as in previous implementation

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    cf02b94 View commit details
    Browse the repository at this point in the history

  3. update readme

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    7aec8d2 View commit details
    Browse the repository at this point in the history

  4. update history

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    7653bf9 View commit details
    Browse the repository at this point in the history

  5. update readme

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    4300fe6 View commit details
    Browse the repository at this point in the history

  6. update readme and add newline personalization tests

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    4772061 View commit details
    Browse the repository at this point in the history

  7. change documentation

    @Kebniss
    Kebniss committed Sep 20, 2018
    Configuration menu
    Copy the full SHA
    05b979a View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2018

  1. DOC cleanup README

    @kmike
    kmike committed Sep 21, 2018
    Configuration menu
    Copy the full SHA
    ad95bff View commit details
    Browse the repository at this point in the history

  2. DOC cleanup function docstring

    @kmike
    kmike committed Sep 21, 2018
    Configuration menu
    Copy the full SHA
    59d2d54 View commit details
    Browse the repository at this point in the history

  3. revert formatting change

    @kmike
    kmike committed Sep 21, 2018
    Configuration menu
    Copy the full SHA
    51947d4 View commit details
    Browse the repository at this point in the history

  4. minor cleanup

    @kmike
    kmike committed Sep 21, 2018
    Configuration menu
    Copy the full SHA
    1370647 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2018

  1. add pytest files to gitignore

    @kmike
    kmike committed Sep 24, 2018
    Configuration menu
    Copy the full SHA
    ab3f776 View commit details
    Browse the repository at this point in the history

  2. refactor _html_to_text function for readability: 

    * prev is always a string now, never a 1-element list
* unify newline and text handling between text and tail
* another workaround for mutable variable in the outer scope (Context class)
* append to a list instead of using a generator
    @kmike
    kmike committed Sep 24, 2018
    Configuration menu
    Copy the full SHA
    22a7fa1 View commit details
    Browse the repository at this point in the history

  3. bikeshedding: rename guess_page_layout option to guess_layout, for br… 

    …evity
    @kmike
    kmike committed Sep 24, 2018
    Configuration menu
    Copy the full SHA
    e161e92 View commit details
    Browse the repository at this point in the history

  4. TST mark test as xfail, change desired output 

    guess_punct_space doesn't provide good output in this case => xfail
    @kmike
    kmike committed Sep 24, 2018
    Configuration menu
    Copy the full SHA
    8b466f8 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2018

  1. cleanup: comments in unclear places

    @kmike
    kmike committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    729e11a View commit details
    Browse the repository at this point in the history

  2. cleanup: remove unnecessary escaping in regex

    @kmike
    kmike committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    2973ee0 View commit details
    Browse the repository at this point in the history

  3. backwards incompatible: make guess_layout=True by default

    @kmike
    kmike committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    607b04a View commit details
    Browse the repository at this point in the history

  4. typo fix in comment

    @kmike
    kmike committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    13394ba View commit details
    Browse the repository at this point in the history

  5. make it clear "\n" and "\n\n" are constants which can't come from ele… 

    …ment text directly
    @kmike
    kmike committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    7a1b57b View commit details
    Browse the repository at this point in the history

  6. remove PY3-only assert 

    not worths it to use six.string_types
    @kmike
    kmike committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    732c87d View commit details
    Browse the repository at this point in the history