Version 3.10.2rc0

Author: mborsetti

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 3.10.1
+current_version = 3.10.2rc0
 message = Release {new_version}
 parse = ^
 	(?P<major>\d+)

CHANGELOG.rst

@@ -31,13 +31,69 @@ can check out the `wish list <https://github.com/mborsetti/webchanges/blob/main/
    Internals, for changes that don't affect users. [triggers a minor patch]
 
 
+Version 3.10.2rc0
+===================
+Unreleased
+
+⚠ Breaking Changes
+------------------
+* Due to a fix to the ``html2text`` filter (see below), the first time you run this new version **you may get a change
+  report with deletions and additions of lines that look identical. This will happen one time only**.
+
+Added
+-----
+* You can now run the command line argument ``--test`` without specifying a JOB to check the config and job files for
+  syntax errors.
+* Improved error messages for url jobs failing due to HTTP reason codes of 400 and higher, to include any text returned
+  by the website (e.g. "Rate exceeded.", "upstream request timeout", etc.).  Not implemented in jobs with
+  ``use_browser: true`` due to limitations in Playwright.
+* New command line argument ``--check-new`` to check if a new version is available.
+
+Changed
+-------
+* On Linux and macOS systems, for security reasons we now check that the hooks file **and** the directory it is located
+  in are **owned** and **writeable** by **only** the user who is running the job (and not by its group or by other
+  users), identical to what we do with the jobs file if any job uses the ``shellpipe`` filter. An ImportWarning message
+  will be issued if the permissions are not correct and the hooks module is therefore not imported.
+* The command line argument ``-v`` or ``--verbose`` has reduced verbose output while the new ``-vv`` has full verbosity.
+
+Fixed
+-----
+* The ``html2text`` filter is no longer retaining any eventual spaces found in the HTML after *the end of the text* on
+  a line, which are not displayed in HTML and therefore a bug in the conversion library used. This was causing a change
+  report to be issued whenever the number of such invisible spaces changed.
+* For job with ``browser: true``, the ``cookies`` directive did not add cookies correctly to the header.
+* Configuration file entries relating to job kinds declared in custom JobBase classes in the hooks file no longer issue
+  the warning of being unrecognized and potential typos.
+* For filters ``execute`` and  ``shellpipe`` and for  reporter ``run_command`` the message of the FileNotFoundError
+  exception is more descriptive and helpful.
+* The ``wait_for_timeout`` job directive was not accepting integers (only floats). Reported by `Markus Weimar
+  <https://github.com/Markus00000>`__ in `#39 <https://github.com/mborsetti/webchanges/issues/39>`__.
+* Fixed an issue in the parser used by the ``xpath`` ``filter`` that under specific conditions caused more html than
+  expected to be returned.
+* Fixed how we determine if a new version has been released due to a change in the data received from PyPi.
+
+Internals
+---------
+* ``urllib3`` is now an explicit dependency due to the refactoring of the requests package (we previously used
+  ``requests.packages.urllib3``). Has no effect since ``urllib3`` is already being installed as a dependency of
+  ``requests``.
+* Updated the vendored version of ``packaging.version.parse()`` to 21.3, released on 2021-11-27.
+* Changed the import logic so if packaging is found to be installed, its ``packaging.version.parse()`` function will be
+  used instead of the vendored one.
+* Changed loading logic so that with ``-vv`` registration of internal classes shows up in the logs.
+* Improved execution speed of certain command line arguments.
+* Added typed.py file to implement `PEP 561 <https://peps.python.org/pep-0561/>`__.
+
+
+
 Version 3.10.1
 ===================
 20220503
 
 Fixed
 -----
-* ``KeyError: 'indent' error`` when using ``beautify`` filter.  Reported by `César de Tassis Filho
+* ``KeyError: 'indent'`` error when using ``beautify`` filter. Reported by `César de Tassis Filho
   <https://github.com/CTassisF>`__ in `#37 <https://github.com/mborsetti/webchanges/issues/37>`__.
 
 

LICENSE

@@ -27,8 +27,12 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 This software redistributes source code of release 2.21 of urlwatch
 https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f
-which is subject to the following license, hereby retained with the source code,
-of which this file is part of:
+which is subject to the following copyright notice and license from
+https://raw.githubusercontent.com/thp/urlwatch/346b25914b0418342ffe2fb0529bed702fddc01f/COPYING
+hereby retained and redistributed with the source code (of which this license
+file is part of), in binary form, and in the documentation. The appearance of
+the name of the author below does not constitute an endorsement or promotion of
+this software by such author.
 
 Copyright (c) 2008-2020 Thomas Perl <[email protected]>
 All rights reserved.

README.rst

@@ -9,6 +9,7 @@
 webchanges
 ==========
 
+|downloads|
 **webchanges** checks web content and notifies you via e-mail (or one of many other supported services) if a change is
 detected. **webchanges** can also check the output of local commands. The notification includes the changed URL or
 command and a summary of what has changed.
@@ -28,10 +29,11 @@ only the latest bug and security fix version (3.x.y) is supported.
 
 Installation
 ============
-Install **webchanges** |pypi_version| |format| |status| with::
+Install **webchanges** |pypi_version| |format| |status| with:
 
-   pip install webchanges
+.. code-block:: bash
 
+   pip install webchanges
 
 
 Documentation
@@ -90,10 +92,10 @@ The code and issues tracker are hosted on `GitHub <https://github.com/mborsetti/
 
 Contributing
 ============
-We welcome any contribution, e.g. documentation, bug reports, new features, etc., as both pull requests and
-`issue reports <https://github.com/mborsetti/webchanges/issues>`__.
+We welcome any contribution no matter how small as both pull requests and `issue reports
+<https://github.com/mborsetti/webchanges/issues>`__.
 
-More information for developers and documentation editors is `here
+More information for developers and documentation contributors is `here
 <https://github.com/mborsetti/webchanges/blob/main/CONTRIBUTING.rst>`__, and our wishlist is `here
 <https://github.com/mborsetti/webchanges/blob/main/WISHLIST.md>`__.
 
@@ -102,16 +104,17 @@ License
 =======
 |license|
 
-Released under the `MIT License <https://opensource.org/licenses/MIT>`__, but including
-`code <https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ licensed under the
-`BSD 3-Clause License <https://opensource.org/licenses/BSD-3-Clause>`__. See the full license `here
-<https://github.com/mborsetti/webchanges/blob/main/LICENSE>`__.
+Released under the `MIT License <https://opensource.org/licenses/MIT>`__ but redistributing modified source code from
+`urlwatch 2.21 <https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ licensed under a
+`BSD 3-Clause License
+<https://raw.githubusercontent.com/thp/urlwatch/346b25914b0418342ffe2fb0529bed702fddc01f/COPYING>`__. See the complete
+license `here <https://github.com/mborsetti/webchanges/blob/main/LICENSE>`__.
 
 
 Compatibility with **urlwatch**
 ================================
 
-This project is based on code of`urlwatch <https://github.com/thp/urlwatch>`__ 2.21. You can easily upgrade from
+This project is based on `urlwatch <https://github.com/thp/urlwatch>`__ 2.21. You can easily upgrade from
 **urlwatch** 2.25 (see `here <https://webchanges.readthedocs.io/en/stable/migration.html>`__) using the same job and
 configuration files and benefit from many HTML-focused improvements, including:
 
@@ -153,9 +156,9 @@ Examples:
 .. |format| image:: https://img.shields.io/pypi/format/webchanges.svg
     :target: https://pypi.org/project/webchanges/
     :alt: Kit format
-.. |downloads| image:: https://img.shields.io/pypi/dw/webchanges.svg
-    :target: https://pypi.org/project/webchanges/
-    :alt: Weekly PyPI downloads
+.. |downloads| image:: https://pepy.tech/badge/requests
+    :target: https://pepy.tech/badge/webchanges
+    :alt: PyPI downloads
 .. |license| image:: https://img.shields.io/pypi/l/webchanges.svg
     :target: https://pypi.org/project/webchanges/
     :alt: License at https://pypi.org/project/webchanges/

RELEASE.rst

@@ -1,4 +1,49 @@
+⚠ Breaking Changes
+------------------
+* Due to a fix to the ``html2text`` filter (see below), the first time you run this new version **you may get a change
+  report with deletions and additions of lines that look identical. This will happen one time only**.
+
+Added
+-----
+* You can now run the command line argument ``--test`` without specifying a JOB to check the config and job files for
+  syntax errors.
+* Improved error messages for url jobs failing due to HTTP reason codes of 400 and higher, to include any text returned
+  by the website (e.g. "Rate exceeded.", "upstream request timeout", etc.).  Not implemented in jobs with
+  ``use_browser: true`` due to limitations in Playwright.
+* New command line argument ``--check-new`` to check if a new version is available.
+
+Changed
+-------
+* On Linux and macOS systems, for security reasons we now check that the hooks file **and** the directory it is located
+  in are **owned** and **writeable** by **only** the user who is running the job (and not by its group or by other
+  users), identical to what we do with the jobs file if any job uses the ``shellpipe`` filter. An ImportWarning message
+  will be issued if the permissions are not correct and the hooks module is therefore not imported.
+* The command line argument ``-v`` or ``--verbose`` has reduced verbose output while the new ``-vv`` has full verbosity.
+
 Fixed
 -----
-* ``KeyError: 'indent' error`` when using ``beautify`` filter.  Reported by `César de Tassis Filho
-  <https://github.com/CTassisF>`__ in `#37 <https://github.com/mborsetti/webchanges/issues/37>`__.
+* The ``html2text`` filter is no longer retaining any eventual spaces found in the HTML after *the end of the text* on
+  a line, which are not displayed in HTML and therefore a bug in the conversion library used. This was causing a change
+  report to be issued whenever the number of such invisible spaces changed.
+* For job with ``browser: true``, the ``cookies`` directive did not add cookies correctly to the header.
+* Configuration file entries relating to job kinds declared in custom JobBase classes in the hooks file no longer issue
+  the warning of being unrecognized and potential typos.
+* For filters ``execute`` and  ``shellpipe`` and for  reporter ``run_command`` the message of the FileNotFoundError
+  exception is more descriptive and helpful.
+* The ``wait_for_timeout`` job directive was not accepting integers (only floats). Reported by `Markus Weimar
+  <https://github.com/Markus00000>`__ in `#39 <https://github.com/mborsetti/webchanges/issues/39>`__.
+* Fixed an issue in the parser used by the ``xpath`` ``filter`` that under specific conditions caused more html than
+  expected to be returned.
+* Fixed how we determine if a new version has been released due to a change in the data received from PyPi.
+
+Internals
+---------
+* ``urllib3`` is now an explicit dependency due to the refactoring of the requests package (we previously used
+  ``requests.packages.urllib3``). Has no effect since ``urllib3`` is already being installed as a dependency of
+  ``requests``.
+* Updated the vendored version of ``packaging.version.parse()`` to 21.3, released on 2021-11-27.
+* Changed the import logic so if packaging is found to be installed, its ``packaging.version.parse()`` function will be
+  used instead of the vendored one.
+* Changed loading logic so that with ``-vv`` registration of internal classes shows up in the logs.
+* Improved execution speed of certain command line arguments.
+* Added typed.py file to implement `PEP 561 <https://peps.python.org/pep-0561/>`__.

docs/cli.rst

@@ -12,10 +12,10 @@ Command line arguments
    positional arguments:
      JOB                   job(s) to run (by index as per --list) (default: run all jobs)
 
-   optional arguments:
+   options:
      -h, --help            show this help message and exit
      -V, --version         show program's version number and exit
-     -v, --verbose         show logging output
+     -v, --verbose         show logging output; use -vv for maximum verbosity
 
    override file defaults:
      --jobs FILE, --urls FILE
@@ -27,17 +27,18 @@ Command line arguments
    job management:
      --list                list jobs and their index number
      --errors              list jobs with errors or no data captured
-     --test JOB, --test-filter JOB
-                           test a job (by index or URL/command) and show filtered output
+     --test [JOB], --test-filter [JOB]
+                           test a job (by index or URL/command) and show filtered output; if no JOB,
+                           check config and job files
      --no-headless         turn off browser headless mode (for jobs using a browser)
      --test-diff JOB, --test-diff-filter JOB
                            test and show diff using existing saved snapshots of a job (by index or
                            URL/command)
      --dump-history JOB    print all saved snapshot history for a job (by index or URL/command)
-     --add JOB             add job (key1=value1,key2=value2,...). WARNING: all remarks are deleted from
-                           jobs file; use --edit instead!
-     --delete JOB          delete job by URL/command or index number. WARNING: all remarks are deleted
-                           from jobs file; use --edit instead!
+
+   backward compatibility (WARNING: all remarks are deleted from jobs file; use --edit instead):
+     --add JOB             add a job (key1=value1,key2=value2,...) [use --edit instead]
+     --delete JOB          delete a job (by index or URL/command) [use --edit instead]
 
    reporters:
      --test-reporter REPORTER
@@ -59,17 +60,16 @@ Command line arguments
      --rollback-cache TIMESTAMP
                            delete recent snapshots since TIMESTAMP (backup the database before using!)
      --delete-snapshot JOB
-                           delete the last saved snapshot of job (URL/command)
+                           delete the last saved snapshot of job (index or URL/command)
      --database-engine {sqlite3,redis,minidb,textfiles}
                            database engine to use (default: sqlite3, unless redis URI in --cache)
      --max-snapshots NUM_SNAPSHOTS
-                           maximum number of snapshots to retain in sqlite3 database (default: 4)
+                           sqlite3 only: maximum number of snapshots to retain in database (default: 4)
 
    miscellaneous:
-     --install-chrome      install or update Google Chrome browser for use with 'use_browser: true' jobs
-     --features            list supported job types, filters and reporters
-     --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
-                           level of logging output when -v is selected (default: DEBUG)
+     --check-new           check if new release is available
+     --install-chrome      install or update Google Chrome browser (for jobs using a browser)
+     --features            list supported job types, filters and reporters (including those loaded by hooks)
 
 .. _job_subset:
 
@@ -92,21 +92,32 @@ be monitoring resources as expected. No snapshots are saved from this run.
 
 .. _test:
 
-Test run a job
---------------
+Test run a job or check config and job files for errors
+-------------------------------------------------------
 You can test a job and its filter by using the command line argument ``--test`` followed by the job index number
 (from ``--list``) or its URL/command; :program:`webchanges` will display the filtered output. This allows to easily
 test changes in filters. Use a negative index number to select a job from the bottom of your job list (i.e. -1 is the
 last job, -2 is the second to last job, etc.).  Combine ``--test`` with ``--verbose`` to get more information, for
-example the text returned from a website with a 4xx (client error) status code::
+example the text returned from a website with a 4xx (client error) status code:
+
+.. code-block:: bash
 
    webchanges --verbose --test 1
 
 Please note that ``max_tries`` will be ignored by ``--test``.
 
+To only check the config and job files for errors, specify --test without a JOB:
+
+.. code-block:: bash
+
+   webchanges --test
+
+
 .. versionchanged:: 3.8
    Accepts negative indices.
 
+.. versionchanged:: 3.10.2
+   JOB no longer required (will only check the config and job files for errors).
 
 .. _test-diff:
 
@@ -140,9 +151,9 @@ Test a reporter
 ---------------
 You can test a reporter by using the command line argument ``--test-reporter`` followed by the reporter name;
 :program:`webchanges` will create a dummy report and send it through the selected reporter. This will help in
-debugging issues, especially when used in conjunction with ``--verbose``::
+debugging issues, especially when used in conjunction with ``-vv``::
 
-   webchanges --verbose --test-reporter telegram
+   webchanges -vv --test-reporter telegram
 
 
 .. versionchanged:: 3.9

docs/examples.rst

@@ -254,23 +254,27 @@ or ignore all HTTP errors if you like by using :ref:`ignore_http_error_codes`
    ignore_http_error_codes: 4xx, 5xx
 
 
-Only receive short notifications containing the URL
+Receive short notifications only containing the URL
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-When using a reporter that uses text, the report -> text -> details setting can be set to false to avoid details and
-report just a summary. You can also set report -> text -> footer to false to make the report even shorter.  You can also
-use the directive ``user_visible_url`` to customize the URL that is reported visible (e.g. watching a REST API
-endpoint, but wanting to show the "web-visible" URL in the report).
+If you only want to be alerted that there is a change without any information about the change itself, you can use a
+a reporter that uses text and set report -> text -> details to false to avoid details being sent; you can also set
+report -> text -> footer to false to make the report even shorter.
+
+Don't forget that you can also use the directive :ref:`user_visible_url` to customize the URL that is reported visible
+(e.g. watching a REST API endpoint, but wanting to show the "web-visible" URL in the report).
+
+If you want the alert for one job only (of many), consider using the :ref:`sha1sum` filter instead.
 
 For example, for email set these in the configuration file (``webchanges --edit-config``):
 
 .. code-block:: yaml
 
    report:
-     ...
+     # ...
      text:
        details: false
        footer: false
-       ...
+       # ...
      email:
        html: false
-       ...
+       # ...

docs/filters.rst

@@ -756,7 +756,7 @@ detail, or saving large snapshots of data.
 
 .. code-block:: yaml
 
-   name: "Calculate SHA-1 hash test"
+   name: "Calculate SHA-1 hash"
    url: https://example.com/sha.html
    filter:
      - sha1sum:
@@ -1160,16 +1160,15 @@ Example:
    filter:
      - shellpipe: echo TEST
 
-.. warning::
-   On Linux and macOS systems, due to security reasons the ``shellpipe`` filter will not run unless **both** the config
-   directory **and** the jobs file are both **owned** and **writeable** by **only** the user who is running the job
-   (and not by its group or by other users). To set this up:
+.. important:: On Linux and macOS systems, due to security reasons the ``shellpipe`` filter will not run unless **both**
+   the jobs file **and** the directory it is located in are **owned** and **writeable** by **only** the user who is
+   running the job (and not by its group or by other users). To set this up:
 
    .. code-block:: bash
 
       cd ~/.config/webchanges  # could be different
-      sudo chown $USER:$(id -g -n) *.yaml
-      sudo chmod go-w *.yaml
+      sudo chown $USER:$(id -g -n) . *.yaml
+      sudo chmod go-w . *.yaml
 
    * ``sudo`` may or may not be required;
    * If making the change from a different account than the one you run :program:`webchanges` from, replace