Version 3.10.2rc3

Author: mborsetti

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 3.10.2rc2
+current_version = 3.10.2rc3
 message = Release {new_version}
 parse = ^
 	(?P<major>\d+)

.deploy.bat

@@ -36,6 +36,9 @@ if %r% EQU n exit /b
 set /p r=Did you run local tests (tox) and ensure CI passed? [N/y] || set r=n
 if %r% EQU N r=n
 if %r% EQU n exit /b
+set /p r=Did you check https://readthedocs.org/projects/webchanges/builds/ and ensure docs built there? [N/y] || set r=n
+if %r% EQU N r=n
+if %r% EQU n exit /b
 set /p r=Are you in the unreleased branch? [N/y] || set r=n
 if %r% EQU N r=n
 if %r% EQU n exit /b

.github/ISSUE_TEMPLATE/bug_report.md

@@ -18,7 +18,8 @@ bug.
 A clear and concise description of what you expected to happen.
 
 **Screen scrape/screenshots**
-If applicable, add screen scrape or screenshots to help explain your problem.
+If applicable, add screen scrape or screenshots to help explain the bug.  Use ``-v`` or ``-vv`` to capture 
+logging.
 
 **Version info**
 Please run ``webchanges -v`` and paste the version information as follows (first 3 lines):

.github/ISSUE_TEMPLATE/feature_request.md

@@ -8,7 +8,7 @@ assignees: ''
 ---
 
 **Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+A clear and concise description of what is the problem to solve. Ex. I'm always frustrated when [...]
 
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.

CHANGELOG.rst

@@ -31,7 +31,7 @@ can check out the `wish list <https://github.com/mborsetti/webchanges/blob/main/
    Internals, for changes that don't affect users. [triggers a minor patch]
 
 
-Version 3.10.2rc2
+Version 3.10.2rc3
 ===================
 Unreleased
 
@@ -43,12 +43,14 @@ Unreleased
 
 Added
 -----
-* You can now run the command line argument ``--test`` without specifying a JOB; this will run a check of the config and
-  job files for syntax errors.
+* You can now run the command line argument ``--test`` without specifying a JOB; this will check the config
+  (default: ``config.yaml``) and job (default: ``job.yaml``) files for syntax errors.
+* New job directive ``compared_versions`` allows change detection to be made against multiple saved snapshots;
+  useful for monitoring websites that change between a set of states (e.g. they are running A/B testing).
+* New command line argument ``--check-new`` to check if a new version of **webchanges** is available.
 * Error messages for url jobs failing with HTTP reason codes of 400 and higher now include any text returned by the
   website (e.g. "Rate exceeded.", "upstream request timeout", etc.). Not implemented in jobs with ``use_browser: true``
   due to limitations in Playwright.
-* New command line argument ``--check-new`` to check if a new version of **webchanges** is available.
 
 Changed
 -------

RELEASE.rst

@@ -6,12 +6,14 @@
 
 Added
 -----
-* You can now run the command line argument ``--test`` without specifying a JOB; this will run a check of the config and
-  job files for syntax errors.
+* You can now run the command line argument ``--test`` without specifying a JOB; this will check the config
+  (default: ``config.yaml``) and job (default: ``job.yaml``) files for syntax errors.
+* New job directive ``compared_versions`` allows change detection to be made against multiple saved snapshots;
+  useful for monitoring websites that change between a set of states (e.g. they are running A/B testing).
+* New command line argument ``--check-new`` to check if a new version of **webchanges** is available.
 * Error messages for url jobs failing with HTTP reason codes of 400 and higher now include any text returned by the
   website (e.g. "Rate exceeded.", "upstream request timeout", etc.). Not implemented in jobs with ``use_browser: true``
   due to limitations in Playwright.
-* New command line argument ``--check-new`` to check if a new version of **webchanges** is available.
 
 Changed
 -------

docs/examples.rst

@@ -44,6 +44,20 @@ crontab)::
 
 .. _always_report:
 
+
+Comparing with several latest snapshots
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+If a webpage frequently changes between several known stable states (e.g. A/B layout testing), it may be desirable to
+have changes reported only if the webpage changes into a new unknown state. You can use compared_versions to do this.
+
+.. code-block:: yaml
+
+   url: https://example.com/
+   compared_versions: 3
+
+In this example, changes are only reported if the webpage becomes different from the latest three distinct states.
+The differences are shown relative to the closest match.
+
 Receiving a report for every run
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 If you are watching pages that change seldomly, but you still want to be notified every time :program:`webchanges`

docs/jobs.rst

@@ -168,6 +168,20 @@ Whether to use a Chrome web browser (true/false). Defaults to false.
 If true, it renders the URL via a JavaScript-enabled web browser and extracts the HTML after rendering (see
 :ref:`above <use_browser>` for important information).
 
+compared_versions
+^^^^^^^^^^^^^^^^^
+Number of saved snapshots to compare against (int). Defaults to 1.
+
+If set to a number greater than 1, instead of comparing the current data to only the very last snapshot captured, it
+is matched against any of *n* snapshots. This is very useful when a webpage frequently changes between several known
+stable states (e.g. they're doing A/B testing), as changes will be reported only when the content changes to a new
+unknown state, in which case the differences are shown relative to the closest match.
+
+Refer to the command line argument ``--max-snapshots`` to ensure that you are saving the number of snapshots you need
+for this directive to run successfully (default is 4) (see :ref:`here<max-snapshots>`).
+
+.. versionadded:: 3.10.2
+
 cookies
 ^^^^^^^
 Cookies to send with the request (a dict).
@@ -194,7 +208,7 @@ http_proxy
 Proxy server to use for HTTP requests (a string). If unspecified or null/false, the system environment variable
 ``HTTP_PROXY``, if defined, will be used.
 
-E.g. ``\http://username:[email protected]:8080``.
+E.g. ``http://username:[email protected]:8080``.
 
 .. versionchanged:: 3.0
    Works for all ``url`` jobs, including those with ``use_browser: true``.
@@ -204,7 +218,7 @@ https_proxy
 Proxy server to use for HTTPS (i.e. secure) requests (a string). If unspecified or null/false, the system environment
 variable ``HTTPS_PROXY``, if defined, will be used.
 
-E.g. ``\https://username:[email protected]:8080``.
+E.g. ``https://username:[email protected]:8080``.
 
 .. versionchanged:: 3.0
    Works for all ``url`` jobs, including those with ``use_browser: true``.
@@ -332,7 +346,32 @@ The following directives are available only for ``url`` jobs without ``use_brows
 
 no_redirects
 ^^^^^^^^^^^^
-Disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection (true/false). Defaults to false.
+Disables GET, OPTIONS, POST, PUT, PATCH, DELETE, HEAD redirection (true/false). Defaults to false (i.e. redirection
+is enabled) for all methods except HEAD. See more `here
+<https://requests.readthedocs.io/en/latest/user/quickstart/#redirection-and-history>`__.  Redirection takes place
+whenever an HTTP status code of 301, 302, 303, 307 or 308 is returned.
+
+Example:
+
+.. code-block:: yaml
+
+   url: "https://donneespubliques.meteofrance.fr/donnees_libres/bulletins/BCM/203001.pdf"
+   no_redirects: true
+   filter:
+     - html2text:
+
+Returns:
+
+.. code-block::
+
+   302 Found
+   ---------
+
+   # Found
+   The document has moved [here](https://donneespubliques.meteofrance.fr/?fond=donnee_indisponible).
+   * * *
+   Apache/2.2.15 (CentOS) Server at donneespubliques.meteofrance.fr Port 80
+
 
 .. versionadded:: 3.2.7
 

webchanges/__init__.py

@@ -18,7 +18,7 @@
 # * MINOR version when you add functionality in a backwards compatible manner, and
 # * MICRO or PATCH version when you make backwards compatible bug fixes. We no longer use '0'
 # If unsure on increments, use pkg_resources.parse_version to parse
-__version__ = '3.10.2rc2'
+__version__ = '3.10.2rc3'
 __description__ = (
     'Check web (or commands) for changes since last run and notify.\n\nAnonymously alerts you of webpage changes.'
 )

webchanges/handler.py

@@ -56,6 +56,7 @@ class JobState(ContextManager):
     _generated_diff_html: Optional[str] = None
     error_ignored: Union[bool, str]
     exception: Optional[Exception] = None
+    history_data: Dict[str, float] = {}
     new_data: str
     new_etag: str
     new_timestamp: float
@@ -121,6 +122,7 @@ def load(self) -> None:
         """Loads form the database the last snapshot for the job."""
         guid = self.job.get_guid()
         self.old_data, self.old_timestamp, self.tries, self.old_etag = self.cache_storage.load(guid)
+        self.history_data = self.cache_storage.get_history_data(guid, self.job.compared_versions)
 
     def save(self, use_old_data: bool = False) -> None:
         """Saves new data retrieved by the job into the snapshot database.

webchanges/jobs.py

@@ -100,6 +100,7 @@ class JobBase(object, metaclass=TrackSubClasses):
     additions_only: Optional[bool] = None
     block_elements: List[str] = []
     chromium_revision: Optional[Union[Dict[str, int], Dict[str, str], str, int]] = None  # deprecated
+    compared_versions: int = 1
     contextlines: Optional[int] = None
     cookies: Optional[Dict[str, str]] = None
     data: Union[str, Dict[str, str]] = None  # type: ignore[assignment]
@@ -426,6 +427,7 @@ class Job(JobBase):
         'name',
         'note',
         'additions_only',
+        'compared_versions',
         'contextlines',
         'deletions_only',
         'diff_filter',
@@ -840,6 +842,7 @@ def _playwright_retrieve(self, job_state: JobState, headless: bool = True) -> Tu
         :raises BrowserResponseError: If a browser error or an HTTP response code between 400 and 599 is received.
         """
         try:
+            from playwright._repo_version import version as playwright_version
             from playwright.sync_api import Error as PlaywrightError
             from playwright.sync_api import ProxySettings, Route, sync_playwright
         except ImportError:
@@ -983,8 +986,9 @@ def _playwright_retrieve(self, job_state: JobState, headless: bool = True) -> Tu
                     ignore_https_errors=self.ignore_https_errors,  # type: ignore[arg-type]
                     extra_http_headers=dict(headers),
                 )
-                logger.debug(
-                    f'Job {self.index_number}: Pyppeteer launched {browser_name} browser ' f'{browser.version}'
+                logger.info(
+                    f'Job {self.index_number}: Pyppeteer {playwright_version} launched {browser_name} browser'
+                    f' {browser.version}'
                 )
             else:
                 context = p.chromium.launch_persistent_context(
@@ -999,9 +1003,9 @@ def _playwright_retrieve(self, job_state: JobState, headless: bool = True) -> Tu
                     ignore_https_errors=self.ignore_https_errors,  # type: ignore[arg-type]
                     extra_http_headers=dict(headers),
                 )
-                logger.debug(
-                    f'Job {self.index_number}: Pyppeteer launched {browser_name} browser version'
-                    f' {context.browser.version} with user data directory '  # type: ignore[union-attr]
+                logger.info(
+                    f'Job {self.index_number}: Pyppeteer {playwright_version} launched {browser_name} browser '
+                    f'{context.browser.version} from user data directory '  # type: ignore[union-attr]
                     f'{self.user_data_dir}'
                 )
 

webchanges/main.py

@@ -93,7 +93,7 @@ def load_hooks(self) -> None:
             )
         else:
             import_module_from_source('hooks', self.urlwatch_config.hooks)
-            logger.info(f'Loaded hooks from {self.urlwatch_config.hooks}')
+            logger.info(f'Imported hooks module from {self.urlwatch_config.hooks}')
 
     def load_jobs(self) -> None:
         """Load jobs from the file into self.jobs.
@@ -102,7 +102,6 @@ def load_jobs(self) -> None:
         """
         if self.urlwatch_config.jobs.is_file():
             jobs = self.jobs_storage.load_secure()
-            logger.info(f'Loaded {len(jobs)} jobs from {self.urlwatch_config.jobs}')
         else:
             print(f'Jobs file not found: {self.urlwatch_config.jobs}')
             raise SystemExit(1)

webchanges/storage.py

@@ -564,6 +564,7 @@ def is_shell_job(job: JobBase) -> bool:
             )
             jobs = [job for job in jobs if job not in removed_jobs]
 
+        logger.info(f'Loaded {len(jobs)} jobs from {self.filename}')
         return jobs
 
 
@@ -1060,7 +1061,7 @@ def __init__(self, filename: Union[str, Path], max_snapshots: int = 4) -> None:
         self.lock = threading.RLock()
 
         self.db = sqlite3.connect(filename, check_same_thread=False)
-        logger.info(f'Using sqlite3 database at {filename}')
+        logger.info(f'Using sqlite3 {sqlite3.sqlite_version} database at {filename}')
         self.cur = self.db.cursor()
         self.cur.execute('PRAGMA temp_store = MEMORY;')
         tables = self._execute("SELECT name FROM sqlite_master WHERE type='table';").fetchone()

webchanges/worker.py

@@ -4,6 +4,7 @@
 
 from __future__ import annotations
 
+import difflib
 import logging
 import os
 import random
@@ -113,14 +114,22 @@ def job_runner(
                     urlwatcher.report.error(job_state)
                 else:
                     logger.info(f'Job {job_state.job.index_number}: Job finished with no exceptions')
-            elif job_state.old_data != '' or job_state.old_timestamp != 0:
+            elif len(job_state.old_data) or job_state.old_timestamp != 0:
                 # This is not the first time running this job (we have snapshots)
-                if job_state.new_data == job_state.old_data:
+                if job_state.history_data.get(job_state.new_data):
+                    # exactly matches one of the previous snapshots
                     if job_state.tries > 0:
                         job_state.tries = 0
                         job_state.save()
                     urlwatcher.report.unchanged(job_state)
                 else:
+                    # no match
+                    if len(job_state.history_data) > 1:
+                        # replace old with best "good enough" previous snapshot
+                        close_matches = difflib.get_close_matches(job_state.new_data, job_state.history_data, n=1)
+                        if close_matches:
+                            job_state.old_data = close_matches[0]
+                            job_state.old_timestamp = job_state.history_data[close_matches[0]]
                     job_state.tries = 0
                     job_state.save()
                     urlwatcher.report.changed(job_state)