jsonschema RefResolutionError for 0.6.3 or lower

[masonproffitt]

Summary

The newest version of jsonschema (4.15.0, released today) breaks the current release version of pyhf (0.6.3).

OS / Environment

NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.8"

Steps to Reproduce

pip install 'jsonschema==4.15.0' 'pyhf==0.6.3'

import pyhf
model = pyhf.simplemodels.uncorrelated_background(
    signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
)

File Upload (optional)

No response

Expected Results

I expected not to get an error because this is the Hello World example. I confirmed that there's no error with jsonschema==4.14.0 (the previous version) in the exact same environment otherwise.

Actual Results

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 889, in resolve_from_url
    document = self.store[url]
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/_utils.py", line 28, in __getitem__
    return self.store[self.normalize(uri)]
KeyError: 'file:///home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/schemas/defs.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 1505, in open_local_file
    stats = os.stat(localfile)
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/schemas/defs.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 892, in resolve_from_url
    document = self.resolve_remote(url)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 1000, in resolve_remote
    with urlopen(uri) as url:
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 1483, in file_open
    return self.open_local_file(req)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/urllib/request.py", line 1522, in open_local_file
    raise URLError(exp)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/schemas/defs.json'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/simplemodels.py", line 141, in uncorrelated_background
    return Model(spec, batch_size=batch_size)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/pdf.py", line 682, in __init__
    utils.validate(self.spec, self.schema, version=self.version)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/utils.py", line 62, in validate
    return validator.validate(spec)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 302, in validate
    for error in self.iter_errors(*args, **kwargs):
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 277, in iter_errors
    for error in errors:
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/_validators.py", line 294, in ref
    scope, resolved = validator.resolver.resolve(ref)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 880, in resolve
    return url, self._remote_cache(url)
  File "/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/jsonschema/validators.py", line 894, in resolve_from_url
    raise exceptions.RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '/home/user/miniconda3/envs/abcd_pyhf/lib/python3.10/site-packages/pyhf/schemas/defs.json'>

pyhf Version

pyhf, version 0.6.3

Code of Conduct

• I agree to follow the Code of Conduct

[kratsg]

Something is off, there should be a 1.0.0 in there (for the version).

[masonproffitt]

Yeah, I'm not sure what's going on there. Just confirmed I see it in 0.7.0rc1, 0.7.0rc2, and master as well.

[kratsg]

Looks like the 4.15.0 changelog contains

* A specific API Reference page is now present in the documentation.
* ``$ref`` on earlier drafts (specifically draft 7 and 6) has been "fixed" to
  follow the specified behavior when present alongside a sibling ``$id``.
  Specifically the ID is now properly ignored, and references are resolved
  against whatever resolution scope was previously relevant.

python-jsonschema/jsonschema@v4.14.0...v4.15.0#diff-2c623f3c6a917be56c59d43279244996836262cb1e12d9d0786c9c49eef6b43c

We were following the jsonschema spec, so it seems the python jsonschema changed how it interprets paths.

[kratsg]

I'm checking to see if we catch this regression first in our head-of-dependencies (https://github.com/scikit-hep/pyhf/actions/runs/2966841468).

[kratsg]

@masonproffitt see if #1976 fixes things for you.

[kratsg]

I also opened up a discussion python-jsonschema/jsonschema#995 to understand where this behavior comes from:

[matthewfeickert]

Yeah, I'm not sure what's going on there. Just confirmed I see it in 0.7.0rc1, 0.7.0rc2, and master as well.

@masonproffitt I've made a v0.7.0rc3 pre-release that has the fix from PR #1976 and PR #1979 in it, which should make testing easier.

[kratsg]

The short of it is a few issues:

• jsonschema on older drafts was using the $id to determine where to load other relative schemas, ignoring base_uri which is incorrect behavior and buggy
• we were not using jsonschema validation correctly (being fixed in feat: HiFa v1.1.0 schema development #1978)
• newer drafts have better behavior

jsonschema==4.15.0 fixes the bug in older drafts and moves the behavior to more closely align with the spec.