CKAN Versions support going forward

[amercader]

We are planning on significantly extending and improving the current features and for that we need to focus our efforts. New features are python3 dependent, so users will need to choose the right ckanext-validation version depending on their CKAN version, with the goal that eventually everybody will be using CKAN 2.9+ on Python 3.

• ckanext-validation 0.0.8 supports CKAN <= 2.8
• ckanext-validation 1.0.0 will support CKAN 2.8 and 2.9 (py2 and py3). This is almost done in CKAN 2.9 / Python 3 #55
• ckanext-validation 2.0.0 will support CKAN >= 2.9 with Python 3 support only. Here's where we will do the planned work, including replacing goodtables with frictionless-py, which is Python 3 only

Once #55 is done I'll release the new version and update the README with these guidelines

cc @aivuk @ThrawnCA

edit: added support for CKAN 2.8 in 1.0.0

[ThrawnCA]

Are you sure it's really that difficult to keep supporting multiple versions? We're already doing it in the qld-gov-au fork and it doesn't seem to be all that hard.

[amercader]

ooookay ookay, we'll include support for 2.8 in 1.0.0 :)
Personally I think compatibility hacks like the ones to support calls to Flask or Pylons test apps make the test harder to read (and extend!) but we'll make an effort to keep up with the core support policy

[ThrawnCA]

I like the idea of a version that supports all CKAN versions. It means you can update components one at a time, instead of having to simultaneously update CKAN and the plugin.

The flipside of tests being harder to read is that making use of helper functions can actually make the tests easier to maintain. It encourages good refactoring and DRY code.

[ThrawnCA]

This extension is only now becoming compatible with newer CKAN+Python. CKAN 2.9 has been out for a while, 2.8 hasn't been accepting pull requests except serious bug fixes, but if the plugins aren't compatible, how can end users upgrade? Dropping 2.8 compatibility when people haven't even had the chance to properly try out 2.9 yet is too quick. There needs to be a transition period.

We recently updated to CKAN 2.9 because of the lack of support for 2.8, but we had to update a bunch of extensions ourselves in order to make it work, and we're still dealing with a lot of bugs in the fallout, things that were overlooked in different extensions, etc. It's nice that the upstream for different plugins is now being updated, but it would have been nicer if that happened before people were being pushed to drop 2.8.

Anyway, rant over, I'm happy to see that 2.8 compatibility is being retained for 1.0.0.

[amercader]

This extension is only now becoming compatible with newer CKAN+Python. CKAN 2.9 has been out for a while, 2.8 hasn't been accepting pull requests except serious bug fixes, but if the plugins aren't compatible, how can end users upgrade? Dropping 2.8 compatibility when people haven't even had the chance to properly try out 2.9 yet is too quick. There needs to be a transition period.

We (CKAN team) make an effort before a new CKAN version is released to upgrade the officially supported extensions (this was done for 2.9 and it is being done for 2.10). Of course we can not cover every single extension out there, specially others maintained by other organizations. In this case, we (OKF) just didn't have the time or resources to update this particular extension until now (or previously, when I started #55 but other priorities came in). To the question of how can end users upgrade in this situation I'd say that one way to help is to contribute as many compatibility fixes as possible to the upstream repos to help bring up extensions up to speed.

We recently updated to CKAN 2.9 because of the lack of support for 2.8, but we had to update a bunch of extensions ourselves in order to make it work, and we're still dealing with a lot of bugs in the fallout, things that were overlooked in different extensions, etc.

As I said, support for 2.9 has been in place for a while in most official extensions. If there are bugs or things overlooked in the migration again the best thing to do is to send the patch upstream (which I'm aware you do in many cases). CKAN is mostly run by volunteers and the community.

It's nice that the upstream for different plugins is now being updated, but it would have been nicer if that happened before people were being pushed to drop 2.8.

Again, we'd love to keep all things always up to date but that's just not possible. Users are encouraged to drop 2.8 (released on 2018, 4 years ago) because there are more modern versions that are safer and the CKAN team can actually maintain, that's just how open source software works. It's unfortunate that this particular extension could not keep up with the core release cycle for a while but hopefully it will from now on.

[duttonw]

Hi @amercader,
Per ckan supported versions documentations.
At any one time, the CKAN Tech Team will support the latest patch release of the last released minor version plus the last patch release of the previous minor version.

2.9.x is the latest offical release at this time, as such 2.8.x still needs to be supported. Once 2.10.x drops, then you could possibly drop py2 support at that time also. Though when things like ckan/ckan#7095 happen. It does not seem like 2.10 is yet viable.

[amercader]

@duttonw yes, while 2.10 is not out (because it's in active development) 2.8 is still supported. py2 support will be dropped at the end of the year (which will hopefully coincide with the 2.10 release). Here's all the plan laid out: https://ckan.org/blog/new-supported-versions-policy-going-forward

[rantoniuk]

I came here after looking at the Compatibility table in README.md and looking for support for current CKAN version: 2.11. Just making sure that 2.11 as stated there is unsupported as it sounds somehow contradictory to this sentence:

ckanext-validation 2.0.0 will support CKAN >= 2.9 with Python 3 support only. Here's where we will do the planned work, including replacing goodtables with frictionless-py, which is Python 3 only

[ThrawnCA]

It is slightly confusing wording, particularly the use of >=, but I'm pretty sure it was never intended to mean "Version 2.0.0 will definitely support every future CKAN version." More likely it's: "Version 2.0.0 will support CKAN 2.9, and will possibly work on later versions depending on what they change."

[duttonw]

Hi @rantoniuk,

Per @ThrawnCA, 2.10 is fully supported on the qld-gov-au branch, this upstream repo has recently been transferred to the CKAN org and the qld-gov-au maintainers have been granted access to continue to improve it.

2.10 + 2.11 for this upstream repo is coming soon. You can see the progress here: https://github.com/ckan/ckanext-validation/actions/runs/12346855699

The qld-gov-au maintainer group is working on this but sadly the code bases between the two forks has drifted and needs heaps more TLC then we previously thought. The readme also needs updating, currently I've put in what is 'currently' supported by this repo after it was moved.

With all the improvements now getting done on this repo version including auto pypi version releases, the qld-gov-au fork should only be requiredfor alpha testing or feature enhancements our business folk request in the near future.

---

2.11's biggest hurdle atm is correcting db startup for this plugin as in 2.11 it now throws
sqlalchemy.exc.UnboundExecutionError: Table object 'validation' is not bound to an Engine or Connection. Execution can not proceed without a database to execute against. before we even get to testing.

[rantoniuk]

2.11's biggest hurdle atm is correcting db startup for this plugin as in 2.11 it now throws
sqlalchemy.exc.UnboundExecutionError: Table object 'validation' is not bound to an Engine or Connection. Execution can not proceed without a database to execute against. before we even get to testing.

Yep, that's exactly where I got after adding the plugin in my dockerized instance via:

RUN  pip3 install -e git+https://github.com/ckan/[email protected]#egg=ckanext-validation && \
     pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-validation/2.2.0/requirements.txt

on a 2.11 setup, just to see if it works. Took me a while to decipher that the paster validation init-db -c ../path/to/ini/file should have been replaced by ckan, but then... I guess there is a chicken-egg problem

• validation cannot be added to the list of loaded plugins because the tables were not created for it
• tables cannot be created, because the plugin is not loaded, so ckan validation init-db cannot be run

Am I guessing correctly?

[duttonw]

@ThrawnCA suggest i look into ckanext-datarequests as he fixed it up for 2.11

I'll get to that after incorporating the keitaroinc/ckan-2.10 pr.