February 2021

Fixing issue with /manuscripts/ view.

There is a "invalid ReturnList" error. It seems to be related with the restframework. I managed to fix it with a hack I found in this issue: https://github.com/encode/django-rest-framework/issues/5236

Customizing the TemplateHTMLRenderer class in cantusdata/renderers/__init__.py

Fixing issue with /admin/map_folios/ view

Apparently, there is a limit of fields that can be submitted now (1000), and we hit that limit with our current implementation of the map_folios view.

The maximum number can be set in the settings.py. Adding this line solves the problem:

DATA_UPLOAD_MAX_NUMBER_FIELDS = 10000

Nevertheless, this fix is only apparently working. Once the backend is trying to process the request, an obscure new error crashes the backend:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/home/vagrant/public/cantusdata/views/map_folios.py", line 189, in _save_mapping
    call_command("refresh_solr", "chants", str(manuscript_id))
  File "/home/vagrant/public/app_env/lib/python3.6/site-packages/django/core/management/__init__.py", line 110, in call_command
    command = load_command_class(app_name, command_name)
  File "/home/vagrant/public/app_env/lib/python3.6/site-packages/django/core/management/__init__.py", line 36, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/vagrant/public/cantusdata/management/commands/refresh_solr.py", line 10, in <module>
    class Command(BaseCommand):
  File "/home/vagrant/public/cantusdata/management/commands/refresh_solr.py", line 26, in Command
    option_list = BaseCommand.option_list + (
AttributeError: type object 'BaseCommand' has no attribute 'option_list'

The last problem I had when running map_folios

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/home/vagrant/public/cantusdata/views/map_folios.py", line 189, in _save_mapping
    call_command("refresh_solr", "chants", str(manuscript_id))
  File "/home/vagrant/public/app_env/lib/python3.6/site-packages/django/core/management/__init__.py", line 110, in call_command
    command = load_command_class(app_name, command_name)
  File "/home/vagrant/public/app_env/lib/python3.6/site-packages/django/core/management/__init__.py", line 36, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/vagrant/public/cantusdata/management/commands/refresh_solr.py", line 10, in <module>
    class Command(BaseCommand):
  File "/home/vagrant/public/cantusdata/management/commands/refresh_solr.py", line 26, in Command
    option_list = BaseCommand.option_list + (
AttributeError: type object 'BaseCommand' has no attribute 'option_list'

was related to a deprecated way of handling CLI arguments in the django management system.

The new way is to overload a add_arguments function instead of appending data to the BaseCommand.option_list variable.

This had to be done at least in solr_refresh.py and import_folio_mappings.py. Probably more scripts will require this change.

For now, 15181 is able to render correctly in django==2.7

Today, I went over the Salzinnes and 15181 manuscripts in the django==2.2 version of the website.

The error reported by Anna is gone when using the automatic mapping script. Both manuscripts seem to be working okay.

I think it is a good moment to clean the code (removed commented lines, etc.) and merge this branch as the head of develop.

The next step will be to move from django==2.2.X to django==3.1.X. I don't think this process is going to be as complicated as it was to get to 2.2, mostly because we are already running on the latest djangorestframework and the latest psycopg2. It's all about catching up with django updates.

Toward django==3.1.X

The process for going to django==3.1.X was not that difficult after all. I moved to that version and everything works as expected.

I remember afterward I tried to move to Ubuntu 20.04 and that was a more complicated problem.

• Ubuntu 20.04 does not include python2 anymore, it seems that python2 is required by some packages installed with npm install for the frontend. I think all of these frontend packages have to be updated too before we move forward in versions.
• Node 8 is no longer supported in Ubuntu 20.04, so I need to get a newer version in Ubuntu 20.04 and that might also break compatibility by itself.

First thing I am going to do is try to get things running on Ubuntu 18.04. If that works, at least we are one step closer to 20.04.

I want to see a couple things:

• Whether Ubuntu 18.04 will also require me to get a more recent Node version
• If so, will that recent Node break the front end compilation, regardless of python2
• As far as I remember, python2 was still available in Ubuntu 18.04

Ubuntu 18.04

Ubuntu 18.04 worked plug-and-play. No changes were necessary except one line in the Vagrantfile.

15181 loads correctly and shows queries/images/volpiano/etc.

Admin interface and mapping interface work as expected.

Furthermore, Ubuntu 18.04 is officially supported by ComputeCanada until March 2023.

Thus, we do not need to touch this code soon.

Cantus will officially be running on Ubuntu 18.04 until the frontend dependencies are updated/revisited.

Deploying staging version of the current container

Launcing from my computer works great. The machine is provisioned with a ComputeCanada-provided image and goes through the full process. Everything seems to be fine. I couldn't actually testdrive the application because I need to access with the hosts' ip and it's not included in the settings.

Nevertheless, all of that workflow of launching from my computer is useless anyway. What needs to happen is that the instance should be launched within the Staging-Builds security network. So the next step is launching from my lab computer.

When launching from my lab computer, vagrant is attempting to create the shared folder ('.', '/vagrant') through SMB (SAMBA). It asks for credentials. I've never experienced that before and I have no idea what credentials is it asking for.

It seems a workaround is to set type: 'nfs'.

This bypassed the first issue, but later threw another error when trying to ssh into the machine to provision.