Fix Folder.DoesNotExist error in local deployment

[cmsetzer]

#3563 may have introduced a regression affecting a locally deployed vanilla Perma setup (as far as I can tell, this does not affect prod). After re-deploying Perma from scratch with all Docker images, volumes, and caches cleared, the following error occurs when you submit any URL for capture:

  File "/perma/perma_web/api/views.py", line 512, in post
    link.move_to_folder_for_user(folder, request.user)  # also sets link.organization
  File "/perma/perma_web/perma/models.py", line 1794, in move_to_folder_for_user
    Folder.objects.select_for_update().get(pk=folder.tree_root_id)
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
perma.models.Folder.DoesNotExist: Folder matching query does not exist.

If we change the get(pk=folder.tree_root_id) to a filter(id=folder.tree_root_id).first(), the query just retrieves 0 rows and then moves on without throwing an exception.

Any concerns with this solution? Or does this suggest a deeper issue?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.03%. Comparing base (b3b0ad8) to head (98eca7d).
Report is 22 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3580      +/-   ##
===========================================
+ Coverage    68.99%   69.03%   +0.03%     
===========================================
  Files           48       48              
  Lines         6969     6978       +9     
===========================================
+ Hits          4808     4817       +9     
  Misses        2161     2161              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rebeccacremona]

Hmmmmm. Yes, I think that does suggest a deeper issue. Nice catch!!

All folders need to have tree_root_id populated, and if there are any folders that don't, those folders are in a bad state.

Here's what I'm going to bet is going on:

• since locally I've been using the same dev database for a long while, this data migration took care of it for me
• during test set up, we use this utility to fix up the installation of our JSON fixtures
• during local dev setup.... we do neither: we just install the fixtures

These old JSON fixtures are a total pain in the butt, and we've migrated... maybe half?... of the suite from using them. I can't wait until we are done with that project and can lose them.

In the meantime, I think probably the quickest fix is to run _fix_json_fixtures inside invoke dev.init_db?

I think we don't want to tweak the code so this isn't an error; if a production folder finds itself in this state, I think it ought to be noisy about it... though, now that we have migrated the production data to use this new field, we might be able to improve the way in which it is noisy about it (e.g., make the field non-nullable).

[cmsetzer]

Thanks @rebeccacremona! I had a hunch it might be something along those lines.

I agree about fixing the root issue in init_db rather than suppressing the exception. I'll try it and push up the change once everything checks out.

[rebeccacremona]

@cmsetzer wonderful, thanks!

if it isn't working for any reason... we can figure out how to update the json files so that they just have all the correct field values. i initially thought that would be too annoying, but could be that was the wrong call 🙂

[cmsetzer]

I took a swing at using _fix_json_fixtures in the DB init task but it did not like importing from conftest.py. Rather than deal with that, I just went ahead and corrected the source data in folders.json so no fixer function is needed.

This takes care of the local issue on my end and tests are passing. Want to take a look?

[rebeccacremona]

Ughh, sorry for that hassle!! Thanks for doing this 🙏!