media/documents/archive/ folder not persisted

[totti4ever]

Paperless-NG v1.1.1 gives this error, 1.1.0 starts without warnings or so (similar to #534 maybe)

Paperless-ng docker container starting...
Installing languages...
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [267 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [9504 B]
Fetched 8422 kB in 3s (2665 kB/s)
Reading package lists...
package tesseract-ocr-deu already installed!
package tesseract-ocr-eng already installed!
Mapping UID and GID for paperless:paperless to 1006:1012
creating directory /tmp/paperless
Waiting for PostgreSQL to start...
Apply database migrations...
Operations to perform:
  Apply all migrations: admin, auth, authtoken, contenttypes, django_q, documents, paperless_mail, sessions
Running migrations:
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 245, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/src/paperless/src/documents/migrations/1012_fix_archive_files.py", line 244, in move_old_to_new_locations
    f"Archived document ID:{doc.id} does not exist at: "
ValueError: Archived document ID:580 does not exist at: /usr/src/paperless/src/../media/documents/archive/bankenversicherungen/Versicherung A/2021-02-03 Dokumententitel - Typ none.pdf
  Applying documents.1012_fix_archive_files...

The it starts over again Paperless-ng docker container starting...

---

Apparently the problem is the archive folder, could that be? I have mapped the following folders:

• /usr/src/paperless/data
• /usr/src/paperless/media/documents/thumbnails
• /usr/src/paperless/media/documents/originals
• /usr/src/paperless/consume

But the document is reported missing at /usr/src/paperless/media/documents/archive/- so I guess I have to map that too to have it persisted? What is in there? The original files? Never tinkered with those. Any idea on how to fix the system, as I'd reckon that
all the originals are missing as they have not been mapped...

[jonaswinkler]

Oh well. First of all, we'll be able to fix this up. Let's go through this step by step.

Some explanation on what the archive is.

• Paperless accepts many different file formats (PDF, images, Office documents, text files, and possibly more in the future). When adding documents to paperless, it converts them into PDF documents, and stores both the original file and the converted PDF document ("archive").
• Reason 1: Display in the web front end. Making viewers for all different kinds of file types is hard, so having a PDF document of any file available for display is helpful.
• Reason 2: PDF documents are much more portable. Especially the PDF/A documents that paperless creates. You can just send them to someone, and they'll be able to open and read it.
• Reason 3: Some PDF documents are just images and don't have embedded text - the OCR library converts these documents into PDF documents with embedded text, so you can select and copy/paste from the document.

There's also a reason I don't just store the archived file and discard the original:

• The libraries may fail and screw up the document. If that happens, the original is still available.
• Especially for office documents, you may want to have access to the editable file in the future.

Originals and archive versions are stored in the respective folders.

Your issue is that no volume is mounted at the archive location. The official docker-compose files mount the entire media folder as a volume to have everything persisted. Therefore, everytime you restarted paperless the archived versions would disappear, but paperless still expects them to be there, since it has information about them stored in the database.

This is also the reason the migration fails. It requires archived files to be present, but they are not.

Paperless also uses a lock file in the root of the media folder to coordinate concurrent modifications to the media folder.

---

First of all, modify your setup so that the entire media folder is mounted as a volume. You'll have to move files from the originals and thumbail volume around. The alternative approach would be to just mount the archive folder as a volume as well, but then you might run into trouble when running operations from two different containers both modifying files in the media directory. A rare occurence, but it might happen.

Second, paperless remembers that it has created archived versions for your documents in the past. For each document, it remembers the checksum of the archived file, if present.

Two alternative approaches:

1. Remove the archive checksum from all documents. If you only have PDF documents in your collection, this is fine, but any other file (images / office documents) won't display in the web frontend.
2. Regenerate missing archive documents.

I can make some detailed instructions for both, depending on what you want to do.

[totti4ever]

Thanks a lot!
I opted for 1 and everything is fine now.

I think I had the mapping running from early versions and didn't check for changes on every update