FIX : Temporary files not cleaned after use

[unichronic]

This fixes issue #731

Earlier the tempfiles and directories created during activities in a project were not cleaned up and led to piling up of files causing the folder to bloat upto several gigabytes.

Changes Made:

• Added a TempfileManager Class within utils.py which tracks new tempfile creation and cleans them up when the the project or application is closed.
• Some files are not being able to be cleaned due to file-locking issues during shut down. I have added a clean_up_old_files() within TempfilManager to check for older tempfiles/directories (older than 1 hour or can be changed according to suggestions) and remove them.
• Implemented the tempfile registration and cleanup with several project objects like masks and surfaces.
• Fixed a few other smaller issues already existing in the code such as NoneType AttributeErrors, circular import errors, project_path errors by adding checks.

Features:

• __init__() initialises the TempfileManager, starts the ThreadPoolExecutor and calls the _startup_background_cleanup() function on application startup.
• _startup_background_cleanup() starts a thread for the _cleanup_old_files to clean up older tempfiles/directories in the background.
• _cleanup_old_files() pattern matches and finds out project related tempfiles/directories which are older than OLD_FILE_THRESHOLD from previous sessions and adds them to the cleanup_queue.
• register_temp_file() tracks tempfiles creation across various part of the codebase and adds to their reference count. The tracked files are added to cleanup_queue once the reference count drops to zero.
• decrement_refs() decreases the reference count of tracked file objects every time they are dropped or deleted.
• _cleanup_worker() fetches items from the cleanup_queue and removes them using _safe_remove_file().
• _safe_remove_file() safely handles and removes tempfiles once their reference counts drop to zero i.e. they are not being used by any part of the program and properly handles errors.
• shutdown() stops the running threads, calls cleanup_all() and properly shuts down TempfileManager.
• cleanup_all() cleans up any leftover tempfiles and starts their cleanup.

I have tested the changes with some basic operations like creating/removing a mask and creating/removing a surface, then saving and reopening.

It would be great if further testing is done.

Please take a look @tfmoraes @paulojamorim @rmatsuda @Rakhesis

P.S. I have commented out the logging statements, as I had added them for testing. They can be removed if needed.

[tfmoraes]

Hi @unichronic I didn't checked this PR yet. But it has some modifications to plugins, please remove those modifications.

[unichronic]

Hi @unichronic I didn't checked this PR yet. But it has some modifications to plugins, please remove those modifications.

Hello @tfmoraes I think the modifications to plugin folder have been removed now.

[unichronic]

The effectiveness of the cleanup can be checked by opening the Temp file directory and seeing the changes. Some files are cleaned up on closing, and .vtp files and temp directories with project data are deleted when Invesalius is restarted.

[unichronic]

Hey @tfmoraes @paulojamorim @rmatsuda, would be great if you could test the changes.

[tfmoraes]

Hi @unichronic I will try to check this in the next days.