Spyder 6 high cpu load when copying many files to a project

[denis-sp]

Problem Description

Spyder 6.0.* has performance issues and the cpu load is high when a project with a lot of files and directories in it is opened.

What steps reproduce the problem?

1. Open Spyder and create an empty project
2. Copy some directory with a lot of files and directories to the project directory
3. The cpu load goes high

What is the expected output? What do you see instead?

The cpu load for Spyder 5.5.* in idle was around 0%.

Versions

• Spyder version: 6.0.1
• Python version: 3.10.15
• Qt version: 5.15.8
• PyQt version: 5.15.9
• Operating System name/version: Windows 10

Dependencies

PASTE DEPENDENCIES HERE

Mandatory:

aiohttp >=3.9.3 : 3.10.5 (OK)
asyncssh >=2.14.0,<3.0.0 : 2.17.0 (OK)
atomicwrites >=1.2.0 : 1.4.1 (OK)
chardet >=2.0.0 : 5.2.0 (OK)
cloudpickle >=0.5.0 : 3.0.0 (OK)
cookiecutter >=1.6.0 : 2.6.0 (OK)
diff_match_patch >=20181111 : 20230430 (OK)
github >=2.3.0 : 2.4.0 (OK)
importlib_metadata >=4.6.0 : 8.5.0 (OK)
intervaltree >=3.0.2 : 3.1.0 (OK)
IPython >=8.13.0,<9.0.0,!=8.17.1 : 8.27.0 (OK)
jedi >=0.17.2,<0.20.0 : 0.19.1 (OK)
jellyfish >=0.7 : 1.1.0 (OK)
jsonschema >=3.2.0 : 4.23.0 (OK)
keyring >=17.0.0 : 25.4.1 (OK)
nbconvert >=4.0 : 7.16.4 (OK)
numpydoc >=0.6.0 : 1.8.0 (OK)
parso >=0.7.0,<0.9.0 : 0.8.4 (OK)
pexpect >=4.4.0 : 4.9.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
psutil >=5.3 : 6.0.0 (OK)
pygments >=2.0 : 2.18.0 (OK)
pylint >=3.1,<4 : 3.3.0 (OK)
pylint_venv >=3.0.2 : 3.0.3 (OK)
pyls_spyder >=0.4.0 : 0.4.0 (OK)
pylsp >=1.12.0,<1.13.0 : 1.12.0 (OK)
pylsp_black >=2.0.0,<3.0.0 : 2.0.0 (OK)
pyuca >=1.2 : 1.2 (OK)
qdarkstyle >=3.2.0,<3.3.0 : 3.2.3 (OK)
qstylizer >=0.2.2 : 0.2.3 (OK)
qtawesome >=1.3.1,<1.4.0 : 1.3.1 (OK)
qtconsole >=5.6.0,<5.7.0 : 5.6.0 (OK)
qtpy >=2.4.0 : 2.4.1 (OK)
rtree >=0.9.7 : 1.3.0 (OK)
setuptools >=49.6.0 : 74.1.2 (OK)
sphinx >=0.6.6 : 8.0.2 (OK)
spyder_kernels >=3.0.0,<3.1.0 : 3.0.0 (OK)
superqt >=0.6.2,<1.0.0 : 0.6.7 (OK)
textdistance >=4.2.0 : 4.6.2 (OK)
three_merge >=0.1.1 : 0.1.1 (OK)
watchdog >=0.10.3 : 5.0.2 (OK)
yarl >=1.9.4 : 1.11.1 (OK)
zmq >=24.0.0 : 26.2.0 (OK)

Optional:

cython >=0.21 : 3.0.11 (OK)
matplotlib >=3.0.0 : 3.9.2 (OK)
numpy >=1.7 : 1.26.4 (OK)
pandas >=1.1.1 : 2.2.2 (OK)
scipy >=0.17.0 : 1.14.1 (OK)
sympy >=0.7.3 : 1.13.2 (OK)

Spyder plugins:

spyder_notebook.notebookplugin 0.6.1 : 0.6.1 (OK)

[ccordoba12]

Hey @denis-sp, thanks for reporting. A couple of questions for you:

• Could you let us know if your project is in a directory part of a network drive?
• Does Spyder freeze due to that high CPU load? Or does it work normally?

[ccordoba12]

@dalthviz, could you check if this happens for you on Windows as well?

[denis-sp]

Hey @denis-sp, thanks for reporting. A couple of questions for you:

* Could you let us know if your project is in a directory part of a network drive?

* Does Spyder freeze due to that high CPU load? Or does it work normally?

1. No, it's not a part of a network drive. It's just a directory with all my projects. I don't create separate projects and keep everything in one project but in separate directories.
2. Mostly it works fine, but sometimes it freezes. So working in this version is inconvenient so I did downgrade to 5.5.6. But I can upgrade it back and check whatever is needed.

[dalthviz]

Checked with the Spyder 6.0.1 Windows installer and I got CPU usage up to 85%. The Python process related with Spyder got up to 19-20% max CPU usage at some points but I would say it was closer to 9-10% most of the time. Other process with high CPU usage were Windows Explorer and the Antimalware Service Executable process:

From what I was able to test, seems like Spyder was not freezing/was able to respond. Probably having more specific steps to reproduce could be worthy (I checked this by copying my Spyder repo clone to a newly created directory/project).

Thinking about possible elements that could be causing extra CPU usage while changing project files, could this be related with fzf usage @ccordoba12 ? Maybe the way it is being used triggers the Windows antimalware service?

Following that, @denis-sp did you install Spyder using conda or have fzf installed over your setup? If you have it installed, what happens if you uninstall that package?

Any other info to better understand what could be happening is greatly appreciated. Let us know!

[ccordoba12]

Probably having more specific steps to reproduce could be worthy (I checked this by copying my Spyder repo clone to a newly created directory/project).

I agree. @denis-sp, you said you have all your projects in a single directory. How many files and subdirectories does that directory have?

Thinking about possible elements that could be causing extra CPU usage while changing project files, could this be related with fzf usage @ccordoba12 ? Maybe the way it is being used triggers the Windows antimalware service?

It could be, good idea. Please uninstall it and try again.

[denis-sp]

Probably having more specific steps to reproduce could be worthy (I checked this by copying my Spyder repo clone to a newly created directory/project).

I agree. @denis-sp, you said you have all your projects in a single directory. How many files and subdirectories does that directory have?

Thinking about possible elements that could be causing extra CPU usage while changing project files, could this be related with fzf usage @ccordoba12 ? Maybe the way it is being used triggers the Windows antimalware service?

It could be, good idea. Please uninstall it and try again.

Well... it's huge:

[denis-sp]

Following that, @denis-sp did you install Spyder using conda or have fzf installed over your setup? If you have it installed, what happens if you uninstall that package?

Any other info to better understand what could be happening is greatly appreciated. Let us know!

It is installed from miniforge distribution. And I have Windows antimalware service disabled.
fzf is installed indeed.

[ccordoba12]

Well... it's huge:

Totally! We were not expecting users to have such a massive project directory, but if things were working for you in Spyder 5, we should be able to make them work again.

fzf is installed indeed.

Ok, perhaps uninstallation of that package is not so easy because it's a Spyder dependency, and Conda/Mamba will try to also uninstall Spyder if one of its dependencies is uninstalled too.

Instead, @denis-sp please follow my advice in #22556 (comment) to disable fzf in Spyder's source code. And if that doesn't work, follow my second advice to disable the file system watcher.

[denis-sp]

Setting _fzf to None doesn't work. But commenting the watcher works.

[ccordoba12]

Thanks for the confirmation @denis-sp. We'll try to fix this problem in 6.0.3 or 6.0.4, both to be released before the end of the year.