EOFError: Ran out of input

[VengeRuan]

The error message I receive is as follows:

[2024-12-16 23:40:50,408][train-tinyRSNN][INFO] - Pretraining on all loco sessions...
Error executing job with overrides: ['seed=1']
Traceback (most recent call last):
File "E:\neural-decoding-RSNN-main\train-tinyRSNN.py", line 263, in
train_all()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self._return_value
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "E:\neural-decoding-RSNN-main\train-tinyRSNN.py", line 89, in train_all
model, history = train_validate_model(
File "E:\neural-decoding-RSNN-main\challenge\train.py", line 121, in train_validate_model
fix, ax = plot_activity_snapshot(
File "E:\neural-decoding-RSNN-main\challenge\utils\plotting.py", line 259, in plot_activity_snapshot
fig, ax = plot_activity_CST(
File "E:\neural-decoding-RSNN-main\challenge\utils\plotting.py", line 171, in plot_activity_CST
scores = model.evaluate(data)
File "e:\neural-decoding-rsnn-main\stork\stork\models.py", line 295, in evaluate
for local_X, local_y in self.data_generator(test_dataset, shuffle=False):
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 433, in iter
self._iterator = self._get_iterator()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 384, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 1048, in init
w.start()
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'configure_model..worker_init_fn'. Did you mean: '_return_value'?
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

[JRBCH]

Hi VengeRuan,

Can you supply some more context for this error message? What are you trying to run? What is your OS? Are you trying to run on CPU or GPU?

From your error message, it looks like there is a problem with pickle serialization that relates to multiprocessing. I have not tried running this code on a windows machine, which you seem to be using. A brief search got me to this thread: pytorch/pytorch#69611, where people seem to report issues with pickle when the num_workers argument for the PyTorch dataloaders is not set to 0.

I would suggest you try setting num_workers=0 in the dataloader and see if that fixed your problem. If not, please provide some more context so we can look into it.

Cheers,
Julian

[VengeRuan]

Thanks for your reply

PyTorch version: 1.12.0+cu116
OS: Microsoft Windows 11
run on GPU
I am trying to run the script of 'python train-tinyRSNN.py seed=1'

I have changed nb_workers in /conf/defaults.yaml,When using num_workers=1 or 0,it still doesn't work and I get the same Error.

[VengeRuan]

(py310) PS E:\neural-decoding-RSNN-main> python train-tinyRSNN.py seed=1
[2024-12-17 13:25:27,385][train-tinyRSNN][INFO] - Starting new simulation...
[2024-12-17 13:25:27,388][train-tinyRSNN][INFO] - Constructing model for loco pretraining...
Loading loco_20170210_03.mat
[2024-12-17 13:25:37,221][challenge.data][INFO] - Extending data...
Loading loco_20170215_02.mat
[2024-12-17 13:25:51,298][challenge.data][INFO] - Extending data...
Loading loco_20170301_05.mat
[2024-12-17 13:26:03,437][challenge.data][INFO] - Extending data...
[2024-12-17 13:26:13,639][challenge.model][INFO] - Initializing with nu = 7.965170860290527
[2024-12-17 13:26:13,642][challenge.model][INFO] - Adding single readout group
[2024-12-17 13:26:13,642][train-tinyRSNN][INFO] - Configuring model...
[2024-12-17 13:26:14,803][train-tinyRSNN][INFO] - Pretraining on all loco sessions...
Error executing job with overrides: ['seed=1']
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
Traceback (most recent call last):
File "E:\neural-decoding-RSNN-main\train-tinyRSNN.py", line 260, in
train_all()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self._return_value
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "E:\neural-decoding-RSNN-main\train-tinyRSNN.py", line 86, in train_all
model, history = train_validate_model(
File "E:\neural-decoding-RSNN-main\challenge\train.py", line 121, in train_validate_model
fix, ax = plot_activity_snapshot(
File "E:\neural-decoding-RSNN-main\challenge\utils\plotting.py", line 259, in plot_activity_snapshot
fig, ax = plot_activity_CST(
File "E:\neural-decoding-RSNN-main\challenge\utils\plotting.py", line 171, in plot_activity_CST
scores = model.evaluate(data)
File "e:\neural-decoding-rsnn-main\stork\stork\models.py", line 295, in evaluate
for local_X, local_y in self.data_generator(test_dataset, shuffle=False):
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 433, in iter
self._iterator = self._get_iterator()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 384, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 1048, in init
w.start()
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "C:\Users\guozh\anaconda3\envs\py310\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'configure_model..worker_init_fn'. Did you mean: '_return_value'?

This is the output While using the num_workers=1

(py310) PS E:\neural-decoding-RSNN-main> python train-tinyRSNN.py seed=1
[2024-12-17 13:38:54,371][train-tinyRSNN][INFO] - Starting new simulation...
[2024-12-17 13:38:54,374][train-tinyRSNN][INFO] - Constructing model for loco pretraining...
Loading loco_20170210_03.mat
[2024-12-17 13:39:03,800][challenge.data][INFO] - Extending data...
Loading loco_20170215_02.mat
[2024-12-17 13:39:17,068][challenge.data][INFO] - Extending data...
Loading loco_20170301_05.mat
[2024-12-17 13:39:28,720][challenge.data][INFO] - Extending data...
[2024-12-17 13:39:38,281][challenge.model][INFO] - Initializing with nu = 7.965170860290527
[2024-12-17 13:39:38,283][challenge.model][INFO] - Adding single readout group
[2024-12-17 13:39:38,284][train-tinyRSNN][INFO] - Configuring model...
[2024-12-17 13:39:39,473][train-tinyRSNN][INFO] - Pretraining on all loco sessions...
Error executing job with overrides: ['seed=1']
Traceback (most recent call last):
File "E:\neural-decoding-RSNN-main\train-tinyRSNN.py", line 260, in
train_all()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self.return_value
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "E:\neural-decoding-RSNN-main\train-tinyRSNN.py", line 86, in train_all
model, history = train_validate_model(
File "E:\neural-decoding-RSNN-main\challenge\train.py", line 121, in train_validate_model
fix, ax = plot_activity_snapshot(
File "E:\neural-decoding-RSNN-main\challenge\utils\plotting.py", line 259, in plot_activity_snapshot
fig, ax = plot_activity_CST(
File "E:\neural-decoding-RSNN-main\challenge\utils\plotting.py", line 171, in plot_activity_CST
scores = model.evaluate(data)
File "e:\neural-decoding-rsnn-main\stork\stork\models.py", line 295, in evaluate
for local_X, local_y in self.data_generator(test_dataset, shuffle=False):
File "e:\neural-decoding-rsnn-main\stork\stork\models.py", line 195, in data_generator
return self.data_generator(dataset, shuffle=shuffle)
File "e:\neural-decoding-rsnn-main\stork\stork\generators.py", line 48, in call
return torch.utils.data.DataLoader(
File "C:\Users\guozh\anaconda3\envs\py310\lib\site-packages\torch\utils\data\dataloader.py", line 235, in init
raise ValueError('persistent_workers option needs num_workers > 0')
ValueError: persistent_workers option needs num_workers > 0

This is the output While using the num_workers=0

[VengeRuan]

After ignore the persistent_workers option,make it.