Error while training PaddleOCR on IAM dataset

[praciouss]

Hi,

I am trying to train PaddleOCR on IAM dataset for handwriting text recognition project that I am developing, I have created a labels.txt file as well for all the images in the dataset but when I am running the command: !python3 tools/train.py -c configs/rec/rec_icdar15_train.yml, the error in attached screenshot is being encountered.

Would be helpful if anyone can help me resolve it. Happy to provide further details as well.

Thank you.

[GreatV]

The error you are encountering while training PaddleOCR on the IAM dataset may be related to a mismatch in the data batch sizes or labels. Based on a similar issue discussed in this discussion, here are some steps you can take to diagnose and fix the problem:

Steps to Resolve:

1. Check the Label File (labels.txt)

   • Ensure that every image has a corresponding label entry.
   • The format should be correct, with image filenames properly mapped to their text.

2. Validate Data Loading

   • Run a small test to verify that images and labels are loaded correctly.
   • Print a few samples using:

     import os
     for image, label in dataset[:10]:
         print(image.shape, label)

   • This helps in detecting any inconsistencies in labels.

3. Batch Size and Data Shape Matching

   • Based on the discussion, the error could be due to a mismatch in batch size and expected shape.
   • Try adjusting batch_size_per_card in yml config:

     loader:
       batch_size_per_card: 4  # Try reducing it to check if error persists

4. Check resize Transformation

   • Ensure all images are properly resized during preprocessing:

     - Resize:
         image_shape: [3, 640, 640]

   • If your dataset has images of varying sizes, apply explicit resizing.

5. Debugging Shape Mismatch

   • Modify the training script to check batch shapes before feeding them into the model:

     for batch in data_loader:
         images, labels = batch
         print(f"Images shape: {images.shape}, Labels shape: {labels.shape}")

   • If there's a mismatch in the number of labels compared to images, it indicates issues in dataset preparation.

Further Debugging:

• Run the training command with logging enabled to find where the failure occurs:

  python3 -m paddle.distributed.launch --log_dir=log --gpus '0' tools/train.py -c configs/rec/rec_icdar15_train.yml

• A detailed log will help you pinpoint the issue.

If the error persists, you may consider attaching the exact error traceback from the terminal rather than a screenshot for better debugging.

Let me know if you need further clarifications! 🚀

---

Response generated by 🤖 feifei-bot | chatgpt-4o-latest

⚠️ This is an AI-generated response. Please verify any critical information.

[MohMehrnia]

i fixed but show me another error
`[2025/03/23 00:20:48] ppocr WARNING: Skipping import of the encryption module.
[2025/03/23 00:20:48] ppocr INFO: Architecture :
[2025/03/23 00:20:48] ppocr INFO: Backbone :
[2025/03/23 00:20:48] ppocr INFO: disable_se : True
[2025/03/23 00:20:48] ppocr INFO: model_name : large
[2025/03/23 00:20:48] ppocr INFO: name : MobileNetV3
[2025/03/23 00:20:48] ppocr INFO: scale : 0.5
[2025/03/23 00:20:48] ppocr INFO: Head :
[2025/03/23 00:20:48] ppocr INFO: fc_decay : 1e-05
[2025/03/23 00:20:48] ppocr INFO: name : CTCHead
[2025/03/23 00:20:48] ppocr INFO: Neck :
[2025/03/23 00:20:48] ppocr INFO: encoder_type : rnn
[2025/03/23 00:20:48] ppocr INFO: hidden_size : 96
[2025/03/23 00:20:48] ppocr INFO: name : SequenceEncoder
[2025/03/23 00:20:48] ppocr INFO: algorithm : CRNN
[2025/03/23 00:20:48] ppocr INFO: model_type : rec
[2025/03/23 00:20:48] ppocr INFO: transforms :
[2025/03/23 00:20:48] ppocr INFO: DecodeImage :
[2025/03/23 00:20:48] ppocr INFO: channel_first : False
[2025/03/23 00:20:48] ppocr INFO: img_mode : RGB
[2025/03/23 00:20:48] ppocr INFO: RecResizeImg :
[2025/03/23 00:20:48] ppocr INFO: image_shape : [3, 640, 640]
[2025/03/23 00:20:48] ppocr INFO: KeepKeys :
[2025/03/23 00:20:48] ppocr INFO: keep_keys : ['image', 'label']
[2025/03/23 00:20:48] ppocr INFO: Eval :
[2025/03/23 00:20:48] ppocr INFO: dataset :
[2025/03/23 00:20:48] ppocr INFO: data_dir : /home/mohammad/MySourceCodes/python/OCRTrain/dataset/val
[2025/03/23 00:20:48] ppocr INFO: label_file_list : ['/home/mohammad/MySourceCodes/python/OCRTrain/dataset/val/label.txt']
[2025/03/23 00:20:48] ppocr INFO: name : SimpleDataSet
[2025/03/23 00:20:48] ppocr INFO: transforms :
[2025/03/23 00:20:48] ppocr INFO: DecodeImage :
[2025/03/23 00:20:48] ppocr INFO: channel_first : False
[2025/03/23 00:20:48] ppocr INFO: img_mode : RGB
[2025/03/23 00:20:48] ppocr INFO: RecResizeImg :
[2025/03/23 00:20:48] ppocr INFO: image_shape : [3, 640, 640]
[2025/03/23 00:20:48] ppocr INFO: KeepKeys :
[2025/03/23 00:20:48] ppocr INFO: keep_keys : ['image', 'label']
[2025/03/23 00:20:48] ppocr INFO: loader :
[2025/03/23 00:20:48] ppocr INFO: batch_size_per_card : 4
[2025/03/23 00:20:48] ppocr INFO: drop_last : False
[2025/03/23 00:20:48] ppocr INFO: num_workers : 4
[2025/03/23 00:20:48] ppocr INFO: shuffle : False
[2025/03/23 00:20:48] ppocr INFO: Global :
[2025/03/23 00:20:48] ppocr INFO: cal_metric_during_train : False
[2025/03/23 00:20:48] ppocr INFO: character_dict_path : /home/mohammad/MySourceCodes/python/OCRTrain/persian_dict.txt
[2025/03/23 00:20:48] ppocr INFO: character_type : ch
[2025/03/23 00:20:48] ppocr INFO: distort : True
[2025/03/23 00:20:48] ppocr INFO: distributed : False
[2025/03/23 00:20:48] ppocr INFO: epoch_num : 100
[2025/03/23 00:20:48] ppocr INFO: eval_batch_step : [0, 2000]
[2025/03/23 00:20:48] ppocr INFO: image_shape : [3, 640, 640]
[2025/03/23 00:20:48] ppocr INFO: label_list : ['train', 'val']
[2025/03/23 00:20:48] ppocr INFO: log_smooth_window : 20
[2025/03/23 00:20:48] ppocr INFO: loss_type : ctc
[2025/03/23 00:20:48] ppocr INFO: max_text_length : 25
[2025/03/23 00:20:48] ppocr INFO: print_batch_step : 10
[2025/03/23 00:20:48] ppocr INFO: save_epoch_step : 3
[2025/03/23 00:20:48] ppocr INFO: save_inference_dir : ./inference/
[2025/03/23 00:20:48] ppocr INFO: save_model_dir : ./output/
[2025/03/23 00:20:48] ppocr INFO: use_gpu : True
[2025/03/23 00:20:48] ppocr INFO: use_parallel : False
[2025/03/23 00:20:48] ppocr INFO: use_visualdl : False
[2025/03/23 00:20:48] ppocr INFO: Loss :
[2025/03/23 00:20:48] ppocr INFO: name : CTCLoss
[2025/03/23 00:20:48] ppocr INFO: Metric :
[2025/03/23 00:20:48] ppocr INFO: main_indicator : acc
[2025/03/23 00:20:48] ppocr INFO: name : RecMetric
[2025/03/23 00:20:48] ppocr INFO: Optimizer :
[2025/03/23 00:20:48] ppocr INFO: lr :
[2025/03/23 00:20:48] ppocr INFO: eta_min : 1e-05
[2025/03/23 00:20:48] ppocr INFO: learning_rate : 0.001
[2025/03/23 00:20:48] ppocr INFO: name : Cosine
[2025/03/23 00:20:48] ppocr INFO: warmup_epoch : 5
[2025/03/23 00:20:48] ppocr INFO: name : Adam
[2025/03/23 00:20:48] ppocr INFO: regularizer :
[2025/03/23 00:20:48] ppocr INFO: factor : 1e-05
[2025/03/23 00:20:48] ppocr INFO: name : L2
[2025/03/23 00:20:48] ppocr INFO: PostProcess :
[2025/03/23 00:20:48] ppocr INFO: name : CTCLabelDecode
[2025/03/23 00:20:48] ppocr INFO: Train :
[2025/03/23 00:20:48] ppocr INFO: dataset :
[2025/03/23 00:20:48] ppocr INFO: data_dir : /home/mohammad/MySourceCodes/python/OCRTrain/dataset/train
[2025/03/23 00:20:48] ppocr INFO: label_file_list : ['/home/mohammad/MySourceCodes/python/OCRTrain/dataset/train/label.txt']
[2025/03/23 00:20:48] ppocr INFO: name : SimpleDataSet
[2025/03/23 00:20:48] ppocr INFO: transforms :
[2025/03/23 00:20:48] ppocr INFO: DecodeImage :
[2025/03/23 00:20:48] ppocr INFO: channel_first : False
[2025/03/23 00:20:48] ppocr INFO: img_mode : RGB
[2025/03/23 00:20:48] ppocr INFO: RecResizeImg :
[2025/03/23 00:20:48] ppocr INFO: image_shape : [3, 640, 640]
[2025/03/23 00:20:48] ppocr INFO: KeepKeys :
[2025/03/23 00:20:48] ppocr INFO: keep_keys : ['image', 'label']
[2025/03/23 00:20:48] ppocr INFO: loader :
[2025/03/23 00:20:48] ppocr INFO: batch_size_per_card : 4
[2025/03/23 00:20:48] ppocr INFO: drop_last : True
[2025/03/23 00:20:48] ppocr INFO: num_workers : 4
[2025/03/23 00:20:48] ppocr INFO: shuffle : True
[2025/03/23 00:20:48] ppocr INFO: profiler_options : None
[2025/03/23 00:20:48] ppocr INFO: train with paddle 2.6.2 and device Place(gpu:0)
[2025/03/23 00:20:48] ppocr INFO: Initialize indexes of datasets:['/home/mohammad/MySourceCodes/python/OCRTrain/dataset/train/label.txt']
[2025/03/23 00:20:48] ppocr INFO: Initialize indexes of datasets:['/home/mohammad/MySourceCodes/python/OCRTrain/dataset/val/label.txt']
W0323 00:20:48.181823 7981 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.6, Runtime API Version: 11.8
W0323 00:20:48.182442 7981 gpu_resources.cc:164] device: 0, cuDNN Version: 9.5.
[2025/03/23 00:20:48] ppocr INFO: train dataloader has 2000 iters
[2025/03/23 00:20:48] ppocr INFO: valid dataloader has 500 iters
[2025/03/23 00:20:48] ppocr INFO: train from scratch
[2025/03/23 00:20:48] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
Traceback (most recent call last):
File "/home/mohammad/MySourceCodes/python/PaddleOCR/tools/train.py", line 272, in
main(config, device, logger, vdl_writer, seed)
File "/home/mohammad/MySourceCodes/python/PaddleOCR/tools/train.py", line 225, in main
program.train(
File "/home/mohammad/MySourceCodes/python/PaddleOCR/tools/program.py", line 366, in train
preds = model(images)
^^^^^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mohammad/MySourceCodes/python/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 85, in forward
x = self.backbone(x)
^^^^^^^^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mohammad/MySourceCodes/python/PaddleOCR/ppocr/modeling/backbones/rec_mobilenet_v3.py", line 153, in forward
x = self.blocks(x)
^^^^^^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/layer/container.py", line 614, in forward
input = layer(input)
^^^^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mohammad/MySourceCodes/python/PaddleOCR/ppocr/modeling/backbones/det_mobilenet_v3.py", line 251, in forward
x = self.expand_conv(inputs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1429, in call
return self.forward(*inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mohammad/MySourceCodes/python/PaddleOCR/ppocr/modeling/backbones/det_mobilenet_v3.py", line 191, in forward
x = F.relu(x)
^^^^^^^^^
File "/home/mohammad/anaconda3/envs/paddleocr/lib/python3.12/site-packages/paddle/nn/functional/activation.py", line 765, in relu
return _C_ops.relu(x)
^^^^^^^^^^^^^^
MemoryError:

---

C++ Traceback (most recent call last):

0 paddle::pybind::eager_api_relu(_object*, _object*, _object*)
1 relu_ad_func(paddle::Tensor const&)
2 paddle::experimental::relu(paddle::Tensor const&)
3 void phi::ReluKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor*)
4 void phi::ActivationGPUImpl<float, phi::GPUContext, phi::funcs::CudaReluFunctor >(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor*, phi::funcs::CudaReluFunctor const&)
5 float* phi::DeviceContext::Alloc(phi::TensorBase*, unsigned long, bool) const
6 phi::DeviceContext::Impl::Alloc(phi::TensorBase*, phi::Place const&, phi::DataType, unsigned long, bool, bool) const
7 phi::DenseTensor::AllocateFrom(phi::Allocator*, phi::DataType, unsigned long, bool)
8 paddle::memory::allocation::Allocator::Allocate(unsigned long)
9 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long)
10 paddle::memory::allocation::Allocator::Allocate(unsigned long)
11 paddle::memory::allocation::Allocator::Allocate(unsigned long)
12 paddle::memory::allocation::Allocator::Allocate(unsigned long)
13 paddle::memory::allocation::Allocator::Allocate(unsigned long)
14 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long)
15 std::string phi::enforce::GetCompleteTraceBackString<std::string >(std::string&&, char const*, int)
16 phi::enforce::GetCurrentTraceBackStringabi:cxx11

---

Error Message Summary:

ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 50.000000MB memory on GPU 0, 3.622803GB memory has been allocated and available memory is only 8.937500MB.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.
   (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:86)

`

[praciouss]

Hi,

Apologies for the delay in getting back but I tried the solutions that you have mentioned, reduced the batch size as well and running it on CPU but still I am facing the same error.

Basically I'm trying to train the IAM dataset and the labels are being loaded correctly with the corresponding images as shown below.

Thats the piece of code I am using to extract image size and labels -

import os
from PIL import Image

data_dir = '/Users/IAM/resized'
label_file = '/Users/IAM/resized/train.txt'

with open(label_file, 'r') as f:
lines = f.readlines()

for i in range(10):
line = lines[i].strip()

file_name, label = line.split(maxsplit=1)  

img_path = os.path.join(data_dir, file_name)

img = Image.open(img_path)

print(f"Image shape: {img.size}, Label: {label}")

Output:
Image shape: (800, 1143), Label: The Minister regards the recent trend of recruiting figures as encouraging . He is confident the aim can be achieved . While the possibility of an eventual return to National Service in some form cannot be entirely discounted , he does not agree that the point has been reached where this need even be considered . If National Service is ever resorted to , engagement would probably have to be for three years .

Still the error that I'm encountering is:

I0327 16:52:25.405913 3941876288 kernel_dispatch.h:102] Get BackendSet from tensor
I0327 16:52:25.412482 3941876288 kernel_dispatch.h:102] Get BackendSet from tensor
[2025/03/27 16:52:25] ppocr INFO: train dataloader has 12 iters
[2025/03/27 16:52:25] ppocr INFO: train from scratch
[2025/03/27 16:52:25] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations
Exception in thread Thread-1 (_thread_loop):
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/paddle/io/dataloader/dataloader_iter.py", line 235, in _thread_loop
batch = self._dataset_fetcher.fetch(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/paddle/io/dataloader/fetcher.py", line 85, in fetch
data = self.collate_fn(data)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/paddle/io/dataloader/collate.py", line 66, in default_collate_fn
return {
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/paddle/io/dataloader/collate.py", line 67, in
key: default_collate_fn([d[key] for d in batch]) for key in sample
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/paddle/io/dataloader/collate.py", line 77, in default_collate_fn
raise TypeError(
TypeError: batch data con only contains: tensor, numpy.ndarray, dict, list, number, but got <class 'NoneType'>

Thanks for the help in advance!