Epoch 98

[baloboba]

svc train -t
[00:14:03] INFO     [00:14:03] Server binary (from Python package v0.7.2):       server_ingester.py:290
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-pa                       
                    ckages/tensorboard_data_server/bin/server                                          

NOTE: Using experimental fast data loading logic. To disable, pass
    "--load_fast=false" and report issues on GitHub. More details:
    https://github.com/tensorflow/tensorboard/issues/4784

[00:14:10] INFO     [00:14:10] Using strategy: auto                                         train.py:98
INFO: GPU available: True (cuda), used: True
           INFO     [00:14:10] GPU available: True (cuda), used: True                   rank_zero.py:63
INFO: TPU available: False, using: 0 TPU cores
           INFO     [00:14:10] TPU available: False, using: 0 TPU cores                 rank_zero.py:63
INFO: HPU available: False, using: 0 HPUs
           INFO     [00:14:10] HPU available: False, using: 0 HPUs                      rank_zero.py:63
           WARNING  [00:14:10]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    so_vits_svc_fork/modules/synthesizers.py:81: UserWarning: Unused                   
                    arguments: {'n_layers_q': 3, 'use_spectral_norm': False,                           
                    'pretrained': {'D_0.pth':                                                          
                    'https://huggingface.co/datasets/ms903/sovits4.0-768vec-layer12/res                
                    olve/main/sovits_768l12_pre_large_320k/clean_D_320000.pth',                        
                    'G_0.pth':                                                                         
                    'https://huggingface.co/datasets/ms903/sovits4.0-768vec-layer12/res                
                    olve/main/sovits_768l12_pre_large_320k/clean_G_320000.pth'}}                       
                      warnings.warn(f"Unused arguments: {kwargs}")                                     
                                                                                                       
           INFO     [00:14:10] Decoder type: hifi-gan                               synthesizers.py:100
           WARNING  [00:14:10]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    torch/nn/utils/weight_norm.py:143: FutureWarning:                                  
                    `torch.nn.utils.weight_norm` is deprecated in favor of                             
                    `torch.nn.utils.parametrizations.weight_norm`.                                     
                      WeightNorm.apply(module, name, dim)                                              
                                                                                                       
[00:14:12] WARNING  [00:14:12]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    so_vits_svc_fork/utils.py:246: UserWarning: Keys not found in                      
                    checkpoint state dict:['emb_g.weight']                                             
                      warnings.warn(f"Keys not found in checkpoint state dict:"                        
                    f"{not_in_from}")                                                                  
                                                                                                       
           WARNING  [00:14:12]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    so_vits_svc_fork/utils.py:264: UserWarning: Shape mismatch:                        
                    ['dec.cond.weight: torch.Size([512, 256, 1]) -> torch.Size([512,                   
                    768, 1])', 'enc_q.enc.cond_layer.weight_v: torch.Size([6144, 256,                  
                    1]) -> torch.Size([6144, 768, 1])',                                                
                    'flow.flows.0.enc.cond_layer.weight_v: torch.Size([1536, 256, 1])                  
                    -> torch.Size([1536, 768, 1])',                                                    
                    'flow.flows.2.enc.cond_layer.weight_v: torch.Size([1536, 256, 1])                  
                    -> torch.Size([1536, 768, 1])',                                                    
                    'flow.flows.4.enc.cond_layer.weight_v: torch.Size([1536, 256, 1])                  
                    -> torch.Size([1536, 768, 1])',                                                    
                    'flow.flows.6.enc.cond_layer.weight_v: torch.Size([1536, 256, 1])                  
                    -> torch.Size([1536, 768, 1])', 'f0_decoder.cond.weight:                           
                    torch.Size([192, 256, 1]) -> torch.Size([192, 768, 1])']                           
                      warnings.warn(                                                                   
                                                                                                       
           INFO     [00:14:12] Loaded checkpoint 'logs/44k/G_0.pth' (epoch 0)              utils.py:307
           INFO     [00:14:12] Loaded checkpoint 'logs/44k/D_0.pth' (epoch 0)              utils.py:307
INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
           INFO     [00:14:12] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]                     cuda.py:61
┏━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃   ┃ Name  ┃ Type                     ┃ Params ┃ Mode  ┃
┡━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ 0 │ net_g │ SynthesizerTrn           │ 45.6 M │ train │
│ 1 │ net_d │ MultiPeriodDiscriminator │ 46.7 M │ train │
└───┴───────┴──────────────────────────┴────────┴───────┘
Trainable params: 92.4 M                                                                               
Non-trainable params: 0                                                                                
Total params: 92.4 M                                                                                   
Total estimated model params size (MB): 369                                                            
Modules in train mode: 486                                                                             
Modules in eval mode: 0                                                                                
           WARNING  [00:14:12]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    lightning/pytorch/trainer/connectors/data_connector.py:425: The                    
                    'val_dataloader' does not have many workers which may be a                         
                    bottleneck. Consider increasing the value of the `num_workers`                     
                    argument` to `num_workers=7` in the `DataLoader` to improve                        
                    performance.                                                                       
                                                                                                       
           WARNING  [00:14:12]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    lightning/pytorch/loops/fit_loop.py:310: The number of training                    
                    batches (1) is smaller than the logging interval                                   
                    Trainer(log_every_n_steps=50). Set a lower value for                               
                    log_every_n_steps if you want to see logs for the training epoch.                  
                                                                                                       
           INFO     [00:14:12] Setting current epoch to 0                                  train.py:311
           INFO     [00:14:12] Setting total batch idx to 0                                train.py:327
           INFO     [00:14:12] Setting global step to 0                                    train.py:317
[00:14:14] WARNING  [00:14:14]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    torch/functional.py:704: UserWarning: stft with                                    
                    return_complex=False is deprecated. In a future pytorch release,                   
                    stft will return complex tensors for all inputs, and                               
                    return_complex=False will raise an error.                                          
                    Note: you can still call torch.view_as_real on the complex output                  
                    to recover the old return format. (Triggered internally at                         
                    ../aten/src/ATen/native/SpectralOps.cpp:873.)                                      
                      return _VF.stft(  # type: ignore[attr-defined]                                   
                                                                                                       
[00:14:15] WARNING  [00:14:15]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    so_vits_svc_fork/train.py:447: FutureWarning:                                      
                    `torch.cuda.amp.autocast(args...)` is deprecated. Please use                       
                    `torch.amp.autocast('cuda', args...)` instead.                                     
                      with autocast(enabled=False):                                                    
                                                                                                       
[00:14:18] WARNING  [00:14:18]                                                          warnings.py:109
                    /home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/                
                    so_vits_svc_fork/train.py:523: FutureWarning:                                      
                    `torch.cuda.amp.autocast(args...)` is deprecated. Please use                       
                    `torch.amp.autocast('cuda', args...)` instead.                                     
                      with autocast(enabled=False):                                                    
                                                                                                       
Epoch 98/189999 ━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/1 0:00:00 • 0:00:00 0.00it/s v_num: 0.000 loss/g/total:  
                                                                           31.428 loss/g/fm: 6.895     
                                                                           loss/g/mel: 21.074          
                                                                           loss/g/kl: 0.897 loss/g/lf0:
                                                                           0.000 loss/d/total: 2.785   
Traceback (most recent call last):
  File "/home/zeus/miniconda3/envs/cloudspace/bin/svc", line 8, in <module>
    sys.exit(cli())
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/so_vits_svc_fork/__main__.py", line 128, in train
    train(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/so_vits_svc_fork/train.py", line 149, in train
    trainer.fit(model, datamodule=datamodule)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 539, in fit
    call._call_and_handle_interrupt(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 575, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 982, in _run
    results = self._run_stage()
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1026, in _run_stage
    self.fit_loop.run()
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 216, in run
    self.advance()
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 455, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 150, in run
    self.advance(data_fetcher)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 322, in advance
    batch_output = self.manual_optimization.run(kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/manual.py", line 94, in run
    self.advance(kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/manual.py", line 114, in advance
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 323, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 391, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/so_vits_svc_fork/train.py", line 483, in training_step
    "slice/mel_org": utils.plot_spectrogram_to_numpy(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/so_vits_svc_fork/utils.py", line 403, in plot_spectrogram_to_numpy
    data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep="")
AttributeError: 'FigureCanvasAgg' object has no attribute 'tostring_rgb'. Did you mean: 'tostring_argb'?
⚡ ~ 

[dani007200964]

Hi!

I had the same issue on Colab, the following fied it for me:

!pip install matplotlib==3.7.0