TypeError: tuple indices must be integers or slices, not str

[sebastiz]

Hi many thanks for your useful blog and notebook. I have been following the instructions and when I get to

%%time
! python nlp_finetuning_lightning_google.py \
  --pretrained distilbert-base-uncased \
  --gpus 1 \
  --auto_select_gpus true \
  --nr_frozen_epochs 5 \

I get the error as below. Do you have any suggestions? Thanks

2021-07-08 11:50:31.201778: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
args: Namespace(accumulate_grad_batches=1, amp_backend='native', amp_level='O2', auto_lr_find=False, auto_scale_batch_size=False, auto_select_gpus=True, batch_size=32, benchmark=False, check_val_every_n_epoch=1, checkpoint_callback=True, default_root_dir=None, deterministic=False, distributed_backend=None, early_stop_callback=False, fast_dev_run=False, frac=1, gpus=1, gradient_clip_val=0, learning_rate=2e-05, limit_test_batches=1.0, limit_train_batches=1.0, limit_val_batches=1.0, log_gpu_memory=None, log_save_interval=100, logger=True, max_epochs=1000, max_steps=None, min_epochs=1, min_steps=None, nr_frozen_epochs=5, num_nodes=1, num_processes=1, num_sanity_val_steps=2, overfit_batches=0.0, overfit_pct=None, precision=32, prepare_data_per_node=True, pretrained='distilbert-base-uncased', process_position=0, profiler=None, progress_bar_refresh_rate=1, reload_dataloaders_every_epoch=False, replace_sampler_ddp=True, resume_from_checkpoint=None, row_log_interval=50, sync_batchnorm=False, terminate_on_nan=False, test_percent_check=None, tpu_cores=<function Trainer._gpus_arg_default at 0x7f940221d0e0>, track_grad_norm=-1, train_percent_check=None, training_portion=0.9, truncated_bptt_steps=None, val_check_interval=1.0, val_percent_check=None, weights_save_path=None, weights_summary='top')
kwargs: {}
Loading BERT tokenizer
PRETRAINED:distilbert-base-uncased
Type tokenizer: <class 'transformers.tokenization_distilbert.DistilBertTokenizer'>
Setting up dataset
Number of training sentences: 15,746

14,171 training samples
1,575 validation samples
DistilBertConfig {
  "activation": "gelu",
  "architectures": [
    "DistilBertForMaskedLM"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4"
  },
  "initializer_range": 0.02,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2,
    "LABEL_3": 3,
    "LABEL_4": 4
  },
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "vocab_size": 30522
}

Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Model Type <class 'transformers.modeling_distilbert.DistilBertForSequenceClassification'>
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0]
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: Could not log computational graph since the `model.example_input_array` attribute is not set or `input_array` was not given
  warnings.warn(*args, **kwargs)

  | Name  | Type                                | Params
--------------------------------------------------------------
0 | model | DistilBertForSequenceClassification | 66 M  
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: Your val_dataloader has `shuffle=True`, it is best practice to turn this off for validation and test dataloaders.
  warnings.warn(*args, **kwargs)
Validation sanity check: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "nlp_finetuning_lightning_google.py", line 467, in <module>
    trainer.fit(m, d)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1073, in fit
    results = self.accelerator_backend.train(model)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/gpu_backend.py", line 51, in train
    results = self.trainer.run_pretrain_routine(model)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1224, in run_pretrain_routine
    self._run_sanity_check(ref_model, model)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 1257, in _run_sanity_check
    eval_results = self._evaluate(model, self.val_dataloaders, max_batches, False)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 333, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 687, in evaluation_forward
    output = model.validation_step(*args)
  File "nlp_finetuning_lightning_google.py", line 140, in validation_step
    loss, logits = self(batch)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "nlp_finetuning_lightning_google.py", line 110, in forward
    loss, logits = res['loss'], res['logits']
TypeError: tuple indices must be integers or slices, not str
CPU times: user 167 ms, sys: 37.7 ms, total: 204 ms
Wall time: 22 s