NotImplementedError

[xiang-xiang-zhu]

When I run the program directly, the transformers library will report an error

Traceback (most recent call last):
  File "main.py", line 620, in <module>
    main()
  File "main.py", line 576, in main
    tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path, do_lower_case=False)
  File "/home/hx/anaconda3/envs/torch_cpu/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1787, in from_pretrained
    **kwargs,
  File "/home/hx/anaconda3/envs/torch_cpu/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1915, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/hx/github项目/OTTers/scripts/tokenization_gpt2.py", line 119, in __init__
    self.max_len_single_sentence = self.max_len # no default special tokens - you can update this value if you add special tokens
AttributeError: 'GPT2Tokenizer' object has no attribute 'max_len'

After I checked on the Internet, I changed max_len to model_max_length. I don't know if this is the reason for the version. At least max_len no longer exists in the latest version.

self.max_len_single_sentence = self.max_len
self.max_len_single_sentence = self.model_max_length

But after the change, there will be a new error

Traceback (most recent call last):
  File "main.py", line 620, in <module>
    main()
  File "main.py", line 588, in main
    model.resize_token_embeddings(len(tokenizer))
  File "/home/hx/anaconda3/envs/torch_cpu/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1015, in resize_token_embeddings
    model_embeds = self._resize_token_embeddings(new_num_tokens)
  File "/home/hx/anaconda3/envs/torch_cpu/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1029, in _resize_token_embeddings
    old_embeddings = self.get_input_embeddings()
  File "/home/hx/anaconda3/envs/torch_cpu/lib/python3.7/site-packages/transformers/modeling_utils.py", line 855, in get_input_embeddings
    return base_model.get_input_embeddings()
  File "/home/hx/anaconda3/envs/torch_cpu/lib/python3.7/site-packages/transformers/modeling_utils.py", line 857, in get_input_embeddings
    raise NotImplementedError
NotImplementedError