Trouble using custom dataset for finetuning mistral with qlora #1161

PhilipAmadasun · 2024-01-22T00:54:09Z

PhilipAmadasun
Jan 22, 2024

EDIT
OS: Linux (Ubuntu 22.04)
2 GPUs of: NVIDIA Corporation GP100GL [Tesla P100 PCIE 16GB] (rev a1)
Python 3.10.12

Transformers version 4.37.0.dev0
Torch.version.cuda -- 11.7
nvcc --version -- CUDA version 12.2
NVIDIA Driver Version: 525.147.05

I am trying to fine-tune mistral with qlora, but I'm making some mistake with custom dataset formatting and/or setting dataset parameters in my qlora.yml file. I have read the ReadME for the repo but I'm definitely confused about correct setup. I will go over my trials and errors.

My dataset:
train.txt. The contents of this .txt file are the same as the .jsonl file. I actually tried with both file types and got the same error.

First qlora.yaml configuration

base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: json
    type:
      system_prompt: ""
      field_system: system
      field_instruction: input
      field_output: output
      format: "[INST] {instruction} [/INST]"
      no_input_format: "[INST] {instruction} [/INST]"
    data_files: /home/philip/fine-tune-mistral/data/train.txt
dataset_prepared_path: last_run_prepared
val_set_size: 0
output_dir: ./qlora-out

adapter: qlora
lora_model_dir:

sequence_len: 8192
sample_packing: false
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 8 #2
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: false
fp16: true
tf32: false
bfloat16: false
float16: true
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: false

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
chat_template: inst
hub_model_id: uyiosa/d_mistral
eval_sample_packing: False

Running accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml --deepspeed deepspeed/zero3.json
error:

The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `2`
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
/home/philip/.local/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
/home/philip/.local/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
[2024-01-22 21:20:01,414] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-22 21:20:01,414] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-22 21:20:10,968] [INFO] [axolotl.normalize_config:163] [PID:1922267] [RANK:0] GPU memory usage baseline: 0.000GB (+1.279GB misc)
[2024-01-22 21:20:10,971] [INFO] [axolotl.normalize_config:163] [PID:1922268] [RANK:1] GPU memory usage baseline: 0.000GB (+1.166GB misc)
                                 dP            dP   dP 
                                 88            88   88 
      .d8888b. dP.  .dP .d8888b. 88 .d8888b. d8888P 88 
      88'  `88  `8bd8'  88'  `88 88 88'  `88   88   88 
      88.  .88  .d88b.  88.  .88 88 88.  .88   88   88 
      `88888P8 dP'  `dP `88888P' dP `88888P'   dP   dP 
                                                       
                                                       

[2024-01-22 21:20:11,573] [DEBUG] [axolotl.load_tokenizer:210] [PID:1922267] [RANK:0] EOS: 2 / </s>
[2024-01-22 21:20:11,573] [DEBUG] [axolotl.load_tokenizer:211] [PID:1922267] [RANK:0] BOS: 1 / <s>
[2024-01-22 21:20:11,573] [DEBUG] [axolotl.load_tokenizer:212] [PID:1922267] [RANK:0] PAD: 2 / </s>
[2024-01-22 21:20:11,573] [DEBUG] [axolotl.load_tokenizer:213] [PID:1922267] [RANK:0] UNK: 0 / <unk>
[2024-01-22 21:20:11,574] [INFO] [axolotl.load_tokenized_prepared_datasets:157] [PID:1922267] [RANK:0] Loading prepared dataset from disk at last_run_prepared/f38869ae770a296d535926b169de60ca...
[2024-01-22 21:20:11,577] [INFO] [axolotl.load_tokenized_prepared_datasets:159] [PID:1922267] [RANK:0] Prepared dataset loaded from disk...
[2024-01-22 21:20:11,713] [DEBUG] [axolotl.load_tokenizer:210] [PID:1922268] [RANK:1] EOS: 2 / </s>
[2024-01-22 21:20:11,713] [DEBUG] [axolotl.load_tokenizer:211] [PID:1922268] [RANK:1] BOS: 1 / <s>
[2024-01-22 21:20:11,713] [DEBUG] [axolotl.load_tokenizer:212] [PID:1922268] [RANK:1] PAD: 2 / </s>
[2024-01-22 21:20:11,713] [DEBUG] [axolotl.load_tokenizer:213] [PID:1922268] [RANK:1] UNK: 0 / <unk>
[2024-01-22 21:20:13,688] [INFO] [axolotl.load_tokenized_prepared_datasets:157] [PID:1922268] [RANK:1] Loading prepared dataset from disk at last_run_prepared/f38869ae770a296d535926b169de60ca...
[2024-01-22 21:20:13,692] [DEBUG] [axolotl.log:61] [PID:1922267] [RANK:0] total_num_tokens: 2285
[2024-01-22 21:20:13,693] [DEBUG] [axolotl.log:61] [PID:1922267] [RANK:0] `total_supervised_tokens: 1271`
[2024-01-22 21:20:13,693] [DEBUG] [axolotl.log:61] [PID:1922267] [RANK:0] total_num_steps: 1
[2024-01-22 21:20:13,694] [INFO] [axolotl.load_tokenized_prepared_datasets:159] [PID:1922268] [RANK:1] Prepared dataset loaded from disk...
[2024-01-22 21:20:13,702] [DEBUG] [axolotl.train.log:61] [PID:1922267] [RANK:0] loading tokenizer... mistralai/Mistral-7B-v0.1
[2024-01-22 21:20:14,135] [DEBUG] [axolotl.load_tokenizer:210] [PID:1922267] [RANK:0] EOS: 2 / </s>
[2024-01-22 21:20:14,136] [DEBUG] [axolotl.load_tokenizer:211] [PID:1922267] [RANK:0] BOS: 1 / <s>
[2024-01-22 21:20:14,136] [DEBUG] [axolotl.load_tokenizer:212] [PID:1922267] [RANK:0] PAD: 2 / </s>
[2024-01-22 21:20:14,136] [DEBUG] [axolotl.load_tokenizer:213] [PID:1922267] [RANK:0] UNK: 0 / <unk>
[2024-01-22 21:20:14,136] [DEBUG] [axolotl.train.log:61] [PID:1922267] [RANK:0] loading model and peft_config...
[2024-01-22 21:20:14,147] [DEBUG] [axolotl.load_tokenizer:210] [PID:1922268] [RANK:1] EOS: 2 / </s>
[2024-01-22 21:20:14,147] [DEBUG] [axolotl.load_tokenizer:211] [PID:1922268] [RANK:1] BOS: 1 / <s>
[2024-01-22 21:20:14,147] [DEBUG] [axolotl.load_tokenizer:212] [PID:1922268] [RANK:1] PAD: 2 / </s>
[2024-01-22 21:20:14,147] [DEBUG] [axolotl.load_tokenizer:213] [PID:1922268] [RANK:1] UNK: 0 / <unk>
[2024-01-22 21:20:17,260] [ERROR] [axolotl.load_model:571] [PID:1922267] [RANK:0] CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.89 GiB total capacity; 13.99 GiB already allocated; 7.88 MiB free; 14.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.89 GiB total capacity; 13.99 GiB already allocated; 7.88 MiB free; 14.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-01-22 21:20:17,261] [ERROR] [axolotl.load_model:571] [PID:1922268] [RANK:1] CUDA out of memory. Tried to allocate 500.00 MiB (GPU 1; 15.89 GiB total capacity; 13.99 GiB already allocated; 139.88 MiB free; 14.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 1; 15.89 GiB total capacity; 13.99 GiB already allocated; 139.88 MiB free; 14.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 65, in train
    model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 572, in load_model
    raise err
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.89 GiB total capacity; 13.99 GiB already allocated; 7.88 MiB free; 14.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 65, in train
    model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 572, in load_model
    raise err
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 1; 15.89 GiB total capacity; 13.99 GiB already allocated; 139.88 MiB free; 14.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1922268 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1922267) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1014, in launch_command
    multi_gpu_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 672, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
axolotl.cli.train FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-22_21:20:18
  host      : arnold.ailab.internal
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1922267)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Running accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml (I will skip to the traceback)

2024-01-22 21:22:04,760] [INFO] [axolotl.train.log:61] [PID:1922988] [RANK:0] Starting trainer...
  0%|                                                                        | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/philip/axolotl/src/axolotl/core/trainer_builder.py", line 330, in compute_loss
    return super().compute_loss(model, inputs, return_outputs=return_outputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 687, in forward
    return model_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 675, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/peft_model.py", line 1071, in forward
    return self.base_model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 108, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1154, in forward
    outputs = self.model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1029, in forward
    layer_outputs = self._gradient_checkpointing_func(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 251, in checkpoint
    return _checkpoint_without_reentrant(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 432, in _checkpoint_without_reentrant
    output = function(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 754, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 283, in forward
    attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB (GPU 1; 15.89 GiB total capacity; 9.00 GiB already allocated; 4.48 GiB free; 9.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/philip/axolotl/src/axolotl/core/trainer_builder.py", line 330, in compute_loss
    return super().compute_loss(model, inputs, return_outputs=return_outputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 687, in forward
    return model_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 675, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/peft_model.py", line 1071, in forward
    return self.base_model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 108, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1154, in forward
    outputs = self.model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1029, in forward
    layer_outputs = self._gradient_checkpointing_func(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 251, in checkpoint
    return _checkpoint_without_reentrant(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 432, in _checkpoint_without_reentrant
    output = function(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 754, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 283, in forward
    attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB (GPU 0; 15.89 GiB total capacity; 9.00 GiB already allocated; 4.37 GiB free; 9.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  0%|                                                                        | 0/1 [00:01<?, ?it/s]
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1922988) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1014, in launch_command
    multi_gpu_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 672, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
axolotl.cli.train FAILED

Running CUDA_VISIBLE_DEVICES=0 accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml

 0%|                                                | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/philip/axolotl/src/axolotl/core/trainer_builder.py", line 330, in compute_loss
    return super().compute_loss(model, inputs, return_outputs=return_outputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 687, in forward
    return model_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 675, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/peft_model.py", line 1071, in forward
    return self.base_model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 108, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1154, in forward
    outputs = self.model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1029, in forward
    layer_outputs = self._gradient_checkpointing_func(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 251, in checkpoint
    return _checkpoint_without_reentrant(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 432, in _checkpoint_without_reentrant
    output = function(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 754, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 283, in forward
    attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB (GPU 0; 15.89 GiB total capacity; 8.68 GiB already allocated; 4.18 GiB free; 9.87 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  0%|                                                | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1023, in launch_command
    simple_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 643, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'axolotl.cli.train', 'examples/mistral/qlora.yml']' returned non-zero exit status 1.

Second qlora.yaml configuration

sample_packing is set to true, everything else remains the same.

Running accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml --deepspeed deepspeed/zero3.json

Same error as with previous config

Running accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2730, in training_step
    inputs = self._prepare_inputs(inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2683, in _prepare_inputs
    if len(inputs) == 0:
TypeError: object of type 'NoneType' has no len()
  0%|                                                                                                                                                                                | 0/1 [00:00<?, ?it/s][2024-01-22 21:33:40,692] [INFO] [axolotl.utils.samplers.multipack._len_est:178] [PID:1929049] [RANK:0] packing_efficiency_estimate: 0.04 total_num_tokens per device: 1142
[2024-01-22 21:33:40,692] [INFO] [axolotl.utils.samplers.multipack._len_est:178] [PID:1929049] [RANK:0] packing_efficiency_estimate: 0.04 total_num_tokens per device: 1142
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2730, in training_step
    inputs = self._prepare_inputs(inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2683, in _prepare_inputs
    if len(inputs) == 0:
TypeError: object of type 'NoneType' has no len()
  0%|                                                                                                                                                                                | 0/1 [00:00<?, ?it/s]
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1929049) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1014, in launch_command
    multi_gpu_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 672, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Running CUDA_VISIBLE_DEVICES=0 accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/philip/axolotl/src/axolotl/core/trainer_builder.py", line 330, in compute_loss
    return super().compute_loss(model, inputs, return_outputs=return_outputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 687, in forward
    return model_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/utils/operations.py", line 675, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/peft_model.py", line 1071, in forward
    return self.base_model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 108, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1154, in forward
    outputs = self.model(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1029, in forward
    layer_outputs = self._gradient_checkpointing_func(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 251, in checkpoint
    return _checkpoint_without_reentrant(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 432, in _checkpoint_without_reentrant
    output = function(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 754, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 300, in forward
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/functional.py", line 1845, in softmax
    ret = input.softmax(dim, dtype=dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 15.89 GiB total capacity; 9.12 GiB already allocated; 4.81 GiB free; 9.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  0%|                                                                                                                                                                                | 0/1 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1023, in launch_command
    simple_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 643, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'axolotl.cli.train', 'examples/mistral/qlora.yml']' returned non-zero exit status 1.

Third qlora.yaml configuration

sequence_len reduced from 8192 to 2000. sample_packing and eval_sample_packing set to true. Everything else remains the same.

Running accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml --deepspeed deepspeed/zero3.json

[2024-01-22 21:54:59,404] [ERROR] [axolotl.load_model:571] [PID:1940781] [RANK:0] CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.89 GiB total capacity; 13.99 GiB already allocated; 7.88 MiB free; 14.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.89 GiB total capacity; 13.99 GiB already allocated; 7.88 MiB free; 14.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-01-22 21:54:59,405] [ERROR] [axolotl.load_model:571] [PID:1940782] [RANK:1] CUDA out of memory. Tried to allocate 500.00 MiB (GPU 1; 15.89 GiB total capacity; 13.99 GiB already allocated; 139.88 MiB free; 14.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 1; 15.89 GiB total capacity; 13.99 GiB already allocated; 139.88 MiB free; 14.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 65, in train
    model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 572, in load_model
    raise err
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.89 GiB total capacity; 13.99 GiB already allocated; 7.88 MiB free; 14.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 65, in train
    model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 572, in load_model
    raise err
  File "/home/philip/axolotl/src/axolotl/utils/models.py", line 529, in load_model
    model = getattr(transformers, model_type).from_pretrained(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3494, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1083, in __init__
    self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 509, in wrapper
    f(module, *args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
  File "/home/philip/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 246, in wrapped_fn
    tensor.data = tensor.data.to(target_fp_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB (GPU 1; 15.89 GiB total capacity; 13.99 GiB already allocated; 139.88 MiB free; 14.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1940781) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1014, in launch_command
    multi_gpu_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 672, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Running accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml
This time I got some warnings

**[2024-01-22 21:47:52,712] [WARNING] [axolotl.load_tokenized_prepared_datasets:164] [PID:1936853] [RANK:0] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset**
[2024-01-22 21:47:52,712] [INFO] [axolotl.load_tokenized_prepared_datasets:171] [PID:1936853] [RANK:0] No seed provided, using default seed of 42
[2024-01-22 21:47:52,743] [DEBUG] [axolotl.load_tokenizer:210] [PID:1936854] [RANK:1] EOS: 2 / </s>
[2024-01-22 21:47:52,743] [DEBUG] [axolotl.load_tokenizer:211] [PID:1936854] [RANK:1] BOS: 1 / <s>
[2024-01-22 21:47:52,743] [DEBUG] [axolotl.load_tokenizer:212] [PID:1936854] [RANK:1] PAD: 2 / </s>
[2024-01-22 21:47:52,744] [DEBUG] [axolotl.load_tokenizer:213] [PID:1936854] [RANK:1] UNK: 0 / <unk>
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 85/85 [00:00<00:00, 245196.59it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 215/215 [00:00<00:00, 217662.41it/s]
Map (num_proc=21): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 59.06 examples/s]
[2024-01-22 21:48:04,561] [INFO] [axolotl.load_tokenized_prepared_datasets:380] [PID:1936853] [RANK:0] merging datasets
Map (num_proc=21): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 103.10 examples/s]
[2024-01-22 21:48:05,061] [DEBUG] [axolotl.log:61] [PID:1936853] [RANK:0] max_input_len: 136
Filter (num_proc=21): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 111.68 examples/s]
[2024-01-22 21:48:07,483] [INFO] [axolotl.load_tokenized_prepared_datasets:390] [PID:1936853] [RANK:0] Saving merged prepared dataset to disk... last_run_prepared/dd6984a131a493e0e451e19b224220ba
[2024-01-22 21:48:07,484] [INFO] [axolotl.load_tokenized_prepared_datasets:161] [PID:1936854] [RANK:1] Unable to find prepared dataset in last_run_prepared/dd6984a131a493e0e451e19b224220ba
[2024-01-22 21:48:07,484] [INFO] [axolotl.load_tokenized_prepared_datasets:162] [PID:1936854] [RANK:1] Loading raw datasets...
**[2024-01-22 21:48:07,484] [WARNING] [axolotl.load_tokenized_prepared_datasets:164] [PID:1936854] [RANK:1] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset**
[2024-01-22 21:48:07,484] [INFO] [axolotl.load_tokenized_prepared_datasets:171] [PID:1936854] [RANK:1] No seed provided, using default seed of 42
Saving the dataset (1/1 shards): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████
..................
..........
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2730, in training_step
    inputs = self._prepare_inputs(inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2683, in _prepare_inputs
    if len(inputs) == 0:
TypeError: object of type 'NoneType' has no len()
  0%|                                                                                                                                                                                | 0/1 [00:00<?, ?it/s][2024-01-22 21:48:50,073] [INFO] [axolotl.utils.samplers.multipack._len_est:178] [PID:1936853] [RANK:0] packing_efficiency_estimate: 0.15 total_num_tokens per device: 1142
[2024-01-22 21:48:50,074] [INFO] [axolotl.utils.samplers.multipack._len_est:178] [PID:1936853] [RANK:0] packing_efficiency_estimate: 0.15 total_num_tokens per device: 1142
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/philip/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/philip/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/philip/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2730, in training_step
    inputs = self._prepare_inputs(inputs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2683, in _prepare_inputs
    if len(inputs) == 0:
TypeError: object of type 'NoneType' has no len()
  0%|                                                                                                                                                                                | 0/1 [00:00<?, ?it/s]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1936853 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 1936854) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1014, in launch_command
    multi_gpu_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 672, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/philip/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Running CUDA_VISIBLE_DEVICES=0 accelerate launch -m axolotl.cli.train examples/mistral/qlora.yml (relevant snippet)

.......
File "/home/philip/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/philip/.local/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 300, in forward
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 15.89 GiB total capacity; 13.13 GiB already allocated; 1023.88 MiB free; 14.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  0%|                                                                                                                                                                                | 0/1 [00:07<?, ?it/s]
Traceback (most recent call last):
  File "/home/philip/.local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1023, in launch_command
    simple_launcher(args)
  File "/home/philip/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 643, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'axolotl.cli.train', 'examples/mistral/qlora.yml']' returned non-zero exit status 1.

If I have forgotten to provide more context for my problem please inform me.
Is there something wrong with my system setup or the config set up or dataset or all of the above?

Answered by NanoCode012

Jan 23, 2024

Hey, in the last error, you had a OOM. Could you please try adapting the config? The readme covers about OOM near the bottom.

View full answer

NanoCode012 · 2024-01-23T04:24:24Z

NanoCode012
Jan 23, 2024
Collaborator

Hey, in the last error, you had a OOM. Could you please try adapting the config? The readme covers about OOM near the bottom.

1 reply

PhilipAmadasun Feb 22, 2024
Author

Thanks! Yeah the GPU is just bad? So I followed the ReadMe and did not use the accelerate option. I think that's what worked for me.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trouble using custom dataset for finetuning mistral with qlora #1161

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 1 reply

{{title}}

{{title}}

Select a reply

Trouble using custom dataset for finetuning mistral with qlora #1161

PhilipAmadasun Jan 22, 2024

First qlora.yaml configuration

Second qlora.yaml configuration

Third qlora.yaml configuration

Replies: 1 comment · 1 reply

NanoCode012 Jan 23, 2024 Collaborator

PhilipAmadasun Feb 22, 2024 Author

PhilipAmadasun
Jan 22, 2024

Replies: 1 comment 1 reply

NanoCode012
Jan 23, 2024
Collaborator

PhilipAmadasun Feb 22, 2024
Author