Skip to content

Commit

Permalink
fix: minor fixes in config lorem ipsum
Browse files Browse the repository at this point in the history
  • Loading branch information
le1nux committed Mar 13, 2024
1 parent 662e68f commit cb6e816
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions config_files/config_lorem_ipsum.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ model:
variant_key: gpt2
config:
sample_key: ${settings.referencing_keys.sample_key}
poe_type: "NOPE"
poe_type: NOPE
block_size: ${settings.training.sequence_length}
prediction_key: ${loss_fn.config.prediction_key}
vocab_size: 50304 # GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency
Expand All @@ -197,7 +197,7 @@ model:
n_embd: 128
dropout: 0.0
bias: true # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster
attention:
attention_config:
attention_type: pytorch_flash_attention
scaling_factor: 3
qkv_transforms:
Expand Down

0 comments on commit cb6e816

Please sign in to comment.