changed tokenizers to tokenizer_options due to naming conflict #8
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Install Then Test Finetuning With prev_run | |
| on: [push, pull_request] | |
| jobs: | |
| Install-And-Test-Quantization: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Check out repository code | |
| uses: actions/checkout@v4 | |
| - run: echo "${{ github.repository }} repository has been cloned to the runner." | |
| - run: echo "Currently on ${{ github.ref }} branch" | |
| - name: ls of directory | |
| run: | | |
| ls ${{ github.workspace }} | |
| - name: Install CPU Dependencies | |
| run: | | |
| python3 -m venv venv | |
| source venv/bin/activate | |
| python3 -m pip install --upgrade pip | |
| python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu | |
| python3 -m pip install numpy transformers datasets tiktoken wandb tqdm tensorboard | |
| python3 -m pip install -r requirements_cpu.txt | |
| python3 -m pip install ml_dtypes | |
| - name: Test finetuning | |
| run: | | |
| source venv/bin/activate | |
| python3 data/shakespeare_char/prepare.py | |
| cd tests | |
| source test_finetuning_cpu.sh | |