-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
M3 Expansion #51
Merged
Merged
M3 Expansion #51
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,15 +1,45 @@ | ||
# Challenges | ||
# M3 Challenges | ||
|
||
## Challenge 1 | ||
|
||
Something simple to start off. Create a bash script called `hello.sh` that prints "Hello World" to the screen. Submit this job to the queue using `sbatch`. Check the status of the job using `squeue`. Once the job has finished, check the output using `cat`. You can find the output file in the directory you submitted the job from. | ||
Navigate to your scratch directory and, using vim (or your chosen in-terminal editor) create a file called `hello.txt` that contains the text "Hello World". Once you have created the file, use the `cat` command to print the contents of the file to the screen. | ||
|
||
## Challenge 2 | ||
|
||
Something a bit more involved. Clone your [challenges repository](https://github.com/MonashDeepNeuron/HPC-Training-Challenges.git) into your personal folder in the scratch directory. Then, in this same directory, create a submission script that will install python 3.10 using miniconda, create a virtual environment, install the necessary dependencies, and clone and run the `alexnet_stl10.py` script in the M3 section. Remember, don't directly load python using module, follow the instructions in the [software tooling](./software-tooling.md#python) chapter. | ||
Once completed, commit and push your changes as well as the output. | ||
Write a bash script that prints the contents of the above hello.txt file to the screen and run it locally (on your login node). | ||
|
||
## Challenge 3 | ||
|
||
A continuation of challenge 2. Edit your submission script so that you get a gpu node, and run the script using the gpu. | ||
Commit and push your changes as well as the output. | ||
Submit the above script to the queue by writing another SLURM bash script. Check the status of the job using `squeue`. Once the job has finished, check the output using `cat`. You can find the output file in the directory you submitted the job from. | ||
|
||
## Challenge 4 | ||
|
||
Request an interactive node and attach to it. Once you have done this, install python 3.7 using conda. | ||
|
||
## Challenge 5 | ||
|
||
Clone and run [this](./dl_on_m3/alexnet_stl10.py) script. You will need to first install the dependencies for it. You don't need to wait for it to finish, just make sure it is working. You will know its working if it starts listing out the loss and accuracy for each epoch. You can stop it by pressing `ctrl + c`. | ||
|
||
Once you have confirmed that it is working, deactivate and delete the conda environment, and then end the interactive session. | ||
|
||
> Hint: I have included the dependencies and their versions (make sure you install the right version) in the `requirements.txt` file. You will need python 3.7 to run this script. | ||
|
||
## Challenge 6 | ||
|
||
Go back to the login node. Now you are going to put it all together. Write a bash script that does the following: | ||
|
||
- (1) requests a compute node | ||
- (2) installs python using conda | ||
- (3) clones and runs the above script | ||
|
||
Let this run fully. Check the output of the script to make sure it ran correctly. Does it match the output of the script you ran in challenge 5? | ||
> Hint: You can check the output of the script at any time by `cat`ing the output file. The script does not need to have finished running for you to do this. | ||
|
||
## Challenge 7 | ||
|
||
Edit your submission script so that you get a gpu node, and run the script using the gpu. | ||
> Hint: Use the m3h partition | ||
|
||
## Challenge 8 | ||
|
||
Now you want to clean up your working directory. First, push your solutions to your challenges repo. Then, delete the challenges directory, as well as the conda environment you created in challenge 6. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,3 +19,30 @@ To request to join the MDN project, again from the Home page click on Join Exiti | |
In the text box type `vf38` and click search. This is the project code for MDN. Then select the project and click submit. You will now have to wait for the project admins to approve your request. Once they have done this, you will be able to access the project. This should not take longer than a few days, and you will get an email telling you when you have access. | ||
|
||
Once you have access to everything, you are ready to get started with M3. Good job!! | ||
|
||
## Git SSH setup | ||
|
||
In order to reliably clone git repos in M3, in particular private ones, it is best practice to use SSH cloning. This is a bit more complicated to set up, but once it is done, it is much more streamlined. There are few steps involved. First, you will need to generate an SSH key on M3. Login to M3, and run the following command: | ||
|
||
```bash | ||
ssh-keygen -t ed25519 -C "[email protected]" | ||
``` | ||
|
||
This will then prompt you to enter a file location. Just press enter to use the default location. It will then ask you to enter a passphrase. This is optional, but recommended. | ||
|
||
Once you have generated your key, you need to add it to the ssh agent. Do this by running: | ||
|
||
```bash | ||
eval "$(ssh-agent -s)" | ||
ssh-add ~/.ssh/id_ed25519 | ||
``` | ||
|
||
You will then need to copy the public key to your clipboard. You can do this by running: | ||
|
||
```bash | ||
cat ~/.ssh/id_ed25519.pub | ||
``` | ||
|
||
Then, go to your github account, go to settings, and click on the SSH and GPG keys tab. Click on New SSH key, and paste the key into the box. Give it a name, and click Add SSH key. | ||
|
||
You should now be able to clone repos using SSH. To do this, go to the repo you want to clone, but instead of copying the HTTP link, copy the SSH link, and then its regular git cloning. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# Strudel | ||
|
||
STRUDEL is a web application used to connect to M3. There are two main benefits to this over regular ssh. Firstly, you are able to access a desktop session, so you can interact easier with M3, look at graphs, etc.. STRUDEL also enables the use of Jupyter notebooks, which are especially useful for data science and machine learning. | ||
|
||
## Accessing STRUDEL | ||
|
||
First, go to the [STRUDEL](https://beta.desktop.cvl.org.au/) website. You should see something like this: | ||
|
||
![strudel select cvl](imgs/strudel1.png) | ||
|
||
Select the CVL option, and you should be taken to another page, where you choose how to log in. | ||
|
||
![strudel login](imgs/strudel2.png) | ||
|
||
Select AAF. On the next page, search for and select Monash University. | ||
|
||
![AAF Login](imgs/aaf_strudel.png) | ||
|
||
You will now be taken to the Monash login page. Once you have logged in, it will show one last page, asking permission to use your details. Click allow, and you will be taken to the STRUDEL home page. | ||
|
||
![strudel home page](imgs/strudel_home.png) | ||
|
||
## Desktop Session | ||
|
||
To start a desktop session using STRUDEL, click on the **Desktop** tab on the side, select your desired options, and click launch. Once the session has started, you will be able to attach to it by clicking on the connect button in the *Pending / Running Desktops* section. | ||
|
||
## Jupyter Notebooks | ||
|
||
Similar to Desktops, if you want a basic Jupyter notebook, click on the **Jupyter Lab** tab, choose how much compute you want, and click launch. | ||
|
||
If you want to have a more customised Jupyter notebook, you can do this by first sshing into M3, and activate conda. Then activate the conda environment `jupyterlab`. Install you desired packages in this environment. Once you have done this, go back to STRUDEL, and launch a **Jupyter Lab - BYO** session. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add temp files like
.DS_Store
to the .gitignore so they don't get tracked as they are a per system meta file.