- Azure subscription. This can be a free trial subscription, MSDN, or the one you use for other work.
- Azure Python SDK and azure-mgmt-batchai, if you like to run recipes using Python Jupyter notebook. See How to install Azure SDK. Install Batch AI management client using the following command:
pip install azure-mgmt-batchai- Azure CLI 2.0, if you like to run recipes using Azure CLI - See Install Azure CLI 2.0 for instructions.
- Azure Storage Account in East US (required for all recipes). See How to create Azure storage accounts
- Register with Microsoft.BatchAI and Microsoft.Batch providers using Azure CLI 2.0 (you can use Cloud Shell):
az provider register -n Microsoft.BatchAI
az provider register -n Microsoft.BatchNote, a provider registration can take up to 15 minutes.
To start, please Clone or download this repo
Use the following links for a quick navigation:
- Rename configuration.json.template to configuration.json.
- Populate it with your Batch AI credentials. Please see this page on how to get your Azure AD credentials.
- Leave the "base_url" filed as empty.
- Our recipe will automatically create resource group if not exist. You need to specify the name of your resource group to create.
- Specify your Azure Storage account name and key, Please see this page.
- Batch AI creates administrator user account on every compute node and enables ssh. You need to specify user name and at least a password or ssh public key for this account.
For your convenience, we provide a collection of helper functions in utilities.py used for each recipes:
- Read parameters from configuration file
- Create python client object (BatchAIManagementClient) to access Azure Batch AI service
- Create/Update resource group
- Download file with given shared access signature (SAS)
- Print Job/Cluster status
- File Streaming
Please install Jupyter Notebook from https://jupyter.org/ or run
python -m pip install jupyter- Route into the root your cloned recipe directory
cd <your clone root>/BatchAI/recipes- Launch the Jupyter Notebook by
jupyter notebook- In the prompted brower brower, navigate into the recipe of interest, and start the *.ipynb file.
The easiest way to start using Azure CLI 2.0 is to launch Shell Console as described in these instructions.
If you prefer to install Azure CLI 2.0 on your computer, please follow these instructions to install or update Azure CLI 2.0 to the latest version.
If you are using Cloud Shell you are already logged in Azure. Otherwise, please execute az login command and follow instructions.
If you have multiple subscriptions, select Batch AI enabled subscription as default one by running the following command:
az account set -s <your subscription>Creation of Clusters, Jobs, File Servers and other resources requires you to specify location where they should be created, the location can be provided via --location parameter or can be added into default Azure CLI 2.0 configuration. To reduce the length of commands the recipes expect you to setup default location using the following command:
az configure --defaults location=eastusClusters, Jobs and File Servers are created under a resource group. It's recommended to create a dedicated resource group for running recipes because it will simplify resource management for you.
Create a resource group batchaitests (or choose your own resource name) and make it default for Azure CLI 2.0 using the following commands:
az group create --name batchaitests --location eastus
az configure --defaults group=batchaitestsEach recipe requires you to have a storage account in a region where Batch AI enabled (currently, eastus). Please you the following commands to create a new storage account and make it default for Azure CLI 2.0:
For GNU/Linux users:
az storage account create --name <unique storage account name> --sku Standard_LRS
export AZURE_STORAGE_ACCOUNT=mystorageaccount
export AZURE_STORAGE_KEY=$(az storage account keys list --account-name <unique storage account name> -o tsv --query [0].value)
export AZURE_BATCHAI_STORAGE_ACCOUNT=mystorageaccount
export AZURE_BATCHAI_STORAGE_KEY=$(az storage account keys list --account-name <unique storage account name> -o tsv --query [0].value)For Windows users:
az storage account create --name <unique storage account name> --sku Standard_LRS
az storage account keys list --account-name mystorageaccount -o tsv --query [0].value > temp.txt
set /p AZURE_STORAGE_KEY=< temp.txt
set AZURE_BATCHAI_STORAGE_ACCOUNT=mystorageaccount
set /p AZURE_BATCHAI_STORAGE_KEY=< temp.txt
del temp.txtDuring Cluster and File Server creation you will need to specify a name and authentication method for administrator account which will be created on each compute node (you can use this account to ssh to the node).
You can provide a password and/or ssh public key as authentication method via --password (-p) and --ssh-public-key (-k) parameters.
GNU/Linux users (including Cloud Shell users) can generate authentication key for ssh using ssh-keygen command.
Note, GNU/Linux part of recipes expects you to have a public ssh key at ~/.ssh/id_rsa.pub, if you prefer to use different ssh key, please update -k parameter value.
Training data used in recipes is compressed in zip archives and requires unzip utility to be installed on the host, please install it using your distribution package manager.
Cloud Shell has unzip already installed.
Each recipe contains cli-instructions.md file which describes input data, cluster and job configuration and provides instructions for cluster and job creation.
If you have any problems or questions, you can reach the Batch AI team at [email protected] or you can create an issue on GitHub.
We also welcome your contributions of additional sample notebooks, scripts, or other examples of working with Batch AI.