notebooks/sample_submit_dsm2_linux_hydro_version.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dmsbatch import create_batch_client, create_blob_client\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First create a batch client from the config file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = create_batch_client('../tests/data/dmsbatch.config')\n",
    "blob_client = create_blob_client('../tests/data/dmsbatch.config')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application packages\n",
    "To copy large files and programs it is best to zip (or targz) them and upload them as application packages\n",
    "\n",
    "Application packages are setup separately in either azure management apis or from the web console or cli tool\n",
    "\n",
    "These are referenced here by their name and version\n",
    "e.g. DSM2, python and other programs\n",
    "\n",
    "One extra field (last one) is the path within the zip file where the executables can be found. These are used later to setup the PATH varible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_pkgs = [('dsm2linux', '8.2.8449db2', 'DSM2-8.2.8449db2-Linux/bin')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show vms available\n",
    "\n",
    "https://docs.microsoft.com/en-us/azure/virtual-machines/fsv2-series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#display(client.skus_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create or resize existing pool\n",
    "If the pool doesn't exist it will create it\n",
    "If the pool exists, it will resize to the second arg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.set_path_to_apps(app_pkgs,ostype='linux')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pool_start_cmds = ['printenv',\n",
    "'yum install -y glibc.i686 libstdc++.i686 glibc.x86_64 libstdc++.x86_64',# --setopt=protected_multilib=false',\n",
    "'yum-config-manager --add-repo https://yum.repos.intel.com/2019/setup/intel-psxe-runtime-2019.repo',\n",
    "'rpm --import https://yum.repos.intel.com/2019/setup/RPM-GPG-KEY-intel-psxe-runtime-2019',\n",
    "'yum install -y intel-icc-runtime-32bit intel-ifort-runtime-32bit']\n",
    "client.wrap_commands_in_shell(pool_start_cmds,ostype='linux')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "created_pool = client.create_pool('dsm2linuxpool',\n",
    "                   1,\n",
    "                   app_packages=[(app,version) for app,version,_ in app_pkgs], \n",
    "                   vm_size='standard_f2s_v2', \n",
    "                   tasks_per_vm=2,\n",
    "                   os_image_data=('openlogic', 'centos', '7_8'),\n",
    "                    start_task_cmd=client.wrap_commands_in_shell(pool_start_cmds,ostype='linux'),\n",
    "                    start_task_admin=True,\n",
    "                    elevation_level='admin'\n",
    "                  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not created_pool:\n",
    "    client.resize_pool('dsm2linuxpool',1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create job on pool or fail if it exists\n",
    "Jobs are containers of tasks (things that run on nodes (machines) in the pool). If this exists, the next line will fail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    client.create_job('dsm2linuxjobs','dsm2linuxpool')\n",
    "except Exception as exc:\n",
    "    print('Job already exists? Its ok, we will continue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a task\n",
    "This uses the application package as pre -set up. If not, create one https://docs.microsoft.com/en-us/azure/batch/batch-application-packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsnow = str(datetime.datetime.now().timestamp()).split('.')[0]\n",
    "task_name = f'hydro_version_{tsnow}'\n",
    "cmd_string = client.wrap_cmd_with_app_path(\n",
    "    \"\"\"\n",
    "source /opt/intel/psxe_runtime/linux/bin/compilervars.sh ia32;\n",
    "%s;\n",
    "hydro -v; \n",
    "unzip -v;\n",
    "\"\"\"%(client.set_path_to_apps(app_pkgs,ostype='linux')),app_pkgs,ostype='linux')\n",
    "print(task_name)\n",
    "print(cmd_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blob_client.blob_client.account_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dmsbatch\n",
    "permissions = dmsbatch.commands.BlobSasPermissions(write=True)\n",
    "# |helpers.azureblob.BlobPermissions.ADD|helpers.azureblob.BlobPermissions.CREATE\n",
    "output_dir_sas_url = blob_client.get_container_sas_url(\n",
    "                                                   'dsm2linuxjobs',\n",
    "                                                   permissions)\n",
    "print(output_dir_sas_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std_out_files = client.create_output_file_spec(\n",
    "    '../std*.txt', output_dir_sas_url, blob_path=f'{task_name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hydro_v_task = client.create_task(task_name,cmd_string,output_files=[std_out_files])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next submit the task and wait "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.submit_tasks('dsm2linuxjobs',[hydro_v_task])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.wait_for_tasks_to_complete('dsm2linuxjobs', datetime.timedelta(seconds=100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finally resize the pool to 0 to save costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.resize_pool('dsm2linuxpool',0)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9e539dc02fc56d55a22d79a0646788fa38cb6ebb3e85aa69306aae5c2f643a8f"
  },
  "kernelspec": {
   "display_name": "Python [conda env:dev_azure]",
   "language": "python",
   "name": "conda-env-dev_azure-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}