notebooks/sample_submit_python_linux_ptm_batch_postpro.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import dmsbatch\n",
    "from dmsbatch import create_batch_client, create_blob_client\n",
    "import datetime\n",
    "import logging\n",
    "#logger = logging.getLogger()\n",
    "#logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First create a batch client from the config file\n",
    "\n",
    "The config file is described in the [README](../README.md)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = create_batch_client('../tests/data/dmsbatch.config')\n",
    "blob_client = create_blob_client('../tests/data/dmsbatch.config')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application packages\n",
    "To copy large files and programs it is best to zip (or targz) them and upload them as application packages\n",
    "\n",
    "Application packages are setup separately in either azure management apis or from the web console or cli tool\n",
    "\n",
    "These are referenced here by their name and version\n",
    "e.g. DSM2, python and other programs\n",
    "\n",
    "One extra field (last one) is the path within the zip file where the executables can be found. These are used later to setup the PATH varible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_pkgs = [('dsm2linux', '8.2.8449db2', 'DSM2-8.2.8449db2-Linux/bin')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show vms available\n",
    "\n",
    "https://docs.microsoft.com/en-us/azure/virtual-machines/fsv2-series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#display(client.skus_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create or resize existing pool\n",
    "If the pool doesn't exist it will create it\n",
    "If the pool exists, it will resize to the second arg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pool_name = 'pydelmodlinuxpool'\n",
    "container_name='ptmbatch'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.create_pool(pool_name,\n",
    "                    1,\n",
    "                    app_packages=[(app,version) for app,version,_ in app_pkgs], \n",
    "                    vm_size='Standard_D11_v2', #standard_f2s_v2' -- too small disk size for task\n",
    "                    tasks_per_vm=1,\n",
    "                    os_image_data=('openlogic', 'centos', '7_8'),\n",
    "                    start_task_cmd=client.wrap_commands_in_shell(['printenv'], ostype='linux'),\n",
    "                    start_task_admin=True,\n",
    "                    elevation_level='admin'\n",
    "                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autoscaling formula for this pool\n",
    "\n",
    "This can be added manually via the console or batch explorer in the resizing section.\n",
    "```\n",
    "// In this example, the pool size is adjusted based on the number of tasks in the queue. Note that both comments and line breaks are acceptable in formula strings.\n",
    "\n",
    "// Get pending tasks for the past 15 minutes.\n",
    "$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);\n",
    "// If we have fewer than 70 percent data points, we use the last sample point, otherwise we use the maximum of last sample point and the history average.\n",
    "$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) : max( $ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));\n",
    "// If number of pending tasks is not 0, set targetVM to pending tasks, otherwise half of current dedicated nodes.\n",
    "$targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes / 2);\n",
    "// The pool size is capped at 20, if target VM value is more than that, set it to 20. This value should be adjusted according to your use case.\n",
    "cappedPoolSize = 2;\n",
    "$TargetLowPriorityNodes = max(0, min($targetVMs, cappedPoolSize));\n",
    "// Set node deallocation mode - keep nodes active only until tasks finish\n",
    "$NodeDeallocationOption = taskcompletion;\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create job on pool or fail if it exists\n",
    "Jobs are containers of tasks (things that run on nodes (machines) in the pool). If this exists, the next line will fail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "UPLOAD=False\n",
    "if UPLOAD:\n",
    "    blob_client.upload_file_to_container(container_name,'pydelmod-linux.tar.gz','../tests/data/pydelmod-linux.tar.gz',30,max_connections=10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "if UPLOAD:\n",
    "    local_file_script = 'd:/dev/ptm_fate_postpro/ptm_fate_postpro.py'\n",
    "    blob_client.upload_file_to_container(container_name,os.path.basename(local_file_script),local_file_script,30)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shared_file = client.create_input_file_spec(container_name,blob_prefix='pydelmod-linux.tar.gz',file_path='.')\n",
    "# don't end command with ; and don't add extra line at end \n",
    "commands = [\"printenv\",\n",
    "    \"mkdir -p ${AZ_BATCH_NODE_SHARED_DIR}/pydelmod\",\n",
    "    \"mv pydelmod-linux.tar.gz ${AZ_BATCH_NODE_SHARED_DIR}/pydelmod\",\n",
    "    \"cd ${AZ_BATCH_NODE_SHARED_DIR}/pydelmod\",\n",
    "    \"tar xvzf pydelmod-linux.tar.gz\",\n",
    "    'echo \"Done setting up pydelmod!\"']\n",
    "startup_task = client.create_prep_task('startup_task',commands, resource_files=[shared_file],ostype='linux')\n",
    "job_name = 'pydelmodjob'\n",
    "client.create_job(job_name,pool_name,prep_task=startup_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a task\n",
    "This uses the application package as pre -set up. If not, create one https://docs.microsoft.com/en-us/azure/batch/batch-application-packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_ptm_fate_postpro_task(task_name, blob_prefix, envvars):\n",
    "    input_file_script = client.create_input_file_spec(\n",
    "        container_name, 'ptm_fate_postpro.py', file_path='.')\n",
    "    input_file_model_output = client.create_input_file_spec(\n",
    "        container_name, blob_prefix=blob_prefix, file_path='.')\n",
    "    permissions = dmsbatch.commands.azureblob.BlobPermissions.WRITE\n",
    "    output_dir_sas_url = blob_client.get_container_sas_url(container_name, permissions)\n",
    "    std_out_files = client.create_output_file_spec(\n",
    "        '../std*.txt', output_dir_sas_url, blob_path=f'{task_name}')\n",
    "    output_dat_files = client.create_output_file_spec(\n",
    "        '**/*.dat', output_dir_sas_url, blob_path=f'{task_name}')\n",
    "\n",
    "    set_path_string = 'export PATH=${AZ_BATCH_NODE_SHARED_DIR}/pydelmod/bin:$PATH'\n",
    "    cmd_string = client.wrap_cmd_with_app_path(\n",
    "        f\"\"\"\n",
    "        {set_path_string};\n",
    "        mv ptm_fate_postpro.py {blob_prefix};\n",
    "        cd {blob_prefix};\n",
    "        python ptm_fate_postpro.py --years 1923 2015 --days 92 --months 1 2 3 4 5 6;\"\"\", app_pkgs, ostype='linux')\n",
    "    # print(cmd_string)\n",
    "    ptm_task = client.create_task(task_name + '_' + os.path.basename(blob_prefix), cmd_string,\n",
    "                                  resource_files=[input_file_script, input_file_model_output],\n",
    "                                  output_files=[\n",
    "                                      std_out_files, output_dat_files],\n",
    "                                  env_settings=envvars)\n",
    "    return ptm_task\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create all tasks\n",
    "This function looks at the insertion location file and the simulation years and months to create an array of tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "postpro_task = create_ptm_fate_postpro_task(\n",
    "    'ptm_fate_postpro', 'neutrally_buoyant_particles/ex_2020/psandhu_1639109550', {})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next submit the task and wait \n",
    "Azure batch limits to submitting 100 tasks at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.submit_tasks(job_name,[postpro_task])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finally resize the pool to 0 to save costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#client.resize_pool(pool_name,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9e539dc02fc56d55a22d79a0646788fa38cb6ebb3e85aa69306aae5c2f643a8f"
  },
  "kernelspec": {
   "display_name": "Python [conda env:dev_azure]",
   "language": "python",
   "name": "conda-env-dev_azure-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}