finetune/PyTorch/notebooks/Pretrained-BERT-NER.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Copyright (c) Microsoft Corporation. All rights reserved.\n\nLicensed under the MIT License.\n\n### PyTorch Pretrained BERT on AzureML for NER \nThis notebook contains an end-to-end walkthrough of using Azure Machine Learning Service to run PyTorch reimplementation of Google's TensorFlow repository for the BERT model developed by Hugging Face. The code is based on this [amazing tutorial](https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/?unapproved=907&moderation-hash=0d668913f6fe8b07d55eb1c6b009dbe0#comment-907) by Tobias Sterbak\n\nYou will find the following contents:\n\nDownload NER dataset on the remote compute and store them in Azure storage\nSpeep-up fine-tuning BERT for NER dataset on AzureML GPU clusters\nFurther fine-tune NER wtih AzureML hyperparameter optimizer\n### Prerequisites\n\nUnderstand the architecture and terms introduced by Azure Machine Learning (AML)\n\nInstall the Python SDK: make sure to install notebook, and contrib\n\nconda create -n azureml -y Python=3.6\nsource activate azureml\npip install --upgrade azureml-sdk[notebooks,contrib] \nconda install ipywidgets\njupyter nbextension install --py --user azureml.widgets\njupyter nbextension enable azureml.widgets --user --py\nYou will need to restart jupyter after this Detailed instructions are [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python/?WT.mc_id=bert-notebook-abornst)\n\nIf you need a free trial account to get started you can get one [here](https://azure.microsoft.com/en-us/offers/ms-azr-0044p/?WT.mc_id=bert-notebook-abornst)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Initialize workspace\n\nTo create or access an Azure ML Workspace, you will need to import the AML library and the following information:\n* A name for your workspace\n* Your subscription id\n* The resource group name\n\nInitialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace/?WT.mc_id=bert-notebook-abornst) object from the existing workspace you created in the Prerequisites step or create a new one. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core import Workspace\n\n# subscription_id = ''\n# resource_group  = ''\n# workspace_name  = ''\n#     ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace_name)\n#     ws.write_config()\n\ntry:\n    ws = Workspace.from_config()\n    print(ws.name, ws.location, ws.resource_group, ws.location, sep='\\t')\n    print('Library configuration succeeded')\nexcept:\n    print('Workspace not found')",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Performing interactive authentication. Please follow the instructions on the terminal.\nTo sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code FCMPHUU4B to authenticate.\nInteractive authentication successfully completed.\nariaiwork\teastus\tai_work\teastus\nLibrary configuration succeeded\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Compute"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "There are two computer option run once(preview) and persistent computer for this demo we will use persistent compute to learn more about run once compute check out the docs."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core.compute import ComputeTarget, AmlCompute\nfrom azureml.core.compute_target import ComputeTargetException\n\n# Choose a name for your CPU cluster\ncluster_name = \"cluster\"\n\n# Verify that cluster does not exist already\ntry:\n    cluster = ComputeTarget(workspace=ws, name=cluster_name)\n    print('Found existing cluster, use it.')\nexcept ComputeTargetException:\n    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_NC6',\n                                                           min_nodes=1,\n                                                           max_nodes=4)\n    cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n\ncluster.wait_for_completion(show_output=True)",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Found existing cluster, use it.\nSucceeded\nAmlCompute wait for completion finished\nMinimum number of nodes requested have been provisioned\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Upload Data\n\nThe dataset we are using comes from Kaggle it can be downloaded [here](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus). To tag on your own data set check out the work done with the open source [Doccanno](https://towardsdatascience.com/text-annotation-on-a-budget-with-azure-web-apps-doccano-b29f479c0c54) which has one click deployment to Azure. Be sure to export to [IOB format](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)) also be sure to  replace **tag_vals** and **tag2idx** in the train script below with your own custom IOB tag values. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ds = ws.get_default_datastore()",
      "execution_count": 3,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "ds. upload_files([\"ner_dataset.csv\"], relative_root='.')",
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Target already exists. Skipping upload for ner_dataset.csv\n",
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "execution_count": 4,
          "data": {
            "text/plain": "$AZUREML_DATAREFERENCE_workspaceblobstore"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Train File"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%writefile train.py\n\nimport argparse\nimport os\nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm, trange\n\nimport torch\nfrom torch.optim import Adam\nfrom torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\nfrom keras.preprocessing.sequence import pad_sequences\nfrom sklearn.model_selection import train_test_split\nfrom pytorch_pretrained_bert import BertTokenizer, BertConfig\nfrom pytorch_pretrained_bert import BertForTokenClassification, BertAdam\n\nfrom seqeval.metrics import f1_score\n\n\nfrom azureml.core import Run\n\nclass SentenceGetter(object):\n    \n    def __init__(self, data):\n        self.n_sent = 1\n        self.data = data\n        self.empty = False\n        agg_func = lambda s: [(w, p, t) for w, p, t in zip(s[\"Word\"].values.tolist(),\n                                                           s[\"POS\"].values.tolist(),\n                                                           s[\"Tag\"].values.tolist())]\n        self.grouped = self.data.groupby(\"Sentence #\").apply(agg_func)\n        self.sentences = [s for s in self.grouped]\n    \n    def get_next(self):\n        try:\n            s = self.grouped[\"Sentence: {}\".format(self.n_sent)]\n            self.n_sent += 1\n            return s\n        except:\n            return None\n\n\n# let user feed in 2 parameters, the location of the data files (from datastore), and the regularization rate of the logistic regression model\nparser = argparse.ArgumentParser()\nparser.add_argument('--data-folder', type=str, dest='data_folder', help='data folder mounting point')\nparser.add_argument('--learning_rate', type=float, default=3e-5, help='learning rate')\nparser.add_argument('--epochs', type=int, default=5)\nargs = parser.parse_args()\n\ndata_folder = args.data_folder\nprint('Data folder:', data_folder)\n\n# load data\ndata = pd.read_csv(os.path.join(data_folder, \"ner_dataset.csv\"), encoding=\"latin1\").fillna(method=\"ffill\")\ngetter = SentenceGetter(data)\nsentences = [\" \".join([s[0] for s in sent]) for sent in getter.sentences]\nlabels = [[s[2] for s in sent] for sent in getter.sentences]\n\n# For a custom Dataset replace tag_vals and tag2idx with your own custom IOB tag values  \ntags_vals = ['B-art',\n 'B-gpe',\n 'B-per',\n 'B-org',\n 'B-tim',\n 'B-nat',\n 'B-eve',\n 'I-geo',\n 'I-per',\n 'B-geo',\n 'I-art',\n 'I-gpe',\n 'I-eve',\n 'I-org',\n 'I-tim',\n 'O',\n 'I-nat',\n 'X']\n\ntag2idx = {'B-art': 0,\n 'B-eve': 6,\n 'B-geo': 9,\n 'B-gpe': 11,\n 'B-nat': 5,\n 'B-org': 3,\n 'B-per': 2,\n 'B-tim': 4,\n 'I-art': 10,\n 'I-eve': 12,\n 'I-geo': 7,\n 'I-gpe': 1,\n 'I-nat': 16,\n 'I-org': 13,\n 'I-per': 8,\n 'I-tim': 15,\n 'O': 14,\n 'X': 17}\n\n# Apply Bert\n\nMAX_LEN = 75\nbs = 32\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nn_gpu = torch.cuda.device_count()\n\ntorch.cuda.get_device_name(0)\n\n# Now we tokenize all sentences\n\ntokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\ntokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n\n# Pad the subword tokens\n\nnum_sent = len(labels)\nfor sent_id in range(num_sent):\n    tokens_len = len(tokenized_texts[sent_id])\n    for i in range(tokens_len):\n      if tokenized_texts[sent_id][i][:2] == \"##\":\n        labels[sent_id].insert(i, \"X\")\n\n\n# Next, we cut and pad the token and label sequences to our desired length.\n\ninput_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n                          maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n\ntags = pad_sequences([[tag2idx.get(l) for l in lab] for lab in labels],\n                     maxlen=MAX_LEN, value=tag2idx[\"O\"], padding=\"post\",\n                     dtype=\"long\", truncating=\"post\")\n\n# The Bert model supports something called attention_mask, which is similar to the masking in keras. So here we create the mask to ignore the padded elements in the sequences.\n\nattention_masks = [[float(i>0) for i in ii] for ii in input_ids]\n\n# Now we split the dataset to use 10% to validate the model.\n\ntr_inputs, val_inputs, tr_tags, val_tags = train_test_split(input_ids, tags, \n                                                            random_state=2018, test_size=0.1)\ntr_masks, val_masks, _, _ = train_test_split(attention_masks, input_ids,\n                                             random_state=2018, test_size=0.1)\n\n# Since we’re operating in pytorch, we have to convert the dataset to torch tensors.\n\ntr_inputs = torch.tensor(tr_inputs)\nval_inputs = torch.tensor(val_inputs)\ntr_tags = torch.tensor(tr_tags)\nval_tags = torch.tensor(val_tags)\ntr_masks = torch.tensor(tr_masks)\nval_masks = torch.tensor(val_masks)\n\n# The last step is to define the dataloaders. We shuffle the data at training time with the RandomSampler and at test time we just pass them sequentially with the SequentialSampler.\"\"\"\n\ntrain_data = TensorDataset(tr_inputs, tr_masks, tr_tags)\ntrain_sampler = RandomSampler(train_data)\ntrain_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=bs)\n\nvalid_data = TensorDataset(val_inputs, val_masks, val_tags)\nvalid_sampler = SequentialSampler(valid_data)\nvalid_dataloader = DataLoader(valid_data, sampler=valid_sampler, batch_size=bs)\n\n# Setup the Bert model for finetuning\n# The pytorch-pretrained-bert package provides a BertForTokenClassification class for token-level predictions. \n# BertForTokenClassification is a fine-tuning model that wraps BertModel and adds token-level classifier on top of the BertModel. \n# The token-level classifier is a linear layer that takes as input the last hidden state of the sequence.\n# We load the pre-trained bert-base-uncased model and provide the number of possible labels.\n\nmodel = BertForTokenClassification.from_pretrained(\"bert-base-uncased\", num_labels=len(tag2idx))\n\n# Now we have to pass the model parameters to the GPU\n\nmodel.cuda();\n\n# Before we can start the fine-tuning process, we have to setup the optimizer and add the parameters it should update. A common choice is the Adam optimizer. \n# We also add some weight_decay as regularization to the main weight matrices. \n# If you have limited resources, you can also try to just train the linear classifier on top of Bert and keep all other weights fixed.\n# This will still give you a good performance.\n\nFULL_FINETUNING = True\nif FULL_FINETUNING:\n    param_optimizer = list(model.named_parameters())\n    no_decay = ['bias', 'gamma', 'beta']\n    optimizer_grouped_parameters = [\n        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n         'weight_decay_rate': 0.01},\n        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n         'weight_decay_rate': 0.0}\n    ]\nelse:\n    param_optimizer = list(model.classifier.named_parameters()) \n    optimizer_grouped_parameters = [{\"params\": [p for n, p in param_optimizer]}]\noptimizer = Adam(optimizer_grouped_parameters, lr=args.learning_rate)\n\n# Finetune Bert\n# First we define some metrics, we want to track while training. \n# We use the f1_score from the seqeval package. You can find more details here.\n# And we use simple accuracy on a token level comparable to the accuracy in keras.\n\n\ndef flat_accuracy(preds, labels):\n    pred_flat = np.argmax(preds, axis=2).flatten()\n    labels_flat = labels.flatten()\n    x_indecies = [n for n,x in enumerate(labels_flat) if x==tag2idx[\"X\"]]\n    pred_flat = np.asarray([pred_flat[i] for i in range(len(pred_flat)) if i not in x_indecies])\n    labels_flat = np.asarray([labels_flat[i] for i in range(len(labels_flat)) if i not in x_indecies])\n        \n    return np.sum(pred_flat == labels_flat) / len(labels_flat)\n\n# Finally, we can fine-tune the model. A few epochs should be enough. The paper suggest 3-4 epochs.\"\"\"\n\nepochs = args.epochs\nmax_grad_norm = 1.0\n\nfor _ in trange(epochs, desc=\"Epoch\"):\n    # TRAIN loop\n    model.train()\n    tr_loss = 0\n    nb_tr_examples, nb_tr_steps = 0, 0\n    for step, batch in enumerate(train_dataloader):\n        # add batch to gpu\n        batch = tuple(t.to(device) for t in batch)\n        b_input_ids, b_input_mask, b_labels = batch\n        # forward pass\n        loss = model(b_input_ids, token_type_ids=None,\n                     attention_mask=b_input_mask, labels=b_labels)\n        # backward pass\n        loss.backward()\n        # track train loss\n        tr_loss += loss.item()\n        nb_tr_examples += b_input_ids.size(0)\n        nb_tr_steps += 1\n        # gradient clipping\n        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)\n        # update parameters\n        optimizer.step()\n        model.zero_grad()\n    # print train loss per epoch\n    print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n    # VALIDATION on validation set\n    model.eval()\n    eval_loss, eval_accuracy = 0, 0\n    nb_eval_steps, nb_eval_examples = 0, 0\n    predictions , true_labels = [], []\n    for batch in valid_dataloader:\n        batch = tuple(t.to(device) for t in batch)\n        b_input_ids, b_input_mask, b_labels = batch\n        \n        with torch.no_grad():\n            tmp_eval_loss = model(b_input_ids, token_type_ids=None,\n                                  attention_mask=b_input_mask, labels=b_labels)\n            logits = model(b_input_ids, token_type_ids=None,\n                           attention_mask=b_input_mask)\n        logits = logits.detach().cpu().numpy()\n        label_ids = b_labels.to('cpu').numpy()\n        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n        true_labels.append(label_ids)\n        \n        tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n        \n        eval_loss += tmp_eval_loss.mean().item()\n        eval_accuracy += tmp_eval_accuracy\n        \n        nb_eval_examples += b_input_ids.size(0)\n        nb_eval_steps += 1\n    eval_loss = eval_loss/nb_eval_steps\n    print(\"Validation loss: {}\".format(eval_loss))\n    print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n    pred_tags = [tags_vals[p_i] for p in predictions for p_i in p]\n    valid_tags = [tags_vals[l_ii] for l in true_labels for l_i in l for l_ii in l_i]\n    \n    x_indecies = [n for n,x in enumerate(valid_tags) if x==\"X\"]\n    pred_tags = [pred_tags[i] for i in range(len(pred_tags)) if i not in x_indecies]\n    valid_tags = [valid_tags[i] for i in range(len(valid_tags)) if i not in x_indecies]\n\n\nmodel.eval()\n\n# save model\nos.makedirs('outputs', exist_ok=True)\ntorch.save(model.state_dict(), 'outputs/bert_ner.model')\n\npredictions = []\ntrue_labels = []\neval_loss, eval_accuracy = 0, 0\nnb_eval_steps, nb_eval_examples = 0, 0\nfor batch in valid_dataloader:\n    batch = tuple(t.to(device) for t in batch)\n    b_input_ids, b_input_mask, b_labels = batch\n\n    with torch.no_grad():\n        tmp_eval_loss = model(b_input_ids, token_type_ids=None,\n                              attention_mask=b_input_mask, labels=b_labels)\n        logits = model(b_input_ids, token_type_ids=None,\n                       attention_mask=b_input_mask)\n        \n    logits = logits.detach().cpu().numpy()\n    predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n    label_ids = b_labels.to('cpu').numpy()\n    true_labels.append(label_ids)\n    tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n\n    eval_loss += tmp_eval_loss.mean().item()\n    eval_accuracy += tmp_eval_accuracy\n\n    nb_eval_examples += b_input_ids.size(0)\n    nb_eval_steps += 1\n    \npred_tags = [[tags_vals[p_i] for p_i in p] for p in predictions]\nvalid_tags = [[tags_vals[l_ii] for l_ii in l_i] for l in true_labels for l_i in l]\nx_indecies = [[n for n,x in enumerate(t) if x==\"X\"] for t in valid_tags] \npred_tags  = [[pred_tags[p][i] for i in range(len(pred_tags[p])) if i not in x_indecies[p]] for p in range(len(pred_tags))]\nvalid_tags = [[valid_tags[t][i] for i in range(len(valid_tags[t])) if i not in x_indecies[t]] for t in range(len(valid_tags))]\n\n# get hold of the current run\nrun = Run.get_context()\nrun.log('loss', eval_loss/nb_eval_steps)\nrun.log('accuracy', eval_accuracy/nb_eval_steps)\nrun.log('f1', f1_score(pred_tags, valid_tags))\nrun.log('tag2idx', tag2idx)\n",
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Overwriting train.py\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Create An Expierment\n\nCreate an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment/?WT.mc_id=bert-notebook-abornst) to track all the runs in your workspace for this distributed PyTorch tutorial. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core import Experiment\nexperiment_name = 'bert_test'\n\nexp = Experiment(workspace=ws, name=experiment_name)",
      "execution_count": 6,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.train.dnn import PyTorch\n\nscript_params = {\n    '--data-folder': ds\n}\nconda = ['tensorflow', 'keras', 'pytorch','scikit-learn']\npip = [\"seqeval[gpu]\",\"pytorch-pretrained-bert==0.4.0\",\"pandas\"]\npt_est = PyTorch(source_directory='.',\n                 script_params=script_params,\n                 compute_target=cluster,\n                 conda_packages=conda,\n                 pip_packages=pip,\n                 entry_script='train.py',\n                 use_gpu=True)",
      "execution_count": 7,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "run = exp.submit(pt_est)",
      "execution_count": 8,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.widgets import RunDetails\n\nRunDetails(run).show()",
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4222b5b8a02f4efcaa766696083d7627",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', '…"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "run.id",
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 11,
          "data": {
            "text/plain": "'bert_test_1560586302_22363fe0'"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model = run.register_model(model_name='bert_ner', model_path='outputs/bert_ner.model')\nprint(model.name, model.id, model.version, sep='\\t')\n",
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": "bert_ner\tbert_ner:3\t3\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Fine-Tuning BERT with Hyperparameter Tuning\n\nWe would also like to optimize our hyperparameter, `learning rate`, using Azure Machine Learning's hyperparameter tuning capabilities.\n\nFor more information on the available tuning algorithms see [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters#log-metrics-for-hyperparameter-tuning/?WT.mc_id=bert-notebook-abornst)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Start a hyperparameter sweep\nFirst, we will define the hyperparameter space to sweep over. In this example we will use random sampling to try different configuration sets of hyperparameter to minimize our primary metric, the evaluation accuracy (`eval_accuracy`)."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.train.hyperdrive import *\nimport math\n\nparam_sampling = RandomParameterSampling( {\n        'learning_rate': loguniform(math.log(1e-4), math.log(1e-6)),\n    }\n)\n\nhyperdrive_run_config = HyperDriveRunConfig(estimator=pt_est,\n                                            hyperparameter_sampling=param_sampling, \n                                            primary_metric_name='f1',\n                                            primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n                                            max_total_runs=16,\n                                            max_concurrent_runs=4)\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Finally, lauch the hyperparameter tuning job."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "hyperdrive_run = experiment.submit(hyperdrive_run_config)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Monitor HyperDrive runs\nWe can monitor the progress of the runs with the following Jupyter widget. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.widgets import RunDetails\n\nRunDetails(hyperdrive_run).show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Find and register the best model\nOnce all the runs complete, we can find the run that produced the model with the highest evaluation f1."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "best_run = hyperdrive_run.get_best_run_by_primary_metric()\nbest_run_metrics = best_run.get_metrics()\nprint(best_run)\nprint('Best Run is:\\n  F1: {0:.5f} \\n  Learning rate: {1:.8f}'.format(\n        best_run_metrics['eval_f1'][-1],\n        best_run_metrics['lr']\n     ))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "# Deploy as web service\nOnce you've tested the model and are satisfied with the results, deploy the model as a web service hosted in [Azure Container Instances](https://azure.microsoft.com/en-us/services/container-instances/?WT.mc_id=bert-notebook-abornst).\n\nTo build the correct environment for ACI, provide the following:\n\nA scoring script to show how to use the model\nAn environment file to show what packages need to be installed\nA configuration file to build the ACI\nThe model you trained before\n\n## Create scoring script\nCreate the scoring script, called score.py, used by the web service call to show how to use the model.\n\nYou must include two required functions into the scoring script:\n\nThe init() function, which typically loads the model into a global object. This function is run only once when the Docker container is started.\n\nThe run(input_data) function uses the model to predict a value based on the input data. Inputs and outputs to the run typically use JSON for serialization and de-serialization, but other formats are supported."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%writefile score.py\nimport os, json\nimport numpy as np\nfrom tqdm import tqdm, trange\n\nimport torch\nfrom torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\nfrom keras.preprocessing.sequence import pad_sequences\nfrom pytorch_pretrained_bert import BertTokenizer, BertConfig\nfrom pytorch_pretrained_bert import BertForTokenClassification\n\nfrom azureml.core.model import Model\n\ndef init():\n    global tag2idx, model, device, MAX_LEN, tokenizer, tags_vals\n    \n    tags_vals = ['B-art',\n                'I-gpe',\n                'B-per',\n                'B-org',\n                'B-tim',\n                'B-nat',\n                'B-eve',\n                'I-geo',\n                'I-per',\n                'B-geo',\n                'I-art',\n                'B-gpe',\n                'I-eve',\n                'I-org',\n                'O',\n                'I-tim',\n                'I-nat',\n                'X']\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    \n    MAX_LEN = 75\n    # retrieve the path to the model file using the model name\n    model = BertForTokenClassification.from_pretrained(\"bert-base-uncased\", num_labels=len(tags_vals))\n    model_path = Model.get_model_path('bert_ner')\n    model.load_state_dict(torch.load(model_path, map_location=device))\n    model.eval()\n    \n    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n    \ndef run(raw_data):\n    print(raw_data)\n    # make prediction\n    tokenized_texts = [tokenizer.tokenize(raw_data)]\n    input_ids = torch.tensor(pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n                          maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\"), device = device)\n    attention_masks = torch.tensor([[float(i>0) for i in ii] for ii in input_ids], device = device)\n    with torch.no_grad():\n        logits = model(input_ids, token_type_ids=None, attention_mask=attention_masks)\n    logits = logits.detach().cpu().numpy()\n    predictions = [list(p) for p in np.argmax(logits, axis=2)][0]\n    return [tags_vals[p_i] for p_i in predictions if p_i != 'X']",
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Overwriting score.py\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Create environment file\nNext, create an environment file, called myenv.yml, that specifies all of the script's package dependencies. This file is used to ensure that all of those dependencies are installed in the Docker image. This model needs scikit-learn and azureml-sdk."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "conda = ['tensorflow', 'keras']\npip = [\"azureml-defaults\", \"azureml-monitoring\", \"torch\",\"pytorch-pretrained-bert==0.4.0\"]\n",
      "execution_count": 47,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core.conda_dependencies import CondaDependencies \n\n\nmyenv = CondaDependencies.create(conda_packages=conda, pip_packages=pip)\n\nwith open(\"myenv.yml\",\"w\") as f:\n    f.write(myenv.serialize_to_string())",
      "execution_count": 48,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Review the content of the myenv.yml file.\n"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "with open(\"myenv.yml\",\"r\") as f:\n    print(f.read())",
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": "# Conda environment specification. The dependencies defined in this file will\n# be automatically provisioned for runs with userManagedDependencies=False.\n\n# Details about the Conda environment file format:\n# https://conda.io/docs/user-guide/tasks/manage-environments.html#create-env-file-manually\n\nname: project_environment\ndependencies:\n  # The python interpreter version.\n  # Currently Azure ML only supports 3.5.2 and later.\n- python=3.6.2\n\n- pip:\n  - azureml-defaults\n  - azureml-monitoring\n  - torch\n  - pytorch-pretrained-bert==0.4.0\n- tensorflow\n- keras\n\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Create configuration file\nCreate a deployment configuration file and specify the number of CPUs and gigabyte of RAM needed for your ACI container. While it depends on your model, the default of 1 core and 1 gigabyte of RAM is usually sufficient for many models. If you feel you need more later, you would have to recreate the image and redeploy the service.`"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core.webservice import AciWebservice\n\naciconfig = AciWebservice.deploy_configuration(cpu_cores=1, \n                                               memory_gb=2, \n                                               tags={\"data\": \"Sentence\",  \"method\" : \"bert\"}, \n                                               description='Predict NER with Bert')\n",
      "execution_count": 50,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Deploy in ACI\nEstimated time to complete: about 7-8 minutes\n\nConfigure the image and deploy. The following code goes through these steps:\n\nBuild an image using:\nThe scoring file (score.py)\nThe environment file (myenv.yml)\nThe model file\nRegister that image under the workspace.\nSend the image to the ACI container.\nStart up a container in ACI using the image.\nGet the web service HTTP endpoint."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%time\nfrom azureml.core.model import Model\nfrom azureml.core.webservice import Webservice\nfrom azureml.core.image import ContainerImage\n\nmodel=Model(ws, 'bert_ner')\n\n# configure the image\nimage_config = ContainerImage.image_configuration(execution_script=\"score.py\", \n                                                  runtime=\"python\", \n                                                  conda_file=\"myenv.yml\")\n\nservice = Webservice.deploy_from_model(workspace=ws,\n                                       name='bert-ner-srvc',\n                                       deployment_config=aciconfig,\n                                       models=[model],\n                                       image_config=image_config)\n\nservice.wait_for_deployment(show_output=True)",
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Creating image\nImage creation operation finished for image bert-ner-srvc:18, operation \"Succeeded\"\nCreating service\nRunning.....................................................................\nSucceededACI service creation operation finished, operation \"Succeeded\"\nCPU times: user 4.88 s, sys: 396 ms, total: 5.28 s\nWall time: 15min 24s\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(service.get_logs())\n",
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": "2019-06-15T13:23:50,538429640+00:00 - rsyslog/run \n2019-06-15T13:23:50,538442240+00:00 - iot-server/run \n2019-06-15T13:23:50,539124446+00:00 - gunicorn/run \n2019-06-15T13:23:50,656866486+00:00 - nginx/run \nEdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...\n2019-06-15T13:23:55,663481005+00:00 - iot-server/finish 1 0\n2019-06-15T13:23:55,664661615+00:00 - Exit code 1 is normal. Not restarting iot-server.\nStarting gunicorn 19.6.0\nListening at: http://127.0.0.1:9090 (13)\nUsing worker: sync\nworker timeout is set to 300\nBooting worker with pid: 47\nBetter speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\nInitializing logger\nStarting up app insights client\nStarting up request id generator\nStarting up app insight hooks\nInvoking user's init function\nhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz not found in cache, downloading to /tmp/tmprdr64zmp\nUsing TensorFlow backend.\n\r  0%|          | 0/407873900 [00:00<?, ?B/s]\n\r  1%|          | 2193408/407873900 [00:00<00:18, 21930963.67B/s]\n\r  2%|▏         | 7315456/407873900 [00:00<00:15, 26472070.89B/s]\n\r  3%|▎         | 12830720/407873900 [00:00<00:12, 31363384.47B/s]\n\r  4%|▍         | 18130944/407873900 [00:00<00:10, 35739552.09B/s]\n\r  6%|▌         | 23607296/407873900 [00:00<00:09, 39894967.66B/s]\n\r  7%|▋         | 28937216/407873900 [00:00<00:08, 43146363.91B/s]\n\r  8%|▊         | 34381824/407873900 [00:00<00:08, 46010528.27B/s]\n\r 10%|▉         | 40120320/407873900 [00:00<00:07, 48918614.09B/s]\n\r 11%|█         | 45227008/407873900 [00:00<00:07, 49080626.62B/s]\n\r 12%|█▏        | 50845696/407873900 [00:01<00:06, 51014542.30B/s]\n\r 14%|█▍        | 56402944/407873900 [00:01<00:06, 52300079.47B/s]\n\r 15%|█▌        | 61933568/407873900 [00:01<00:06, 53166474.68B/s]\n\r 17%|█▋        | 67543040/407873900 [00:01<00:06, 53999595.84B/s]\n\r 18%|█▊        | 73289728/407873900 [00:01<00:06, 54994790.83B/s]\n\r 19%|█▉        | 78830592/407873900 [00:01<00:06, 54559964.80B/s]\n\r 21%|██        | 84316160/407873900 [00:01<00:06, 53567359.61B/s]\n\r 22%|██▏       | 89698304/407873900 [00:01<00:05, 53329016.70B/s]\n\r 23%|██▎       | 95307776/407873900 [00:01<00:05, 54127846.38B/s]\n\r 25%|██▍       | 100736000/407873900 [00:01<00:05, 53035645.59B/s]\n\r 26%|██▌       | 106317824/407873900 [00:02<00:05, 53839714.85B/s]\n\r 27%|██▋       | 111716352/407873900 [00:02<00:05, 52610264.42B/s]\n\r 29%|██▊       | 116994048/407873900 [00:02<00:05, 52022116.57B/s]\n\r 30%|███       | 122485760/407873900 [00:02<00:05, 52856940.47B/s]\n\r 31%|███▏      | 127783936/407873900 [00:02<00:05, 49020740.28B/s]\n\r 33%|███▎      | 132750336/407873900 [00:02<00:05, 47301018.14B/s]\n\r 34%|███▍      | 138124288/407873900 [00:02<00:05, 49064228.96B/s]\n\r 35%|███▌      | 143758336/407873900 [00:02<00:05, 51040666.42B/s]\n\r 37%|███▋      | 149292032/407873900 [00:02<00:04, 52256933.08B/s]\n\r 38%|███▊      | 154567680/407873900 [00:02<00:04, 52232062.36B/s]\n\r 39%|███▉      | 159825920/407873900 [00:03<00:04, 50428734.47B/s]\n\r 40%|████      | 164907008/407873900 [00:03<00:04, 50100320.39B/s]\n\r 42%|████▏     | 169944064/407873900 [00:03<00:04, 49591640.29B/s]\n\r 43%|████▎     | 175147008/407873900 [00:03<00:04, 50298491.79B/s]\n\r 44%|████▍     | 180606976/407873900 [00:03<00:04, 51512821.22B/s]\n\r 46%|████▌     | 186170368/407873900 [00:03<00:04, 52682744.69B/s]\n\r 47%|████▋     | 191459328/407873900 [00:03<00:05, 41032499.07B/s]\n\r 48%|████▊     | 195973120/407873900 [00:03<00:05, 40859194.72B/s]\n\r 49%|████▉     | 201553920/407873900 [00:04<00:04, 44428651.61B/s]\n\r 51%|█████     | 206894080/407873900 [00:04<00:04, 46783956.31B/s]\n\r 52%|█████▏    | 212137984/407873900 [00:04<00:04, 48346039.51B/s]\n\r 53%|█████▎    | 217449472/407873900 [00:04<00:03, 49682938.92B/s]\n\r 55%|█████▍    | 223113216/407873900 [00:04<00:03, 51582847.52B/s]\n\r 56%|█████▌    | 228688896/407873900 [00:04<00:03, 52764208.51B/s]\n\r 57%|█████▋    | 234051584/407873900 [00:04<00:03, 52994200.98B/s]\n\r 59%|█████▉    | 239690752/407873900 [00:04<00:03, 53964252.90B/s]\n\r 60%|██████    | 245402624/407873900 [00:04<00:02, 54872163.38B/s]\n\r 62%|██████▏   | 250927104/407873900 [00:04<00:02, 53966687.03B/s]\n\r 63%|██████▎   | 256846848/407873900 [00:05<00:02, 55434552.90B/s]\n\r 64%|██████▍   | 262458368/407873900 [00:05<00:02, 55636036.37B/s]\n\r 66%|██████▌   | 268043264/407873900 [00:05<00:02, 55303478.86B/s]\n\r 67%|██████▋   | 273589248/407873900 [00:05<00:02, 54701291.18B/s]\n\r 68%|██████▊   | 279071744/407873900 [00:05<00:02, 53527671.10B/s]\n\r 70%|██████▉   | 284439552/407873900 [00:05<00:02, 50239456.60B/s]\n\r 71%|███████   | 289515520/407873900 [00:05<00:02, 49309643.61B/s]\n\r 72%|███████▏  | 294842368/407873900 [00:05<00:02, 50433294.41B/s]\n\r 74%|███████▎  | 300065792/407873900 [00:05<00:02, 50959420.71B/s]\n\r 75%|███████▍  | 305187840/407873900 [00:05<00:02, 50959170.28B/s]\n\r 76%|███████▌  | 310635520/407873900 [00:06<00:01, 51965548.64B/s]\n\r 78%|███████▊  | 316106752/407873900 [00:06<00:01, 52759356.10B/s]\n\r 79%|███████▉  | 321397760/407873900 [00:06<00:01, 52247523.10B/s]\n\r 80%|████████  | 326634496/407873900 [00:06<00:01, 52230952.33B/s]\n\r 82%|████████▏ | 332497920/407873900 [00:06<00:01, 53998990.49B/s]\n\r 83%|████████▎ | 338375680/407873900 [00:06<00:01, 55348739.93B/s]\n\r 84%|████████▍ | 344238080/407873900 [00:06<00:01, 56138604.34B/s]\n\r 86%|████████▌ | 349871104/407873900 [00:06<00:01, 55777081.28B/s]\n\r 87%|████████▋ | 355463168/407873900 [00:06<00:00, 55622674.56B/s]\n\r 89%|████████▊ | 361146368/407873900 [00:06<00:00, 55978274.55B/s]\n\r 90%|█████████ | 367100928/407873900 [00:07<00:00, 57000834.72B/s]\n\r 91%|█████████▏| 373022720/407873900 [00:07<00:00, 57648025.88B/s]\n\r 93%|█████████▎| 378876928/407873900 [00:07<00:00, 57909567.27B/s]\n\r 94%|█████████▍| 385101824/407873900 [00:07<00:00, 59145393.74B/s]\n\r 96%|█████████▌| 391170048/407873900 [00:07<00:00, 59597600.10B/s]\n\r 97%|█████████▋| 397191168/407873900 [00:07<00:00, 59767010.39B/s]\n\r 99%|█████████▉| 403174400/407873900 [00:07<00:00, 59368825.35B/s]\n\r100%|██████████| 407873900/407873900 [00:07<00:00, 52649944.89B/s]\ncopying /tmp/tmprdr64zmp to cache at /root/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba\ncreating metadata file for /root/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba\nremoving temp file /tmp/tmprdr64zmp\nloading archive file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz from cache at /root/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba\nextracting archive file /root/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba to temp dir /tmp/tmpnyfqapi_\nModel config {\n  \"attention_probs_dropout_prob\": 0.1,\n  \"hidden_act\": \"gelu\",\n  \"hidden_dropout_prob\": 0.1,\n  \"hidden_size\": 768,\n  \"initializer_range\": 0.02,\n  \"intermediate_size\": 3072,\n  \"max_position_embeddings\": 512,\n  \"num_attention_heads\": 12,\n  \"num_hidden_layers\": 12,\n  \"type_vocab_size\": 2,\n  \"vocab_size\": 30522\n}\n\nWeights of BertForTokenClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\nWeights from pretrained model not used in BertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n2019-06-15 13:24:54,982 | azureml.core.run | DEBUG | Could not load run context Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run., switching offline: False\n2019-06-15 13:24:54,983 | azureml.core.run | DEBUG | Could not load the run context and allow_offline set to False\n2019-06-15 13:24:54,983 | azureml.core.model | DEBUG | RunEnvironmentException: Could not load a submitted run, if outside of an execution context, use experiment.start_logging to initialize an azureml.core.Run.\n2019-06-15 13:24:54,983 | azureml.core.model | DEBUG | version is None. Latest version is 3\n2019-06-15 13:24:54,983 | azureml.core.model | DEBUG | Found model path at azureml-models/bert_ner/3/bert_ner.model\nhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache, downloading to /tmp/tmp2sn7rb9t\n\r  0%|          | 0/231508 [00:00<?, ?B/s]\n\r100%|██████████| 231508/231508 [00:00<00:00, 6196056.09B/s]\ncopying /tmp/tmp2sn7rb9t to cache at /root/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\ncreating metadata file for /root/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\nremoving temp file /tmp/tmp2sn7rb9t\nloading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\nUsers's init has completed successfully\nScoring timeout setting is not found. Use default timeout: 3600000 ms\n\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "service = ws.webservices['bert-ner-srvc']",
      "execution_count": 53,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Get the scoring web service's HTTP endpoint, which accepts REST client calls. This endpoint can be shared with anyone who wants to test the web service or integrate it into an application.\n\n"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "print(service.scoring_uri)",
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": "http://6cbb2059-8488-4f6d-bd93-df25d6647efb.eastus.azurecontainer.io/score\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Test deployed service"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import requests\nimport json\n\n# send a random row from the test set to score\ninput_data = \"Microsoft to launch AI Digital labs in India, will bring benefits to 1.5 lakh students.\"\n\nheaders = {'Content-Type':'application/json'}\n\n# for AKS deployment you'd need to the service key in the header as well\n# api_key = service.get_key()\n# headers = {'Content-Type':'application/json',  'Authorization':('Bearer '+ api_key)} \n\nresp = requests.post(service.scoring_uri, input_data, headers=headers)\nprint(\"prediction:\", \" \".join([f\"{t} ({l})\" for t, l in zip(input_data.split(), json.loads(resp.text))]))",
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": "prediction: Microsoft (B-org) to (O) launch (O) AI (B-org) Digital (I-org) labs (I-org) in (O) India, (B-geo) will (O) bring (O) benefits (O) to (O) 1.5 (O) lakh (O) students. (O)\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "# Next Steps\n\nHope you enjoyed this tutorial you should now have what you need to get started fine tuning your own state of the art BERT NER models.\n\nTo learn more about the developement of contextual word embeddings such as BERT check out the second post on my series [Beyond Word Embeddings](https://towardsdatascience.com/beyond-word-embeddings-part-2-word-vectors-nlp-modeling-from-bow-to-bert-4ebd4711d0ec)"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python36",
      "display_name": "Python 3.6",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}