04-Ensemble-learning-Experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "26d1037b-3887-4842-a0c8-eebb9e8da867",
   "metadata": {},
   "source": [
    "### Preamble\n",
    "\n",
    "> _If you've reached this part of the tutorial, you're a **Star**_ 🌟 🙌\n",
    "> \n",
    "> **Thank you!** I hope you're finding this useful, and interesting!\n",
    "> \n",
    "> _Valerio_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac1398ce",
   "metadata": {},
   "source": [
    "# Ensemble learning to combine multiple ML models\n",
    "\n",
    "In the last notebook of our tutorial, we will explore how to use [Ensemble learning](https://en.wikipedia.org/wiki/Ensemble_learning) as a potential strategy to combine multiple machine learning models, trained on different datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53e6b02a-c638-4041-a82b-b64c90066ef6",
   "metadata": {},
   "source": [
    "## Step 1. Login to datasites as **External Researcher**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fa52e68-66f5-4b00-b724-15e3610765fd",
   "metadata": {},
   "source": [
    "⚠️ First verify that the Datasites are already running. If needed, launch the following command in a new terminal session:\n",
    "\n",
    "```bash\n",
    "$ python launch_datasites.py\n",
    "```\n",
    "\n",
    "**Note**: In Jupyter Lab, you can open a new terminal session via `File >> New >> Terminal`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad4acae5-f788-447f-b366-0c6e15f11691",
   "metadata": {},
   "outputs": [],
   "source": [
    "import syft as sy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccf7693a-195a-46b8-9e44-a7fc5c773279",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasites import DATASITE_URLS\n",
    "\n",
    "datasites = {}\n",
    "for name, url in DATASITE_URLS.items():\n",
    "    datasites[name] = sy.login(url=url, email=\"researcher@openmined.org\", password=\"****\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0d36de7-494f-401b-8f95-444cb5375809",
   "metadata": {},
   "source": [
    "## Step 2. Prepare Ensemble Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a725f05-8e3a-4c8c-81fc-80046bcd5936",
   "metadata": {},
   "outputs": [],
   "source": [
    "mock_data = datasites[\"Cleveland Clinic\"].datasets[\"Heart Disease Dataset\"].assets[\"Heart Study Data\"].mock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d5b9ad8-8ba0-4fbf-9c21-20bf90a1bf48",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load pre-trained models\n",
    "from utils import load_models\n",
    "\n",
    "models = load_models(datasites, root=\"./models\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a1bae21-f681-474e-8832-7ab12b0baf96",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Explore \"Voting Classifier\" as Ensemble model strategy  \n",
    "from sklearn.ensemble import VotingClassifier\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "y = mock_data[\"num\"].map(lambda v: 0 if v == 0 else 1)\n",
    "\n",
    "# assign weights based on performance calculated in Experiment 03\n",
    "voting_model = VotingClassifier(estimators=[(name, model) for name, model in models.items()], \n",
    "                                weights=[2,1,0.2,0.5])\n",
    "\n",
    "# workaround to by-pass re-fit: https://stackoverflow.com/a/54610569\n",
    "voting_model.estimators_ = list(models.values())\n",
    "voting_model.le_ = LabelEncoder().fit(y)\n",
    "voting_model.classes_ = voting_model.le_.classes_\n",
    "\n",
    "voting_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b792ad39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# DS/ML libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import matthews_corrcoef as mcc_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "\n",
    "# ML Data preparation - same strategy as in model training\n",
    "def by_demographics(data):\n",
    "    sex = data[\"sex\"].map(lambda v: '0' if v == 0 else '1')\n",
    "    target = data[\"num\"].map(lambda v: '0' if v == 0 else '1')\n",
    "    return (sex+target).values\n",
    "\n",
    "\n",
    "# 1. get features and labels\n",
    "X = mock_data.drop(columns=[\"age\", \"sex\", \"num\"], axis=1)\n",
    "y = mock_data[\"num\"].map(lambda v: 0 if v == 0 else 1)\n",
    "# 2. partition data\n",
    "_, X_test, _, y_test = train_test_split(\n",
    "    X, y, random_state=12345, stratify=by_demographics(mock_data)\n",
    ")\n",
    "y_pred = voting_model.predict(X_test)\n",
    "print(mcc_score(y_test, y_pred))\n",
    "print(confusion_matrix(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dcf3ab9-5efe-4b80-80b8-77073dd67bbd",
   "metadata": {},
   "source": [
    "## Step 3. Submit Experiment to each datasite"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9344b470-4ad8-4e25-894f-29c364ab7899",
   "metadata": {},
   "source": [
    "_Another evaluation experiment, this time using an `EnsembleVotingClassifier` model on each datasite!_\n",
    "\n",
    "> 🙋 **Note**: The code of the `evaluate` function we are going to create is **identical** to the one we used in our previous evaluation experiment (see [`03-Model-Evaluation-Experiment.ipynb`](./03-Model-Evaluation-Experiment.ipynb)).\n",
    ">\n",
    "> It has been replicated here for clarity - but a **better way** to handle this would've been to import the code from a module, and just change the input parameter of the `syft_function_single_use` decorator!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdb3fd07-67e6-4798-845f-4a84416e27b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import serialize_and_upload\n",
    "\n",
    "remote_models = {}\n",
    "for name, datasite in datasites.items():\n",
    "    print(f\"Datasite: {name}\")\n",
    "    # 1. Get data asset from datasite\n",
    "    data_asset = datasite.datasets[\"Heart Disease Dataset\"].assets[\"Heart Study Data\"]\n",
    "    # 1.1 Upload and serialise the Voting classifier to each Datasite\n",
    "    remote_voting_model = serialize_and_upload(model=voting_model, to=datasite)\n",
    "    remote_models[name] = remote_voting_model\n",
    "    \n",
    "    @sy.syft_function_single_use(data=data_asset, model=remote_voting_model)\n",
    "    def evaluate(data, model):\n",
    "        # DS/ML libraries\n",
    "        from sklearn.model_selection import train_test_split\n",
    "        from sklearn.metrics import matthews_corrcoef as mcc_score\n",
    "        from sklearn.metrics import confusion_matrix\n",
    "        import joblib  # to load serialised input model\n",
    "        \n",
    "        # ML Data preparation - same strategy as in model training\n",
    "        def by_demographics(data):\n",
    "            sex = data[\"sex\"].map(lambda v: '0' if v == 0 else '1')\n",
    "            target = data[\"num\"].map(lambda v: '0' if v == 0 else '1')\n",
    "            return (sex+target).values\n",
    "        \n",
    "        # 1. get features and labels\n",
    "        X = data.drop(columns=[\"age\", \"sex\", \"num\"], axis=1)\n",
    "        y = data[\"num\"].map(lambda v: 0 if v == 0 else 1)\n",
    "        # 2. partition data\n",
    "        _, X_test, _, y_test = train_test_split(\n",
    "            X, y, random_state=12345, stratify=by_demographics(data)\n",
    "        )\n",
    "        # 3. Get trained model\n",
    "        classifier = joblib.load(model)  # load model\n",
    "        # 4. Evaluate Metrics (MCC and Confusion Matrix)\n",
    "        y_pred = classifier.predict(X_test)\n",
    "        return mcc_score(y_test, y_pred), confusion_matrix(y_test, y_pred)\n",
    "    \n",
    "    ensemble_ml_project = sy.Project(\n",
    "        name=\"Evaluate Ensable Voting Classifier on Heart Study Data\",\n",
    "        description=\"\"\"This time I would like to evaluate the performance of an\n",
    "        Ensemble Voting Models, using pre-trained models on each datasite.\"\"\",\n",
    "        members=[datasite],\n",
    "    )\n",
    "    ensemble_ml_project.create_code_request(evaluate, datasite)\n",
    "    project = ensemble_ml_project.send()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06e608be-ecf0-42b0-b41c-066b2adffc8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import check_status_last_code_requests\n",
    "\n",
    "check_status_last_code_requests(datasites)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d6ecd12-70e6-4a0d-b497-d0e2fbb7dcac",
   "metadata": {},
   "source": [
    "## Step 4. Evaluate Models on all datasites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dcb4799-1f90-4b51-ba23-45934ab87c76",
   "metadata": {},
   "outputs": [],
   "source": [
    "mcc_scores, confusion_matrices = {}, {}\n",
    "for name, datasite in datasites.items():\n",
    "    print(f\"Datasite: {name}\")\n",
    "    data_asset = datasite.datasets[\"Heart Disease Dataset\"].assets[\"Heart Study Data\"]\n",
    "    remote_model = remote_models[name]\n",
    "    results = datasite.code.evaluate(data=data_asset, model=remote_model).get_from(datasite)\n",
    "    mcc_scores[name], confusion_matrices[name] = results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b12f2cf-ab2a-4e74-a4ce-987fad66488d",
   "metadata": {},
   "source": [
    "**Note**: \n",
    "MCC Classification report of the `RandomForest` classifiers trained independently on each Datasites\n",
    "\n",
    "```\n",
    "'Cleveland Clinic': 0.766,\n",
    "'Hungarian Inst. of Cardiology': 0.46,\n",
    "'Univ. Hospitals Zurich and Basel': 0.0,\n",
    "'V.A. Medical Center': 0.258}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0600ae7",
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, score in mcc_scores.items():\n",
    "    print(f\"{name}: {score:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57142cd3-775f-40ef-a2b0-6212009cec50",
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "from utils import plot_all_confusion_matrices\n",
    "\n",
    "plot_all_confusion_matrices(confusion_matrices)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bfbbce0-923d-42c8-a495-d2576576ffdb",
   "metadata": {},
   "source": [
    "## Conclusions 🎉"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a4cb45f-abdf-45a2-8771-7413c28a3e42",
   "metadata": {},
   "source": [
    "Using an Ensemble of all models, we have been able to _improve_ the results of our classifier on **all** the four datasets by _indirectly_ incorporating the knowledge gathered from each (non-public) dataset, independently!\n",
    "\n",
    "And we did this by **never, ever** _seeing_ the true private data stored in each datasite!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}