Day-4-Practice-Raw.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"view-in-github"},"source":["<a href=\"https://colab.research.google.com/github/stefanlessmann/ASE-ML/blob/main/Day-4-Practice-Raw.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"]},{"cell_type":"markdown","source":["# Wecome to our Section 14 Deep Learning Practice Session\n","Today, we revisit the session on explainable AI (XAI) and interpretable machine learning. We start by re-creating the neural network-based credit risk model from the previous session. Neural networks are famous for being black-box models. Thus, our credit risk model is just a perfect example for a non-explainable model that we can diagnose and eventually interpret using the explanation methods discussed in the session.  \n","\n","The outline of today's session is as follows:\n","- Create black-box neural network classifier (code already available)\n","- Explain the black-box using linear regression surrogate model\n","- Compute permutation-based feature importance\n","- Examine partial dependence"],"metadata":{"id":"grvYBSqLwztN"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"h6vhKogUqLQY"},"outputs":[],"source":["# Standard libraries for data data handling and plotting\n","import numpy as np \n","import seaborn as sns\n","import matplotlib.pyplot as plt\n","import pandas as pd\n","from sklearn.preprocessing import MinMaxScaler"]},{"cell_type":"markdown","source":["Below, we provide the code from the previous practice session, which you can run to load our credit data, train a neural network classifier, and produce test set predictions.\n"],"metadata":{"id":"7ES6apIFFxqd"}},{"cell_type":"code","source":["# Load data and create NN-based credit risk model:\n","\n","# 1. Load data from GitHub\n","data_url = \"https://raw.githubusercontent.com/stefanlessmann/ASE-ML/master/hmeq.csv\"\n","df = pd.read_csv(data_url)\n","\n","# 2. Perform some minimal preprocessing to facilitate training a NN\n","df = df.drop(columns=[\"REASON\", \"JOB\"], axis=1)  # delete categorical features\n","y = df.pop(\"BAD\")  # Remove the target variable so that it remains unchanged\n","df = df.fillna(value=df.mean())  # Mean-impute missing values\n","scaler = MinMaxScaler(feature_range=(0,1))  # Scale feature values\n","X = pd.DataFrame(scaler.fit_transform(df), columns=df.columns)\n","\n","# 3. Partition data into training (70%) and test (30%) sets\n","from sklearn.model_selection import train_test_split\n","Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.3)\n","\n","# 4. Train neural network\n","from sklearn.neural_network import MLPClassifier\n","nn = MLPClassifier(hidden_layer_sizes=(100, 10), learning_rate_init=0.01, max_iter=1000)\n","nn.fit(Xtrain, ytrain)\n","\n","# 5. Compute NN predictions for the test data\n","yhat = nn.predict(Xtest)\n","\n","# Compute a performance indicator over the test set predictions. \n","from sklearn.metrics import ConfusionMatrixDisplay\n","print(\"Classification accuracy: \", nn.score(X=Xtest, y=ytest))\n","ConfusionMatrixDisplay.from_predictions(y_true=ytest, y_pred=yhat, labels=[0, 1], display_labels=['Good(0)', 'Bad(1)']);\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":467},"id":"rJcyttMeFsGC","executionInfo":{"status":"ok","timestamp":1680183653357,"user_tz":-120,"elapsed":5812,"user":{"displayName":"Stefan Lessmann","userId":"06342662613942148717"}},"outputId":"dc58e5ef-75fb-477c-8def-cb80f125f88a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Classification accuracy:  0.9010067114093959\n"]},{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 2 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjMAAAGwCAYAAABcnuQpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLoElEQVR4nO3deVhUZfsH8O8MOICsigJOIoIogntihphLobhk+mqZvVRolKVQLrmWEmpqqblvqeVS+EMrtSQzcckNRCVxA8mFBBewYhOUbeb8/uDl1AkdGWaAM/j9XNe5rnjO85xzDyHePqtCEAQBRERERCZKWdsBEBERERmCyQwRERGZNCYzREREZNKYzBAREZFJYzJDREREJo3JDBEREZk0JjNERERk0sxrO4DHmVarxa1bt2BrawuFQlHb4RARkR4EQcDdu3ehVquhVFZf30BhYSGKi4sNfo5KpYKlpaURIpIfJjO16NatW3B1da3tMIiIyADp6elo2rRptTy7sLAQ7m42yLijMfhZLi4uSE1NrZMJDZOZWmRrawsAuP5rc9jZcMSP6qb/tGpX2yEQVYtSlOAY9oi/y6tDcXExMu5ocD2hOexsq/73RN5dLdw6/47i4mImM2Rc5UNLdjZKg35IieTMXFGvtkMgqh7/OwyoJqYJ2NgqYGNb9fdoUbenMjCZISIikjmNoIXGgJMUNYLWeMHIEJMZIiIimdNCgBZVz2YMaWsKOLZBREREJo09M0RERDKnhRaGDBQZ1lr+mMwQERHJnEYQoBGqPlRkSFtTwGEmIiIiMmnsmSEiIpI5TgDWjckMERGRzGkhQMNk5qE4zEREREQmjT0zREREMsdhJt2YzBAREckcVzPpxmEmIiIiMmnsmSEiIpI57f8uQ9rXZUxmiIiIZE5j4GomQ9qaAiYzREREMqcRYOCp2caLRY44Z4aIiIhMGntmiIiIZI5zZnRjMkNERCRzWiiggcKg9nUZh5mIiIjIpLFnhoiISOa0QtllSPu6jMkMERGRzGkMHGYypK0p4DATERERmTT2zBAREckce2Z0YzJDREQkc1pBAa1gwGomA9qaAg4zERERkUljzwwREZHMcZhJNyYzREREMqeBEhoDBlM0RoxFjpjMEBERyZxg4JwZgXNmiIiIiOSLPTNEREQyxzkzujGZISIikjmNoIRGMGDOTB0/zoDDTERERGTS2DNDREQkc1oooDWg/0GLut01w2SGiIhI5jhnRjcOMxEREZFJY88MERGRzBk+AZjDTERERFSLyubMGHDQJIeZiIiIiOSLPTNEREQypzXwbCauZiIiIqJaxTkzujGZISIikjktlNxnRgfOmSEiIiKTxp4ZIiIimdMICmgEAzbNM6CtKWAyQ0REJHMaAycAazjMRERERCRf7JkhIiKSOa2ghNaA1UzaOr6aiT0zREREMlc+zGTIpa8jR45g0KBBUKvVUCgU2LVrl3ivpKQEU6dORbt27WBtbQ21Wo3XX38dt27dkjwjKysLQUFBsLOzg4ODA0JCQpCfny+pc+7cOTzzzDOwtLSEq6srFixYoHesTGaIiIiogoKCAnTo0AGrVq2qcO/evXv49ddfMXPmTPz666/YsWMHUlJS8MILL0jqBQUF4eLFi4iJiUF0dDSOHDmC0aNHi/fz8vLQt29fuLm5ISEhAQsXLkRERATWrVunV6wcZiIiIpI5LQxbkaStQpv+/fujf//+D7xnb2+PmJgYSdnKlSvx1FNPIS0tDc2aNUNycjL27t2LU6dOwdfXFwCwYsUKDBgwAIsWLYJarUZkZCSKi4vx5ZdfQqVSoU2bNkhMTMTixYslSc+jsGeGiIhI5so3zTPkAsp6Qv55FRUVGS3G3NxcKBQKODg4AADi4uLg4OAgJjIAEBAQAKVSifj4eLFOjx49oFKpxDqBgYFISUlBdnZ2pd/NZIaIiOgx4erqCnt7e/GaP3++UZ5bWFiIqVOn4pVXXoGdnR0AICMjA05OTpJ65ubmaNiwITIyMsQ6zs7OkjrlX5fXqQwOMxEREcmc4WczlbVNT08Xkw0AsLCwMDi2kpISDB8+HIIgYM2aNQY/ryqYzBAREcmcFgpoYcicmbK2dnZ2kmTGUOWJzPXr13Hw4EHJs11cXHDnzh1J/dLSUmRlZcHFxUWsk5mZKalT/nV5ncrgMBMREZHMlffMGHIZW3kic/nyZezfvx+Ojo6S+35+fsjJyUFCQoJYdvDgQWi1WnTt2lWsc+TIEZSUlIh1YmJi4OXlhQYNGlQ6FiYzREREVEF+fj4SExORmJgIAEhNTUViYiLS0tJQUlKCF198EadPn0ZkZCQ0Gg0yMjKQkZGB4uJiAIC3tzf69euHt956CydPnsTx48cRFhaGESNGQK1WAwD++9//QqVSISQkBBcvXsS2bduwbNkyTJw4Ua9YOcxEREQkc4afzaR/29OnT6N3797i1+UJRnBwMCIiIvDDDz8AADp27Chpd+jQIfTq1QsAEBkZibCwMDz33HNQKpUYNmwYli9fLta1t7fHvn37EBoais6dO6NRo0YIDw/Xa1k2wGSGiIhI9rSCAlpD9pmpQttevXpB0HEMgq575Ro2bIitW7fqrNO+fXscPXpU7/j+icNMREREZNLYM0NERCRzWgOHmbR1vO+CyQwREZHMGX5qdt1OZur2pyMiIqI6jz0zREREMqeBAhoDNs0zpK0pYDJDREQkcxxm0q1ufzoiIiKq89gzQ0REJHMaGDZUpDFeKLLEZIaIiEjmOMykG5MZIiIimTP0sMjqOGhSTur2pyMiIqI6jz0zREREMidAAa0Bc2YELs0mIiKi2sRhJt3q9qcjIiKiOo89M0RERDKnFRTQClUfKjKkrSlgMkNERCRzGgNPzTakrSmo25+OiIiI6jz2zBAREckch5l0YzJDREQkc1oooTVgMMWQtqagbn86IiIiqvPYM0NERCRzGkEBjQFDRYa0NQVMZoiIiGSOc2Z0YzJDREQkc4KBp2YL3AGYiIiISL7YM0NERCRzGiigMeCwSEPamgImM0RERDKnFQyb96IVjBiMDHGYiYiIiEwae2bIpJw/YY1vVjvh8vn6yMqsh4++SEW3/rni/a8WueCX7x3wx616qKcS4NnuPkZNu43WT94T61w+Z4Uv5qrx29n6UJoJ6D4gB29H3IKVtVbyrn3bGmLHusa4cc0C9W006PF8DsLm36yxz0r0IC+HZcJ/QC5cPYtQXKhE0un6+GJuE9y4ainWadC4BG/OvI0ne9xFfRst0q9aIGqZE47tcai9wMkgWgMnABvS1hTU7U/3CAqFArt27ZKUffHFF+jbt69ez3n66afx3XffGTEyepjCe0p4tLmPsHk3Hnj/CY9ChM69gc8PpuCzXVfg4lqM6a+0QM5fZgCAvzLMMW1EC6jdi7As+jfMjbyK6ymWWDS+meQ5333eGJs+dcHw0EysO3QJn2y7is697lb75yN6lPZ+Bdi9qRHGP98S00d4wMxcwLz/uwYLK41YZ/LyNLi2KETESHe8/WwrHN9jjw8+v44Wbe/peDLJmRYKg6+6rNaTmYyMDIwbNw6enp6wtLSEs7Mz/P39sWbNGty7V7N/8AoLCzFz5kx89NFHkvJvvvkGrVu3hqWlJdq1a4c9e/ZI7s+YMQPTpk2DViv9lz0ZX5dn72Lk1Az4/6M35p+eHZqDJ3vko4lbMZp7FWJ0xE3cu2uG1CQrAED8fnuYmwsIm3cDrp5F8Op4H+99egPHfnTAzVQVAOBujhk2f9oEk5el4dmhOVA3L4aHTyH8AvNq7HMSPcyHQR6I2d4Q13+zxLUkK3w2vhmcm5agZfv7Yh0f33v4/stGSEmsj4w0C/zfMmcU5JpJ6hDVJbWazFy7dg2dOnXCvn37MG/ePJw5cwZxcXGYMmUKoqOjsX///hqN59tvv4WdnR38/f3FstjYWLzyyisICQnBmTNnMGTIEAwZMgQXLlwQ6/Tv3x93797FTz/9VKPxkm4lxQrs+doR1nYaePiU/RIvKVLAvJ4A5T9+8lWWZUnoxZM2AIBfj9hCKwB/ZtTDmz1aI6izDz5+2w13btar8c9A9CjWdmU9MndzzMSypNP10fOFHNg6lEKhENBzcDZUlgLOxdrUVphkoPIdgA256rJaTWbGjh0Lc3NznD59GsOHD4e3tzc8PDwwePBg/Pjjjxg0aBAAIC0tDYMHD4aNjQ3s7OwwfPhwZGZmSp61Zs0atGjRAiqVCl5eXvjqq68k9y9fvowePXrA0tISPj4+iImJqRBPVFSU+M5yy5YtQ79+/TB58mR4e3tjzpw5ePLJJ7Fy5UqxjpmZGQYMGICoqChjfWvIACdi7DDYsx0GubfHzvWNMT/qCuwdy37hd+iej+w/6uGb1Y1RUqzA3RwzfDlPDQDIulM2hSzjugqCFoha7ox3Zt/EjHW/4262OaaPaIGS4rr9C4FMi0Ih4J1ZN3HhZH1cT7ESy+e+3Rxm9QR8m3QR0b+fw7hPb2BWSHPc+t2iFqMlQ5TPmTHkqstq7dP99ddf2LdvH0JDQ2Ftbf3AOgqFAlqtFoMHD0ZWVhYOHz6MmJgYXLt2DS+//LJYb+fOnRg3bhzef/99XLhwAW+//TZGjRqFQ4cOAQC0Wi2GDh0KlUqF+Ph4rF27FlOnTq3wvmPHjsHX11dSFhcXh4CAAElZYGAg4uLiJGVPPfUUjh49qvMzFxUVIS8vT3KR8XX0z8fqmBQs+eEyfHvdxdy3myPnz7JEpblXISYtvY7vPnfCCy3a45WObeDiWowGjUug+F+eohWA0hIlxs65Cd9ed+Hd+R6mr/kdt1ItcJb/siUZCZt3E26tCzF/jJukPHjKbdjYaTF1uAfe7d8K361rjA/X/o7mrTnMRHVTra1munLlCgRBgJeXl6S8UaNGKCwsBACEhoYiICAA58+fR2pqKlxdXQEAW7ZsQZs2bXDq1Cl06dIFixYtwsiRIzF27FgAwMSJE3HixAksWrQIvXv3xv79+3Hp0iX8/PPPUKvL/hU+b9489O/fX3xvTk4OcnNzxfvlMjIy4OzsLClzdnZGRkaGpEytViM9PR1arRZK5YNzxPnz52PWrFn6fqtIT5b1tXjCvRhPuBfDu/M9jPL3xt7/a4gR794BUDav5tmhOcj+wxyW9bVQKIAd6xqjiVsRAKChUykAoFmrQvGZDo4a2DUs5VATyUbo3Bvo2icP7/+nBf68rRLLm7gVYfAbf2F0Ly9c/61shdO1JCu061qAF0b+heXTmtZWyGQALQw8m4kTgGvWyZMnkZiYiDZt2qCoqAjJyclwdXUVExkA8PHxgYODA5KTkwEAycnJknkuAODv7y+57+rqKklU/Pz8JPXv3y/7F4ulpSWqwsrKClqtFkVFRQ+tM336dOTm5opXenp6ld5F+hG0QElRxR/1Bo1LYWWtxeHvHVDPQosne+QDANp0KQAA3Lj6d5d8XrYZ8rLM4fxESc0ETfRQAkLn3kC3frmY8lILZKZLh44srMrmgP17PYJGAyiUdXzntDpMMHAlk1DHk5la65nx9PSEQqFASkqKpNzDwwNAWXJQkxwdHaFQKJCdnS0pd3FxqTA/JzMzEy4uLpKyrKwsWFtb64zbwsICFhYcszbE/QIlbqX+/T3MSFfh6gUr2DqUwq6hBluXOcOvby4aOpcgL8scP2xshD8z6uGZQTlim++/bAQf3wJYWWvx6xFbbJijxhsf3IKNfdm8mqYtiuAXmIs14U9g3IJ0WNtq8eW8JmjqWYgO/lyeTbUrbN5N9P5PNiJGueN+vhINGpcl2AV3zVBcqET6FUvcvKbCuAU3sH62GnnZZujWLxdP9shH+OvutRw9VRVPzdat1npmHB0d0adPH6xcuRIFBQUPreft7Y309HRJL0ZSUhJycnLg4+Mj1jl+/Lik3fHjxyX309PTcfv2bfH+iRMnJPVVKhV8fHyQlJQkKffz88OBAwckZTExMRV6di5cuIBOnTo96mOTgX47Wx9j+3phbN+y4cnPI57A2L5e2LKoCZRKATeuWGDOW80R0t0b4cEeyMs2x2c7L6O5199DRimJ9TH9lRZ45zkv7Il0xHsL0jHkzT8l75m8/Dpad7qH8Nc9MGmYJ8zrCZgbeQ3mHGWiWjZo5F+wsddi0Y6riDqbJF49X8gBAGhKFZjxmgdy/zLHrM2pWHvgNwS8mI1F41xx6qBd7QZPVE1qdQfg1atXw9/fH76+voiIiED79u2hVCpx6tQpXLp0CZ07d0ZAQADatWuHoKAgLF26FKWlpRg7dix69uwpTtadPHkyhg8fjk6dOiEgIAC7d+/Gjh07xKXdAQEBaNWqFYKDg7Fw4ULk5eXhww8/rBBPYGAgjh07hvHjx4tl48aNQ8+ePfHZZ59h4MCBiIqKwunTp7Fu3TpJ26NHj+q92R7pr0O3fPx8K/Gh98O/+P2Rz5iyPO2RdaxttZi4OB0TF3MokOQlUN3hkXVupZYl9VR3cAdg3Wr107Vo0QJnzpxBQEAApk+fjg4dOsDX1xcrVqzApEmTMGfOHCgUCnz//fdo0KABevTogYCAAHh4eGDbtm3ic4YMGYJly5Zh0aJFaNOmDT7//HNs3LgRvXr1AgAolUrs3LkT9+/fx1NPPYU333wTc+fOrRBPSEgI9uzZg9zcvzdk69atG7Zu3Yp169ahQ4cO+Pbbb7Fr1y60bdtWrHPz5k3ExsZi1KhR1ffNIiKix1b5MJMhV12mEASBM8L+4aWXXsKTTz6J6dOnV7rN1KlTkZ2dXaG35lHy8vJgb2+P7N88YGdbt7NmenwFqjvWdghE1aJUKMEv+B65ubmws6ueIbzyvycG73sD9axVj27wECUFxfi+75fVGmtt4t+g/7Jw4ULY2Oi3l4iTkxPmzJlTTREREdHjjmcz6cZTs/+lefPmePfdd/Vq8/7771dTNERERFzN9CjsmSEiIiKTxmSGiIhI5mpjAvCRI0cwaNAgqNVqKBQK7Nq1S3JfEASEh4ejSZMmsLKyQkBAAC5fviypk5WVhaCgINjZ2cHBwQEhISHIz8+X1Dl37hyeeeYZWFpawtXVFQsWLNA7ViYzREREMlcbyUxBQQE6dOiAVatWPfD+ggULsHz5cqxduxbx8fGwtrZGYGCgeCQRAAQFBeHixYuIiYlBdHQ0jhw5gtGjR4v38/Ly0LdvX7i5uSEhIQELFy5ERESE3gtqOGeGiIiIKujfv7/kDMN/EgQBS5cuxYwZMzB48GAAZecmOjs7Y9euXRgxYgSSk5Oxd+9enDp1StwXbsWKFRgwYAAWLVoEtVqNyMhIFBcX48svv4RKpUKbNm2QmJiIxYsXS5KeR2HPDBERkcwZq2cmLy9Pcuk6T1CX1NRUZGRkICAgQCyzt7dH165dERcXBwCIi4uDg4ODmMgAZZvYKpVKxMfHi3V69OgBlervZeeBgYFISUmpcLyQLkxmiIiIZE6AYcuzyzeUc3V1hb29vXjNnz+/SvFkZGQAAJydnSXlzs7O4r2MjAw4OTlJ7pubm6Nhw4aSOg96xj/fURkcZiIiIpI5Yy3NTk9Pl2yaV1cOP2bPDBER0WPCzs5OclU1mXFxcQEAZGZmSsozMzPFey4uLrhz547kfmlpKbKysiR1HvSMf76jMpjMEBERyZzczmZyd3eHi4sLDhw4IJbl5eUhPj4efn5+AAA/Pz/k5OQgISFBrHPw4EFotVp07dpVrHPkyBGUlJSIdWJiYuDl5YUGDRpUOh4mM0RERDJXG8lMfn4+EhMTkZiYCKBs0m9iYiLS0tKgUCgwfvx4fPzxx/jhhx9w/vx5vP7661Cr1RgyZAgAwNvbG/369cNbb72FkydP4vjx4wgLC8OIESOgVqsBAP/973+hUqkQEhKCixcvYtu2bVi2bBkmTpyoV6ycM0NEREQVnD59Gr179xa/Lk8wgoODsWnTJkyZMgUFBQUYPXo0cnJy0L17d+zduxeWlpZim8jISISFheG5556DUqnEsGHDsHz5cvG+vb099u3bh9DQUHTu3BmNGjVCeHi4XsuyAZ6aXat4ajY9DnhqNtVVNXlqdvcfQmFuXfXJuqUFRTj2wqo6e2o2e2aIiIhkThAUEAyY92JIW1PA7gAiIiIyaeyZISIikrnyze8MaV+XMZkhIiKSOWNtmldXcZiJiIiITBp7ZoiIiGSOE4B1YzJDREQkcxxm0o3JDBERkcyxZ0Y3zpkhIiIik8aeGSIiIpkTDBxmqus9M0xmiIiIZE4AYMjhQ3X93CIOMxEREZFJY88MERGRzGmhgII7AD8UkxkiIiKZ42om3TjMRERERCaNPTNEREQypxUUUHDTvIdiMkNERCRzgmDgaqY6vpyJw0xERERk0tgzQ0REJHOcAKwbkxkiIiKZYzKjG5MZIiIimeMEYN04Z4aIiIhMGntmiIiIZI6rmXRjMkNERCRzZcmMIXNmjBiMDHGYiYiIiEwae2aIiIhkjquZdGMyQ0REJHPC/y5D2tdlHGYiIiIik8aeGSIiIpnjMJNuTGaIiIjkjuNMOjGZISIikjsDe2ZQx3tmOGeGiIiITBp7ZoiIiGSOOwDrxmSGiIhI5jgBWDcOMxEREZFJY88MERGR3AkKwybx1vGeGSYzREREMsc5M7pxmImIiIhMGntmiIiI5I6b5unEZIaIiEjmuJpJt0olMz/88EOlH/jCCy9UORgiIiIifVUqmRkyZEilHqZQKKDRaAyJh4iIiB6kjg8VGaJSyYxWq63uOIiIiOghOMykm0GrmQoLC40VBxERET2MYISrDtM7mdFoNJgzZw6eeOIJ2NjY4Nq1awCAmTNn4osvvjB6gERERFSzNBoNZs6cCXd3d1hZWaFFixaYM2cOhH9sWCMIAsLDw9GkSRNYWVkhICAAly9fljwnKysLQUFBsLOzg4ODA0JCQpCfn2/0ePVOZubOnYtNmzZhwYIFUKlUYnnbtm2xYcMGowZHREREAKAwwlV5n376KdasWYOVK1ciOTkZn376KRYsWIAVK1aIdRYsWIDly5dj7dq1iI+Ph7W1NQIDAyWjNkFBQbh48SJiYmIQHR2NI0eOYPTo0VX+LjyM3snMli1bsG7dOgQFBcHMzEws79ChAy5dumTU4IiIiAhGG2bKy8uTXEVFRQ98XWxsLAYPHoyBAweiefPmePHFF9G3b1+cPHmyLBxBwNKlSzFjxgwMHjwY7du3x5YtW3Dr1i3s2rULAJCcnIy9e/diw4YN6Nq1K7p3744VK1YgKioKt27dMuq3R+9k5ubNm/D09KxQrtVqUVJSYpSgiIiIyPhcXV1hb28vXvPnz39gvW7duuHAgQP47bffAABnz57FsWPH0L9/fwBAamoqMjIyEBAQILaxt7dH165dERcXBwCIi4uDg4MDfH19xToBAQFQKpWIj4836ufSe9M8Hx8fHD16FG5ubpLyb7/9Fp06dTJaYERERPQ/RtoBOD09HXZ2dmKxhYXFA6tPmzYNeXl5aN26NczMzKDRaDB37lwEBQUBADIyMgAAzs7OknbOzs7ivYyMDDg5OUnum5ubo2HDhmIdY9E7mQkPD0dwcDBu3rwJrVaLHTt2ICUlBVu2bEF0dLRRgyMiIiIY7dRsOzs7STLzMNu3b0dkZCS2bt2KNm3aIDExEePHj4darUZwcHDV46gmeg8zDR48GLt378b+/fthbW2N8PBwJCcnY/fu3ejTp091xEhEREQ1aPLkyZg2bRpGjBiBdu3a4bXXXsOECRPEYSkXFxcAQGZmpqRdZmameM/FxQV37tyR3C8tLUVWVpZYx1iqdDbTM888g5iYGKMGQkRERA8mCGWXIe31ce/ePSiV0v4OMzMzcRNdd3d3uLi44MCBA+jYsSOAssnF8fHxGDNmDADAz88POTk5SEhIQOfOnQEABw8ehFarRdeuXav+YR6gygdNnj59GsnJyQDK5tGUB0pERERGVsOnZg8aNAhz585Fs2bN0KZNG5w5cwaLFy/GG2+8AaDs+KLx48fj448/RsuWLeHu7o6ZM2dCrVaLRyB5e3ujX79+eOutt7B27VqUlJQgLCwMI0aMgFqtNuDDVKR3MnPjxg288sorOH78OBwcHAAAOTk56NatG6KiotC0aVOjBkhEREQ1a8WKFZg5cybGjh2LO3fuQK1W4+2330Z4eLhYZ8qUKSgoKMDo0aORk5OD7t27Y+/evbC0tBTrREZGIiwsDM899xyUSiWGDRuG5cuXGz1ehSDo1/nUr18/5OTkYPPmzfDy8gIApKSkYNSoUbCzs8PevXuNHmRdlZeXB3t7e2T/5gE7W4NOliCSrUB1x9oOgahalAol+AXfIzc3t1KTaqui/O+JpstnQ2ll+egGD6G9X4gb74VXa6y1Se+emcOHDyM2NlZMZADAy8sLK1aswDPPPGPU4IiIiAhQCGWXIe3rMr2TGVdX1wdujqfRaIw+BkZERESo8TkzpkbvsY2FCxfi3XffxenTp8Wy06dPY9y4cVi0aJFRgyMiIiJ6lEr1zDRo0AAKxd+b9RQUFKBr164wNy9rXlpaCnNzc7zxxhviLGYiIiIyEiNtmldXVSqZWbp0aTWHQURERA/FYSadKpXMyHHrYiIiIiLAgE3zAKCwsBDFxcWSsrq45IuIiKhWsWdGJ70nABcUFCAsLAxOTk6wtrZGgwYNJBcREREZmWCEqw7TO5mZMmUKDh48iDVr1sDCwgIbNmzArFmzoFarsWXLluqIkYiIiOih9B5m2r17N7Zs2YJevXph1KhReOaZZ+Dp6Qk3NzdERkYiKCioOuIkIiJ6fHE1k05698xkZWXBw8MDQNn8mKysLABA9+7dceTIEeNGR0REROIOwIZcdZneyYyHhwdSU1MBAK1bt8b27dsBlPXYlB88SURERFRT9E5mRo0ahbNnzwIApk2bhlWrVsHS0hITJkzA5MmTjR4gERHRY48TgHXSe87MhAkTxP8OCAjApUuXkJCQAE9PT7Rv396owRERERE9ikH7zACAm5sb3NzcjBELERERPYACBp6abbRI5KlSyczy5csr/cD33nuvysEQERER6atSycySJUsq9TCFQsFkpgpeCugHc6VFbYdBVC2U7a1rOwSiaqHUFAEXauhlXJqtU6WSmfLVS0RERFQLeJyBTnqvZiIiIiKSE4MnABMREVE1Y8+MTkxmiIiIZM7QXXy5AzARERGRjLFnhoiISO44zKRTlXpmjh49ildffRV+fn64efMmAOCrr77CsWPHjBocERERgccZPILeycx3332HwMBAWFlZ4cyZMygqKgIA5ObmYt68eUYPkIiIiEgXvZOZjz/+GGvXrsX69etRr149sdzf3x+//vqrUYMjIiKivycAG3LVZXrPmUlJSUGPHj0qlNvb2yMnJ8cYMREREdE/cQdgnfTumXFxccGVK1cqlB87dgweHh5GCYqIiIj+gXNmdNI7mXnrrbcwbtw4xMfHQ6FQ4NatW4iMjMSkSZMwZsyY6oiRiIiI6KH0HmaaNm0atFotnnvuOdy7dw89evSAhYUFJk2ahHfffbc6YiQiInqscdM83fROZhQKBT788ENMnjwZV65cQX5+Pnx8fGBjY1Md8RERERH3mdGpypvmqVQq+Pj4GDMWIiIiIr3pncz07t0bCsXDZ0UfPHjQoICIiIjoXwxdXs2eGamOHTtKvi4pKUFiYiIuXLiA4OBgY8VFRERE5TjMpJPeycySJUseWB4REYH8/HyDAyIiIiLSh9FOzX711Vfx5ZdfGutxREREVI77zOhktFOz4+LiYGlpaazHERER0f9wabZueiczQ4cOlXwtCAJu376N06dPY+bMmUYLjIiIiKgy9E5m7O3tJV8rlUp4eXlh9uzZ6Nu3r9ECIyIiIqoMvZIZjUaDUaNGoV27dmjQoEF1xURERET/xNVMOuk1AdjMzAx9+/bl6dhEREQ1qHzOjCFXXab3aqa2bdvi2rVr1RELERERkd70TmY+/vhjTJo0CdHR0bh9+zby8vIkFxEREVUDLst+qEonM7Nnz0ZBQQEGDBiAs2fP4oUXXkDTpk3RoEEDNGjQAA4ODpxHQ0REVB1qYZ+Zmzdv4tVXX4WjoyOsrKzQrl07nD59+u+QBAHh4eFo0qQJrKysEBAQgMuXL0uekZWVhaCgINjZ2cHBwQEhISHVssFupScAz5o1C++88w4OHTpk9CCIiIhIPrKzs+Hv74/evXvjp59+QuPGjXH58mVJp8WCBQuwfPlybN68Ge7u7pg5cyYCAwORlJQk7jsXFBSE27dvIyYmBiUlJRg1ahRGjx6NrVu3GjXeSiczglCW1vXs2dOoARAREZFuxto079/TQSwsLGBhYVGh/qeffgpXV1ds3LhRLHN3dxf/WxAELF26FDNmzMDgwYMBAFu2bIGzszN27dqFESNGIDk5GXv37sWpU6fg6+sLAFixYgUGDBiARYsWQa1WV/0D/Ytec2Z0nZZNRERE1cRIw0yurq6wt7cXr/nz5z/wdT/88AN8fX3x0ksvwcnJCZ06dcL69evF+6mpqcjIyEBAQIBYZm9vj65duyIuLg5A2ckADg4OYiIDAAEBAVAqlYiPjzfCN+Vveu0z06pVq0cmNFlZWQYFRERERNUjPT0ddnZ24tcP6pUBgGvXrmHNmjWYOHEiPvjgA5w6dQrvvfceVCoVgoODkZGRAQBwdnaWtHN2dhbvZWRkwMnJSXLf3NwcDRs2FOsYi17JzKxZsyrsAExERETVy1jDTHZ2dpJk5mG0Wi18fX0xb948AECnTp1w4cIFrF27FsHBwVUPpJrolcyMGDGiQpZFRERE1ayGdwBu0qQJfHx8JGXe3t747rvvAAAuLi4AgMzMTDRp0kSsk5mZiY4dO4p17ty5I3lGaWkpsrKyxPbGUuk5M5wvQ0RE9Hjw9/dHSkqKpOy3336Dm5sbgLLJwC4uLjhw4IB4Py8vD/Hx8fDz8wMA+Pn5IScnBwkJCWKdgwcPQqvVomvXrkaNV+/VTERERFTDarhnZsKECejWrRvmzZuH4cOH4+TJk1i3bh3WrVsHoKyDY/z48fj444/RsmVLcWm2Wq3GkCFDAJT15PTr1w9vvfUW1q5di5KSEoSFhWHEiBFGXckE6JHMaLVao76YiIiIKsdYc2Yqq0uXLti5cyemT5+O2bNnw93dHUuXLkVQUJBYZ8qUKSgoKMDo0aORk5OD7t27Y+/eveIeMwAQGRmJsLAwPPfcc1AqlRg2bBiWL19e9Q/yEAqBXS61Ji8vD/b29ghoNhbmygfPKCcydVp769oOgahalGqKcPDCQuTm5lZqUm1VlP894TV+HswsLB/d4CE0RYVIWfpBtcZam/Q+m4mIiIhITvRazURERES1oIbnzJgaJjNEREQyV9NzZkwNh5mIiIjIpLFnhoiISO44zKQTkxkiIiKZ4zCTbhxmIiIiIpPGnhkiIiK54zCTTkxmiIiI5I7JjE4cZiIiIiKTxp4ZIiIimVP87zKkfV3GZIaIiEjuOMykE5MZIiIimePSbN04Z4aIiIhMGntmiIiI5I7DTDoxmSEiIjIFdTwhMQSHmYiIiMiksWeGiIhI5jgBWDcmM0RERHLHOTM6cZiJiIiITBp7ZoiIiGSOw0y6MZkhIiKSOw4z6cRhJiIiIjJp7JkhIiKSOQ4z6cZkhoiISO44zKQTkxkiIiK5YzKjE+fMEBERkUljzwwREZHMcc6MbkxmiIiI5I7DTDpxmImIiIhMGntmiIiIZE4hCFAIVe9eMaStKWAyQ0REJHccZtKJw0xERERk0tgzQ0REJHNczaQbkxkiIiK54zCTThxmIiIiIpPGnhkiIiKZ4zCTbkxmiIiI5I7DTDoxmSEiIpI59szoxjkzREREZNLYM0NERCR3HGbSickMERGRCajrQ0WG4DATERERmTQmM0RERHInCIZfBvjkk0+gUCgwfvx4saywsBChoaFwdHSEjY0Nhg0bhszMTEm7tLQ0DBw4EPXr14eTkxMmT56M0tJSg2J5ECYzREREMle+msmQq6pOnTqFzz//HO3bt5eUT5gwAbt378Y333yDw4cP49atWxg6dKh4X6PRYODAgSguLkZsbCw2b96MTZs2ITw8vOrBPASTGSIiosdEXl6e5CoqKtJZPz8/H0FBQVi/fj0aNGgglufm5uKLL77A4sWL8eyzz6Jz587YuHEjYmNjceLECQDAvn37kJSUhK+//hodO3ZE//79MWfOHKxatQrFxcVG/VxMZoiIiOROMMIFwNXVFfb29uI1f/58na8NDQ3FwIEDERAQIClPSEhASUmJpLx169Zo1qwZ4uLiAABxcXFo164dnJ2dxTqBgYHIy8vDxYsXq/iNeDCuZiIiIpI5hbbsMqQ9AKSnp8POzk4st7CweGibqKgo/Prrrzh16lSFexkZGVCpVHBwcJCUOzs7IyMjQ6zzz0Sm/H75PWNiMkNERPSYsLOzkyQzD5Oeno5x48YhJiYGlpaWNRCZYZjMkElr0/EvDAu6Ck+vXDg2LsKcqb44ccTlgXVDp5zDgP+kYd1SH3y/zUMsb9EqF6NCk9HSOwdarQKxh5pg/XIfFN7nHw+qfcNfToK//w00bXoXxcVmSEpqhC+/bI+bN/7+C6l//6vo1fs6PFtko751KV4c9h8UFKgkz9m0eTecne9Jyr78sj2+2e5dI5+DDFTDm+YlJCTgzp07ePLJJ8UyjUaDI0eOYOXKlfj5559RXFyMnJwcSe9MZmYmXFzKfge7uLjg5MmTkueWr3Yqr2Msj+2cmYiICHTs2FFSVlxcDE9PT8TGxlb6OUlJSWjatCkKCgqMHCFVhqWlBqmX7bDms7Y66/n1vI3WbXLw5x/SLtWGjQoxd8UJ3LphjYlvdkf4hK5o5nEXE2YkVmPURJXXrt0f2L27JSZMCMAH03vC3FyLuXMPw8Li7+WtFhalOH26CaK2+eh81pYtbfHfV14Qrx++b1nd4ZOR1PRqpueeew7nz59HYmKiePn6+iIoKEj873r16uHAgQNim5SUFKSlpcHPzw8A4Ofnh/Pnz+POnTtinZiYGNjZ2cHHR/fPqr5kn8yMHDkSCoVCvBwdHdGvXz+cO3fO6O9au3Yt3N3d0a1bN7Fs7ty56NatG+rXr19hbBAAfHx88PTTT2Px4sVGj4ceLeGEE75a1xpxh5s8tI5j4/t4Z+JFLIzoBE2p9Ef+Kf9MlJYqsGZRW9xMs8HlZAesXNAO3Z/NQJOmTFCp9s2c0RP7Y9yRdt0eqakNsPizp+DsfA8tW2aJdXbt8sI3271x6ZKjzmfdv2eO7Gwr8SoqYu+jyajhfWZsbW3Rtm1byWVtbQ1HR0e0bdsW9vb2CAkJwcSJE3Ho0CEkJCRg1KhR8PPzw9NPPw0A6Nu3L3x8fPDaa6/h7Nmz+PnnnzFjxgyEhobqnKtTFbJPZgCgX79+uH37Nm7fvo0DBw7A3Nwczz//vFHfIQgCVq5ciZCQEEl5cXExXnrpJYwZM+ahbUeNGoU1a9ZUy0ZAZBiFQsD74Yn4LtIDaam2Fe7Xq6dFaYkSgqAQy4qLzAAAbdpnVahPVNvq1y8BANy9q3pEzYpeGn4J27bvxMqVP2PYi5egVBowo5Qee0uWLMHzzz+PYcOGoUePHnBxccGOHTvE+2ZmZoiOjoaZmRn8/Pzw6quv4vXXX8fs2bONHotJpOUWFhaSMbhp06bhmWeewR9//IHGjRtj6tSp2LlzJ27cuAEXFxcEBQUhPDwc9erVE5/xySefYMmSJbh37x6GDx+Oxo0bS96RkJCAq1evYuDAgZLyWbNmAQA2bdr00Pj69OmDrKwsHD58GM8999xD6xUVFUnW9Ofl5VX6e0BV8+JrV6HRKPDDdvcH3j+b0AhvjkvC0KCr+GGbOyytSjFyTDIAoEEj3fsvENU0hULA2++cwcWLjXD9uoNebb//vhWuXGmAu3dV8PH+EyNHnUPDhvexfl2n6gmWjMrQje+Mca7TL7/8Ivna0tISq1atwqpVqx7axs3NDXv27DH85Y9gEj0z/5Sfn4+vv/4anp6ecHQs61K1tbXFpk2bkJSUhGXLlmH9+vVYsmSJ2Gb79u2IiIjAvHnzcPr0aTRp0gSrV6+WPPfo0aNo1aoVbG0r/uv9UVQqFTp27IijR4/qrDd//nzJ+n5XV1e930WV5+mVg8HDU7Hk444AFA+sk5Zqi8VzOmLoK9ew49BP+Dp6PzJv10f2XxYQ+I9WkpnQ0AQ0b56LT+b76d125w4vnD/nhN9THbBnjyc2rO+IF164jHr1NNUQKRmdkfaZqatMomcmOjoaNjY2AICCggI0adIE0dHRUCrLcrEZM2aIdZs3b45JkyYhKioKU6ZMAQAsXboUISEh4hDSxx9/jP3796OwsFBsd/36dajV6irHqFarcf36dZ11pk+fjokTJ4pf5+XlMaGpRm06ZsG+QRE27fx7gpqZuYCQd5Mw+OVUvDG0rBft8L4ncHjfE3BoUITCQjMIAjBkxDVk3KpfW6ETVTBmbAKe6noLkyc9iz//NPxn81KKI8zNBTg5F0hWRhGZIpNIZnr37o01a9YAALKzs7F69Wr0798fJ0+ehJubG7Zt24bly5fj6tWryM/PR2lpqWQdfXJyMt555x3JM/38/HDo0CHx6/v37xu0lt7Kygr37t3TWcfCwsLok57o4Q7+1BSJpxpJymYvjcehn5oi5seKSWROdtn/mz7Pp6Gk2AxnTjauUIeo5gkYM/ZXdOt2E1On9EZmpo1RntrCIxsajQK5OfLfQ4TkMcwkZyaRzFhbW8PT01P8esOGDbC3t8f69esxcOBABAUFYdasWQgMDIS9vT2ioqLw2Wef6fWORo0a4fz581WOMSsrCy1atKhye6oaS6tSqP+x6shFfQ8eLXNxN0+FPzKtcDdPOklSU6pEdpYFbqb9/RfC8y+mIvlcQ9y/b4ZOT/2JN8KSsGm1Nwry64GotoWGJqBX7zTMntUd9++bo0GD+wCAgoJ6KC4u+xXeoMF9NGhQCLX6LgCgefNc3L9vjjt36iM/3wKtvf9Ea6+/cPasE+7frwdv7z8x+u1EHDrohvx8/ScSUy0w9ORrA0/NljuTSGb+TaFQQKlU4v79+4iNjYWbmxs+/PBD8f6/h3u8vb0RHx+P119/XSwrPwirXKdOnbBmzRoIggCF4sHzK3S5cOECXnzxRb3bkWFats7BJ6v//n/51rgkAMD+H5v+b67Mo7XyyUHQm7/BykqD9OvWWPlpexza27Q6wiXS2/ODrgIAFiw8JCn/7LOnsD+mbGL7gIFX8eqrf591s+izg5I6JSVK9OyZhqBXL6JePS0yM6yxc2cr7NzhVUOfgqh6mUQyU1RUJJ7jkJ2djZUrVyI/Px+DBg1CXl4e0tLSEBUVhS5duuDHH3/Ezp07Je3HjRuHkSNHwtfXF/7+/oiMjMTFixfh4fH3LrC9e/dGfn4+Ll68iLZt/96ALS0tDVlZWUhLS4NGo0FiYiIAwNPTU5zH8/vvv+PmzZsVDuKi6nf+TCMM9Kv8Mv3yeTL/tHg2V3OQfPXv9/Ij60R+3RaRXz9848irVxpiwoQ+xgyLahiHmXQziWRm7969aNKkbFM0W1tbtG7dGt988w169eoFAJgwYQLCwsJQVFSEgQMHYubMmYiIiBDbv/zyy7h69SqmTJmCwsJCDBs2DGPGjMHPP/8s1nF0dMR//vMfREZGSk4RDQ8Px+bNm8WvO3Uq+4vv0KFD4vv/7//+D3379oWbm1s1fQeIiOixVsPHGZgahSDU8YE0PZw7dw59+vTB1atXxV6XRykuLkbLli2xdetW+Pv76/W+vLw82NvbI6DZWJgrOTGY6iatvXVth0BULUo1RTh4YSFyc3MrdXhjVZT/PeHXbzbM61V9snZpSSHi9oZXa6y1yeT2malO7du3x6efforU1NRKt0lLS8MHH3ygdyJDRERUWTV9NpOpMYlhppo0cuRIvep7enpKVloREREZnVYouwxpX4cxmSEiIpI7zpnRicNMREREZNLYM0NERCRzChi4NNtokcgTkxkiIiK54w7AOnGYiYiIiEwae2aIiIhkjjsA68ZkhoiISO64mkknDjMRERGRSWPPDBERkcwpBAEKAybxGtLWFDCZISIikjvt/y5D2tdhHGYiIiIik8aeGSIiIpnjMJNuTGaIiIjkjquZdGIyQ0REJHfcAVgnzpkhIiIik8aeGSIiIpnjDsC6MZkhIiKSOw4z6cRhJiIiIjJp7JkhIiKSOYW27DKkfV3GZIaIiEjuOMykE4eZiIiIyKSxZ4aIiEjuuGmeTkxmiIiIZI7HGejGYSYiIiIyaeyZISIikjtOANaJyQwREZHcCQAMWV5dt3MZJjNERERyxzkzunHODBEREZk09swQERHJnQAD58wYLRJZYjJDREQkd5wArBOHmYiIiMiksWeGiIhI7rQAFAa2r8OYzBAREckcVzPpxmEmIiIikpg/fz66dOkCW1tbODk5YciQIUhJSZHUKSwsRGhoKBwdHWFjY4Nhw4YhMzNTUictLQ0DBw5E/fr14eTkhMmTJ6O0tNTo8TKZISIikrvyCcCGXHo4fPgwQkNDceLECcTExKCkpAR9+/ZFQUGBWGfChAnYvXs3vvnmGxw+fBi3bt3C0KFDxfsajQYDBw5EcXExYmNjsXnzZmzatAnh4eFG+7aU4zATERGR3BlpNVNeXp6k2MLCAhYWFhWq7927V/L1pk2b4OTkhISEBPTo0QO5ubn44osvsHXrVjz77LMAgI0bN8Lb2xsnTpzA008/jX379iEpKQn79++Hs7MzOnbsiDlz5mDq1KmIiIiASqWq+uf5F/bMEBERPSZcXV1hb28vXvPnz69Uu9zcXABAw4YNAQAJCQkoKSlBQECAWKd169Zo1qwZ4uLiAABxcXFo164dnJ2dxTqBgYHIy8vDxYsXjfWRALBnhoiISP6M1DOTnp4OOzs7sfhBvTL/ptVqMX78ePj7+6Nt27YAgIyMDKhUKjg4OEjqOjs7IyMjQ6zzz0Sm/H75PWNiMkNERCR3RlqabWdnJ0lmKiM0NBQXLlzAsWPHDAigenGYiYiISObKl2YbclVFWFgYoqOjcejQITRt2lQsd3FxQXFxMXJyciT1MzMz4eLiItb59+qm8q/L6xgLkxkiIiKSEAQBYWFh2LlzJw4ePAh3d3fJ/c6dO6NevXo4cOCAWJaSkoK0tDT4+fkBAPz8/HD+/HncuXNHrBMTEwM7Ozv4+PgYNV4OMxEREcldDZ/NFBoaiq1bt+L777+Hra2tOMfF3t4eVlZWsLe3R0hICCZOnIiGDRvCzs4O7777Lvz8/PD0008DAPr27QsfHx+89tprWLBgATIyMjBjxgyEhoZWaq6OPpjMEBERyZ1WABQGJDNa/dquWbMGANCrVy9J+caNGzFy5EgAwJIlS6BUKjFs2DAUFRUhMDAQq1evFuuamZkhOjoaY8aMgZ+fH6ytrREcHIzZs2dX/XM8BJMZIiIikhAq0ZNjaWmJVatWYdWqVQ+t4+bmhj179hgztAdiMkNERCR3NTzMZGqYzBAREcmegckM6nYyw9VMREREZNLYM0NERCR3HGbSickMERGR3GkFGDRUpOdqJlPDYSYiIiIyaeyZISIikjtBW3YZ0r4OYzJDREQkd5wzoxOTGSIiIrnjnBmdOGeGiIiITBp7ZoiIiOSOw0w6MZkhIiKSOwEGJjNGi0SWOMxEREREJo09M0RERHLHYSadmMwQERHJnVYLwIC9YrR1e58ZDjMRERGRSWPPDBERkdxxmEknJjNERERyx2RGJw4zERERkUljzwwREZHc8TgDnZjMEBERyZwgaCEYcPK1IW1NAZMZIiIiuRMEw3pXOGeGiIiISL7YM0NERCR3goFzZup4zwyTGSIiIrnTagGFAfNe6vicGQ4zERERkUljzwwREZHccZhJJyYzREREMidotRAMGGaq60uzOcxEREREJo09M0RERHLHYSadmMwQERHJnVYAFExmHobDTERERGTS2DNDREQkd4IAwJB9Zup2zwyTGSIiIpkTtAIEA4aZBCYzREREVKsELQzrmeHSbCIiIiLZYs8MERGRzHGYSTcmM0RERHLHYSadmMzUovJMuVRbXMuREFUfrYa/ZqhuKtUUAaiZXo9SlBi0Z14pSowXjAzxt0wtunv3LgDglxsbajkSIiKqqrt378Le3r5anq1SqeDi4oJjGXsMfpaLiwtUKpURopIfhVDXB9JkTKvV4tatW7C1tYVCoajtcOq8vLw8uLq6Ij09HXZ2drUdDpHR8We8ZgmCgLt370KtVkOprL71NIWFhSguNrwHX6VSwdLS0ggRyQ97ZmqRUqlE06ZNazuMx46dnR1/0VOdxp/xmlNdPTL/ZGlpWWeTEGPh0mwiIiIyaUxmiIiIyKQxmaHHhoWFBT766CNYWFjUdihE1YI/4/S44gRgIiIiMmnsmSEiIiKTxmSGiIiITBqTGSIiIjJpTGaIiIjIpDGZIQKgUCiwa9cuSdkXX3yBvn376vWcp59+Gt99950RIyPST0REBDp27CgpKy4uhqenJ2JjYyv9nKSkJDRt2hQFBQVGjpDI+JjMkGxkZGRg3Lhx8PT0hKWlJZydneHv7481a9bg3r17NRpLYWEhZs6ciY8++khS/s0336B169awtLREu3btsGeP9LyUGTNmYNq0adBq6/YJtWS4kSNHQqFQiJejoyP69euHc+fOGf1da9euhbu7O7p16yaWzZ07F926dUP9+vXh4OBQoY2Pjw+efvppLF682OjxEBkbkxmShWvXrqFTp07Yt28f5s2bhzNnziAuLg5TpkxBdHQ09u/fX6PxfPvtt7Czs4O/v79YFhsbi1deeQUhISE4c+YMhgwZgiFDhuDChQtinf79++Pu3bv46aefajReMk39+vXD7du3cfv2bRw4cADm5uZ4/vnnjfoOQRCwcuVKhISESMqLi4vx0ksvYcyYMQ9tO2rUKKxZswalpaVGjYnI6AQiGQgMDBSaNm0q5OfnP/C+VqsVBEEQrl+/LrzwwguCtbW1YGtrK7z00ktCRkaGpO7q1asFDw8PoV69ekKrVq2ELVu2SO7/9ttvwjPPPCNYWFgI3t7ewr59+wQAws6dO8U6AwcOFCZNmiRpN3z4cGHgwIGSsq5duwpvv/22pGzUqFHCq6++qtfnp8dPcHCwMHjwYEnZ0aNHBQDCnTt3BEEQhClTpggtW7YUrKysBHd3d2HGjBlCcXGxpM38+fMFJycnwcbGRnjjjTeEqVOnCh06dBDvnzp1SlAqlUJeXt4D49i4caNgb2//wHtFRUWChYWFsH///ip/TqKawJ4ZqnV//fUX9u3bh9DQUFhbWz+wjkKhgFarxeDBg5GVlYXDhw8jJiYG165dw8svvyzW27lzJ8aNG4f3338fFy5cwNtvv41Ro0bh0KFDAMpOKh86dChUKhXi4+Oxdu1aTJ06tcL7jh07Bl9fX0lZXFwcAgICJGWBgYGIi4uTlD311FM4evRolb4X9PjKz8/H119/DU9PTzg6OgIAbG1tsWnTJiQlJWHZsmVYv349lixZIrbZvn07IiIiMG/ePJw+fRpNmjTB6tWrJc89evQoWrVqBVtbW71jUqlU6NixI3+eSfZ4ajbVuitXrkAQBHh5eUnKGzVqhMLCQgBAaGgoAgICcP78eaSmpsLV1RUAsGXLFrRp0wanTp1Cly5dsGjRIowcORJjx44FAEycOBEnTpzAokWL0Lt3b+zfvx+XLl3Czz//DLVaDQCYN28e+vfvL743JycHubm54v1yGRkZcHZ2lpQ5OzsjIyNDUqZWq5Geng6tVgulkv9eoIeLjo6GjY0NAKCgoABNmjRBdHS0+HMzY8YMsW7z5s0xadIkREVFYcqUKQCApUuXIiQkRBxC+vjjj7F//37xzw0AXL9+vcLPsj7UajWuX79e5fZENYG/aUm2Tp48icTERLRp0wZFRUVITk6Gq6urmMgAZZMUHRwckJycDABITk6WzHMBAH9/f8l9V1dXyS93Pz8/Sf379+8DACwtLasUt5WVFbRaLYqKiqrUnh4fvXv3RmJiIhITE3Hy5EkEBgaif//+YvKwbds2+Pv7w8XFBTY2NpgxYwbS0tLE9snJyejatavkmQ/6ea7qzzJQ9vNc0xPwifTFZIZqnaenJxQKBVJSUiTlHh4e8PT0hJWVVY3G4+joCIVCgezsbEm5i4sLMjMzJWWZmZlwcXGRlGVlZcHa2rrG4ybTY21tDU9PT3h6eqJLly7YsGEDCgoKsH79esTFxSEoKAgDBgxAdHQ0zpw5gw8//BDFxcV6vaNRo0YVfpb1kZWVhcaNG1e5PVFNYDJDtc7R0RF9+vTBypUrde5p4e3tjfT0dKSnp4tlSUlJyMnJgY+Pj1jn+PHjknbHjx+X3E9PT8ft27fF+ydOnJDUV6lU8PHxQVJSkqTcz88PBw4ckJTFxMRU+JfwhQsX0KlTp0d9bKIKFAoFlEol7t+/j9jYWLi5ueHDDz+Er68vWrZsWWG4x9vbG/Hx8ZKyf/88d+rUCZcuXYJQxTOF+fNMpoBzZkgWVq9eDX9/f/j6+iIiIgLt27eHUqnEqVOncOnSJXTu3BkBAQFo164dgoKCsHTpUpSWlmLs2LHo2bOnOFl38uTJGD58ODp16oSAgADs3r0bO3bsEJd2BwQEoFWrVggODsbChQuRl5eHDz/8sEI8gYGBOHbsGMaPHy+WjRs3Dj179sRnn32GgQMHIioqCqdPn8a6deskbY8ePar3Znv0eCoqKhLnXGVnZ2PlypXIz8/HoEGDkJeXh7S0NERFRaFLly748ccfsXPnTkn7cePGYeTIkfD19YW/vz8iIyNx8eJFeHh4iHV69+6N/Px8XLx4EW3bthXL09LSkJWVhbS0NGg0GiQmJgIo6yktn8fz+++/4+bNmxUmvhPJTm0vpyIqd+vWLSEsLExwd3cX6tWrJ9jY2AhPPfWUsHDhQqGgoEAQBOMszU5JSRG6d+8uqFQqoVWrVsLevXsrLM2+ePGiYGVlJeTk5Ejabt++XWjVqpWgUqmENm3aCD/++KPk/o0bN4R69eoJ6enpRvzOUF0UHBwsABAvW1tboUuXLsK3334r1pk8ebLg6Ogo2NjYCC+//LKwZMmSCsuo586dKzRq1EiwsbERgoODhSlTpkiWZgtC2bYC06ZN0/n+8uvQoUNinXnz5gmBgYHG/uhERqcQhCr2PRLVcS+99BKefPJJTJ8+vdJtpk6diuzs7Aq9NUS16dy5c+jTpw+uXr0q9ro8SnFxMVq2bImtW7dWmFRPJDecM0P0EAsXLqz0L/5yTk5OmDNnTjVFRFQ17du3x6efforU1NRKt0lLS8MHH3zARIZMAntmiIiIyKSxZ4aIiIhMGpMZIiIiMmlMZoiIiMikMZkhIiIik8ZkhoiIiEwakxmix9zIkSMxZMgQ8etevXpJdj6uKb/88gsUCgVycnIeWkehUGDXrl2VfmZERAQ6duxoUFy///47FAqFuEMuEckPkxkiGRo5ciQUCgUUCgVUKhU8PT0xe/ZslJaWVvu7d+zYUem9ciqTgBARVTeezUQkU/369cPGjRtRVFSEPXv2IDQ0FPXq1XvgjsTFxcVQqVRGeW/Dhg2N8hwioprCnhkimbKwsICLiwvc3NwwZswYBAQE4IcffgDw99DQ3LlzoVar4eXlBQBIT0/H8OHD4eDggIYNG2Lw4MH4/fffxWdqNBpMnDgRDg4OcHR0xJQpUyqcpvzvYaaioiJMnToVrq6usLCwgKenJ7744gv8/vvv6N27NwCgQYMGUCgUGDlyJABAq9Vi/vz5cHd3h5WVFTp06IBvv/1W8p49e/agVatWsLKyQu/evSVxVtbUqVPRqlUr1K9fHx4eHpg5cyZKSkoq1Pv888/h6uqK+vXrY/jw4cjNzZXc37BhA7y9vWFpaYnWrVtj9erVesdCRLWHyQyRibCyskJxcbH49YEDB5CSkoKYmBhER0ejpKQEgYGBsLW1xdGjR3H8+HHY2NigX79+YrvPPvsMmzZtwpdffoljx44hKyurwknM//b666/j//7v/7B8+XIkJyfj888/h42NDVxdXfHdd98BAFJSUnD79m0sW7YMADB//nxs2bIFa9euxcWLFzFhwgS8+uqrOHz4MICypGvo0KEYNGgQEhMT8eabb2LatGl6f09sbW2xadMmJCUlYdmyZVi/fj2WLFkiqXPlyhVs374du3fvxt69e3HmzBmMHTtWvB8ZGYnw8HDMnTsXycnJmDdvHmbOnInNmzfrHQ8R1ZJaPeaSiB4oODhYGDx4sCAIgqDVaoWYmBjBwsJCmDRpknjf2dlZKCoqEtt89dVXgpeXl6DVasWyoqIiwcrKSvj5558FQRCEJk2aCAsWLBDvl5SUCE2bNhXfJQiC0LNnT2HcuHGCIJSdMA5AiImJeWCchw4dEgAI2dnZYllhYaFQv359ITY2VlI3JCREeOWVVwRBEITp06cLPj4+kvtTp06t8Kx/w79ON/+3hQsXCp07dxa//uijjwQzMzPhxo0bYtlPP/0kKJVK4fbt24IgCEKLFi2ErVu3Sp4zZ84cwc/PTxAEQUhNTRUACGfOnHnoe4modnHODJFMRUdHw8bGBiUlJdBqtfjvf/+LiIgI8X67du0k82TOnj2LK1euwNbWVvKcwsJCXL16Fbm5ubh9+za6du0q3jM3N4evr2+FoaZyiYmJMDMzQ8+ePSsd95UrV3Dv3j306dNHUl5cXIxOnToBAJKTkyVxAICfn1+l31Fu27ZtWL58Oa5evYr8/HyUlpbCzs5OUqdZs2Z44oknJO/RarVISUmBra0trl69ipCQELz11ltindLSUtjb2+sdDxHVDiYzRDLVu3dvrFmzBiqVCmq1Gubm0j+u1tbWkq/z8/PRuXNnREZGVnhW48aNqxSDlZWV3m3y8/MBAD/++KMkiQDK5gEZS1xcHIKCgjBr1iwEBgbC3t4eUVFR+Oyzz/SOdf369RWSKzMzM6PFSkTVi8kMkUxZW1vD09Oz0vWffPJJbNu2DU5OThV6J8o1adIE8fHx6NGjB4CyHoiEhAQ8+eSTD6zfrl07aLVaHD58GAEBARXul/cMaTQasczHxwcWFhZIS0t7aI+Ot7e3OJm53IkTJx79If8hNjYWbm5u+PDDD8Wy69evV6iXlpaGW7duQa1Wi+9RKpXw8vKCs7Mz1Go1rl27hqCgIL3eT0TywQnARHVEUFAQGjVqhMGDB+Po0aNITU3FL7/8gvfeew83btwAAIwbNw6ffPIJdu3ahUuXLmHs2LE694hp3rw5goOD8cYbb2DXrl3iM7dv3w4AcHNzg0KhQHR0NP744w/k5+fD1tYWkyZNwoQJE7B582ZcvXoVv/76K1asWCFOqn3nnXdw+fJlTJ48GSkpKdi6dSs2bdqk1+dt2bIl0tLSEBUVhatXr2L58uUPnMxsaWmJ4OBgnD17FkePHsV7772H4cOHw8XFBQAwa9YszJ8/H8uXL8dvv/2G8+fPY+PGjVi8eLFe8RBR7WEyQ1RH1K9fH0eOHEGzZs0wdOhQeHt7IyQkBIWFhWJPzfvvv4/XXnsNwcHB8PPzg62tLf7zn//ofO6aNWvw4osvYuzYsWjdujXeeustFBQUAACeeOIJzJo1C9OmTYOzszPCwsIAAHPmzMHMmTMxf/58eHt7o1+/fvjxxx/h7u4OoGwey3fffYddu3ahQ4cOWLt2LebNm6fX533hhRcwYcIEhIWFoWPHjoiNjcXMmTMr1PP09MTQoUMxYMAA9O3bF+3bt5csvX7zzTexYcMGbNy4Ee3atUPPnj2xadMmMVYikj+F8LCZf0REREQmgD0zREREZNKYzBAREZFJYzJDREREJo3JDBEREZk0JjNERERk0pjMEBERkUljMkNEREQmjckMERERmTQmM0RERGTSmMwQERGRSWMyQ0RERCbt/wEJd7lVqw65JgAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"cell_type":"markdown","source":["## Surrogate model-based explanation\n","A surrogate model is, by definition, an intrinsically interpretable model, which we train to predict (aka reproduce) the predictions of a black-box model.\n","\n","We just created a black-box model above. As surrogate model, we will use linear regression, and see how well we can linearly approximate our NN. Specifically, fit the surrogate model to the predicted class *probabilities* of the neural network using the original features. To predict probabilities, you will need to use the method `predict_proba()`, which any `sklearn` classifier provides.\n","\n","One question worth discussing in this context is which data set to use for fitting the surroage model, the training or test data. From a programming perspective, it does not make a difference. So first develop your code and then we discuss the subtle differences between the two options."],"metadata":{"id":"9X_bAobqH9qI"}},{"cell_type":"code","source":["# Linear regression as surrogate model\n","from sklearn.linear_model import LinearRegression\n","\n","# Produce (probability) forecasts for the test data\n","\n","\n","# Fit regress original features on the probability forecast\n","\n","\n","\n"],"metadata":{"id":"uBUZ3_vLH3Sp"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Given that the linear regression can give no more than an approximation of the neural network, it is crucial to examine the quality of the approximation. If it is poor, the insights, which we expect the surrogate model to provide may not be reliable. \n","\n","Write some code to assess the quality of the approximation."],"metadata":{"id":"c7rCNGaQ3hLi"}},{"cell_type":"code","source":["# Compare NN predictions to linear regression approximation\n"],"metadata":{"id":"o7CC2AiC155C"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Permutation-based feature importance\n","We introduces the concepts of permutation-based feature importance in the lecture. Now we want to examine how it works in Python. Of course, this will require us finding a Python method that implements the approach. To shortcut your searching, note that there is such method in *the most popular Python library for ML* and it is called `permutation_importance`. "],"metadata":{"id":"kZm4SJeyIgw3"}},{"cell_type":"code","source":["# Permutation-based feature importance\n","\n"],"metadata":{"id":"S8syUjAMK-i5"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["The calculation of feature importance is pretty easy. However, visualizing the estimates importance scores takes a little work. Fortunately, this is a standard task and ready-to-use demo code is easily found on the web. In fact, we only found to browse the [official sklearn website](https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-multicollinear-py) and will find some nice demo code to plot feature importance scores as a boxplot.  "],"metadata":{"id":"1gZA7O03LNVu"}},{"cell_type":"code","source":["# Creating PDPs \n"],"metadata":{"id":"YvW75YwTLBdJ"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Partial dependence\n","Next up, we discussed the concept of partial dependence. A partial dependence plot (PDP) complements feature importance analysis. For example, it is interesting to examine how those features that importance analysis identified as crucial affect model predictions (when averaging over the effect of the other features). Let's try to create a PDP for our NN-classifier. "],"metadata":{"id":"6XKus66XIhEL"}},{"cell_type":"code","source":["# Partial dependence analysis\n"],"metadata":{"id":"D7f9vg2_IhOp"},"execution_count":null,"outputs":[]}]}