Machine_Learning_Part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03679fd3",
   "metadata": {},
   "source": [
    "# Machine learning:  Recap\n",
    "\n",
    "In general, a learning problem considers a set of n samples of data and then tries to predict properties of unknown data. If each sample is more than a single number and, for instance, a multi-dimensional entry (aka multivariate data), it is said to have several attributes or features.\n",
    "\n",
    "# Training set and testing set\n",
    "\n",
    "Machine learning is about learning some properties of a data set and then testing those properties against another data set. A common practice in machine learning is to evaluate an algorithm by splitting a data set into two. We call one of those sets the training set, on which we learn some properties; we call the other set the testing set, on which we test the learned properties.\n",
    "\n",
    "# The problem setting\n",
    "\n",
    "\n",
    "Which approach should I take for my problem?\n",
    "\n",
    "<figure>\n",
    "<img src=\"figs/Scikit.png\" width='1000'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b14970f",
   "metadata": {},
   "source": [
    " We think about our estimator, depending on how the distribution of the data is, what is our desired precision, how big is our data set, how much time and computation effort do we have, ... .\n",
    " \n",
    "\n",
    " \n",
    "\n",
    " \n",
    " \n",
    " ### Many classification methods exist:\n",
    " which method would you choose?\n",
    " \n",
    " \n",
    " #### Note: Classification is different from Regression:\n",
    "\n",
    "- Classify for categorical output\n",
    "- Regression: predicting continuous-valued attribute(s)\n",
    "\n",
    "\n",
    " \n",
    " For example here there are different methods illustration for 3 different  data distributions. \n",
    " \n",
    " \n",
    " \n",
    " \n",
    " \n",
    "<figure>\n",
    "<img src=\"figs/InputData2.png\" width='1100'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n",
    "Let us start with 2 main classification approaches. \n",
    "\n",
    " # Supervied learning: \n",
    " Labeled information is available and can be used for learning.\n",
    "\n",
    "## Example: K-Nearest Neighbours classification\n",
    "\n",
    "KNN is a supervised machine learning algorithm  and one the most simple and helpful techniques for classification.  Within this algorithm the prediction for each new data is made based on the k-most similar data in our stored dateset. The similarity can be any feature like Eucledean distance. Then the k nearest neighbours to our new date are the k nearest points. \n",
    "The k-nearest-neighbor classifier is commonly based on the Euclidean distance between a test sample and the specified training samples. Let $$x_i$$ be an input sample with p features $$(x_{i1},x_{i2},...,x_{ip})$$, and   n be the total number of input samples (i=1,2,...,n). The Euclidean distance between sample $$x_i$$ and $$x_1$$\n",
    "\n",
    " is defined as:\n",
    "\n",
    "\n",
    "$$d(x_i,x_l)=  \\sqrt{(x_{i 1} − x_{l 1})^2+(x_{i 2} − x_{l2})^2 + ... + (x_{ip} − x_{lp})^2}$$\n",
    "\n",
    "### Using an example dataset\n",
    "let us start with a famouse datset of flower species, Iris dataset. It is a multiclass dataset and there are 4 attributes to classify the dataset as follows:\n",
    "\n",
    "- Sepal length \n",
    "- Sepal width \n",
    "- Petal length \n",
    "- Petal width \n",
    "\n",
    "and the class names are :\n",
    "\n",
    "- - Iris-setosa\n",
    "- - Iris-versicolor\n",
    "- - Iris-virginica\n",
    "\n",
    "\n",
    "\n",
    "### Algorithm steps\n",
    "\n",
    "STEP 1: Choose the number K of neighbors\n",
    "\n",
    "STEP 2: Take the K nearest neighbors of the new data point, according to your distance metric\n",
    "\n",
    "STEP 3: Among these K neighbors, count the number of data points to each category\n",
    "\n",
    "STEP 4: Assign the new data point to the category where you counted the most neighbors\n",
    "\n",
    "\n",
    "### Import libraries and load the dataset.\n",
    "\n",
    "\n",
    "Iris dataset includes three iris species with 50 samples each as well as some properties about each flower.\n",
    "\n",
    "\n",
    "\n",
    "<figure>\n",
    "<img src=\"figs/Iris.png\" width='600'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n",
    "### Pandas\n",
    "\n",
    "pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool,\n",
    "built on top of the Python programming language.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8414086a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "dataset = pd.read_csv (\"Data/iris.csv\" )\n",
    "#print (dataset)\n",
    "print (dataset.shape)\n",
    "dataset.head(6)  # The head() function is used to get the first n rows.\n",
    "#dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd274a2",
   "metadata": {},
   "source": [
    "number of instances (rows) of each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9204edd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.groupby('Species').size()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c489743",
   "metadata": {},
   "source": [
    "###  Dividing data into features and labels\n",
    "the dataset contain six columns: Id, SepalLength[cm], SepalWidth [cm], PetalLength [cm], PetalWidth [cm] and Species. The actual features are described by columns 1-4. Last column contains labels of samples. Firstly we need to split data into two arrays: X (features) and y (labels).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5bb485a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dataset.iloc[:, 1:5].values\n",
    "y = dataset.iloc[:, 5].values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ecf9211",
   "metadata": {},
   "source": [
    "### Label encoding\n",
    "The labels are categorical. KNeighborsClassifier does not accept string labels. We need to use LabelEncoder to transform them into numbers. Iris-setosa correspond to 0, Iris-versicolor correspond to 1 and Iris-virginica correspond to 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac56ecf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)  # Fit label encoder and return encoded labels."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c9a4e43",
   "metadata": {},
   "source": [
    "\n",
    "### Spliting dataset into training set and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1620da2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)  \n",
    "# random_state, Controls the shuffling applied to the data before applying the split.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de5a547",
   "metadata": {},
   "source": [
    "###  Data Visualization\n",
    " - Parallel Coordinates\n",
    " \n",
    " Parallel coordinates is a plotting technique for plotting multivariate data. It allows one to see clusters in data and to estimate other statistics visually. Using parallel coordinates points are represented as connected line segments. Each vertical line represents one attribute. One set of connected line segments represents one data point. Points that tend to cluster will appear closer together.\n",
    " \n",
    " #### import data visulization libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66865dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baa9bd52",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandas.plotting import parallel_coordinates\n",
    "plt.figure(figsize=(15,10))\n",
    "parallel_coordinates(dataset.drop(\"Id\", axis=1), \"Species\")\n",
    "plt.title('Parallel Coordinates Plot', fontsize=20, fontweight='bold')\n",
    "plt.xlabel('Features', fontsize=15)\n",
    "plt.ylabel('Features values', fontsize=15)\n",
    "plt.legend(loc=1, prop={'size': 15}, frameon=True,shadow=True, facecolor=\"white\", edgecolor=\"black\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "415fd726",
   "metadata": {},
   "source": [
    "## Using KNN for classification\n",
    "### Making predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9158380e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fitting classifier to the Training set\n",
    "# Loading libraries\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Instantiate learning model (k = 3)\n",
    "classifier = KNeighborsClassifier(n_neighbors=2)\n",
    "\n",
    "# Fitting the model\n",
    "classifier.fit(X_train, y_train)\n",
    "\n",
    "# Predicting the Test set results\n",
    "y_pred = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51a7106c",
   "metadata": {},
   "source": [
    "### Evaluating predictions\n",
    "#### Model accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47d4ac41",
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = accuracy_score(y_test, y_pred)*100\n",
    "print('Accuracy of our model is : ' + str(round(accuracy, 2)) + ' %.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c95b2ccf",
   "metadata": {},
   "source": [
    "\n",
    "#### Confusion matrix generation.\n",
    "\n",
    "Confusion matrix is always built to evaluate the performance of the model on test data. Maximization of the diagonal elements proves the best performance of model. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d54084e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "print (cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd742c50",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    " \n",
    " # Unsupervised learning:\n",
    " No (initial) labels and learning needs to structure data on its own."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ed52857",
   "metadata": {},
   "source": [
    "## Clustering \n",
    "\n",
    "There are several clustering methods:\n",
    "\n",
    "\n",
    "<figure>\n",
    "<img src=\"figs/clustering.png\" width='900'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n",
    "\n",
    "\n",
    "<figure>\n",
    "<img src=\"figs/Cluster_table.png\" width='1100'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "950a999c",
   "metadata": {},
   "source": [
    "## Example :DBSCAN \n",
    "Density-Based Spatial Clusering of Applications with Noice.\n",
    "\n",
    "- 1) Epsilon: The maximum distance (euclidean distance) between a pair of points. The two points are considered as neighbors if and only if they are separated by a distance less than or equal to epsilon.\n",
    "- 2) MinPoints: The minimum number of points required to form a dense cluster.\n",
    "\n",
    "<figure>\n",
    "<img src=\"figs/DBSCAN.png\" width='650'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n",
    "## DBSCAN example\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52278d51",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn import datasets\n",
    "from sklearn.datasets import make_blobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a903636",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configuration options\n",
    "num_samples_total = 1000\n",
    "cluster_centers = [(3,3), (7,7)]\n",
    "num_classes = len(cluster_centers)\n",
    "epsilon = 1.0\n",
    "min_samples = 13\n",
    "\n",
    "# Generate data\n",
    "X, y = make_blobs(n_samples = num_samples_total, centers = cluster_centers, n_features = num_classes, center_box=(0, 1), cluster_std = 0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "773fef29",
   "metadata": {},
   "source": [
    "The generated random points "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60d01f98",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.shape(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1062167-53bd-4f8b-aee5-283fb176ef4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in range(num_samples_total):\n",
    "\n",
    "    plt.plot(X[:,0], X[:,1],'o', \n",
    "             markeredgecolor='k', markersize=5)#markerfacecolor=col,\n",
    "plt.title('The distribution of generated random data')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75d300a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute DBSCAN\n",
    "db = DBSCAN(eps=epsilon, min_samples=min_samples).fit(X)\n",
    "labels = db.labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ceaec94f-4c6f-4f45-8561-05c4884d432f",
   "metadata": {},
   "outputs": [],
   "source": [
    "no_clusters = len(np.unique(labels) )\n",
    "no_noise = np.sum(np.array(labels) == -1, axis=0)\n",
    "\n",
    "print('Estimated no. of clusters: %d' % no_clusters)\n",
    "print('Estimated no. of noise points: %d' % no_noise)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fcc7c39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate scatter plot for training data\n",
    "colors = list(map(lambda x: '#3b4cc0' if x == 1 else '#b40426', labels))\n",
    "plt.scatter(X[:,0], X[:,1], c=colors, marker=\"o\", picker=True)\n",
    "plt.title('Two clusters with data')\n",
    "plt.xlabel('Axis X[0]')\n",
    "plt.ylabel('Axis X[1]')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50bc0bb1",
   "metadata": {},
   "source": [
    "An example of The application of DBSCAN clustering algorithm on the damage sites detection in Dual Phase Steel can be found here.\n",
    "As you can see the damage sites are darker than the rest of microstructure , however there are also some dark thin shadows sites which maybe misinterpreted as damage.  \n",
    "\n",
    "<figure>\n",
    "<img src=\"figs/20.png\" width='540'>\n",
    "<figcaption></figcaption>\n",
    "</figure>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc4e1159",
   "metadata": {},
   "source": [
    "### <span style=\"color: red\"> Excersice :</span>  Load the Mag-Al-Ca alloy from figs folder and apply DBSCAN after thresholding. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}