Lesson106-Imbalanced Data对于算法的影响.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第106讲 Imbalanced Data对于算法的影响\n",
    "\n",
    "## Python学习：https://www.ixigua.com/home/77346806707?utm_source=xiguastudio\n",
    "\n",
    "## Python源文件及数据下载链接: https://github.com/rayc2020/LessonPythonCode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since your question is more about imbalanced dataset, I am assuming we can safely look at only supervised learning algorithms \n",
    "\n",
    "In this, cart, random forest , bagging boosting type of algorithms are fairly resistant to imbalanced data.\n",
    "\n",
    "Logistic, svm, Ann,knn are sensitive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All are these algorithms are using distance as a metric, be it Euclidean distance or Manhattan distance\n",
    "All the ml algo which contains gradient descent.... Are impacted by Imbalanced dataset... Such as ANN, Logistic regression, Linear Regression... Etc and KNN also"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All ML algorithms deals with linear seperability and linear relationship are affect by imbalance dataset Because they are supervised learning. where target varaible are imbalanced that makes over fitting of data\n",
    "\n",
    "i think all the algorithms of classification. Except for the Ensemble Learning algorithms RF, XGboost, Gradient Boost, Ada Boost other algorithms may get affected by the Imbalanced dataset.  \n",
    "\n",
    "why only these not effected\n",
    "1. Ensemble Learning is of creating N number of Weak models to make strong learners in RF. \n",
    "2. XGBoost is a combination of Bagging and boosting, it takes the weighted average of the many weak models and by focusing on weak predictions and iterating through the next models it reduces the error.\n",
    "3. same with gradient boost it starts with a weak prediction as average and builds decision trees by adding learning rate times the outcome of the residual tree to the initial prediction and process repeated.\n",
    "4. Ada boost gives more weight to the incorrectly classified samples in creating further stumps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification Algorithms such as Logistic Regression and Linear SVM (kernel= 'linear') which try to find an optimal hyperplane are generally prone to imbalanced dataset.  \n",
    "\n",
    "The reason behind this is these linear classifiers are actually to solve their respective optimization problems in presence of a tradeoff, which is: Greater Generalization Power Vs Minimal Classification Error. Hence if the dataset itself is imbalanced, it will be only solving the optimization problem for the points of the majority class, given we haven't done any hyper-paramter tuning. Now for some aggressive hyper-parameter tuning, we get these classifiers to perform okayish for imbalanced dataset (tuning for minimising error over maximising generalization power), but then we will be sacrificing the Generalization Power for unseen datasets, which isn't desirable, also hyper-parameter tuning won't work for severely imbalanced datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "algorithms affected by Imbalanced Data:\n",
    "• Linear Regression\n",
    "• Logistic Regression\n",
    "• KNN\n",
    "• SVM\n",
    "• KMeans\n",
    "• Hierarchical clustering\n",
    "• Neural Networks\n",
    "• PCA\n",
    "\n",
    "- Now, for Similarity based algos such as KNN are also prone to imbalanced datasets, as they employ a majority voting at the end to make their classification decision.\n",
    "\n",
    "- Naive Bayes isn't affected by the imbalanced as it calulates the likelihood of each (seen) features during runtime.\n",
    "\n",
    "- Tree based algorithms such as Decision Trees and Random Forest should also not be affected by imbalanced datasets, because they divide the given feature space into axis parallel hyperplanes. (Also, they try to maximise the information gain)\n",
    "\n",
    "Also, I think that MLP based algos would overfit the training dataset if it is imbalanced, hence making them immune to \n",
    "imbalanced datasets."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}