tps-july2021-ridge-regression.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/sabyasachi96/tps-july2021-ridge-regression?scriptVersionId=188649952\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"Starting Data science and this would be my first competition, and have tried exploring Ridge Regression. Any Feedaback or suggestion is very much appreciated to improve my skill and score.  ","metadata":{}},{"cell_type":"markdown","source":"# Importing data and libraries","metadata":{}},{"cell_type":"code","source":"## Importing Libraries and **Data files**\n\nimport pandas as pd\nimport seaborn as sns\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-07-30T11:47:45.018475Z","iopub.execute_input":"2021-07-30T11:47:45.018894Z","iopub.status.idle":"2021-07-30T11:47:45.027332Z","shell.execute_reply.started":"2021-07-30T11:47:45.018861Z","shell.execute_reply":"2021-07-30T11:47:45.02624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/tabular-playground-series-jul-2021/train.csv')\ntest = pd.read_csv('../input/tabular-playground-series-jul-2021/test.csv')\nsample = pd.read_csv('../input/tabular-playground-series-jul-2021/sample_submission.csv')\n\nprint(\"train shape\", train.shape)\nprint(\"test shape\", test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:45.045607Z","iopub.execute_input":"2021-07-30T11:47:45.046107Z","iopub.status.idle":"2021-07-30T11:47:45.091898Z","shell.execute_reply.started":"2021-07-30T11:47:45.046077Z","shell.execute_reply":"2021-07-30T11:47:45.091153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Analysis","metadata":{}},{"cell_type":"markdown","source":"Steps in data analysis will follow these following steps\n\n*   Check out categorical and numerical data\n* Check for duplicate data\n*   Detect Null values \n*   Explore the data with graphs and plots (following may be )\n * distribution\n * Correlation\n * Box plot to get an idea about the outliers\n* Next step would be to get the correlation between these indepnedent and dependent variables\n* changes in the data points of dependent and independent data over time.\n* Graphs will help tell the kind of model is needed\n","metadata":{}},{"cell_type":"markdown","source":"## Extracting information","metadata":{}},{"cell_type":"code","source":"print(\"train shape\", train.shape)\ntrain.describe().T","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:45.277212Z","iopub.execute_input":"2021-07-30T11:47:45.277721Z","iopub.status.idle":"2021-07-30T11:47:45.331278Z","shell.execute_reply.started":"2021-07-30T11:47:45.27769Z","shell.execute_reply":"2021-07-30T11:47:45.33023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Data types of the columns \\n\\n', train.info())\nprint('\\n\\n\\nTotal Null values\\n', train.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:45.33325Z","iopub.execute_input":"2021-07-30T11:47:45.333657Z","iopub.status.idle":"2021-07-30T11:47:45.355463Z","shell.execute_reply.started":"2021-07-30T11:47:45.333615Z","shell.execute_reply":"2021-07-30T11:47:45.354288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Total Duplicate values\\n', train.duplicated().sum())","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:45.356944Z","iopub.execute_input":"2021-07-30T11:47:45.357248Z","iopub.status.idle":"2021-07-30T11:47:45.370889Z","shell.execute_reply.started":"2021-07-30T11:47:45.357219Z","shell.execute_reply":"2021-07-30T11:47:45.369893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### *Inference* 1: \n* No Null values\n* All except time are continuous variables of float type data\n* Obviously there is a regression and time series analysis, which will be dealt accordingly.\n","metadata":{}},{"cell_type":"markdown","source":"## Univariate DA \n- Histogram and BoxPLot","metadata":{}},{"cell_type":"code","source":"hist = train.hist(figsize = (18, 10), bins=50, grid = False, xlabelsize=8, ylabelsize=8, layout = (3,4))","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:45.372258Z","iopub.execute_input":"2021-07-30T11:47:45.372592Z","iopub.status.idle":"2021-07-30T11:47:48.026789Z","shell.execute_reply.started":"2021-07-30T11:47:45.372562Z","shell.execute_reply":"2021-07-30T11:47:48.025715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"box = train.boxplot(figsize = (18,8), rot = 20 )","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:48.028362Z","iopub.execute_input":"2021-07-30T11:47:48.028675Z","iopub.status.idle":"2021-07-30T11:47:48.508589Z","shell.execute_reply.started":"2021-07-30T11:47:48.028644Z","shell.execute_reply":"2021-07-30T11:47:48.50742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### *Inference* 2: \n* From histogram - Data is normally distributed, except the target ones mostly right skewed.\n* From Box-plot - Outliers in all the sensor data, thus a outlier sensitive model has to be picked.\n* From Box-Plot - Scaling has to be perfomrmed, since the data ranges are pretty wide. The data all are in different scales, thus have to be scaled accordingly.\n* Because of outliers, i'll use **Robust scaler** use for scaling the features","metadata":{"execution":{"iopub.status.busy":"2021-07-30T10:56:24.897929Z","iopub.execute_input":"2021-07-30T10:56:24.89831Z","iopub.status.idle":"2021-07-30T10:56:24.906509Z","shell.execute_reply.started":"2021-07-30T10:56:24.898277Z","shell.execute_reply":"2021-07-30T10:56:24.905022Z"}}},{"cell_type":"markdown","source":"## Bivariate DA - \n\n* Comparing the 3 target features with each of the independent features","metadata":{}},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:48.510258Z","iopub.execute_input":"2021-07-30T11:47:48.510897Z","iopub.status.idle":"2021-07-30T11:47:48.518995Z","shell.execute_reply.started":"2021-07-30T11:47:48.510852Z","shell.execute_reply":"2021-07-30T11:47:48.517866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.pairplot(train, size = 5, \n    x_vars=['deg_C', 'relative_humidity', 'absolute_humidity','sensor_1', 'sensor_2', 'sensor_3', 'sensor_4', 'sensor_5'],\n    y_vars=['target_carbon_monoxide', 'target_benzene', 'target_nitrogen_oxides'], \n)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:48.520353Z","iopub.execute_input":"2021-07-30T11:47:48.520809Z","iopub.status.idle":"2021-07-30T11:47:54.464818Z","shell.execute_reply.started":"2021-07-30T11:47:48.520776Z","shell.execute_reply":"2021-07-30T11:47:54.459577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"There is good relation between the target and the other independent variables, so i'll perform a correlation matrix for that.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,9))\nsns.heatmap(train.corr(), \n            annot=True, cmap=\"RdBu\", fmt='.2f', \n            center = 0, linewidths=0.1, \n            cbar_kws={\"shrink\": .8}, square = True) ","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:54.467925Z","iopub.execute_input":"2021-07-30T11:47:54.46843Z","iopub.status.idle":"2021-07-30T11:47:55.504841Z","shell.execute_reply.started":"2021-07-30T11:47:54.468387Z","shell.execute_reply":"2021-07-30T11:47:55.503473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Inference 4 \n* Deg-c; humidity are less correlated with the target varibalesa and less with the other inpdependent variables \n* Surprisingly some variables are negatively correlated with the target variables\n* There is Multi-collinearity among the variables, thus for prediction - I'll perform **Ridge Regression.** \n* I'll compare the model acuuracy  ","metadata":{}},{"cell_type":"markdown","source":"# Model Plan","metadata":{}},{"cell_type":"markdown","source":"\n## Steps\n* Remove the date time column \n* train-test split before scaling to prevent data leakage\n* Scale the test and train data seprately with Robust scalar transformation\n* Perform Ridge and lasso Regression \n* Perform Accuracy with the split data \n* Final prediction ","metadata":{}},{"cell_type":"markdown","source":"### Data Preparation","metadata":{}},{"cell_type":"code","source":"m_train = train.copy()\nm_train.drop(['date_time'], axis = 1, inplace = True)\nm_train.columns","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.507161Z","iopub.execute_input":"2021-07-30T11:47:55.507622Z","iopub.status.idle":"2021-07-30T11:47:55.520691Z","shell.execute_reply.started":"2021-07-30T11:47:55.507573Z","shell.execute_reply":"2021-07-30T11:47:55.519289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Splitting and Scaling Data","metadata":{}},{"cell_type":"code","source":"# separating the Target from the independent varibales\nx_data = m_train.drop(['target_carbon_monoxide', 'target_benzene', 'target_nitrogen_oxides'],axis=1)\ny_data = m_train[['target_carbon_monoxide', 'target_benzene', 'target_nitrogen_oxides']]\n\n\n# Splitting the data into train and test data \nfrom sklearn.model_selection import train_test_split\nx_train, x_test, y_train, y_test = train_test_split(x_data, y_data, test_size=0.2, random_state=10)\nprint(\"X-train shape\", x_train.shape, \"\\nY-test shape\" ,y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.522542Z","iopub.execute_input":"2021-07-30T11:47:55.522921Z","iopub.status.idle":"2021-07-30T11:47:55.53943Z","shell.execute_reply.started":"2021-07-30T11:47:55.522887Z","shell.execute_reply":"2021-07-30T11:47:55.53834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Robust Scaling\n\nfrom sklearn.preprocessing import RobustScaler\n\nscaler = RobustScaler() # class object\nx_train = pd.DataFrame(scaler.fit_transform(x_train), columns = x_train.columns)\nx_test = pd.DataFrame(scaler.fit_transform(x_test), columns = x_test.columns)\n","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.54097Z","iopub.execute_input":"2021-07-30T11:47:55.541317Z","iopub.status.idle":"2021-07-30T11:47:55.567348Z","shell.execute_reply.started":"2021-07-30T11:47:55.541285Z","shell.execute_reply":"2021-07-30T11:47:55.56599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Target Carbon-Monoxide Model Creation ","metadata":{}},{"cell_type":"markdown","source":"Since Ridge regression from Sklearn also has a normalisation section, so i wanted to check how the model would be affected with a different scaling method. ","metadata":{}},{"cell_type":"code","source":"# Ridge Model Creation of Robust scaled data: w/normalisation\nfrom sklearn.linear_model import Ridge\nridge = Ridge(fit_intercept = True, normalize = False)\nridge.fit(x_train,y_train['target_carbon_monoxide'])\nridge_predict = ridge.predict(x_test)\nprint('\\nRidge score: w/ Robust scaled data:', ridge.score(x_test,y_test['target_carbon_monoxide']))\nprint('Mean Absolute % Error: ', np.mean(np.abs(ridge_predict - y_test['target_carbon_monoxide'])/       \n                     np.abs(y_test['target_carbon_monoxide'])))","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.56897Z","iopub.execute_input":"2021-07-30T11:47:55.569299Z","iopub.status.idle":"2021-07-30T11:47:55.598639Z","shell.execute_reply.started":"2021-07-30T11:47:55.569268Z","shell.execute_reply":"2021-07-30T11:47:55.59716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Splitting the data into train and test data for non scaled data modelling\nfrom sklearn.model_selection import train_test_split\nx_train1, x_test1, y_train1, y_test1 = train_test_split(x_data, y_data, test_size=0.2, random_state=10)\nprint(\"X-train1 shape\", x_train1.shape, \"-x--x--x-\" ,  \" Y-test1 shape\" ,y_test1.shape)\n\n\n# Ridge Model Creation of non scaled data\nfrom sklearn.linear_model import Ridge\nridge = Ridge(fit_intercept = False, normalize = False)\nridge.fit(x_train1,y_train1['target_carbon_monoxide'])\nridge_predict = ridge.predict(x_test1)\nprint('\\nRidge score: of non scaled data:', ridge.score(x_test1,y_test1['target_carbon_monoxide']))\nprint('Mean Absolute % Error: ', np.mean(np.abs(ridge_predict - y_test['target_carbon_monoxide'])/       \n                     np.abs(y_test['target_carbon_monoxide'])))\n","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.604339Z","iopub.execute_input":"2021-07-30T11:47:55.604886Z","iopub.status.idle":"2021-07-30T11:47:55.648394Z","shell.execute_reply.started":"2021-07-30T11:47:55.604839Z","shell.execute_reply":"2021-07-30T11:47:55.647157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems the robust scaled has a better ridge score and lower error %. Now i'll continue with the other and check the model.","metadata":{}},{"cell_type":"markdown","source":"## Target Benzene model Creation","metadata":{}},{"cell_type":"code","source":"# Ridge Model Creation of Robust scaled data:\nfrom sklearn.linear_model import Ridge\nridge = Ridge(fit_intercept = True, normalize = False)\nridge.fit(x_train,y_train['target_benzene'])\nridge_predict = ridge.predict(x_test)\nprint('\\nRidge score: of Robust scaled data: ', ridge.score(x_test,y_test['target_benzene']))\nprint('Mean Absolute % Error: ', np.mean(np.abs(ridge_predict - y_test['target_benzene'])/       \n                     np.abs(y_test['target_benzene'])))\n","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.654085Z","iopub.execute_input":"2021-07-30T11:47:55.657022Z","iopub.status.idle":"2021-07-30T11:47:55.690739Z","shell.execute_reply.started":"2021-07-30T11:47:55.656951Z","shell.execute_reply":"2021-07-30T11:47:55.68921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Target Nitrogen Oxide - Model Creation","metadata":{}},{"cell_type":"code","source":"# Ridge Model Creation of Robust scaled data: w/normalisation\nfrom sklearn.linear_model import Ridge\nridge = Ridge(fit_intercept = True, normalize = False)\nridge.fit(x_train,y_train['target_nitrogen_oxides'])\nridge_predict = ridge.predict(x_test)\nprint('\\nRidge score: of Robust scaled data: w/intercept, w/o normalisation', ridge.score(x_test,y_test['target_nitrogen_oxides']))\nprint('Mean Absolute % Error: ', np.mean(np.abs(ridge_predict - y_test['target_nitrogen_oxides'])/       \n                     np.abs(y_test['target_nitrogen_oxides'])))\n","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.696667Z","iopub.execute_input":"2021-07-30T11:47:55.699883Z","iopub.status.idle":"2021-07-30T11:47:55.73207Z","shell.execute_reply.started":"2021-07-30T11:47:55.699806Z","shell.execute_reply":"2021-07-30T11:47:55.730882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final regression model\n\n* parameters are Alpha = 1, which is by default 1; Normalise = false; Fit_intercept = true","metadata":{}},{"cell_type":"code","source":"test.drop(columns = 'date_time', inplace = True)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.737839Z","iopub.execute_input":"2021-07-30T11:47:55.740937Z","iopub.status.idle":"2021-07-30T11:47:55.751897Z","shell.execute_reply.started":"2021-07-30T11:47:55.740865Z","shell.execute_reply":"2021-07-30T11:47:55.750606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#  Robust Scaling of the test data\n\nfrom sklearn.preprocessing import RobustScaler\n\nscaler = RobustScaler() # class object\ntest = pd.DataFrame(scaler.fit_transform(test), columns = test.columns)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.761771Z","iopub.execute_input":"2021-07-30T11:47:55.765545Z","iopub.status.idle":"2021-07-30T11:47:55.789129Z","shell.execute_reply.started":"2021-07-30T11:47:55.765447Z","shell.execute_reply":"2021-07-30T11:47:55.787843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Final Ridge Model; 'target_carbon_monoxide', 'target_benzene', 'target_nitrogen_oxides'\n\nfrom sklearn.linear_model import Ridge\nridge = Ridge(alpha = 1.0, fit_intercept = True, normalize = False)\nridge.fit(x_train,y_train['target_carbon_monoxide'])\nsample['target_carbon_monoxide'] = ridge.predict(test)\n\nridge.fit(x_train,y_train['target_benzene'])\nsample['target_benzene'] = ridge.predict(test)\n\nridge.fit(x_train,y_train['target_nitrogen_oxides'])\nsample['target_nitrogen_oxides'] = ridge.predict(test)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.795459Z","iopub.execute_input":"2021-07-30T11:47:55.798928Z","iopub.status.idle":"2021-07-30T11:47:55.838416Z","shell.execute_reply.started":"2021-07-30T11:47:55.798847Z","shell.execute_reply":"2021-07-30T11:47:55.837117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample.to_csv('submission_Ridge regression.csv', index=False)\nsample","metadata":{"execution":{"iopub.status.busy":"2021-07-30T11:47:55.844569Z","iopub.execute_input":"2021-07-30T11:47:55.848034Z","iopub.status.idle":"2021-07-30T11:47:55.918279Z","shell.execute_reply.started":"2021-07-30T11:47:55.847955Z","shell.execute_reply":"2021-07-30T11:47:55.917014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}