{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <center> Open Machine Learning Course mlcourse.ai. English session #2\n",
    "\n",
    "### <center> Autor: Sudarkova Sveta (@swets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <center> Individual data analysis project </center>\n",
    "### <center> Predict quality of wine </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Research plan**\n",
    " - Feature and data explanation\n",
    " - Primary data analysis\n",
    " - Primary visual data analysis\n",
    " - Insights and found dependencies\n",
    " - Metrics selection\n",
    " - Model selection\n",
    " - Data preprocessing\n",
    " - Cross-validation and adjustment of model hyperparameters\n",
    " - Creation of new features and description of this process\n",
    " - Plotting training and validation curves\n",
    " - Prediction for test or hold-out samples\n",
    " - Conclusions\n",
    " \n",
    "Read more <a href=\"https://mlcourse.ai/roadmap\">here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 1. Feature and data explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This datasets is related to red variants of the Portuguese \"Vinho Verde\" wine. Due to privacy and logistic issues, only physicochemical (inputs) and sensory (the output) variables are available (e.g. there is no data about grape types, wine brand, wine selling price, etc.).\n",
    "\n",
    "The datasets can be viewed as classification or regression tasks. The classes are ordered and not balanced (e.g. there are much more normal wines than excellent or poor ones).\n",
    "\n",
    "This dataset is also available from the UCI machine learning repository, https://archive.ics.uci.edu/ml/datasets/wine+quality , I just shared it to kaggle for convenience. (If I am mistaken and the public license type disallowed me from doing so, I will take this down if requested.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Source & Acknowledgements<br>\n",
    "Original dataset was obtained from https://www.kaggle.com/uciml/red-wine-quality-cortez-et-al-2009 <br>\n",
    "This dataset is also available from the UCI machine learning repository, https://archive.ics.uci.edu/ml/datasets/wine+quality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. **Modeling wine preferences by data mining from physicochemical properties.** *Decision Support Systems*, 47(4):547-553, 2009. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature descriptions in dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**fixed acidity**<br> Most acids involved with wine or fixed or nonvolatile (do not evaporate readily).<br>Type: Numeric<br><br>\n",
    "**volatile acidity**<br> The amount of acetic acid in wine, which at too high of levels can lead to an unpleasant, vinegar taste.<br>Type: Numeric<br><br>\n",
    "**citric acid**<br> Found in small quantities, citric acid can add 'freshness' and flavor to wines.<br>Type: Numeric<br><br>\n",
    "**residual sugar**<br> The amount of sugar remaining after fermentation stops, it's rare to find wines with less than 1 gram/liter and wines with greater than 45 grams/liter are considered sweet.<br>Type: Numeric<br><br>\n",
    "**chlorides**<br> The amount of salt in the wine.<br>Type: Numeric<br><br>\n",
    "**free sulfur dioxide**<br> the free form of SO2 exists in equilibrium between molecular SO2 (as a dissolved gas) and bisulfite ion; it prevents microbial growth and the oxidation of wine.<br>Type: Numeric<br><br>\n",
    "**total sulfur dioxide**<br> Amount of free and bound forms of S02; in low concentrations, SO2 is mostly undetectable in wine, but at free SO2 concentrations over 50 ppm, SO2 becomes evident in the nose and taste of wine.<br>Type: Numeric<br><br>\n",
    "**density**<br> The density of water is close to that of water depending on the percent alcohol and sugar content.<br>Type: Numeric<br><br>\n",
    "**pH**<br> describes how acidic or basic a wine is on a scale from 0 (very acidic) to 14 (very basic); most wines are between 3-4 on the pH scale.<br>Type: Numeric<br><br>\n",
    "**sulphates**<br> a wine additive which can contribute to sulfur dioxide gas (S02) levels, wich acts as an antimicrobial and antioxidant.<br>Type: Numeric<br><br>\n",
    "**alcohol**<br> The percent alcohol content of the wine.<br>Type: Numeric<br><br>\n",
    "**quality**<br> output variable (based on sensory data, score between 0 and 10).<br>Type: Numeric<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 2-3. Primary data and visual data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "import seaborn as sns\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import statsmodels.stats.api as sm\n",
    "\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine = pd.read_csv('winequality-red.csv', sep=',', header=0)\n",
    "wine.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do some plotting to know how the data columns are distributed in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'fixed acidity', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that fixed acidity does not give any specification to classify the quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'volatile acidity', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that its quite a downing trend in the volatile acidity as we go higher the quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'citric acid', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Composition of citric acid go higher as we go higher in the quality of the wine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'residual sugar', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see, that hasn't got trend on sugar and quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'chlorides', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Composition of chloride also go down as we go higher in the quality of the wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'free sulfur dioxide', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see, that hasn't got trend on free sulfur dioxide and quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'total sulfur dioxide', data = wine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'sulphates', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sulphates level goes higher with the quality of wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,6))\n",
    "sns.barplot(x = 'quality', y = 'alcohol', data = wine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alcohol level also goes higher as te quality of wine increases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the distribution of expert assessments of wines in the sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize(8,6))\n",
    "stat = wine.groupby('quality')['quality'].agg(lambda x : float(len(x)) / wine.shape[0])\n",
    "stat.plot(kind='bar', fontsize=14, width=0.9, color=\"red\")\n",
    "plt.xticks(rotation=0)\n",
    "plt.ylabel('Proportion', fontsize=14)\n",
    "plt.xlabel('Quality', fontsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see, that data is normally distributed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see heatmap correlation between features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,6))\n",
    "sns.heatmap(wine.corr(),annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting, we can now see that acidity and pH level of wine are in inverse proportion. <br>\n",
    "This result illustrates a fact from a chemisty: as we know, pH = -lg[H+], whehe [H+] in fact represents acidity. So, bigger acidity - lower pH level. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,8))\n",
    "plt.scatter(wine['fixed acidity'],wine['pH'])\n",
    "plt.xlabel('Acidity').set_size(20)\n",
    "plt.ylabel('pH').set_size(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there is no direct connection between total acidity and pH (it is possible to find wines with a high pH for wine and high acidity).In wine tasting, the term “acidity” refers to the fresh, tart and sour attributes of the wine which are evaluated in relation to how well the acidity balances out the sweetness and bitter components of the wine such as tannins.\n",
    "\n",
    "Acidity in wine is important.\n",
    "\n",
    "As much as modern health has demonized acidic foods, acidity is an essential trait in wine that’s necessary for quality. Great wines are in balance with their 4 fundamental traits (Acidity, tannin, alcohol and sweetness) and as wines age, the acidity acts as a buffer to preserve the wine longer. For example, Sauternes, a wine with both high acidity and sweetness, is known to age several decades."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 4. Insights and found dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarise previous results, we can see that:\n",
    "- data is normally disturbuted;\n",
    "- we can find dependency between volatile acidity/citric acid/chlorides/sulphates/alcohol and quality of wine, we can use that in our model (very good for logistic regression);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 5. Metrics selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the classes are normally balanced. We making a classification task, and we can see, that we got some trends by features, and it's mean, that they can have weights. And it's offers to use MSE — measure of the quality of an estimator — it is always non-negative, and values closer to zero are better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 6. Model selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our task we will consider the following models:\n",
    "- **LogisticRegression**. A simple linear model is a good baseline in almost any classification problem. Its linear coefficients will also allow to assess the importance of a particular feature in the dataset, with the help of L1-regularization it will be possible to get rid of linearly dependent features.\n",
    "- **RandomForest**. A other model, used tree model, good baseline and not easy to interpretation, but in this cases can gives better results, as logistic regression.<br>\n",
    "Also, we can use xgboost model (or catboost, or other busting gradient method) but I just want to give you baseline classification model. <br>\n",
    "In the end I'll show some models without tuning parameters (to the future)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 7. Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check, how much NaN values in dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see, that we don't have NaN values (its very good). So, most of preprocessing is over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 8. Cross-validation and adjustment of model hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start Modeling\n",
    "# Split Training and Testing data (within training data file only)\n",
    "# Hold out method validation\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train = wine.iloc[:,1:len(wine.columns)-2] # this represents the input Features\n",
    "Y_train = wine.loc[:,'quality']\n",
    "\n",
    "\n",
    "# Scaling features (only feature NOT observation)\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "# Scaling - brings to range (0 to 1)\n",
    "ScaleFn = MinMaxScaler()\n",
    "X_Scale = ScaleFn.fit_transform(X_train)\n",
    "# Standardise - brings to Zero mean, Unit variance\n",
    "ScaleFn = StandardScaler()\n",
    "X_Strd = ScaleFn.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Proceed to perform PCA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "pca = PCA()\n",
    "x_pca = pca.fit_transform(X_Strd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the graph to find the principal components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,10))\n",
    "plt.plot(np.cumsum(pca.explained_variance_ratio_), 'ro-')\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AS per the graph, we can see that 7 principal components attribute for 90% of variation in the data. <br>\n",
    "We shall pick the first 7 components for our prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_new = PCA(n_components=8)\n",
    "x_new = pca_new.fit_transform(X_Strd)\n",
    "print(x_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_size = .30\n",
    "seedNo = 11\n",
    "\n",
    "\n",
    "X_train,X_test,Y_train,Y_test = train_test_split(x_new,Y_train,test_size = test_size, random_state = seedNo)\n",
    "\n",
    "print(\"train X\", X_train.shape)\n",
    "print(\"train Y\", Y_train.shape)\n",
    "print(\"test X\", X_test.shape)\n",
    "print(\"test Y\", Y_test.shape)\n",
    "\n",
    "# Choose algorithm and train\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import SVC\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "mymodel = []\n",
    "mymodel.append(('LogReg', LogisticRegression()))\n",
    "mymodel.append(('KNN', KNeighborsClassifier()))\n",
    "mymodel.append(('DeciTree', DecisionTreeClassifier()))\n",
    "mymodel.append(('RandForest', RandomForestClassifier()))\n",
    "mymodel.append(('SVM', SVC()))\n",
    "mymodel.append(('XGBoost', XGBClassifier()))\n",
    "\n",
    "\n",
    "\n",
    "All_model_result = []\n",
    "All_model_name = []\n",
    "for algoname, algorithm in mymodel:    \n",
    "    kfoldFn = KFold(n_splits = 11, random_state = seedNo)\n",
    "    Eval_result = cross_val_score(algorithm, X_train, Y_train, cv = kfoldFn, scoring = 'accuracy')\n",
    "    \n",
    "    All_model_result.append(Eval_result)\n",
    "    All_model_name.append(algoname)\n",
    "    print(\"Modelname and Model accuracy:\", algoname, 100*Eval_result.mean(),\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build rf model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestRegressor(n_estimators=100, min_samples_leaf=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Качество выросло ещё сильнее, хотя модель и переобучилась:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sqrt(mean_squared_error(rf.predict(X_train), Y_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sqrt(mean_squared_error(rf.predict(X_test), Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A true evaluation of wines and their predictions of a random forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize(16,7))\n",
    "plt.subplot(121)\n",
    "pyplot.scatter(Y_train, rf.predict(X_train), color=\"red\", alpha=0.1)\n",
    "pyplot.xlim(2.5,9.5)\n",
    "pyplot.ylim(2.5,9.5)\n",
    "plot(range(11), color='black')\n",
    "grid()\n",
    "pyplot.title('Train set', fontsize=20)\n",
    "pyplot.xlabel('Quality', fontsize=14)\n",
    "pyplot.ylabel('Estimated quality', fontsize=14)\n",
    "\n",
    "plt.subplot(122)\n",
    "pyplot.scatter(Y_test, rf.predict(X_test), color=\"red\", alpha=0.1)\n",
    "pyplot.xlim(2.5,9.5)\n",
    "pyplot.ylim(2.5,9.5)\n",
    "plot(range(11), color='black')\n",
    "grid()\n",
    "pyplot.title('Test set', fontsize=20)\n",
    "pyplot.xlabel('Quality', fontsize=14)\n",
    "pyplot.ylabel('Estimated quality', fontsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficient of determination for the random forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf.score(X_test, Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center> 9. Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In conclusion, I want to say, that we can:\n",
    "- use stacking model;\n",
    "- generate new features;\n",
    "- find more insights;\n",
    "- use other metrics and models.<br>\n",
    "\n",
    "Thank you for listening!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}