{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "related-stream",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import plot_confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unauthorized-damage",
   "metadata": {},
   "source": [
    "# Material classification\n",
    "\n",
    "Neutron scattering can give complementary information about materials, such as the structural information from diffraction and electronic information from spectroscopy. \n",
    "In this exercise, we will look at using a random forest classification of structural and electronic information to create a model that can guess the [space group](https://en.wikipedia.org/wiki/Space_group) for new materials.\n",
    "\n",
    "The first thing that we need is some data, I have obtained data for lithium, sodium, and potassium containing materials of the four most common space groups ([space group numbers](https://en.wikipedia.org/wiki/List_of_space_groups): 2, 14, 15, 19). \n",
    "This data can be found in the file [`materials_data.csv`](https://github.com/arm61/trad_ml_methods/raw/main/materials_data.csv). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "tribal-lying",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('materials_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "first-christian",
   "metadata": {},
   "source": [
    "This data is read in as a `pandas.DataFrame` so it can be visualised as so. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "determined-korea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pretty_formula</th>\n",
       "      <th>spacegroup.number</th>\n",
       "      <th>band_gap</th>\n",
       "      <th>density</th>\n",
       "      <th>volume</th>\n",
       "      <th>formation_energy_per_atom</th>\n",
       "      <th>nsites</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>KAl9O14</td>\n",
       "      <td>14</td>\n",
       "      <td>4.1426</td>\n",
       "      <td>3.142646</td>\n",
       "      <td>534.648960</td>\n",
       "      <td>-3.364229</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>K2Se2O</td>\n",
       "      <td>15</td>\n",
       "      <td>0.0975</td>\n",
       "      <td>3.303683</td>\n",
       "      <td>253.443517</td>\n",
       "      <td>-0.940536</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K2AgF4</td>\n",
       "      <td>14</td>\n",
       "      <td>0.6151</td>\n",
       "      <td>3.343987</td>\n",
       "      <td>260.263084</td>\n",
       "      <td>-2.371859</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Na2TeS3</td>\n",
       "      <td>14</td>\n",
       "      <td>1.7919</td>\n",
       "      <td>2.904619</td>\n",
       "      <td>616.908734</td>\n",
       "      <td>-1.000993</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>K3Zn2Cl7</td>\n",
       "      <td>2</td>\n",
       "      <td>3.9024</td>\n",
       "      <td>2.345705</td>\n",
       "      <td>702.644832</td>\n",
       "      <td>-1.909423</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>537</th>\n",
       "      <td>Na2Cr2O7</td>\n",
       "      <td>2</td>\n",
       "      <td>2.5253</td>\n",
       "      <td>2.654095</td>\n",
       "      <td>655.601817</td>\n",
       "      <td>-1.869509</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>538</th>\n",
       "      <td>K4SnO4</td>\n",
       "      <td>2</td>\n",
       "      <td>2.3423</td>\n",
       "      <td>2.977324</td>\n",
       "      <td>378.252471</td>\n",
       "      <td>-1.839865</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>539</th>\n",
       "      <td>LiTa3O8</td>\n",
       "      <td>15</td>\n",
       "      <td>3.3841</td>\n",
       "      <td>7.889695</td>\n",
       "      <td>285.303789</td>\n",
       "      <td>-3.263699</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>540</th>\n",
       "      <td>LiSmO2</td>\n",
       "      <td>14</td>\n",
       "      <td>3.7655</td>\n",
       "      <td>6.485182</td>\n",
       "      <td>193.881819</td>\n",
       "      <td>-3.185751</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>541</th>\n",
       "      <td>K4UO5</td>\n",
       "      <td>2</td>\n",
       "      <td>2.1910</td>\n",
       "      <td>4.207670</td>\n",
       "      <td>374.454975</td>\n",
       "      <td>-2.645721</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>542 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    pretty_formula  spacegroup.number  band_gap   density      volume  \\\n",
       "0          KAl9O14                 14    4.1426  3.142646  534.648960   \n",
       "1           K2Se2O                 15    0.0975  3.303683  253.443517   \n",
       "2           K2AgF4                 14    0.6151  3.343987  260.263084   \n",
       "3          Na2TeS3                 14    1.7919  2.904619  616.908734   \n",
       "4         K3Zn2Cl7                  2    3.9024  2.345705  702.644832   \n",
       "..             ...                ...       ...       ...         ...   \n",
       "537       Na2Cr2O7                  2    2.5253  2.654095  655.601817   \n",
       "538         K4SnO4                  2    2.3423  2.977324  378.252471   \n",
       "539        LiTa3O8                 15    3.3841  7.889695  285.303789   \n",
       "540         LiSmO2                 14    3.7655  6.485182  193.881819   \n",
       "541          K4UO5                  2    2.1910  4.207670  374.454975   \n",
       "\n",
       "     formation_energy_per_atom  nsites  \n",
       "0                    -3.364229      48  \n",
       "1                    -0.940536      10  \n",
       "2                    -2.371859      14  \n",
       "3                    -1.000993      24  \n",
       "4                    -1.909423      24  \n",
       "..                         ...     ...  \n",
       "537                  -1.869509      44  \n",
       "538                  -1.839865      18  \n",
       "539                  -3.263699      24  \n",
       "540                  -3.185751      16  \n",
       "541                  -2.645721      20  \n",
       "\n",
       "[542 rows x 7 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assisted-better",
   "metadata": {},
   "source": [
    "The first thing to do is separate our data into training and validation data (this process is common in machine learning methods). \n",
    "For this we will use the `scikit-learn` function [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "falling-sailing",
   "metadata": {},
   "outputs": [],
   "source": [
    "train, validate = train_test_split(data, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "immune-surprise",
   "metadata": {},
   "source": [
    "We have split the data, which consisted of data about 542 materials, so that 80 % will be used to train and the remaining 20 % is reserved for validation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "loose-pressure",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pretty_formula</th>\n",
       "      <th>spacegroup.number</th>\n",
       "      <th>band_gap</th>\n",
       "      <th>density</th>\n",
       "      <th>volume</th>\n",
       "      <th>formation_energy_per_atom</th>\n",
       "      <th>nsites</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>KInS2</td>\n",
       "      <td>15</td>\n",
       "      <td>2.0103</td>\n",
       "      <td>3.080923</td>\n",
       "      <td>940.171139</td>\n",
       "      <td>-1.270707</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>479</th>\n",
       "      <td>Na4SiO4</td>\n",
       "      <td>2</td>\n",
       "      <td>3.2269</td>\n",
       "      <td>2.568896</td>\n",
       "      <td>237.930444</td>\n",
       "      <td>-2.400403</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212</th>\n",
       "      <td>Na2(ReSe2)3</td>\n",
       "      <td>15</td>\n",
       "      <td>1.2790</td>\n",
       "      <td>6.859253</td>\n",
       "      <td>1044.230215</td>\n",
       "      <td>-0.603486</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>KC4N3</td>\n",
       "      <td>2</td>\n",
       "      <td>3.7294</td>\n",
       "      <td>1.393763</td>\n",
       "      <td>307.767109</td>\n",
       "      <td>-0.042663</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>Na7In3Se8</td>\n",
       "      <td>2</td>\n",
       "      <td>1.1676</td>\n",
       "      <td>3.824842</td>\n",
       "      <td>987.301832</td>\n",
       "      <td>-0.915512</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>407</th>\n",
       "      <td>K2Al2Sb3</td>\n",
       "      <td>14</td>\n",
       "      <td>0.9992</td>\n",
       "      <td>3.561968</td>\n",
       "      <td>1855.194833</td>\n",
       "      <td>-0.363389</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>Li4P2O7</td>\n",
       "      <td>14</td>\n",
       "      <td>5.6533</td>\n",
       "      <td>2.258198</td>\n",
       "      <td>593.292328</td>\n",
       "      <td>-2.751962</td>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>327</th>\n",
       "      <td>Na2Te2O5</td>\n",
       "      <td>15</td>\n",
       "      <td>3.0831</td>\n",
       "      <td>4.378071</td>\n",
       "      <td>578.299060</td>\n",
       "      <td>-1.757621</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>423</th>\n",
       "      <td>K4ZnO3</td>\n",
       "      <td>2</td>\n",
       "      <td>1.6956</td>\n",
       "      <td>2.756436</td>\n",
       "      <td>650.135219</td>\n",
       "      <td>-1.503639</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>425</th>\n",
       "      <td>KPbO2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.4532</td>\n",
       "      <td>6.172046</td>\n",
       "      <td>598.988692</td>\n",
       "      <td>-1.581835</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>433 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    pretty_formula  spacegroup.number  band_gap   density       volume  \\\n",
       "53           KInS2                 15    2.0103  3.080923   940.171139   \n",
       "479        Na4SiO4                  2    3.2269  2.568896   237.930444   \n",
       "212    Na2(ReSe2)3                 15    1.2790  6.859253  1044.230215   \n",
       "188          KC4N3                  2    3.7294  1.393763   307.767109   \n",
       "171      Na7In3Se8                  2    1.1676  3.824842   987.301832   \n",
       "..             ...                ...       ...       ...          ...   \n",
       "407       K2Al2Sb3                 14    0.9992  3.561968  1855.194833   \n",
       "357        Li4P2O7                 14    5.6533  2.258198   593.292328   \n",
       "327       Na2Te2O5                 15    3.0831  4.378071   578.299060   \n",
       "423         K4ZnO3                  2    1.6956  2.756436   650.135219   \n",
       "425          KPbO2                  2    1.4532  6.172046   598.988692   \n",
       "\n",
       "     formation_energy_per_atom  nsites  \n",
       "53                   -1.270707      32  \n",
       "479                  -2.400403      18  \n",
       "212                  -0.603486      44  \n",
       "188                  -0.042663      16  \n",
       "171                  -0.915512      36  \n",
       "..                         ...     ...  \n",
       "407                  -0.363389      56  \n",
       "357                  -2.751962      52  \n",
       "327                  -1.757621      36  \n",
       "423                  -1.503639      32  \n",
       "425                  -1.581835      32  \n",
       "\n",
       "[433 rows x 7 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "preliminary-marketplace",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pretty_formula</th>\n",
       "      <th>spacegroup.number</th>\n",
       "      <th>band_gap</th>\n",
       "      <th>density</th>\n",
       "      <th>volume</th>\n",
       "      <th>formation_energy_per_atom</th>\n",
       "      <th>nsites</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Na3CoO3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.8575</td>\n",
       "      <td>2.947379</td>\n",
       "      <td>198.203241</td>\n",
       "      <td>-1.453332</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>480</th>\n",
       "      <td>RbLiF2</td>\n",
       "      <td>15</td>\n",
       "      <td>6.1608</td>\n",
       "      <td>3.242957</td>\n",
       "      <td>267.093984</td>\n",
       "      <td>-3.028814</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>509</th>\n",
       "      <td>Na3BS3</td>\n",
       "      <td>15</td>\n",
       "      <td>2.6404</td>\n",
       "      <td>1.964923</td>\n",
       "      <td>297.430370</td>\n",
       "      <td>-1.235997</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>KEu2I5</td>\n",
       "      <td>14</td>\n",
       "      <td>1.3409</td>\n",
       "      <td>4.987760</td>\n",
       "      <td>1301.792889</td>\n",
       "      <td>-1.579361</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348</th>\n",
       "      <td>Na6Sn2S7</td>\n",
       "      <td>15</td>\n",
       "      <td>2.0948</td>\n",
       "      <td>2.597359</td>\n",
       "      <td>766.943648</td>\n",
       "      <td>-1.218262</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>Li3CrF6</td>\n",
       "      <td>15</td>\n",
       "      <td>3.9646</td>\n",
       "      <td>2.839623</td>\n",
       "      <td>655.448686</td>\n",
       "      <td>-3.142304</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>KGaCl4</td>\n",
       "      <td>14</td>\n",
       "      <td>4.6049</td>\n",
       "      <td>2.201932</td>\n",
       "      <td>756.038494</td>\n",
       "      <td>-1.881241</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Na6Ge2Se7</td>\n",
       "      <td>15</td>\n",
       "      <td>1.6000</td>\n",
       "      <td>3.363290</td>\n",
       "      <td>825.447012</td>\n",
       "      <td>-0.846623</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>LiIO5</td>\n",
       "      <td>14</td>\n",
       "      <td>0.2279</td>\n",
       "      <td>3.483491</td>\n",
       "      <td>407.744687</td>\n",
       "      <td>-0.730449</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>Na3AlTe3</td>\n",
       "      <td>14</td>\n",
       "      <td>2.2355</td>\n",
       "      <td>3.517580</td>\n",
       "      <td>904.012910</td>\n",
       "      <td>-0.804205</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>109 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    pretty_formula  spacegroup.number  band_gap   density       volume  \\\n",
       "46         Na3CoO3                  2    0.8575  2.947379   198.203241   \n",
       "480         RbLiF2                 15    6.1608  3.242957   267.093984   \n",
       "509         Na3BS3                 15    2.6404  1.964923   297.430370   \n",
       "251         KEu2I5                 14    1.3409  4.987760  1301.792889   \n",
       "348       Na6Sn2S7                 15    2.0948  2.597359   766.943648   \n",
       "..             ...                ...       ...       ...          ...   \n",
       "368        Li3CrF6                 15    3.9646  2.839623   655.448686   \n",
       "43          KGaCl4                 14    4.6049  2.201932   756.038494   \n",
       "14       Na6Ge2Se7                 15    1.6000  3.363290   825.447012   \n",
       "292          LiIO5                 14    0.2279  3.483491   407.744687   \n",
       "197       Na3AlTe3                 14    2.2355  3.517580   904.012910   \n",
       "\n",
       "     formation_energy_per_atom  nsites  \n",
       "46                   -1.453332      14  \n",
       "480                  -3.028814      16  \n",
       "509                  -1.235997      14  \n",
       "251                  -1.579361      32  \n",
       "348                  -1.218262      30  \n",
       "..                         ...     ...  \n",
       "368                  -3.142304      60  \n",
       "43                   -1.881241      24  \n",
       "14                   -0.846623      30  \n",
       "292                  -0.730449      28  \n",
       "197                  -0.804205      28  \n",
       "\n",
       "[109 rows x 7 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "increasing-journalism",
   "metadata": {},
   "source": [
    "We then define the columns of interest and segment our data into the `X` and `y` elements. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dental-asset",
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = ['band_gap', 'formation_energy_per_atom']\n",
    "X_train = train[columns]\n",
    "y_train = train['spacegroup.number']\n",
    "X_validate = validate[columns]\n",
    "y_validate = validate['spacegroup.number']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mobile-france",
   "metadata": {},
   "source": [
    "With the data split up, we can define our random forest model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "maritime-underground",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RandomForestClassifier(n_estimators=100).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "commercial-decimal",
   "metadata": {},
   "source": [
    "Let's see how well our model managed to classify the validation data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "electric-morrison",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATwAAAEKCAYAAACPJum2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAApoklEQVR4nO3dd5gV5dn48e99yu6yjbK7LF2KgCJ2BNGoaCyYIj9jYizRGI1YMBpLEmOPifoaY4mJ5cUS02yxJOQNAcSoqJGuIh2ks7tsAZaybDvn/v1xZiu7Z8/CnLbn/lzXXJyZeWbmnj3Lvc8zz8wzoqoYY0wq8MQ7AGOMiRVLeMaYlGEJzxiTMizhGWNShiU8Y0zKsIRnjEkZlvCMMQlHRF4UkVIRWdrOehGRJ0VkrYgsEZHjItmvJTxjTCJ6CZgYZv25wHBnmgw8E8lOLeEZYxKOqs4BtocpMgn4k4bMBXqISN+O9utzK8BoSvNlakZ6j3iH4bq0QbXxDiFq6tYnxa9Wp2lN1/zOqtlLrdbIwezjnNOztGJ7IKKyi5bULAOqmy2aqqpTO3G4/sDmZvNbnGXF4TZKit/KjPQenDhqcrzDcF3fpzbFO4SoKb0kL94hREX9ug3xDiEq5um7B72Piu0B5s8cFFFZb9811ao65qAP2klJkfCMMYlPgSDBWB1uKzCw2fwAZ1lYdg3PGOMKRanTQESTC6YBlzu9tScClaoatjkLVsMzxrjIrRqeiLwCTADyRWQLcC/gB1DVZ4HpwNeAtUAV8INI9msJzxjjCkUJuDTcnKpe3MF6BaZ0dr+W8IwxrgmS2ONrWsIzxrhCgYAlPGNMqrAanjEmJShQl+CvjLCEZ4xxhaLWpDXGpAiFQGLnO0t4xhh3hJ60SGyW8IwxLhECHNT4A1FnCc8Y44pQp4UlPGNMCgjdh2cJzxiTIoJWwzPGpAKr4RljUoYiBBJ8xDlLeMYY11iT1hiTEhShVr3xDiMsS3jGGFeEbjy2Jq0xJkVYp0UCOf64Iq67ehEejzLjnWG8/sYRLdaPPqKUa69exJDBO3no1yfz0X+b3sD0r7+/woaN3QEoK8vivl+dFtPYw6n+pJ7Kx6shCJnn+cm5PL3F+sonqqlZFHqPgFYrwR1K39k5ABSdtBvfsNBfZW+hkPebzNgG3wnHj9vG5Ju+wOOBWf83iL/9ZUSL9UccXc7kG5cyZNguHr5vDB+/3y9OkR68MRN2ce0vi/B6lH+/0ovXf18Y75A6pCoE1Gp4LYjIQOBPQCGhWvBUVf1ttI/r8QSZcu1C7rj7DMoruvHkYzOZO28AmzZ3byxTVpbJo0+cyAXnr9hv+9paL1Nu+lq0w+w0DSiVv6km78lMvL2Fsh9UkXGKD/+Qpmsp3X+c0fh5z+u11K1ueomKpEPvP2fFNOYD4fEo192yhLtuPony0m48/vwHzP2oD5s35DaWKduWyeMPHsu3Ll4bx0gPnsejTHlwKz+/aCjlxX5+N30Nc2d2Z9OajI43jrOg1fD2Uw/cqqqLRSQHWCQi76jq8mgedOTwCoqLsynZlg3AB3MOYfy4LS0S3rbS0DpN8J6m5uqWB/EN8ODrH/rL2u0sH9Vz6lskvOb2vVNHzg/T21yXyEYcvoOiLVmUFIWS85zZ/TnxKyUtEl5pSah2qsHk+f7aMvLYKoo2pFGyKfQ9vf+PHow/pzLhE16o0yKxG40xr3+qarGqLnY+7wZWEHpjeFTl5e2jrLypJlNekUleXlXE26elBXjysRk8/shMxp+4ueMNYiRQFsTbu+lr9Pb2EChre4ye+uIggSIlfUxTMtRaKLtiL2VX7WXfB3VRj/dA5RVUU17arXG+vKwbeQXVYbZIXnl96igrSmucLy/2k983cb+bBg2dFpFM8RLXdCwig4FjgXnxjCMSl185iYrtmfQp3MPDD7zLhg09KC7JiXdYnbLvnTq6ne5DvE01oMK3s/D29lC/NUjFlCr8w7z4BiT2dRiTuAIJ3jqK22+2iGQDbwI/VtVdbayfLCILRWRhXX3kNbH2VFR0oyB/b+N8fl4VFRWRX6Cv2B4qW7ItmyVLezNs6I6DjskN3gIPgdKmUcgCpUG8BW3/0u2bXU+3s/0tt3dqh77+HtKO87a4vpdIKsoyyO+9r3E+v2AfFWWJ3cQ7UBUlfgr61TbO5/eto7zYH2aLxNDwpEUkU7zE5cgi4ieU7P6qqm+1VUZVp6rqGFUd4/cdfM/hqjV59Ou3m8LCPfh8AU47dSNz50fWks7OqsXvCyWC3NxqRh1e1uLaXzz5D/dQvzlIfVEQrVP2vVNPxin7V9zrNgTQXYr/yKavPLhL0dpQ8zewM0jtkgC+IYlZu1u9sgf9B+6lsO9efL4gp565lXkf94l3WFGx6rNM+g+ppXBgDT5/kAmTdjJ3VmL8vnUkqJ6IpniJRy+tAC8AK1T1sVgdNxj08PSzY3jgF+/h8SizZg9l46YeXHbpEtas6cXc+QMYMbyCu++YQ052LeNO2Mpll37BNVO+zsCBldw4ZT6qgojy+htHJEzCE5/Q/bYMKm6qCt2W8g0//qFedk2tIe0wLxmnhr7ife/U0+0sP6Eff0j9hiA7H64GARSyL09rt7Mj3oIBD888dhS/fOwTPB7lnX8NYtP6XL531QrWrOzBvI/7MvywHdz14Hyyc+oYe3IJl161kusvOyPeoXdaMCA8dWd/Hnx5HR4vzHq1FxtXJ35tNjR4QGL+wWwgGuO3DInIV4APgS9oGhH6DlWd3t42uVn99MRRk2MRXkz1fWpTvEOImtJL8uIdQlTUr9sQ7xCiYp6+yy7dflAX4IYcma33vXVURGWvGPHJIlUdczDHOxAxr+Gp6keQ4DfrGGM6TRW78dgYkyrEbjw2xqQGxWp4xpgUkuidFpbwjDGuUMQGADXGpIbQaxoTO6UkdnTGmCRiL+I2xqQIhbg+RREJS3jGGNckeg0vsdOxMSZpqIqrz9KKyEQRWSUia0Xk9jbWDxKR90TkUxFZIiIdjtBrNTxjjCtCnRbuPIstIl7gKeAsYAuwQESmtRoo+C7gdVV9RkRGAdOBweH2awnPGOMSV99pMRZYq6rrAETkVWAS0DzhKdAw5HV3oKijnVrCM8a4ItRpEfE1vHwRWdhsfqqqTm023x9oPrT4FmBcq33cB8wSkR8BWcCZHR3UEp4xxjWdeNKi3IXRUi4GXlLVR0VkPPBnERmtqsH2NrCEZ4xxhctPWmwFBjabH+Asa+4qYCKAqn4iIhlAPlDa3k6tl9YY4xoXX+KzABguIkNEJA24CJjWqswm4KsAInI4kAGUhdup1fCMMa5QhbqgO3UoVa0XkRuAmYAXeFFVl4nI/cBCVZ0G3Ao8JyI3E7qEeIV2MKKxJTxjjCtCTVr3Go3OKOjTWy27p9nn5cDJndmnJTxjjGsS/UkLS3jGGFd08raUuLCEZ4xxibtN2miwhGeMcY2908IFUluPZ1O7t9YkrZpAUvz4D0htvx7xDiEqvFvT4x1CdNQcfKIK9dIm5nuNG3Td/3HGmJiyId6NMSnFmrTGmJRgvbTGmJRivbTGmJSgKtRbwjPGpApr0hpjUoJdwzPGpBRLeMaYlGD34RljUordh2eMSQmqUO/SAKDRYgnPGOMaa9IaY1KCXcMzxqQUtYRnjEkV1mlhjEkJqnYNzxiTMoSA9dIaY1KFXcMzxqQEe5bWGJM6NHQdL5FZwjPGuMZ6aY0xKUGt08IYk0qsSZtAjj+pnGt+thqPR5n5dn/+9uLgFut9/iC3PbCMQw/fxe5KPw/99EhKi7oBMHj4bn5090oys+vRINx0yVjqahPjHZy1c+vY+0QVBCHjm+l0uyyjxfq9v62ibnE9AFqj6A6l18wejeuDe5XKSyvxn5JG9q2ZsQw9rDFHb+X6H8zH41H+/e5wXvvHkS3WH3l4Cdd9fwFDD9nBA0+cyofzBjeu++GlCxl33BY8Aou+6MvTfxgLCdTcOv7UnVx37yY8HmXGawW8/my/Fuv9aUFue3Qdw0fvZddOHw/dcCjbtqZz7FcqufKnm/H5lfo64fmHBvH5J7lxOov9JXovbdTqnyLyooiUisjSNtbdKiIqIvnROn5rHo9y/R2ruOf6Y7j2/PGcNrGEgUP3tChzzvlb2bPLxw+/eTJv/2UQV/54bWhbb5CfPLiM3//qMK771nh+dtXxBOoTo+quAWXvo1XkPppNj7/mUjO7lvr1gRZlsm7KpMcfc+nxx1wyLkgn7TR/i/X7ntuH75jE+tvnkSA/umoudzx4Jj+8eRKnn7yeQf13tihTWp7NI0+fzH8+GtJi+agRpYweWco1t53H1beex8hhFRw1alsMow/P41Gm3L+Ru64YweSzj2TCeRUMOnRfizLnXFjGnkovV55+NG+/0Icrb98MwK7tPu794QiuO/dIfnPbUH7y2JfxOIU2qYYSXiRTvETzf+1LwMTWC0VkIHA2sCmKx97PiNGVFG3uRsnWTOrrPcyZUcj4CWUtypx4ehmzp/UF4KN3enP02O2Actz47axfk8361TkA7K5MIxhMjL9k9SsCeAd48Pb3In4h/at+6j6sbbd87exa0s5Ma9p+ZT3B7UH8J/jb3SYeRh5aTlFJLiWlOdQHvLz/3yGcdMLmFmW2lWWzflOv/f4DqYI/LYDPF8TvD+LzBtlZ2bLWG08jj95D8cZ0SjZnUF/n4YN/5jH+rB0tyow/awez3wzVBz78dy+OOWkXoHy5PIvtpaHvb+PqbqRnBPGnBWN9Cu0KqkQ0xUvUEp6qzgG2t7HqceCnhG7biZm83jWUlzT90peXZpBXWLNfmTKnTDDgoWqPj9wedfQ/pAoUfvnMYp58dR7fvmJDLEMPK1gWxNO76Wv09PYQKGv7RxsoCRAoDuI/PlSb06Cy9/f7yLwhcZqxDfJ7VVFWkdU4X16RSX6vvRFtu2JNbz5f1ofXpr7Oa1NfZ+Hn/di0tUeUIu28vD51lBWnN86Xl6SR16flH6m8wqYywYCwd7eX3J71Lcp85dwdrF2aRV1tYrQ2oKGW1/EULzFtx4jIJGCrqn4uEj7Li8hkYDJAhic7BtG1z+tVRh27kx9fMpaaai8PTl3MmuW5fD6/V1zj6qza2XWkT0hDvKGfffVbNaSN9+Pt7aEuzrG5qV/hLgb1r+Tia78DwMN3v8Pow7axdGVhnCNzzyHDq7jyZ5u58/KR8Q6lkSIEE7yXNmbRiUgmcAdwTyTlVXWqqo5R1TFpnm4HffyK0nTy+1Q3zuf3rqZiW/p+ZQqcMh5vkMzsenbt9FNems7SRT3ZtTONmmovCz/K49DDdx10TG7wFHgIljY1aYKlQbwFbf8xqZldS9pZzZqzSwNUv1nNjgsqqXpqH7Uzatj7TFXUY45E+fZMCvKaanT5eVWUb88Ks0WTk8duYsWaAqpr/FTX+FnwaX9GjSjreMMYqSjxU9C3qXWR36eWipK0lmW2NZXxeJWsnAC7dvgay9/9v2v4za1DKd6UOE11CDXbIpniJZbpeBgwBPhcRDYAA4DFItInFgdfvSyXfoP2Udh/Hz5fkFMnbmPuBwUtysx7v4AzzysG4CtnlbJkfk9AWPxxHoOH7yE9I4DHG2T08TvZtC6+tc4GvsO8BLYECRQF0Dql5t06/F9J269cYGMA3a34Rjf1LOfcl0XPt3rQ883uZE7pRtrEdLKuS4zm7aov8+nfdxd9Cnbj8waYcNJ6Plk4IKJtS8uzOOrwbXg8QbzeIEeNKmHT1u5Rjjhyq5Zk029wDYUDavD5g5z2zQrmzu7Roszc2T0584JyAE45d7vTEytk5dRz/4ur+MPDA1m+KCf2wYfjcqeFiEwUkVUislZEbm+nzIUislxElonIyx3tM2ZNWlX9AujdMO8kvTGqWh6L4wcDHp55aCS/euZTPB5l1t/7senLbL53/ZesWZbLvA8KmPl2P257YBnP//Njdu/y8/BPRwOwZ7eft/88iCdeno8qLPwwnwUfxqyDOSzxCVk3Z7Lrlj0QgPRvpOEb6qXquX34DvOSdkoo+dXMriXtTD8dXUpIFMGgh9+/OI6H7pyNxxNk5nvD2bilJ9+/8FNWf5nHJ4sGMWJYOffd9h7ZWbWcePwWLr/wM66+9f/x4dxDOGZ0Cc/9ZhoKLPisP3MXDYz3KTUKBoSn7z2EB/60Eo8HZv2tgI1rMrns5i2s+SKLubN7MuO1An76+Je8+N7n7K708dCPhgFw3ve30e+QGi65sYhLbiwC4I7LR1JZkSCdTi5V30TECzwFnAVsARaIyDRVXd6szHDg58DJqrpDRHq3vbdm+9UoXUEUkVeACUA+sA24V1VfaLZ+AxEmvO7+3jq+17ejEmc89fh7fceFklTZzw6JdwhR4V2wIt4hRMXcmn+zK1hxUH8NM4b114H/c11EZddeePciVR3T3noRGQ/cp6rnOPM/B1DVh5qV+TWwWlWfjzTGdmt4IvI7wuRrVb0x3I5V9eIO1g/uKDhjTPJQ6MztWvkisrDZ/FRVndpsvj/Q/D6kLcC4VvsYASAiHwNeQglyRriDhmvSLgyzzhhjWlIg8nvsysPV8CLkA4YTakkOAOaIyJGqujPcBm1S1T82nxeRTFVNjC48Y0xCcvEK2Vag+YXXAc6y5rYA81S1DlgvIqsJJcAF7e20w15aERkvIsuBlc780SLydCeDN8akAvfuS1kADBeRISKSBlwETGtV5u+Eanc4j6mOANaF22kkt6U8AZwDVACo6ufAqRGFbIxJIZHdkhLJbSmqWg/cAMwEVgCvq+oyEblfRM5zis0EKpwK2XvAT1S1Itx+I7otRVU3t7qdIdBeWWNMCnPxpg9VnQ5Mb7XsnmafFbjFmSISScLbLCInASoifuAmQhnXGGOaKGiCDKrRnkiatNcCUwh1ExcBxzjzxhjTikQ4xUeHNTznxuBLYxCLMSbZJfiIx5H00g4VkX+KSJkzoOc/RGRoLIIzxiSZBB89IJIm7cvA60BfoB/wN+CVaAZljElCDTceRzLFSSQJL1NV/6yq9c70FyCxxqQxxiSEpB0AVEQaRrf8tzM0y6uEcvh3adVVbIwxACR4L224TotFhBJcwxlc02ydEhqWxRhjGkmCd1qEe5Z2SHvrjDFmP/EezjgCET1pISKjgVE0u3anqn+KVlDGmGQU3w6JSHSY8ETkXkIP6I4idO3uXOAjwBKeMaalBK/hRdJL+23gq0CJqv4AOBpInBcEGGMSRzDCKU4iadLuU9WgiNSLSC5QSstxqowxprMDgMZFJAlvoYj0AJ4j1HO7B/gkmkEZY5JT0vbSNlDV652Pz4rIDCBXVZdENyxjTFJK1oQnIseFW6eqi6MTkjHGREe4Gt6jYdYpcIbLsbR/sDQfwUEdvnIy6XxRmuB/Dg9CQWbMXnkcU56amniHEB0uPe+VtE1aVT09loEYY5KcktSPlhljTOckaw3PGGM6K2mbtMYY02kJnvAiGfFYROR7InKPMz9IRMZGPzRjTNLpAiMePw2MBy525ncDT0UtImNMUhKNfIqXSJq041T1OBH5FEBVdzhvAjfGmJa6QC9tnYh4cSqiIlJAXB//NcYkqkTvtIikSfsk8DbQW0QeIDQ01INRjcoYk5wS/BpeJM/S/lVEFhEaIkqA/6eqK6IemTEmucT5+lwkIhkAdBBQBfyz+TJV3RTNwIwxSSjZEx7wL5pe5pMBDAFWAUdEMS5jTBKSBL+6H0mT9sjm884oKte3U9wYYxJWp5+0UNXFIjIuGsEYY5JcsjdpReSWZrMe4DigKGoRGWOSU1fotABymn2uJ3RN783ohGOMSWrJnPCcG45zVPW2GMVjjElmyZrwRMSnqvUicnIsAzLGJCch8Xtpwz1pMd/59zMRmSYil4nItxqmWARnjEkiLg8eICITRWSViKwVkdvDlLtARFRExnS0z0iu4WUAFYTeYdFwP54Cb0UWtjEmZbjUpHUupz0FnAVsARaIyDRVXd6qXA5wEzAvkv2GS3i9nR7apTQlugYJ3lI3xsSFe5lhLLBWVdcBiMirwCRgeatyvwQeBn4SyU7DNWm9QLYz5TT73DAZY0wLnWjS5ovIwmbT5Fa76g9sbja/xVnWdKzQQxADVfVfkcYXroZXrKr3R7qjZHD8cUVcd/UiPB5lxjvDeP2Nlk/HjT6ilGuvXsSQwTt56Ncn89F/BzWu+9ffX2HDxu4AlJVlcd+vTotp7OGkL95D9xdKIKhUndmTPRfk71cm4+NKcl4tAxHqBqez85YBjeukKkDvG7+kemwOlZP7xjL0sE44cgs3XDIXjyfI9DkjeeVfR7dY/+1zvuBrp64mEBQqd2fwyAunsK0idBfV/9w6g1HDyvhidSF3PnF2PMI/KGMm7OLaXxbh9Sj/fqUXr/++MN4hRSbyGl65qnZ4za09IuIBHgOu6Mx24RLeQY3kJyIvAt8ASlV1tLPsPuBqoMwpdoeqTj+Y40TK4wky5dqF3HH3GZRXdOPJx2Yyd94ANm3u3limrCyTR584kQvO338wmNpaL1Nu+losQu2cgNJ9ajEV9x1CIM9PwU/XUT02h/qB6Y1FvEU15LxZQflDQ9BsL56d9S12kfNyGTWjMmMdeVgeCXLTZf/lJ49MpGx7Fs/cO43/fjqIjUU9G8us3ZjHdb+YRE2tj/NOX8HkCxfwy2dCr0t+bfpRZKTX840JK+N1CgfM41GmPLiVn180lPJiP7+bvoa5M7uzaU1GvEMLT13tpd0KDGw2P8BZ1iAHGA28LyIAfYBpInKeqi5sb6fhmrRfPfBYAXgJmNjG8sdV9RhnikmyAxg5vILi4mxKtmVTX+/lgzmHMH7clhZltpVms35DT1QTe9TW5vxr9lHfN41AnzTwC/u+0p2M+btblMl6Zyd7z+2JZnsBCPZo+jvn/3If3sp6ao5JrKsUhw0tY+u2XIrLcqkPePnPvKGcdGzLAXo+W9mPmtrQuSz/soCCXnsb1326oh9V1f6YxuyWkcdWUbQhjZJN6dTXeXj/Hz0Yf05lvMOKjHvj4S0AhovIEGeE9YuAaY2HUa1U1XxVHayqg4G5QNhkB2ESnqpujyis9refAxzUPtyUl7ePsvKsxvnyikzy8qoi3j4tLcCTj83g8UdmMv7EzR1vECPe7fUE8pv+YwfyfHgr6lqWKarFV1RL/s/Xk/+z9aQv3hNaEVRy/7CNyu8nXnMpv2cVpdubfV87Minoubfd8l87dTXzlwxod30yyetTR1lR01sUyov95PetC7NF4nDrthRVrQduAGYCK4DXVXWZiNwvIucdaHzxeE3jDSJyObAQuFVVd7RVyLmIORkgI617W0Vi6vIrJ1GxPZM+hXt4+IF32bChB8UlOR1vmAAkoHiLayn/5WC8FXXk37mB0t8Oo9sHldQcn00wPzlrQg3OHL+WEUPKufmhr8c7FOPi/RtOC3B6q2X3tFN2QiT7jGSIdzc9AwwDjgGKgUfbK6iqU1V1jKqO8fsO/vpSRUU3CvKbagj5eVVUVES+34rtobIl27JZsrQ3w4a2madjLtDLh7e86a+/t6KeQF7LBBbI81N9Qg74hEBhGvX90vAV1ZK2qorM6dvpPXkNuS9to9v7leT8aVusT6FN5Tsy6d2siZrfs4qyHVn7lTtu1FYu/eZn3PXEWdTVe2MZYtRUlPgp6FfbOJ/ft47y4iT4oxRpczbBX9PoGlXdpqoBVQ0CzxG61yYmVq3Jo1+/3RQW7sHnC3DaqRuZO79/xxsC2Vm1+H0BAHJzqxl1eFmLzo54qhveDV9xLd5ttVCndPuokuoTWl6Pqx6XQ/rSUPLw7KrHV1RLfaGfnTcPoPS5EZROHc6uKwrZN6E7uy9PjObtyvUF9C/cRZ/83fi8Ac4Yt45PPh3Uosyhg8q55YqPueu3Z7Fzd7c4Req+VZ9l0n9ILYUDa/D5g0yYtJO5sxLj9y0coWu8ptE1ItJXVYud2fMJ3dQcE8Ggh6efHcMDv3gPj0eZNXsoGzf14LJLl7BmTS/mzh/AiOEV3H3HHHKyaxl3wlYuu/QLrpnydQYOrOTGKfNRFUSU1984ImESHl6h8uo+5P1iU+i2lK/2oH5QBjkvl1J7aDdqxuZQc2wW6Z/toeBHa8EjVH6/EM2Nx9WMyAWDHn73l/E8fNuM0K0ZH45gQ1FPrjh/EavX5/Pfzw7hmu8uICO9jnun/AeA0ops7vrtWQA88fP/Y1DfSrpl1PHaY6/wyIunsHBpclzjCwaEp+7sz4Mvr8PjhVmv9mLj6gTvoXUk+vBQohqdCEXkFWACkA9sA+515o8hVKndAFzTLAG2Kzern544qvV9iclv610J/ttxEAqeSazbXNzinxW2EzBpzdN32aXbD+r2hMzCgTr8ols6LggsefKWRQdzH96BitqfeVW9uI3FL0TreMaYBJDgf8MTu11jjEkeXWTEY2OMiYwlPGNMqkj0AUAt4RljXGNNWmNMaojzTcWRsIRnjHGPJTxjTCpoeNIikVnCM8a4RoKJnfEs4Rlj3GHX8IwxqcSatMaY1GEJzxiTKqyGZ4xJHZbwjDEpwd23lkWFJTxjjCvsPjxjTGqJ0oDCbrGEZ4xxjdXwjDGpwW48NsakEuu0MMakDEt4xpjUoFinhSuqqtFFy+Idheu+GPdZvEOImnPOPybeIZg4sE4LY0zqsIRnjEkFduOxMSZ1qNoAoMaYFJLY+c4SnjHGPdakNcakBgWsSWuMSRmJne/wxDsAY0zXIRrZFNG+RCaKyCoRWSsit7ex/hYRWS4iS0TkXRE5pKN9WsIzxrhGghrR1OF+RLzAU8C5wCjgYhEZ1arYp8AYVT0KeAP4dUf7tYRnjHGHdmLq2FhgraquU9Va4FVgUovDqb6nqlXO7FxgQEc7tWt4xhhXhG48jvgiXr6ILGw2P1VVpzab7w9sbja/BRgXZn9XAf/u6KCW8Iwx7ol8tJRyVR3jxiFF5HvAGOC0jspawjPGuKYTNbyObAUGNpsf4CxreTyRM4E7gdNUtaajndo1PGOMO9y9hrcAGC4iQ0QkDbgImNa8gIgcC/wvcJ6qlkayU6vhGWNc4t6ztKpaLyI3ADMBL/Ciqi4TkfuBhao6DXgEyAb+JiIAm1T1vHD7tYRnjHGPiwOAqup0YHqrZfc0+3xmZ/dpCc8Y4w57EbcxJqXYEO/GmJSR2PnOEp4xxj0STOw2rSU8Y4w7lM7ceBwXlvCMMa4Q1M0bj6PCbjxuZsyEXTz/4Ur+8PEKLrxhW7zDccWjNw/kwiOPYPLpI+Mdiuu64vfVIGnPTTWyKU6ilvBE5EURKRWRpc2WHS0in4jIFyLyTxHJjdbxO8vjUaY8uJW7Lh3C1RNGcvqknQwaXh3vsA7a2d/dzgN/XRfvMFzXVb8vSPJzS9WEB7wETGy17HngdlU9Engb+EkUj98pI4+tomhDGiWb0qmv8/D+P3ow/pzKeId10I48cS85PQPxDsN1XfX7giQ+t4ZreJFMcRK1hKeqc4DtrRaPAOY4n98BLojW8Tsrr08dZUVpjfPlxX7y+9bFMSITTlf+vpL53CQYjGiKl1hfw1tG0yB+36HlaAjGmKQWYXO2izZp23IlcL2ILAJygNr2CorIZBFZKCIL6+hw1JeDVlHip6BfUzj5fesoL/ZH/bjmwHTl7ytpz02xhNecqq5U1bNV9XjgFeDLMGWnquoYVR3jJz3qsa36LJP+Q2opHFiDzx9kwqSdzJ3VPerHNQemK39fSX1uCX4NL6b34YlIb1UtFREPcBfwbCyPH04wIDx1Z38efHkdHi/MerUXG1dnxDusg/bQdYew5JNsKrf7uPT4UVx2awkTL2l9aTX5dNXvC5L73BL9PryoJTwReQWYQGjs+i3AvUC2iExxirwF/CFaxz8QC/6Ty4L/JMydMq74+TMb4x1C1HTF76tB0p5bqiY8Vb24nVW/jdYxjTFxpAqBxH62zB4tM8a4J1VreMaYFGQJzxiTEhRw6Z0W0WIJzxjjEgW1a3jGmFSgWKeFMSaF2DU8Y0zKsIRnjEkN8X1ONhKW8Iwx7lDAXuJjjEkZVsMzxqQGe7TMGJMqFNTuwzPGpAx70sIYkzLsGp4xJiWoWi+tMSaFWA3PGJMaFA0k9juQLeEZY9xhw0MZY1JKgt+WEuv30hpjuigFNKgRTZEQkYkiskpE1orI7W2sTxeR15z180RkcEf7tIRnjHGHOgOARjJ1QES8wFPAucAo4GIRGdWq2FXADlU9FHgceLij/VrCM8a4RgOBiKYIjAXWquo6Va0FXgUmtSozCfij8/kN4KsiIuF2mhTX8Hazo3y2vhGrF6zmA+WxOJC3byyO0ihm5xWyNnaHivm5xUwsz+uQg93BbnbMnK1v5EdYPENEFjabn6qqU5vN9wc2N5vfAoxrtY/GMqpaLyKVQB5hfmZJkfBUtSBWxxKRhao6JlbHi5Wuel7Qdc8t2c5LVSfGO4aOWJPWGJOItgIDm80PcJa1WUZEfEB3oCLcTi3hGWMS0QJguIgMEZE04CJgWqsy04DvO5+/DfxHNfyjHknRpI2xqR0XSUpd9byg655bVz2vDjnX5G4AZgJe4EVVXSYi9wMLVXUa8ALwZxFZC2wnlBTDkg4SojHGdBnWpDXGpAxLeMaYlGEJzyEiA0XkPRFZLiLLROSmeMd0oETkRREpFZGlbay7VURURCK9XyqhtHVuInKfiGwVkc+c6WvxjPFAtHNeR4vIJyLyhYj8U0Ry4xljV2AJr0k9cKuqjgJOBKa08ShLsngJ2O+eKBEZCJwNbIp1QC56iTbODXhcVY9xpukxjskNL7H/eT0P3K6qRwJvAz+JdVBdjSU8h6oWq+pi5/NuYAWhO7mTjqrOIdRr1drjwE8JPeedlMKcW1Jr57xGAHOcz+8AF8Q0qC7IEl4bnFEXjgXmxTkU14jIJGCrqn4e71ii5AYRWeI0DXvGOxiXLKPp+dHv0PJGXHMALOG1IiLZwJvAj1V1V7zjcYOIZAJ3APfEO5YoeQYYBhwDFAOPxjUa91wJXC8ii4AcoDbO8SQ9u/G4GRHxE0p2f1XVt+Idj4uGAUOAz53BJAYAi0VkrKqWxDUyF6jqtobPIvIc8H9xDMc1qrqS0DVXRGQE8PX4RpT8LOE5nGFlXgBWqOpj8Y7HTar6BdC7YV5ENgBjVLVLjDAiIn1VtdiZPR/Yr3c6GYlIb1UtFREPcBfwbLxjSnbWpG1yMnAZcEYy394AICKvAJ8AI0Vki4hcFe+Y3NLOuf3auXVjCXA6cHNcgzwA7ZzXxSKyGlgJFAF/iGeMXYE9WmaMSRlWwzPGpAxLeMaYlGEJzxiTMizhGWNShiU8Y0zKsITXBYhIwLmNZqmI/M15suJA9/WSiHzb+fx8uAEURGSCiJx0AMfY0NZoLe0tb1VmTyePdZ+I3NbZGE3XZAmva9jnjBIymtDjR9c2X+m84KTTVPWHqro8TJEJQKcTnjHxYgmv6/kQONSpfX0oItOA5SLiFZFHRGSB85D9NRB6wkREfi8iq0RkNi2fyHhfRMY4nyeKyGIR+VxE3nUGWLgWuNmpXZ4iIgUi8qZzjAUicrKzbZ6IzHLGGXweCPuyZGebv4vIImebya3WPe4sf1dECpxlw0RkhrPNhyJymCs/TdOl2KNlXYhTkzsXmOEsOg4YrarrnaRRqaoniEg68LGIzCI0KsxIYBRQCCwHXmy13wLgOeBUZ1+9VHW7iDwL7FHV3zjlXiY0Lt1HIjKI0AtYDgfuBT5S1ftF5OtAJE9+XOkcoxuwQETeVNUKIIvQS1xuFpF7nH3fQOiFN9eq6hoRGQc8DZxxAD9G04VZwusauonIZ87nDwk9E3wSMF9V1zvLzwaOarg+R+gdnsOBU4FXVDUAFInIf9rY/4nAnIZ9qWp749GdCYxyBigAyHVGnzkV+Jaz7b9EZEcE53SjiJzvfB7oxFoBBIHXnOV/Ad5yjnES8Ldmx06P4BgmxVjC6xr2qeoxzRc4//H3Nl8E/EhVZ7Yq5+bzwh7gRFWtbiOWiInIBELJc7yqVonI+0BGO8XVOe7O1j8DY1qza3ipYyZwnTMEFiIyQkSyCI2o+13nGl9fQg/ftzYXOFVEhjjb9nKW7yY0TluDWcCPGmZE5Bjn4xzgEmfZuUBHA3R2B3Y4ye4wQjXMBh5CL13G2edHzriF60XkO84xRESO7uAYJgVZwksdzxO6PrdYQi+K+V9CNfy3gTXOuj8RGrGjBVUtAyYTaj5+TlOT8p/A+Q2dFsCNwBinU2Q5Tb3FvyCUMJcRatp29E6NGYBPRFYA/0Mo4TbYC4x1zuEM4H5n+aXAVU58zUcKNqaRjZZijEkZVsMzxqQMS3jGmJRhCc8YkzIs4RljUoYlPGNMyrCEZ4xJGZbwjDEp4/8DHy16jEvlQWsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(model, X_validate, y_validate, normalize='true')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "annoying-fluid",
   "metadata": {},
   "source": [
    "The confusion matrix allows us to compare the actual space group of the validation data, with that from the classification. \n",
    "For a perfect classification, you would have an [identity matrix](https://en.wikipedia.org/wiki/Identity_matrix) with the dimension of the number of labels. \n",
    "\n",
    "## Exercise\n",
    "\n",
    "It is clear from the above confusion matrix, that our current selection of columns of interest is not sufficient to accurately classify the data. \n",
    "In this exercise, you should try different combinations of the columns of interest to see which give the best classification of the space group for these materials. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unauthorized-association",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}