{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Päivitetty 2023-05-02 / Aki Taanila\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "print(f'Päivitetty {datetime.now().date()} / Aki Taanila')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Puuttuvien (missing) arvojen käsittely\n",
    "\n",
    "Puuttuvat arvot eivät ole ongelma kuvailevassa analytiikassa. Selittävässä analytiikassakaan puuttuvista arvoista ei tarvitse huolehtia muuta kuin merkitsevyystestauksen yhteydessä. Useimmille ennakoivan analytiikan malleille puuttuvat arvot ovat myrkkyä.\n",
    "\n",
    "Tärkeitä työkaluja puuttuvien arvojen käsittelyyn ovat **isnull()**, **notnull()**, **dropna()** ja **fillna()**.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nro</th>\n",
       "      <th>sukup</th>\n",
       "      <th>ikä</th>\n",
       "      <th>perhe</th>\n",
       "      <th>koulutus</th>\n",
       "      <th>palveluv</th>\n",
       "      <th>palkka</th>\n",
       "      <th>johto</th>\n",
       "      <th>työtov</th>\n",
       "      <th>työymp</th>\n",
       "      <th>palkkat</th>\n",
       "      <th>työteht</th>\n",
       "      <th>työterv</th>\n",
       "      <th>lomaosa</th>\n",
       "      <th>kuntosa</th>\n",
       "      <th>hieroja</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>3587</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2963</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1989</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>36</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2144</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2183</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1598</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>79</td>\n",
       "      <td>1</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1638</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>80</td>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2612</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>81</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2808</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>35</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2183</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>82 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    nro  sukup  ikä  perhe  koulutus  palveluv  palkka  johto  työtov  työymp  \\\n",
       "0     1      1   38      1       1.0      22.0    3587      3     3.0       3   \n",
       "1     2      1   29      2       2.0      10.0    2963      1     5.0       2   \n",
       "2     3      1   30      1       1.0       7.0    1989      3     4.0       1   \n",
       "3     4      1   36      2       1.0      14.0    2144      3     3.0       3   \n",
       "4     5      1   24      1       2.0       4.0    2183      2     3.0       2   \n",
       "..  ...    ...  ...    ...       ...       ...     ...    ...     ...     ...   \n",
       "77   78      1   22      1       3.0       0.0    1598      4     4.0       4   \n",
       "78   79      1   33      1       1.0       2.0    1638      1     3.0       2   \n",
       "79   80      1   27      1       2.0       7.0    2612      3     4.0       3   \n",
       "80   81      1   35      2       2.0      16.0    2808      3     4.0       3   \n",
       "81   82      2   35      2       3.0      15.0    2183      3     4.0       4   \n",
       "\n",
       "    palkkat  työteht  työterv  lomaosa  kuntosa  hieroja  \n",
       "0         3        3      NaN      NaN      NaN      NaN  \n",
       "1         1        3      NaN      NaN      NaN      NaN  \n",
       "2         1        3      1.0      NaN      NaN      NaN  \n",
       "3         3        3      1.0      NaN      NaN      NaN  \n",
       "4         1        2      1.0      NaN      NaN      NaN  \n",
       "..      ...      ...      ...      ...      ...      ...  \n",
       "77        3        4      NaN      1.0      1.0      NaN  \n",
       "78        1        2      1.0      NaN      NaN      NaN  \n",
       "79        3        3      1.0      NaN      1.0      NaN  \n",
       "80        3        3      NaN      NaN      NaN      NaN  \n",
       "81        3        4      1.0      NaN      NaN      NaN  \n",
       "\n",
       "[82 rows x 16 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Avaan datan\n",
    "df = pd.read_excel('https://taanila.fi/data1.xlsx')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nro</th>\n",
       "      <th>sukup</th>\n",
       "      <th>ikä</th>\n",
       "      <th>perhe</th>\n",
       "      <th>koulutus</th>\n",
       "      <th>palveluv</th>\n",
       "      <th>palkka</th>\n",
       "      <th>johto</th>\n",
       "      <th>työtov</th>\n",
       "      <th>työymp</th>\n",
       "      <th>palkkat</th>\n",
       "      <th>työteht</th>\n",
       "      <th>työterv</th>\n",
       "      <th>lomaosa</th>\n",
       "      <th>kuntosa</th>\n",
       "      <th>hieroja</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>82 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      nro  sukup    ikä  perhe  koulutus  palveluv  palkka  johto  työtov  \\\n",
       "0   False  False  False  False     False     False   False  False   False   \n",
       "1   False  False  False  False     False     False   False  False   False   \n",
       "2   False  False  False  False     False     False   False  False   False   \n",
       "3   False  False  False  False     False     False   False  False   False   \n",
       "4   False  False  False  False     False     False   False  False   False   \n",
       "..    ...    ...    ...    ...       ...       ...     ...    ...     ...   \n",
       "77  False  False  False  False     False     False   False  False   False   \n",
       "78  False  False  False  False     False     False   False  False   False   \n",
       "79  False  False  False  False     False     False   False  False   False   \n",
       "80  False  False  False  False     False     False   False  False   False   \n",
       "81  False  False  False  False     False     False   False  False   False   \n",
       "\n",
       "    työymp  palkkat  työteht  työterv  lomaosa  kuntosa  hieroja  \n",
       "0    False    False    False     True     True     True     True  \n",
       "1    False    False    False     True     True     True     True  \n",
       "2    False    False    False    False     True     True     True  \n",
       "3    False    False    False    False     True     True     True  \n",
       "4    False    False    False    False     True     True     True  \n",
       "..     ...      ...      ...      ...      ...      ...      ...  \n",
       "77   False    False    False     True    False    False     True  \n",
       "78   False    False    False    False     True     True     True  \n",
       "79   False    False    False    False     True    False     True  \n",
       "80   False    False    False     True     True     True     True  \n",
       "81   False    False    False    False     True     True     True  \n",
       "\n",
       "[82 rows x 16 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# isnull() kertoo mitkä ovat puuttuvia arvoja (True) ja mitkä eivät (False)\n",
    "df.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nro          0\n",
       "sukup        0\n",
       "ikä          0\n",
       "perhe        0\n",
       "koulutus     1\n",
       "palveluv     2\n",
       "palkka       0\n",
       "johto        0\n",
       "työtov       1\n",
       "työymp       0\n",
       "palkkat      0\n",
       "työteht      0\n",
       "työterv     35\n",
       "lomaosa     62\n",
       "kuntosa     73\n",
       "hieroja     60\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Puuttuvien arvojen lukumäärät\n",
    "# Laskenta perustuu siihen että True=1 ja False=0\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nro</th>\n",
       "      <th>sukup</th>\n",
       "      <th>ikä</th>\n",
       "      <th>perhe</th>\n",
       "      <th>koulutus</th>\n",
       "      <th>palveluv</th>\n",
       "      <th>palkka</th>\n",
       "      <th>johto</th>\n",
       "      <th>työtov</th>\n",
       "      <th>työymp</th>\n",
       "      <th>palkkat</th>\n",
       "      <th>työteht</th>\n",
       "      <th>työterv</th>\n",
       "      <th>lomaosa</th>\n",
       "      <th>kuntosa</th>\n",
       "      <th>hieroja</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>82 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     nro  sukup   ikä  perhe  koulutus  palveluv  palkka  johto  työtov  \\\n",
       "0   True   True  True   True      True      True    True   True    True   \n",
       "1   True   True  True   True      True      True    True   True    True   \n",
       "2   True   True  True   True      True      True    True   True    True   \n",
       "3   True   True  True   True      True      True    True   True    True   \n",
       "4   True   True  True   True      True      True    True   True    True   \n",
       "..   ...    ...   ...    ...       ...       ...     ...    ...     ...   \n",
       "77  True   True  True   True      True      True    True   True    True   \n",
       "78  True   True  True   True      True      True    True   True    True   \n",
       "79  True   True  True   True      True      True    True   True    True   \n",
       "80  True   True  True   True      True      True    True   True    True   \n",
       "81  True   True  True   True      True      True    True   True    True   \n",
       "\n",
       "    työymp  palkkat  työteht  työterv  lomaosa  kuntosa  hieroja  \n",
       "0     True     True     True    False    False    False    False  \n",
       "1     True     True     True    False    False    False    False  \n",
       "2     True     True     True     True    False    False    False  \n",
       "3     True     True     True     True    False    False    False  \n",
       "4     True     True     True     True    False    False    False  \n",
       "..     ...      ...      ...      ...      ...      ...      ...  \n",
       "77    True     True     True    False     True     True    False  \n",
       "78    True     True     True     True    False    False    False  \n",
       "79    True     True     True     True    False     True    False  \n",
       "80    True     True     True    False    False    False    False  \n",
       "81    True     True     True     True    False    False    False  \n",
       "\n",
       "[82 rows x 16 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# notnull() toimii päinvastoin kuin isnull()\n",
    "df.notnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nro         82\n",
       "sukup       82\n",
       "ikä         82\n",
       "perhe       82\n",
       "koulutus    81\n",
       "palveluv    80\n",
       "palkka      82\n",
       "johto       82\n",
       "työtov      81\n",
       "työymp      82\n",
       "palkkat     82\n",
       "työteht     82\n",
       "työterv     47\n",
       "lomaosa     20\n",
       "kuntosa      9\n",
       "hieroja     22\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Arvojen lukumäärät\n",
    "df.notnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "79"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Poistan (dropna) rivit, joilla oli puuttuva arvo muuttujalle koulutus, palveluv  tai työtov\n",
    "df_dropna = df.dropna(subset = ['koulutus', 'palveluv', 'työtov'])\n",
    "\n",
    "# Montako riviä jäi jäljelle?\n",
    "df_dropna.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>työterv</th>\n",
       "      <th>lomaosa</th>\n",
       "      <th>kuntosa</th>\n",
       "      <th>hieroja</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>79 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    työterv  lomaosa  kuntosa  hieroja\n",
       "0       0.0      0.0      0.0      0.0\n",
       "1       0.0      0.0      0.0      0.0\n",
       "2       1.0      0.0      0.0      0.0\n",
       "3       1.0      0.0      0.0      0.0\n",
       "4       1.0      0.0      0.0      0.0\n",
       "..      ...      ...      ...      ...\n",
       "77      0.0      1.0      1.0      0.0\n",
       "78      1.0      0.0      0.0      0.0\n",
       "79      1.0      0.0      1.0      0.0\n",
       "80      0.0      0.0      0.0      0.0\n",
       "81      1.0      0.0      0.0      0.0\n",
       "\n",
       "[79 rows x 4 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Korvaan (fillna) dikotomisten muuttujien puuttuvat arvot nollilla\n",
    "df_fillna = df_dropna[['työterv', 'lomaosa', 'kuntosa', 'hieroja']].fillna(value=0)\n",
    "df_fillna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nro</th>\n",
       "      <th>sukup</th>\n",
       "      <th>ikä</th>\n",
       "      <th>perhe</th>\n",
       "      <th>koulutus</th>\n",
       "      <th>palveluv</th>\n",
       "      <th>palkka</th>\n",
       "      <th>johto</th>\n",
       "      <th>työtov</th>\n",
       "      <th>työymp</th>\n",
       "      <th>palkkat</th>\n",
       "      <th>työteht</th>\n",
       "      <th>työterv</th>\n",
       "      <th>lomaosa</th>\n",
       "      <th>kuntosa</th>\n",
       "      <th>hieroja</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>3119</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    nro  sukup  ikä  perhe  koulutus  palveluv  palkka  johto  työtov  työymp  \\\n",
       "56   57      1   61      2       2.0      36.0    3119      2     NaN       2   \n",
       "\n",
       "    palkkat  työteht  työterv  lomaosa  kuntosa  hieroja  \n",
       "56        1        5      1.0      NaN      NaN      1.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Voin suodattaa puuttuvat arvot näkyviin\n",
    "# Suodatan ne rivit, joilla työtov-muuttuja sisältää puuttuvan arvon\n",
    "df[df['työtov'].isnull()]\n",
    "# työtov-muuttujalla on puuttuva arvo indeksinumeron 56 kohdalla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.061728395061729"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Korvaan alkuperäisessä datassa työtov-muuttujan puuttuvan arvon keskiarvolla\n",
    "df['työtov'] = df['työtov'].fillna(df['työtov'].mean())\n",
    "\n",
    "# Tarkistan miten korvaaminen onnistui (aiemmin selvisi, että puuttuva arvo on indeksinumeron 56 kohdalla)\n",
    "df.loc[56, 'työtov']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lisätietoa\n",
    "\n",
    "Data-analytiikka Pythonilla https://tilastoapu.wordpress.com/python/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}