{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img align=\"center\" src=\"https://raw.githubusercontent.com/FUlyankin/Parsers/master/images%20/cats.jpg\" height=\"1200\" width=\"1200\"> \n",
"\n",
"# Домашка №3: основы статистики\n",
"\n",
"\n",
"У всех нас есть датасет по контакту. В нём лежит информация про всех нас. Эту информацию надо как следует проанализировать. Именно этим мы и подолжаем заниматься.\n",
"\n",
"Грамотно расчитывайте свои силы и делайте тот объём заданий, который позволит вам получить желаемую оценку :) \n",
"\n",
"__Важно:__ за циклы в любом из пунктов вы получаете ноль баллов.\n",
"\n",
"----------------------\n",
"\n",
"Подгрузим данные и посмотрим на первые пять строчек из таблицы.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>city</th>\n",
" <th>country</th>\n",
" <th>first_name</th>\n",
" <th>home_town</th>\n",
" <th>in_hse_memes_group</th>\n",
" <th>is_bmm</th>\n",
" <th>is_closed</th>\n",
" <th>last_name</th>\n",
" <th>likes_memes</th>\n",
" <th>uid</th>\n",
" <th>...</th>\n",
" <th>photo_month_mean</th>\n",
" <th>photo_repost_cnt</th>\n",
" <th>photo_repost_max</th>\n",
" <th>photo_repost_mean</th>\n",
" <th>photo_repost_median</th>\n",
" <th>photo_text_len_cnt</th>\n",
" <th>photo_ava_change_cnt</th>\n",
" <th>photo_text_url_len_cnt</th>\n",
" <th>friends_from_course_cnt</th>\n",
" <th>friends_mail_from_course_pct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Москва</td>\n",
" <td>Россия</td>\n",
" <td>Александра</td>\n",
" <td>Москва</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>Абашкова</td>\n",
" <td>60.0</td>\n",
" <td>182152789</td>\n",
" <td>...</td>\n",
" <td>1.333333</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>42.0</td>\n",
" <td>0.428571</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Рязань</td>\n",
" <td>Россия</td>\n",
" <td>Анастасия</td>\n",
" <td>Рязань</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>Чуфистова</td>\n",
" <td>0.0</td>\n",
" <td>148020433</td>\n",
" <td>...</td>\n",
" <td>2.375000</td>\n",
" <td>2.0</td>\n",
" <td>1.0</td>\n",
" <td>0.105263</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>32.0</td>\n",
" <td>0.281250</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Москва</td>\n",
" <td>Россия</td>\n",
" <td>Александр</td>\n",
" <td>Омск</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>Головачев</td>\n",
" <td>0.0</td>\n",
" <td>138413935</td>\n",
" <td>...</td>\n",
" <td>1.400000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>32.0</td>\n",
" <td>0.406250</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>Анна</td>\n",
" <td>NaN</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>Лобанова</td>\n",
" <td>0.0</td>\n",
" <td>366261055</td>\n",
" <td>...</td>\n",
" <td>4.166667</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>45.0</td>\n",
" <td>0.333333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>NaN</td>\n",
" <td>Россия</td>\n",
" <td>Алексей</td>\n",
" <td>NaN</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>Пузырный</td>\n",
" <td>21.0</td>\n",
" <td>111252392</td>\n",
" <td>...</td>\n",
" <td>3.181818</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>41.0</td>\n",
" <td>0.341463</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 98 columns</p>\n",
"</div>"
],
"text/plain": [
" city country first_name home_town in_hse_memes_group is_bmm \\\n",
"0 Москва Россия Александра Москва True True \n",
"1 Рязань Россия Анастасия Рязань True True \n",
"2 Москва Россия Александр Омск False True \n",
"3 NaN NaN Анна NaN False True \n",
"4 NaN Россия Алексей NaN True True \n",
"\n",
" is_closed last_name likes_memes uid ... photo_month_mean \\\n",
"0 False Абашкова 60.0 182152789 ... 1.333333 \n",
"1 False Чуфистова 0.0 148020433 ... 2.375000 \n",
"2 False Головачев 0.0 138413935 ... 1.400000 \n",
"3 False Лобанова 0.0 366261055 ... 4.166667 \n",
"4 False Пузырный 21.0 111252392 ... 3.181818 \n",
"\n",
" photo_repost_cnt photo_repost_max photo_repost_mean photo_repost_median \\\n",
"0 0.0 0.0 0.000000 0.0 \n",
"1 2.0 1.0 0.105263 0.0 \n",
"2 0.0 0.0 0.000000 0.0 \n",
"3 0.0 0.0 0.000000 0.0 \n",
"4 0.0 0.0 0.000000 0.0 \n",
"\n",
" photo_text_len_cnt photo_ava_change_cnt photo_text_url_len_cnt \\\n",
"0 0.0 0.0 0.0 \n",
"1 0.0 0.0 0.0 \n",
"2 0.0 0.0 0.0 \n",
"3 0.0 0.0 0.0 \n",
"4 0.0 0.0 0.0 \n",
"\n",
" friends_from_course_cnt friends_mail_from_course_pct \n",
"0 42.0 0.428571 \n",
"1 32.0 0.281250 \n",
"2 32.0 0.406250 \n",
"3 45.0 0.333333 \n",
"4 41.0 0.341463 \n",
"\n",
"[5 rows x 98 columns]"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"df = pd.read_csv('../data/vk_download/vk_main.csv', sep='\\t')\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__[1] У скольких человек с маркетинга нет авы?__\n",
"\n",
"Используй переменные `is_bmm` и `has_ava_dummy` (значение `True` можно воспринимать как $1$, а `False` — как $0$). "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"98"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[(df.is_bmm == 1) & (df.has_ava_dummy == 0)].shape[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__[2] Сколько парней/девушек с инстой/без указали в профиле, что знают английский?__ \n",
"\n",
"Эта информация лежит в переменной `english_dummy`. Значение $1$ означает, что английский язык указан в профиле. По аналогии в `instagram_dummy` $1$ означают, что человек оставил ссылку на свою инсту. __Проинтерпретируйте получившиеся цифры.__ "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"male_dummy instagram_dummy\n",
"0 0 0.126437\n",
" 1 0.214286\n",
"1 0 0.142857\n",
" 1 0.185185\n",
"Name: english_dummy, dtype: float64"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_agg = df.groupby(['male_dummy', 'instagram_dummy'])['english_dummy']\n",
"df_agg.sum()/df_agg.count()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Люди, которые указали инстаграмм, указывают и знание английского языка. Этот вовсе не означает, что люди с инстаграммом лучше знают язык. Скорее всего, они просто более открыты миру и более подробно заполняют свой профиль в социальной сетке. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__[1] Вывести имена людей, у которых на стенках больше всего эмодзи (топ 5%)__\n",
"\n",
" Все эмодзи, которые были оставлены на стенке у студента, лежат в колонке `wall_emoji_trace`. Отсортируйте всех людей из этого топа по числу эмодзи. При работе с табличкой не забудьте заполнить пропуски."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>first_name</th>\n",
" <th>wall_emoji_cnt</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Анастасия</td>\n",
" <td>1975</td>\n",
" </tr>\n",
" <tr>\n",
" <th>231</th>\n",
" <td>Григорий</td>\n",
" <td>1914</td>\n",
" </tr>\n",
" <tr>\n",
" <th>296</th>\n",
" <td>Юлия</td>\n",
" <td>1904</td>\n",
" </tr>\n",
" <tr>\n",
" <th>113</th>\n",
" <td>Elizabeth</td>\n",
" <td>1456</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176</th>\n",
" <td>Анжелика</td>\n",
" <td>1076</td>\n",
" </tr>\n",
" <tr>\n",
" <th>196</th>\n",
" <td>Алик</td>\n",
" <td>1058</td>\n",
" </tr>\n",
" <tr>\n",
" <th>161</th>\n",
" <td>Оля</td>\n",
" <td>751</td>\n",
" </tr>\n",
" <tr>\n",
" <th>214</th>\n",
" <td>Дуся</td>\n",
" <td>737</td>\n",
" </tr>\n",
" <tr>\n",
" <th>180</th>\n",
" <td>Аня</td>\n",
" <td>473</td>\n",
" </tr>\n",
" <tr>\n",
" <th>253</th>\n",
" <td>Стася</td>\n",
" <td>446</td>\n",
" </tr>\n",
" <tr>\n",
" <th>198</th>\n",
" <td>Иван</td>\n",
" <td>441</td>\n",
" </tr>\n",
" <tr>\n",
" <th>395</th>\n",
" <td>Тимур</td>\n",
" <td>402</td>\n",
" </tr>\n",
" <tr>\n",
" <th>414</th>\n",
" <td>Дима</td>\n",
" <td>357</td>\n",
" </tr>\n",
" <tr>\n",
" <th>208</th>\n",
" <td>Ден</td>\n",
" <td>357</td>\n",
" </tr>\n",
" <tr>\n",
" <th>199</th>\n",
" <td>Дарья</td>\n",
" <td>309</td>\n",
" </tr>\n",
" <tr>\n",
" <th>209</th>\n",
" <td>Диана</td>\n",
" <td>306</td>\n",
" </tr>\n",
" <tr>\n",
" <th>369</th>\n",
" <td>Полина</td>\n",
" <td>293</td>\n",
" </tr>\n",
" <tr>\n",
" <th>261</th>\n",
" <td>Ксения</td>\n",
" <td>282</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154</th>\n",
" <td>Виктория</td>\n",
" <td>274</td>\n",
" </tr>\n",
" <tr>\n",
" <th>174</th>\n",
" <td>Андрей</td>\n",
" <td>239</td>\n",
" </tr>\n",
" <tr>\n",
" <th>375</th>\n",
" <td>Сандрик</td>\n",
" <td>223</td>\n",
" </tr>\n",
" <tr>\n",
" <th>387</th>\n",
" <td>Соня</td>\n",
" <td>213</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" first_name wall_emoji_cnt\n",
"1 Анастасия 1975\n",
"231 Григорий 1914\n",
"296 Юлия 1904\n",
"113 Elizabeth 1456\n",
"176 Анжелика 1076\n",
"196 Алик 1058\n",
"161 Оля 751\n",
"214 Дуся 737\n",
"180 Аня 473\n",
"253 Стася 446\n",
"198 Иван 441\n",
"395 Тимур 402\n",
"414 Дима 357\n",
"208 Ден 357\n",
"199 Дарья 309\n",
"209 Диана 306\n",
"369 Полина 293\n",
"261 Ксения 282\n",
"154 Виктория 274\n",
"174 Андрей 239\n",
"375 Сандрик 223\n",
"387 Соня 213"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['wall_emoji_cnt'] = df['wall_emoji_trace'].fillna('').apply(len)\n",
"\n",
"q = df['wall_emoji_cnt'].quantile(0.95)\n",
"\n",
"df.loc[df['wall_emoji_cnt'] >= q,\n",
" ['first_name', 'wall_emoji_cnt']].sort_values('wall_emoji_cnt', ascending=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__[3] Давайте проанализируем колонку со средним кол-во фото в месяц (`photo_month_mean`)__\n",
"\n",
"* Постройте на одной картинке гистограмы для распределения этого показателя по разным полам. \n",
"* Правда ли, что типичная девушка выкладывает значительно больше фотографий, чем типичный мужчина? (подумайте какой именно показатель типичности нужно выбрать для сравнения и обоснуйте почему)\n",
"* Для какого пола показатель оказывается более непредсказуемым? (подумайте как именно корректно эту непредсказуемость оценить, обычное стандартное отклонение явно не подходит)\n",
"\n",
"Не забывайте подгрузить пакет `matplotlib`! __Все свои рассуждения пишите прямо по ходу кода! Нет рассуждений => нет баллов!__ "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAD8CAYAAAC4lecIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG6FJREFUeJzt3X2QXXWd5/H3FwJJsIEIaC+SlOkZQxCDEGh5kHWqG1TAUUP5VLHcNVG2UiDrwAw+gOjKqji4k9LxYUWjsIEdTKuIkk2JI8H0qOUSSCQgkGSIGKFZBHm0OySYkO/+cU9ik/RT7r3dfe7t96uqq+8553fO+X1zkvvJ+Z1zz43MRJIkldN+490BSZI0OINakqQSM6glSSoxg1qSpBIzqCVJKjGDWpKkEjOoJUkqMYNakqQSM6glSSqxSePdAYAjjjgiZ86cWfN2tmzZwkte8pLaO1QyzVhXM9YE1tVorKtxNGNNa9eufSIzXzZcu1IE9cyZM1mzZk3N2+nu7qajo6P2DpVMM9bVjDWBdTUa62oczVhTRPx+JO0c+pYkqcQMakmSSsygliSpxEpxjVqS1Pi2b99OT08P27Ztq/u2Dz30UNavX1/37Y6FKVOmMH36dA444ICq1jeoJUl10dPTw8EHH8zMmTOJiLpuu7e3l4MPPriu2xwLmcmTTz5JT08PbW1tVW3DoW9JUl1s27aNww8/vO4h3cgigsMPP7ymUQaDWpJUN4b03mr9MzGoJUkqMa9RS5JGxcr7H6vbtrZu3crbThrda9Td3d0sXryYFStWjOp+9lVzBvXGW4ZvM/uc0e+HJEk1cuhbktQ0Nm/ezDHHHMPChQs5+uijed/73sfKlSs5/fTTmTVrFnfccQd33HEHp512GnPnzuX1r389Gzdu3Gs7W7Zs4YMf/CAnn3wyc+fO5eabbx6HaiqGDeqIuDYiHo+Ie/eY/+GI2BAR90XE/+g3/7KI2BQRGyPirNHotCRJg9m0aROXXHIJGzZsYMOGDXznO9/hl7/8JYsXL+bzn/88xxxzDL/4xS+46667+MxnPsMnPvGJvbZx5ZVXcsYZZ3DHHXewatUqPvrRj7Jly5ZxqGZkQ99Lga8B1++aERGdwDzg+Mx8PiJeXsw/FpgPvAZ4BbAyIo7OzBfq3XFJkgbS1tbGcccdB8BrXvMazjzzTCKC4447js2bN/Pss8+yYMECHnjgASKC7du377WNn/70pyxfvpzFixcDlY+ePfTQQ7z61a8e01pgBEGdmT+PiJl7zL4AuCozny/aPF7Mnwd0FfN/FxGbgJOB/1u3HkuSNITJkyfvfr3ffvvtnt5vv/3YsWMHn/rUp+js7OSHP/whmzdvHvBbuTKTH/zgB8yePXusuj2oaq9RHw28ISJWR8S/RcTrivlHAQ/3a9dTzJMkqRSeffZZjjqqEk1Lly4dsM1ZZ53FV7/6VTITgLvuumusureXau/6ngQcBpwKvA74XkT81b5sICIWAYsAWltb6e7urrIrf9HX11fZzvN7D2Ps5dHa9zdWdtfVRJqxJrCuRmNd9XXooYfS29u7e/qUGQfVbdsvvDD5RdseTF9fHzt37tzddvv27WzdupXe3t7dyy688ELOP/98PvOZz/DmN7+ZzKS3t5fnnnuOHTt20Nvby8UXX8yll17KnDlz2LlzJ6985Sv5/ve/X3X/t23bVvUxiV3/WxiyUWXoe0VmzimmfwJ8ITNXFdO/pRLa/wUgM/+xmP+vwBWZOeTQd3t7e65Zs6aqAvrb/cXiTfbxrGb8wvRmrAmsq9FYV32tX79+1K7hNuqzvncZ6M8mItZmZvtw61Y79P0joLPY0dHAgcATwHJgfkRMjog2YBZwR5X7kCRpwht26DsilgEdwBER0QN8GrgWuLb4yNafgQVZOTW/LyK+B9wP7AAu9I5vSZKqN5K7vt87yKL/NEj7K4Era+mUJEmq8MlkkiSVmEEtSVKJGdSSJJVYc357liRp/I3ko7IjtP/WrXDCO4Zt95WvfIWrr76aE088kRtuuKFu+9/liiuuoKWlhY985CN13/ZgDGpJUtP4+te/zsqVK5k+ffp4d6VuHPqWJDWF888/nwcffJBzzjmHK6+8csCvqVy6dCnnnnsub3rTm5g5cyZf+9rX+OIXv8jcuXM59dRTeeqppwD41re+xete9zqOP/543vnOd/Lcc8/ttb/f/va3nH322Zx00km84Q1vYMOGDaNSl0EtSWoK3/jGN3jFK17BqlWr2LJly6BfU3nvvfdy0003ceedd3L55Zdz0EEHcdddd3Haaadx/fWVL4p8xzvewZ133sndd9/Nq1/9aq655pq99rdo0SK++tWvsnbtWhYvXsyHPvShUanLoW9JUtMZ7GsqATo7Ozn44IM5+OCDOfTQQ3nb294GwHHHHcc999wDVML8k5/8JM888wx9fX2cddZZL9p+X18fv/rVr3j3u9+9e97zzz8/KrUY1JKkpjPY11SuXr162K/BBFi4cCE/+tGPOP7441m6dOleX6ixc+dOpk2bxrp160a3EBz6liQ1oVq/prK3t5cjjzyS7du3D3j3+CGHHEJbW9vub9TKTO6+++7aOz4Az6glSaOjjt9S+MIIvuKyv0996lNcfPHFvPa1r2Xnzp20tbWxYsWKEa//2c9+llNOOYWXvexlnHLKKQN+xeYNN9zABRdcwOc+9zm2b9/O/PnzOf744/epnyNhUEuSmsbmzZt3v/7mN7+51/KFCxeycOHCAdv3X3bBBRdwwQUX7LX+FVdcsft1W1sbP/nJT2rt8rAc+pYkqcQMakmSSsygliTVza6bt/QXtf6ZGNSSpLqYMmUKTz75pGHdT2by5JNPMmXKlKq3MezNZBFxLfBW4PHMnLPHskuAxcDLMvOJiAjgy8BbgOeAhZn566p7J0lqGNOnT6enp4c//vGPdd/2tm3bagq78TRlypSanj0+kru+lwJfA67vPzMiZgBvBh7qN/scYFbxcwpwdfFbktTkDjjgANra2kZl293d3cydO3dUtl12ww59Z+bPgacGWPQl4GNA/zGOecD1WXE7MC0ijqxLTyVJmoCqukYdEfOARzJzz8ewHAU83G+6p5gnSZKqECO56B8RM4EVmTknIg4CVgFvzsxnI2Iz0F5co14BXJWZvyzWuw34eGauGWCbi4BFAK2trSd1dXXVXExfXx8tLS3w/J+Gbzz5kJr3N1Z219VEmrEmsK5GY12Noxlr6uzsXJuZ7cO1q+bJZH8NtAF3V+4dYzrw64g4GXgEmNGv7fRi3l4ycwmwBKC9vT07Ojqq6MqLdXd309HRARtvGb7x7Nr3N1Z219VEmrEmsK5GY12NoxlrGql9HvrOzN9k5sszc2ZmzqQyvH1iZv4BWA68PypOBZ7NzEfr22VJkiaOkXw8axnQARwRET3ApzNz72/QrvgxlY9mbaLy8awP1Kmf+2Tdw88MufyEGdPGqCeSJNVm2KDOzPcOs3xmv9cJXFh7tyRJEvhkMkmSSs2gliSpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkps2KCOiGsj4vGIuLffvH+KiA0RcU9E/DAipvVbdllEbIqIjRFx1mh1XJKkiWAkZ9RLgbP3mHcrMCczXwv8O3AZQEQcC8wHXlOs8/WI2L9uvZUkaYIZNqgz8+fAU3vM+2lm7igmbwemF6/nAV2Z+Xxm/g7YBJxcx/5KkjShRGYO3yhiJrAiM+cMsOz/AN/NzH+JiK8Bt2fmvxTLrgFuycwbB1hvEbAIoLW19aSurq5a6gCgr6+PlpYWtvY+NWS7qQdOgsmH1Ly/sbKrrmbSjDWBdTUa62oczVhTZ2fn2sxsH67dpFp2EhGXAzuAG/Z13cxcAiwBaG9vz46Ojlq6AkB3dzcdHR2sW7lsyHYnzJgKs2vf31jZVVczacaawLoajXU1jmasaaSqDuqIWAi8FTgz/3Ja/ggwo1+z6cU8SZJUhao+nhURZwMfA96emc/1W7QcmB8RkyOiDZgF3FF7NyVJmpiGPaOOiGVAB3BERPQAn6Zyl/dk4NaIgMp16fMz876I+B5wP5Uh8Qsz84XR6rwkSc1u2KDOzPcOMPuaIdpfCVxZS6ckSVKFTyaTJKnEDGpJkkrMoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcQMakmSSmzYoI6IayPi8Yi4t9+8wyLi1oh4oPj90mJ+RMRXImJTRNwTESeOZuclSWp2IzmjXgqcvce8S4HbMnMWcFsxDXAOMKv4WQRcXZ9uSpI0MQ0b1Jn5c+CpPWbPA64rXl8HnNtv/vVZcTswLSKOrFdnJUmaaCIzh28UMRNYkZlziulnMnNa8TqApzNzWkSsAK7KzF8Wy24DPp6ZawbY5iIqZ920trae1NXVVXMxfX19tLS0sLV3z/9XvNjUAyfB5ENq3t9Y2VVXM2nGmsC6Go11NY5mrKmzs3NtZrYP125SrTvKzIyI4dN+7/WWAEsA2tvbs6Ojo9au0N3dTUdHB+tWLhuy3QkzpsLs2vc3VnbV1UyasSawrkZjXY2jGWsaqWrv+n5s15B28fvxYv4jwIx+7aYX8yRJUhWqDerlwILi9QLg5n7z31/c/X0q8GxmPlpjHyVJmrCGHfqOiGVAB3BERPQAnwauAr4XEecBvwfeUzT/MfAWYBPwHPCBUeizJEkTxrBBnZnvHWTRmQO0TeDCWjslSZIqfDKZJEklZlBLklRiBrUkSSVmUEuSVGIGtSRJJWZQS5JUYga1JEklZlBLklRiBrUkSSVmUEuSVGIGtSRJJWZQS5JUYga1JEklZlBLklRiBrUkSSVmUEuSVGI1BXVE/H1E3BcR90bEsoiYEhFtEbE6IjZFxHcj4sB6dVaSpImm6qCOiKOAvwPaM3MOsD8wH/gC8KXMfBXwNHBePToqSdJEVOvQ9yRgakRMAg4CHgXOAG4sll8HnFvjPiRJmrAiM6tfOeIi4EpgK/BT4CLg9uJsmoiYAdxSnHHvue4iYBFAa2vrSV1dXVX3Y5e+vj5aWlrY2vvUkO2mHjgJJh9S8/7Gyq66mkkz1gTW1Wisq3E0Y02dnZ1rM7N9uHaTqt1BRLwUmAe0Ac8A3wfOHun6mbkEWALQ3t6eHR0d1XZlt+7ubjo6Oli3ctmQ7U6YMRVm176/sbKrrmbSjDWBdTUa62oczVjTSNUy9P1G4HeZ+cfM3A7cBJwOTCuGwgGmA4/U2EdJkiasWoL6IeDUiDgoIgI4E7gfWAW8q2izALi5ti5KkjRxVR3Umbmayk1jvwZ+U2xrCfBx4B8iYhNwOHBNHfopSdKEVPU1aoDM/DTw6T1mPwicXMt2JUlShU8mkySpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkqspqCOiGkRcWNEbIiI9RFxWkQcFhG3RsQDxe+X1quzkiRNNLWeUX8Z+ElmHgMcD6wHLgVuy8xZwG3FtCRJqkLVQR0RhwJ/A1wDkJl/zsxngHnAdUWz64Bza+2kJEkTVWRmdStGnAAsAe6ncja9FrgIeCQzpxVtAnh61/Qe6y8CFgG0trae1NXVVVU/+uvr66OlpYWtvU8N2W7qgZNg8iE172+s7KqrmTRjTWBdjca6Gkcz1tTZ2bk2M9uHa1dLULcDtwOnZ+bqiPgy8Cfgw/2DOSKezswhr1O3t7fnmjVrqupHf93d3XR0dLBu5bIh250wYxrMPqfm/Y2VXXU1k2asCayr0VhX42jGmiJiREFdyzXqHqAnM1cX0zcCJwKPRcSRRSeOBB6vYR+SJE1oVQd1Zv4BeDgiZhezzqQyDL4cWFDMWwDcXFMPJUmawCbVuP6HgRsi4kDgQeADVML/exFxHvB74D017kOSpAmrpqDOzHXAQOPrZ9ayXUmSVOGTySRJKjGDWpKkEjOoJUkqMYNakqQSM6glSSoxg1qSpBIzqCVJKjGDWpKkEjOoJUkqMYNakqQSM6glSSoxg1qSpBIzqCVJKjGDWpKkEjOoJUkqMYNakqQSqzmoI2L/iLgrIlYU020RsToiNkXEdyPiwNq7KUnSxFSPM+qLgPX9pr8AfCkzXwU8DZxXh31IkjQh1RTUETEd+Fvg28V0AGcANxZNrgPOrWUfkiRNZLWeUf8z8DFgZzF9OPBMZu4opnuAo2rchyRJE1ZkZnUrRrwVeEtmfigiOoCPAAuB24thbyJiBnBLZs4ZYP1FwCKA1tbWk7q6uqrqR399fX20tLSwtfepIdtNPXASTD6k5v2NlV11NZNmrAmsq9FYV+Noxpo6OzvXZmb7cO0m1bCP04G3R8RbgCnAIcCXgWkRMak4q54OPDLQypm5BFgC0N7enh0dHTV0paK7u5uOjg7WrVw2ZLsTZkyF2bXvb6zsqquZNGNNYF2NxroaRzPWNFJVD31n5mWZOT0zZwLzgZ9l5vuAVcC7imYLgJtr7qUkSRPUaHyO+uPAP0TEJirXrK8ZhX1IkjQh1DL0vVtmdgPdxesHgZPrsV1JkiY6n0wmSVKJGdSSJJWYQS1JUokZ1JIklZhBLUlSiRnUkiSVmEEtSVKJGdSSJJWYQS1JUokZ1JIklVhdHiHakDbeMvTy2eeMTT8kSRqCZ9SSJJWYQS1JUokZ1JIklZhBLUlSiRnUkiSVWNVBHREzImJVRNwfEfdFxEXF/MMi4taIeKD4/dL6dVeSpImlljPqHcAlmXkscCpwYUQcC1wK3JaZs4DbimlJklSFqoM6Mx/NzF8Xr3uB9cBRwDzguqLZdcC5tXZSkqSJqi7XqCNiJjAXWA20ZuajxaI/AK312IckSRNRZGZtG4hoAf4NuDIzb4qIZzJzWr/lT2fmXtepI2IRsAigtbX1pK6urpr6AdDX10dLSwtbe58ast3UA0fwQLbJh9Tcn3rZVVczacaawLoajXU1jmasqbOzc21mtg/XrqZHiEbEAcAPgBsy86Zi9mMRcWRmPhoRRwKPD7RuZi4BlgC0t7dnR0dHLV0BoLu7m46ODtatXDZkuxNmTB1+Y7Nr70+97KqrmTRjTWBdjca6Gkcz1jRStdz1HcA1wPrM/GK/RcuBBcXrBcDN1XdPkqSJrZYz6tOB/wz8JiLWFfM+AVwFfC8izgN+D7ynti5KkjRxVR3UmflLIAZZfGa125UkSX/hk8kkSSoxg1qSpBIzqCVJKjGDWpKkEjOoJUkqsZoeeNKo1j38zKDLTpgxbdBlkiSNNc+oJUkqMYNakqQSM6glSSoxg1qSpBIzqCVJKrEJedf3iGy8Zejls88Zm35IkiY0z6glSSoxg1qSpBJz6Hsf9H9QyhMvPPaiZW88tnWsuyNJmgA8o5YkqcRG7Yw6Is4GvgzsD3w7M68arX3V01CPF5UkaayNSlBHxP7A/wTeBPQAd0bE8sy8fzT216xW3l8ZXt+6bcfu1yMxZsPw3hkvSaNutIa+TwY2ZeaDmflnoAuYN0r7kiSpaY3W0PdRwMP9pnuAU0ZpX+PiiP/3sxfP2H/wb90aaDj9iVecMez2jyheP7p98t77G8oQfXmROp3xDna54IkXHhvy7H6oUQJvzpPG0D6MjjXtv9sSjxCO213fEbEIWFRM9kXExjps9gjgiTpsp2yasa5mrAmsq9FYV+NoxppeOZJGoxXUjwAz+k1PL+btlplLgCX13GlErMnM9npuswyasa5mrAmsq9FYV+NoxppGarSuUd8JzIqItog4EJgPLB+lfUmS1LRG5Yw6M3dExH8F/pXKx7Ouzcz7RmNfkiQ1s1G7Rp2ZPwZ+PFrbH0Rdh9JLpBnrasaawLoajXU1jmasaUQiM8e7D5IkaRA+QlSSpBJriqCOiLMjYmNEbIqIS8e7P9WKiBkRsSoi7o+I+yLiomL+YRFxa0Q8UPx+6Xj3tRoRsX9E3BURK4rptohYXRy37xY3HjaUiJgWETdGxIaIWB8RpzX68YqIvy/+/t0bEcsiYkqjHquIuDYiHo+Ie/vNG/D4RMVXihrviYgTx6/ngxukpn8q/g7eExE/jIhp/ZZdVtS0MSLOGp9eD2+guvotuyQiMiKOKKYb4ljVS8MHdb/HlZ4DHAu8NyKOHd9eVW0HcElmHgucClxY1HIpcFtmzgJuK6Yb0UXA+n7TXwC+lJmvAp4GzhuXXtXmy8BPMvMY4Hgq9TXs8YqIo4C/A9ozcw6Vm0Hn07jHailw9h7zBjs+5wCzip9FwNVj1Md9tZS9a7oVmJOZrwX+HbgMoHj/mA+8pljn68V7ZhktZe+6iIgZwJuBh/rNbpRjVRcNH9Q00eNKM/PRzPx18bqXypv+UVTqua5odh1w7vj0sHoRMR34W+DbxXQAZwA3Fk0arq6IOBT4G+AagMz8c2Y+Q+Mfr0nA1IiYBBwEPEqDHqvM/Dnw1B6zBzs+84Drs+J2YFpEHDk2PR25gWrKzJ9m5o5i8nYqz66ASk1dmfl8Zv4O2ETlPbN0BjlWAF8CPgb0v6GqIY5VvTRDUA/0uNKjxqkvdRMRM4G5wGqgNTMfLRb9AWjE5/T9M5V/bDuL6cOBZ/q9uTTicWsD/gj8r2JI/9sR8RIa+Hhl5iPAYipnL48CzwJrafxj1d9gx6dZ3ks+COx6HmZD1xQR84BHMvPuPRY1dF37qhmCuulERAvwA+DizPxT/2VZuU2/oW7Vj4i3Ao9n5trx7kudTQJOBK7OzLnAFvYY5m6041Vcr51H5T8hrwBewgDDkc2i0Y7PcCLiciqX0G4Y777UKiIOAj4B/Lfx7st4a4agHvZxpY0kIg6gEtI3ZOZNxezHdg3rFL8fH6/+Vel04O0RsZnKpYkzqFzbnVYMr0JjHrceoCczVxfTN1IJ7kY+Xm8EfpeZf8zM7cBNVI5fox+r/gY7Pg39XhIRC4G3Au/Lv3zutpFr+msq/2G8u3jvmA78OiL+A41d1z5rhqBumseVFtdtrwHWZ+YX+y1aDiwoXi8Abh7rvtUiMy/LzOmZOZPK8flZZr4PWAW8q2jWiHX9AXg4ImYXs84E7qexj9dDwKkRcVDx93FXTQ19rPYw2PFZDry/uKP4VODZfkPkpRYRZ1O5tPT2zHyu36LlwPyImBwRbVRuvrpjPPq4rzLzN5n58sycWbx39AAnFv/uGvZYVSUzG/4HeAuVOx1/C1w+3v2poY7/SGUY7h5gXfHzFirXc28DHgBWAoeNd19rqLEDWFG8/isqbxqbgO8Dk8e7f1XUcwKwpjhmPwJe2ujHC/jvwAbgXuB/A5Mb9VgBy6hca99O5Y3+vMGODxBUPkHyW+A3VO58H/caRljTJirXbHe9b3yjX/vLi5o2AueMd//3pa49lm8GjmikY1WvH59MJklSiTXD0LckSU3LoJYkqcQMakmSSsygliSpxAxqSZJKzKCWJKnEDGpJkkrMoJYkqcT+P2WATiRtM+3hAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 576x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df[df.male_dummy == 1].photo_month_mean.hist(figsize=(8,4), alpha=0.3, label=\"male\", bins=50)\n",
"df[df.male_dummy == 0].photo_month_mean.hist(figsize=(8,4), alpha=0.3, label=\"female\", bins=50)\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"У распределений длинные хвосты, есть выбросы в данных. Среднее чувствительно к выбросам, значит более корректно сравнивать \"типичных\" мальчиков и девочек с помощью медиан. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"male_dummy\n",
"0 1.75\n",
"1 1.50\n",
"Name: photo_month_mean, dtype: float64"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.groupby('male_dummy')['photo_month_mean'].agg('median')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Дисперсия, а значит и среднее квадратичное отклонение, чувствительны к выбросам. Чтобы полечить выборку от их тлетворного влияния, нужно сделать срез. Как отсечь выбросы? Ну например, можно считать выбросом всё, что пробивает $95\\%$ квантиль. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"10.837499999999999"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"q = df['photo_month_mean'].quantile(0.95)\n",
"q"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAD8CAYAAABEgMzCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEsFJREFUeJzt3W+MXfV95/H3JxgTlyk11N0rB6w1UpErNqs4yyhKilTN8GcV2qrwoEKpditrhTT7oJumu5W6dKU+WGl3RaVV2zxYrWSFNJY26wmiibCiJi02TKtKXTZxMtsmdhGUhmIwdih446HeBJLvPphDOwWbe2d87vw8975fknXPOfO7937nC/jD73fOPTdVhSRJauc9rQuQJGnaGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNbdvMN9u1a1ft3bt3M9/yivD6669z7bXXti5jYtjP/tjLftnPfk1CP48fP/5KVf3YsHGbGsZ79+7lq1/96ma+5RVhaWmJubm51mVMDPvZH3vZL/vZr0noZ5LnRxnnMrUkSY0ZxpIkNTY0jJPsS7K85s93kvxKkhuSPJ7kme7x+s0oWJKkSTM0jKvq6araX1X7gduAvwW+ADwIHKuqW4Bj3b4kSVqn9S5T3wn8ZVU9D9wLHOqOHwLu67MwSZKmxXrD+GPA4W57UFWnu+2XgUFvVUmSNEVSVaMNTLYDLwH/pKrOJDlXVTvX/Py1qnrHeeMkC8ACwGAwuG1xcbGfyreQlZUVZmZmWpcxMexnf+xlv+xnvyahn/Pz88eranbYuPV8zvge4GtVdabbP5Nkd1WdTrIbOHuxJ1XVQeAgwOzsbG31z4xtxCR8Vu5KYj/7Yy/7ZT/7NU39XM8y9S/w90vUAEeAA932AeCxvoqSJGmajDQzTnItcDfwr9ccfgh4JMkDwPPA/f2X9+6Wjx4ePmiN/Xu6VfV994yhGkmSNmakMK6q14Effduxv2H16mpJknQZvAOXJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNjRTGSXYmeTTJXyQ5meQjSW5I8niSZ7rH68ddrCRJk2jUmfEngS9X1U8AHwBOAg8Cx6rqFuBYty9JktZpaBgn+RHgp4CHAarqe1V1DrgXONQNOwTcN64iJUmaZKPMjG8Gvg38bpKvJ/lUkmuBQVWd7sa8DAzGVaQkSZMsVfXuA5JZ4H8Bt1fVU0k+CXwH+HhV7Vwz7rWqesd54yQLwALAYDC4bXFxsbfiL5x/dV3jd2zftrpxzXW91TCKlZUVZmZmNvU9J5n97I+97Jf97Nck9HN+fv54Vc0OG7dthNc6BZyqqqe6/UdZPT98JsnuqjqdZDdw9mJPrqqDwEGA2dnZmpubG6X+kSwfPbyu8fv37Fjd2NdfDaNYWlqiz9972tnP/tjLftnPfk1TP4cuU1fVy8ALSfZ1h+4ETgBHgAPdsQPAY2OpUJKkCTfKzBjg48Bnk2wHngP+FatB/kiSB4DngfvHU6IkSZNtpDCuqmXgYmved/ZbjiRJ08c7cEmS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1JhhLElSY4axJEmNGcaSJDVmGEuS1Ni2UQYl+RZwHvg+8GZVzSa5AfgcsBf4FnB/Vb02njIlSZpc65kZz1fV/qqa7fYfBI5V1S3AsW5fkiSt0+UsU98LHOq2DwH3XX45kiRNn1HDuIA/THI8yUJ3bFBVp7vtl4FB79VJkjQFUlXDByU3VtWLSf4R8DjwceBIVe1cM+a1qrr+Is9dABYABoPBbYuLi70Vf+H8q+sav2N7d4r8mut6q2EUKysrzMzMbOp7TjL72R972S/72a9J6Of8/PzxNad3L2mkC7iq6sXu8WySLwAfAs4k2V1Vp5PsBs5e4rkHgYMAs7OzNTc3N+KvMNzy0cPrGr9/z47VjX391TCKpaUl+vy9p5397I+97Jf97Nc09XPoMnWSa5P88FvbwD8HvgEcAQ50ww4Aj42rSEmSJtkoM+MB8IUkb43/n1X15SRfAR5J8gDwPHD/+MqUJGlyDQ3jqnoO+MBFjv8NcOc4ipIkaZp4By5JkhozjCVJaswwliSpMcNYkqTGDGNJkhozjCVJaswwliSpMcNYkqTGDGNJkhozjCVJaswwliSpMcNYkqTGDGNJkhozjCVJaswwliSpMcNYkqTGDGNJkhozjCVJaswwliSpMcNYkqTGDGNJkhobOYyTXJXk60m+2O3fnOSpJM8m+VyS7eMrU5KkybWemfEngJNr9n8T+O2q+nHgNeCBPguTJGlajBTGSW4Cfgb4VLcf4A7g0W7IIeC+cRQoSdKkG3Vm/DvArwE/6PZ/FDhXVW92+6eAG3uuTZKkqbBt2IAkPwucrarjSebW+wZJFoAFgMFgwNLS0npf4pIuXLhmXePPfeuN1Y3T/dUwipWVlV5/72lnP/tjL/tlP/s1Tf0cGsbA7cDPJflp4L3AdcAngZ1JtnWz45uAFy/25Ko6CBwEmJ2drbm5uT7qBmD56OF1jd+/Z8fqxr7+ahjF0tISff7e085+9sde9st+9mua+jl0mbqqfr2qbqqqvcDHgCeq6l8ATwI/3w07ADw2tiolSZpgl/M5438P/Lskz7J6DvnhfkqSJGm6jLJM/XeqaglY6rafAz7Uf0mSJE0X78AlSVJjhrEkSY0ZxpIkNWYYS5LUmGEsSVJjhrEkSY0ZxpIkNWYYS5LUmGEsSVJjhrEkSY0ZxpIkNWYYS5LUmGEsSVJjhrEkSY0ZxpIkNWYYS5LUmGEsSVJjhrEkSY0ZxpIkNWYYS5LUmGEsSVJjQ8M4yXuT/O8k/yfJN5P8x+74zUmeSvJsks8l2T7+ciVJmjyjzIy/C9xRVR8A9gMfTfJh4DeB366qHwdeAx4YX5mSJE2uoWFcq1a63au7PwXcATzaHT8E3DeWCiVJmnAjnTNOclWSZeAs8Djwl8C5qnqzG3IKuHE8JUqSNNlSVaMPTnYCXwB+A/hMt0RNkj3Al6rq/Rd5zgKwADAYDG5bXFzso24ALpx/dV3jd2zftrpxzXW91TCKlZUVZmZmNvU9J5n97I+97Jf97Nck9HN+fv54Vc0OG7dtPS9aVeeSPAl8BNiZZFs3O74JePESzzkIHASYnZ2tubm59bzlu1o+enhd4/fv2bG6sa+/GkaxtLREn7/3tLOf/bGX/bKf/Zqmfo5yNfWPdTNikuwA7gZOAk8CP98NOwA8Nq4iJUmaZKPMjHcDh5JcxWp4P1JVX0xyAlhM8p+ArwMPj7FOSZIm1tAwrqo/Az54kePPAR8aR1GSJE0T78AlSVJjhrEkSY0ZxpIkNWYYS5LUmGEsSVJjhrEkSY0ZxpIkNbau22FudcsvnAPgle+fGfk5d906GFc5kiQBzowlSWrOMJYkqTHDWJKkxgxjSZIaM4wlSWrMMJYkqTHDWJKkxgxjSZIaM4wlSWrMMJYkqTHDWJKkxqbq3tRv2fXSE6MPvmrn6uO+e8ZTjCRp6jkzliSpsaFhnGRPkieTnEjyzSSf6I7fkOTxJM90j9ePv1xJkibPKDPjN4FfrapbgQ8Dv5TkVuBB4FhV3QIc6/YlSdI6DQ3jqjpdVV/rts8DJ4EbgXuBQ92wQ8B94ypSkqRJtq5zxkn2Ah8EngIGVXW6+9HLwKDXyiRJmhKpqtEGJjPAHwH/uao+n+RcVe1c8/PXquod542TLAALAIPB4LbFxcV+KgcunH+1t9e6lB3buwvOr7luw6+xsrLCzMxMTxXJfvbHXvbLfvZrEvo5Pz9/vKpmh40b6aNNSa4Gfg/4bFV9vjt8JsnuqjqdZDdw9mLPraqDwEGA2dnZmpubG+UtR7J89HBvr3Up+/fsWN3YN7fh11haWqLP33va2c/+2Mt+2c9+TVM/R7maOsDDwMmq+q01PzoCHOi2DwCP9V+eJEmTb5SZ8e3ALwJ/nmS5O/YfgIeAR5I8ADwP3D+eEiVJmmxDw7iq/gTIJX58Z7/lSJI0fbwDlyRJjRnGkiQ1ZhhLktSYYSxJUmOGsSRJjRnGkiQ1ZhhLktSYYSxJUmOGsSRJjRnGkiQ1NtK3Nk2z5RfOAfDK98+MNP6uW/1aZ0nS+jgzliSpMcNYkqTGXKYet6e/BN99Y/VxVPvuGV89kqQrjjNjSZIaM4wlSWrMZeqeHT3xD6+63vXSOS587xqWX3j9ouP379m5GWVJkq5gzowlSWrMMJYkqTGXqRt766Yia73bDUa8qYgkTR5nxpIkNTY0jJN8OsnZJN9Yc+yGJI8neaZ7vH68ZUqSNLlGmRl/Bvjo2449CByrqluAY92+JEnagKFhXFV/DLz6tsP3Aoe67UPAfT3XJUnS1NjoOeNBVZ3utl8GvKpIkqQNSlUNH5TsBb5YVe/v9s9V1c41P3+tqi563jjJArAAMBgMbltcXOyh7FUXzr99wj4+b1593Yaet+2N7/DGD97D1e/5Qc8VvdNGa3y7H37vlX2R/crKCjMzM63LmAj2sl/2s1+T0M/5+fnjVTU7bNxG/9Y9k2R3VZ1Oshs4e6mBVXUQOAgwOztbc3NzG3zLd1o+eri31xrmlff90w09b9dLT3D6wjXs3vHdnit6p43W+HZzV/jHp5aWlujz36NpZi/7ZT/7NU393Ogy9RHgQLd9AHisn3IkSZo+o3y06TDwp8C+JKeSPAA8BNyd5Bngrm5fkiRtwNBl6qr6hUv86M6ea5EkaSp5By5Jkhq7si+bvYLseumJ1iUMtZEaX3nfHWOoRJK0Hs6MJUlqzDCWJKkxw1iSpMYMY0mSGjOMJUlqzKupJeDoiTPrGn/XFX7LUElbizNjSZIaM4wlSWrMMJYkqTHDWJKkxgxjSZIa82pqbczTX/q7zeUXzo30lL7ug33h/735rlc/T/SVzmv6vtawfwYX6/1E90naYpwZS5LUmGEsSVJjLlNPuYt+7eJVOzfvvUYwsV/zeIkl581y9MSZoUv+l8NlcGl0zowlSWrMMJYkqTGXqfUOo14dfaUa17Lr5bzHNC7ZjvufwyT0tPd/jzZy6mPfPet/jnrnzFiSpMYuK4yTfDTJ00meTfJgX0VJkjRNNrxMneQq4L8BdwOngK8kOVJVJ/oqTtpsG73ie5jll8bysmO3kX5s1tXvR0+cWXd9+/d0nxQY09Ls+Ytcnf5uNe66xPFN/QTBRq/q73p4xZ6yuczfa7Ndzsz4Q8CzVfVcVX0PWATu7acsSZKmx+WE8Y3AC2v2T3XHJEnSOoz9auokC8BCt7uS5Olxv+cVaBfwSusiJoj97I+97Jf97Nck9PMfjzLocsL4RWDPmv2bumP/QFUdBA5exvtseUm+WlWzreuYFPazP/ayX/azX9PUz8tZpv4KcEuSm5NsBz4GHOmnLEmSpseGZ8ZV9WaSfwP8AXAV8Omq+mZvlUmSNCUu65xxVf0+8Ps91TLJpnqZfgzsZ3/sZb/sZ7+mpp+pqtY1SJI01bwdpiRJjRnGY5RkT5Ink5xI8s0kn2hd01aX5KokX0/yxda1bHVJdiZ5NMlfJDmZ5COta9qqkvzb7r/xbyQ5nOS9rWvaSpJ8OsnZJN9Yc+yGJI8neaZ7vL5ljeNmGI/Xm8CvVtWtwIeBX0pya+OatrpPACdbFzEhPgl8uap+AvgA9nVDktwI/DIwW1XvZ/WC1o+1rWrL+Qzw0bcdexA4VlW3AMe6/YllGI9RVZ2uqq912+dZ/cvOu5RtUJKbgJ8BPtW6lq0uyY8APwU8DFBV36uqrf3dmW1tA3Yk2Qb8ELBF70beRlX9MfDq2w7fCxzqtg8B921qUZvMMN4kSfYCHwSealvJlvY7wK8BP2hdyAS4Gfg28Lvdsv+nklzbuqitqKpeBP4r8NfAaeD/VtUftq1qIgyq6nS3/TKw9b/A+l0YxpsgyQzwe8CvVNV3WtezFSX5WeBsVR1vXcuE2Ab8M+C/V9UHgdeZ8GXAcenOZd7L6v/gvA+4Nsm/bFvVZKnVj/1M9Ed/DOMxS3I1q0H82ar6fOt6trDbgZ9L8i1WvyHsjiT/o21JW9op4FRVvbVS8yir4az1uwv4q6r6dlW9AXwe+MnGNU2CM0l2A3SPZxvXM1aG8RglCavn5E5W1W+1rmcrq6pfr6qbqmovqxfHPFFVzj42qKpeBl5Isq87dCfgd5FvzF8DH07yQ91/83fixXB9OAIc6LYPAI81rGXsDOPxuh34RVZnccvdn59uXZTU+Tjw2SR/BuwH/kvjerakbnXhUeBrwJ+z+vfq1Nw5qg9JDgN/CuxLcirJA8BDwN1JnmF19eGhljWOm3fgkiSpMWfGkiQ1ZhhLktSYYSxJUmOGsSRJjRnGkiQ1ZhhLktSYYSxJUmOGsSRJjf1/Q/wm2yqlkEQAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 576x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df[df.photo_month_mean < q].groupby('male_dummy')['photo_month_mean'].hist(figsize=(8,4), alpha=0.3, bins=30);"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>mean</th>\n",
" <th>std</th>\n",
" <th>median</th>\n",
" </tr>\n",
" <tr>\n",
" <th>male_dummy</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2.420398</td>\n",
" <td>1.815699</td>\n",
" <td>1.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.130265</td>\n",
" <td>1.811635</td>\n",
" <td>1.500000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" mean std median\n",
"male_dummy \n",
"0 2.420398 1.815699 1.666667\n",
"1 2.130265 1.811635 1.500000"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[df.photo_month_mean < q].groupby('male_dummy')['photo_month_mean'].agg(['mean', 'std', 'median'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Разброс между группами не различается. Если вы сразу догадались, что данные кишат выбросами и обрезали их, вы вполне себе можете теперь сравнивать типичных представителей по средним. Такой ответ тоже считается правильным. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__[3] Немного сегментации__\n",
"\n",
"Давайте попробуем разбить всех людей из нашей таблички на равные сегменты по тому сколько у них есть друзей с потока (`friends_from_course_cnt`). Для этого: \n",
"\n",
"* Заполните пропуски в переменной `friends_from_course_cnt` нулями.\n",
"* Найдите для переменной `friends_from_course_cnt` квантили уровня $0.25, 0.5, 0.75$. \n",
"* Создайте колонку `segment` и заполните её по следующему принципу: \n",
" - Поставьте в неё $1$, если у человека друзей меньше, чем квантиль уровня $0.25$ \n",
" - Поставьте $2$, если у человека друзей от квантиля $0.25$ до $0.5$\n",
" - Поставьте $3$, если у человека друзей от квантиля $0.5$ до $0.75$\n",
" - Поставьте $4$, если у человека друзей от квантиля $0.75$\n",
" \n",
"Получившиеся группы - это разные сегменты курса. Одни из них более социальные, другие менее социальные. Используя полученную колонку ответьте на вопросы: \n",
"\n",
"- Какое направление (БММ или УБ) оказалось более социальным? \n",
"- Правда ли, что более социальные люди чаще лайкают мемы в вышкинской группе? \n",
"- Чётко опишите все выводы, которые вы сделали. \n",
"\n",
"__Hint:__ для выявления группы для человека можно написать функцию, которая сравнит его число друзей со всеми квантилями и выдаст группу. Потом эту функцию можно применить к колонке с помощью `apply`."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"df.friends_from_course_cnt.fillna(0, inplace=True)\n",
"q = df.friends_from_course_cnt.quantile([0.25, 0.5, 0.75]).values"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"def find_segm(friends):\n",
" if friends <= q[0]:\n",
" return 1\n",
" elif (friends > q[0])&(friends <= q[1]):\n",
" return 2\n",
" elif (friends > q[1])&(friends <= q[2]):\n",
" return 3\n",
" else:\n",
" return 4\n",
" \n",
"df['segment'] = df.friends_from_course_cnt.apply(find_segm)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th></th>\n",
" <th>segment</th>\n",
" </tr>\n",
" <tr>\n",
" <th>is_bmm</th>\n",
" <th>segment</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th rowspan=\"4\" valign=\"top\">False</th>\n",
" <th>1</th>\n",
" <td>63</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>68</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>60</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>72</td>\n",
" </tr>\n",
" <tr>\n",
" <th rowspan=\"4\" valign=\"top\">True</th>\n",
" <th>1</th>\n",
" <td>45</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>42</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>48</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>27</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" segment\n",
"is_bmm segment \n",
"False 1 63\n",
" 2 68\n",
" 3 60\n",
" 4 72\n",
"True 1 45\n",
" 2 42\n",
" 3 48\n",
" 4 27"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Дальше есть какие-нибудь групп баи вроде таких и пояснения: \n",
"df.groupby(['is_bmm', 'segment']).agg({'segment': 'count'})"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>likes_memes</th>\n",
" </tr>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>22.753623</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>17.550459</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>28.121495</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>29.252525</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" likes_memes\n",
"segment \n",
"1 22.753623\n",
"2 17.550459\n",
"3 28.121495\n",
"4 29.252525"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.groupby('segment').agg({'likes_memes': 'mean'})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__Ваши выводы и их обоснование:__ \n",
"\n",
"- Если тут не будет написано ваших пояснений, за задание вы получаете ноль баллов!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"--------"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__[n] Удиви нас. Попробуй найти в данных какую-то классную особенность. Если у тебя это получится, мы поставим дополнительные баллы.__ Если вы найдёте полную фигню (сколько всего друзей у Маши или типа того), баллов не будет. Найденный факт реально должен выносить мозг и сносить крышу."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Попробуй меня на вкус!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Настрадался? Выскажи всё, что думаешь обо всём этом [в анонимке по третьему дз.](https://docs.google.com/forms/d/e/1FAIpQLSf5IFDJv8YsZDdkeLN4KXNU64zL9oXMtG5Rp36rsitOYOwYwQ/viewform) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": false,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}