{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "source": [ "DQ_birthdates_advanced_stats_v1.0.ipynb\n", "\n", "**Analysis approach of the data quality of dates using z-score (with Python)**" ], "metadata": { "id": "ZC8Y0xZhw2j9" } }, { "cell_type": "markdown", "source": [ "As a continuation of the article *Analysis approach of the data quality of dates using basic statistical methods (with Python)*, we are going to use z-score, a statistical calculation based on the standard deviation.\n", "\n", "**Z-score**\n", "After importing and exploring the dates of birth dataset (from the article related) we need to transform all dates to numbers (dates to simple number and dates to ages from now).\n", "Please note that we have previously reported a potential issue with the date 01/01/2000 (mm/dd/YYYY) possibly due to a technical error or the use of the date as a dummy date, so we will remove again these dates for this analysis." ], "metadata": { "id": "qpMUDYkLxB4t" } }, { "cell_type": "code", "source": [ "import pandas as pd\n", "from datetime import date\n", "\n", "def years_from_now(d):\n", " today = d.today()\n", " age = today.year - d.year - ((today.month, today.day) < (d.month, d.day))\n", " \n", " return age\n", "\n", "#Import dataset dates from github\n", "url = \"https://raw.githubusercontent.com/mabrotons/datasets/master/birthdates.csv\"\n", "\n", "\n", "df = pd.read_csv(url, index_col=0, parse_dates=['birthdates'])\n", "\n", "#transforming dates in numbers\n", "df['birthdates_num'] = [int(d.strftime(\"%Y%m%d\")) for d in df['birthdates']] \n", "df = df.loc[df['birthdates_num'] != 20000101]\n", "\n", "#transforming dates in ages old\n", "df['ages'] = [years_from_now(d) for d in df['birthdates']] " ], "metadata": { "id": "sQHn2cRVxHX5" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "We will build a couple of plots to represent birthdates, in order to have a first view of the dataset. As the plots show, the data represented is identical in both plots (symmetrically), and we have to decide which date format will be most useful for analysis." ], "metadata": { "id": "03XxFY6AvUch" } }, { "cell_type": "code", "source": [ "import matplotlib.pyplot as plt\n", "from matplotlib import ticker\n", "\n", "fig, axes =plt.subplots(1, 2, figsize=(20,5))\n", "\n", "dates_num = df['birthdates_num']\n", "ages = df['ages']\n", "\n", "axes[0].hist(dates_num, bins=50, edgecolor='black')\n", "axes[1].hist(ages, bins=50, edgecolor='black')\n", "plt.xticks(rotation=30)\n", "\n", "#formating the xticks labels (year) for first subplot()\n", "axes[0].xaxis.set_major_formatter(ticker.FuncFormatter(lambda x,pos: format(x/10000,'1.0f')))\n", "axes[0].title.set_text('Dates (num format)')\n", "axes[1].title.set_text('Ages (from now)')\n", "\n", "plt.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 323 }, "id": "zFtR17L6vVdq", "outputId": "7ca072db-a9ec-4582-97f6-2e61491b241f" }, "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "
" ], "image/png": "iVBORw0KGgoAAAANSUhEUgAABIQAAAFHCAYAAADQlpeLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhlZ1Uv/u8iHRBkCJCGdNLpJCIogzJFQFTggjJGw08RiAgB0cjkgFxJov4AUZSIEwjijQQTFBkFiREvRkZFAoSZBJCIhE6TJs2MTCZh3T/27qTSdFdXd53qU1X783me89TZ0zmrzlvn1Dprv++7q7sDAAAAwHRca94BAAAAAHBgKQgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATIyCELBkVfWLVfWn844jSarq8VX1mar676q66bzjWaiqrlNVH62qjfOOBQBYf6pq45hrXHdcvnlVva2qvlJVfzTv+GZt/P0+UlXXmXcssJ4oCMEqUlWfrKqvj//Mv1hV/15Vj6uqJb1Xq+roquqq2rACsV07yW8lec6sH3s/Yjk4yR8nuW93X7+7PzfneN5SVT+/c7m7v5nkxUlOmV9UAMBKGf/3f2GOBYpTkpzZ3V8fl09K8tkkN+zup8wpphXT3Z9J8uYMvycwIwpCsPr8eHffIMlRSZ6d5OQkZ8w3pCTJ8Uk+2t3b5h1Ikpsn+Y4kF+zrgTU4EJ99f5vkRGeyAGB9qaqjk/xIkk7yE3N4/uskOTHJ3yxYfVSSC7u793DMzE8WzsFLk/zivIOA9URBCFap7v5Sd5+d5GEZCgu3S5KqelBVva+qvlxVW6vqGQsOe9v484vjUKofHI/5ubGb7Req6g1VddS4vqrqT6rqsvHxPrTzeXbjAUneunNhQW+kE6vqU1X12ar6zQXbz6yq312wfK+qumTB8ier6ter6oNV9dWqOmPsDvxPYw+pf6mqG+8aRFXdKsnHFvyebxrX372q3l1VXxp/3n3BMW+pqmdV1duTfC3Jd42xP6GqPj4+3+9U1S3GXllfrqpXjr2iUlU3rqpzqmrH+BqeU1Wbx23PypAUPn98zZ8/tt8lSb6Q5G57eD0BgLXpUUnOS3JmhsLMVarqplX1D2Mu8e6q+t2q+rcF27+3qs6tqs9X1ceq6qELtj2wqi4c85JtVfW/9/D8d03yxTHXSFXtjOOpYy7yo1X1jKp6dVX9TVV9Ocmjq+rwqjp7fO6LquoXFjz3M6rqVeP+XxlzwltV1aljnri1qu67pxdkzOv+95jXfamqXlFV37Fg+y+Mz/n5MYbDx/W/XVV/Nt4/eMwJnzMuX7eqvlFVNxkf5p0ZcrijFm8eYKkUhGCV6+53JbkkQ9EhSb6aIRE5JMmDkjy+qh48brvH+POQcSjVO6rq+CS/keQnk2xM8q9JXjbud9/xmFsluVGShybZ0/Cr78vVhZiFfjjJ9yS5T5KnVdWt9+HX+6kkPzY+/48n+acx1o0ZPp9+edcDuvs/ktx2we957zFR+Mckz0ty0wzDyf6xrjm30CMzdDO+QZKLx3X3S3LnDEWbpyY5PcnPJjkyye2SnDDud60kf5Xh7NuWJF9PsrPw85sZXtMnja/5kxY850eS3H4fXg8AYPV7VIbeKi9Ncr+quvmCbS/IkKsdlqFIc1XBqKq+M8m5GXoR3yzJw5P8eVXdZtzljCS/OPYUv12SN+3h+a+Rk3X3o8dY/mDMRf5l3HR8kldnyBlfmuTlGXLKw5M8JMnvVdW9Fzzujyf56yQ3TvK+JG/IkAMdkeSZSf7PXl6Xhya5f5Jjknx/kkePv/e9k/z+uH1Thjzs5eMxb01yr/H+DyTZnqvz2R9M8rHu/vz4e16R5KLIrWBmFIRgbfh0kpskSXe/pbs/1N3f6u4PZiju3HORYx+X5Pe7+yPjP9LfS3KH8ezK5RkKJN+bpMZ9Lt3D4xyS5Cu7Wf/b3f317v5Akg9k3/5J/1l3f2YchvavSd7Z3e/r7m8keW2SOy7xcR6U5OPd/dfdfUV3vyzJRzMkNjud2d0XjNsvH9f9QXd/ubsvSPLhJP/c3Z/o7i9lKE7dMUm6+3Pd/Xfd/bXu/kqSZ2Xx13ynr2R43QCAdaCqfjjDCaJXdvd7kvxnkp8Ztx2U4WTX08ec4cIkZy04/Lgkn+zuvxrzkfcl+bskPz1uvzzJbarqht39he5+7x7C2FNOtqt3dPffd/e3khya5IeSnNzd3+ju9yd5UYbi1k7/2t1vGPPFV2U4QffsMW96eZKjq2qxvOZ53f3psYDzD0nuMK5/RJIXd/d7x3kWT03ygzUMvXtHkluOJ/HukaEodkRVXT9DrvXWXZ5DbgUzpCAEa8MRST6fJFV116p68zh86UsZCj6HLnLsUUmeW8Mk1V8cH6eSHNHdb8rQ0+UFSS6rqtOr6oZ7eJwvZCge7Wr7gvtfS3L9ffi9PrPg/td3s7zUxzo8V/f62eniDK/bTlv39/mr6npV9X+q6uKx2/XbkhwyJn6LuUGSLy4hfgBgbTgxwwmkz47Lf5urewFtTLIh18w5Ft4/Ksldd+ZkY172iAy9iZKhmPTAJBdX1VtrHPq/G3vKyXa18LkPT/L58cTWTrvmSrvmQZ/t7isXLCeL52Z7ygmvkad1939n6JF+xDgp9vkZij/3yFAA+vcMxavdFYTkVjBDCkKwylXVD2T4Z71z/PnfJjk7yZHdfaMkf5GhwJMMkxvuamuG7seHLLhdt7v/PUm6+3ndfeckt8kwdOvX9xDKB8ftS/XVJNdbsHzYnnacgU9nSLIW2pJk4QTYu51kcYmekmFY3F27+4a5uivzYq97ktw6Q68pAGCNq+ES7w9Ncs+q2l5V25M8Ocntq+r2SXYkuSLJ5gWHHbng/tYkb90lJ7t+dz8+Sbr73d19fIbhZH+f5JV7CGWpOdnC/OTTSW5SVQsLSbvmSivlGnnaOHTupgue+61J7p2hZ/a7x+X7JblLrp4fc+fE2N8duRXMjIIQrFJVdcOqOi5DF92/6e4PjZtukOEMzzeq6i4ZuymPdiT5VpLvWrDuL5KcWlW3HR/3RlX10+P9Hxh7HB2coYDzjfH43Xl9ljZMaqf3J3lgVd2kqg5L8qv7cOy+en2SW1XVz1TVhqp6WIYC1zkzevwbZDgz9sVxvqKn77L9M7nma56qOiLDML/zZhQDADBfD05yZYYc4w7j7dYZhr0/auxN85okzxh7F39vrjkk65wM+cojxwmUDx5zsVtX1bWr6hFVdaNxiNaXs+ec7F0ZeiofsYft36a7t2boefP7VfUdVfX9SR6ba16pbKW8LMljquoONVwh7fcyTBPwyXH7WzO8Thd29/8keUuSn0/yX929Y8Hj3CXDkLtde4UD+0lBCFaff6iqr2Q4i/SbGSZIfsyC7U9I8sxxn6dlwdmj7v5ahvlt3j52Rb5bd782yWlJXj4Od/pwhiuGJckNk/xlhq7HF2fovvucPcWV5Ht3XhViCf46wxmcTyb55ySvWOJx+6y7P5dhXP5TMvwOT01y3ILu3Mv1p0mum+SzGQo8/3eX7c9N8pAarkD2vHHdzyQ5axwrDwCsfScm+avu/lR3b995yzD8/hFjD5YnZbhQx/YMudDLknwzScbhWvfNMJn0p8d9TktynfHxH5nkk2O+9rgMw8m+zVg0OTPDhTD2xQlJjh6f+7UZ5jr6l0WPmIHxOf7/DPMlXZrkFhleg53+PUOetbM30IUZTlK+Ldf0iAwnOoEZqe7ljKIApqSqTkpym+5eyd4+a9549usDSe7R3ZfNOx4AYD6q6rQkh3X3iXvded8ed+eVY+84zsOzrlXVzTL0JLrjePERYAYUhAAAAGZgHCZ27SQfynAZ9dcn+fnu/vu5BgawGxvmHQAAAMA6cYMMw8QOzzDH4B8led1cIwLYAz2EAAAAACbGpNIAAAAAE6MgBAAAADAxq2IOoUMPPbSPPvroeYcBAKyQ97znPZ/t7o3zjoNrkoMBwPq2WA62KgpCRx99dM4///x5hwEArJCqunjeMfDt5GAAsL4tloMZMgYAAAAwMQpCAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMQpCAEzCps1bUlWL3jZt3jLvMIF1xOcOAKvZhnkHAAAHwvZtW3PUyecsus/Fpx13gKIBpsDnDgCrmR5CAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMQpCAAAAABOjIAQAAAAwMXstCFXVi6vqsqr68G62PaWquqoOHZerqp5XVRdV1Qer6k4rETQAwHonBwMAVtJSegidmeT+u66sqiOT3DfJpxasfkCSW463k5K8cPkhAgBM0pmRgwEAK2SvBaHufluSz+9m058keWqSXrDu+CQv6cF5SQ6pqk0ziRQAYELkYADAStqvOYSq6vgk27r7A7tsOiLJ1gXLl4zrAABYJjkYADArG/b1gKq6XpLfyNBVeb9V1UkZujRny5Yty3koAIB1Tw4GAMzS/vQQukWSY5J8oKo+mWRzkvdW1WFJtiU5csG+m8d136a7T+/uY7v72I0bN+5HGAAAkyIHAwBmZp8LQt39oe6+WXcf3d1HZ+iSfKfu3p7k7CSPGq90cbckX+ruS2cbMgDA9MjBWMymzVtSVYveNm3WIwyAq+11yFhVvSzJvZIcWlWXJHl6d5+xh91fn+SBSS5K8rUkj5lRnACsM5s2b8n2bVsX3eewI47MpZd8atF9YL2Sg7Evtm/bmqNOPmfRfS4+7bgDFA0Aa8FeC0LdfcJeth+94H4neeLywwJgvfPlBRYnBwMAVtJ+XWUMAAAAgLVLQQgAAABgYhSEAAAAACZGQQgAAABgYhSEAAAAACZGQQgAAABgYhSEAAAAACZGQQiAfbJp85ZU1aK3TZu3zDtMAABgERvmHQAAa8v2bVtz1MnnLLrPxacdd4CiAQAA9oceQgAAAAAToyAEAAAAMDEKQgAAAAAToyAEAABcxcUDAKbBpNIAAMBVXDwAYBr0EAIAAACYGAUhAAAAgIlREAIAgHk56GDz9QAwF+YQAgCAebnycvP1ADAXeggBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDE7LUgVFUvrqrLqurDC9Y9p6o+WlUfrKrXVtUhC7adWlUXVdXHqup+KxU4AMB6JgeDq23avCVVteht0+Yt8w4TYE3ZsIR9zkzy/CQvWbDu3CSndvcVVXVaklOTnFxVt0ny8CS3TXJ4kn+pqlt195WzDRsAYN07M3IwSJJs37Y1R518zqL7XHzacQcoGoD1Ya89hLr7bUk+v8u6f+7uK8bF85JsHu8fn+Tl3f3N7v6vJBclucsM4wWYFGdEYbrkYADASlpKD6G9+bkkrxjvH5EhOdnpknEdAPvBGVFgEXIwAGC/LWtS6ar6zSRXJHnpfhx7UlWdX1Xn79ixYzlhAABMihwMAFiu/S4IVdWjkxyX5BHd3ePqbUmOXLDb5nHdt+nu07v72O4+duPGjfsbBgDApMjBAIBZ2K+CUFXdP8lTk/xEd39twaazkzy8qq5TVcckuWWSdy0/TADYvaXMs1RV8w4TZkIOBgDMyl7nEKqqlyW5V5JDq+qSJE/PcEWL6yQ5d0yyz+vux3X3BVX1yiQXZujG/ERXtwBgJS1lnqXEXEusPXIwAGAl7bUg1N0n7Gb1GYvs/6wkz1pOUAAAUycHAwBW0rImlQYAAABg7VEQAgAAAJiYvQ4ZA4C5OehgE0IDAMAKUBACYPW68vK9ThhtsmgAANh3howBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAABr30EHp6oWvW3avGXeUQKsGhvmHQAAAMCyXXl5jjr5nEV3ufi04w5QMACrnx5CAAAAABOjIAQAAAAwMQpCAHOwafOWvc5zUFXzDhMAAFinzCEEMAfbt23d6zwHibkOAACAlaGHEAAAAMDE6CEEAACr2Xg5dQCYJQUhAABYzVxOHYAVYMgYAAAAwMQoCAEAAABMjIIQAAAAwMQoCAGsdeNko4vdNm3eMu8oAQCAVcSk0gBrnclGAQCAfbTXHkJV9eKquqyqPrxg3U2q6tyq+vj488bj+qqq51XVRVX1waq600oGD8AqtYReS8Di5GDM3BI+mw/05/OmzVtWXUwAU7GUHkJnJnl+kpcsWHdKkjd297Or6pRx+eQkD0hyy/F21yQvHH8CMCV6LcEsnBk5GLO0hM/m5MB+Pm/ftnXVxQQwFXvtIdTdb0vy+V1WH5/krPH+WUkevGD9S3pwXpJDqmrTrIIFYD+ZZwjWHDkYa91Sev8AMD/7O4fQzbv70vH+9iQ3H+8fkWTrgv0uGdddGgDmR48dWC/kYKwZS+n9438PwPws+ypj3d1Jel+Pq6qTqur8qjp/x44dyw0DAGBS5GAAwHLsb0HoMzu7IY8/LxvXb0ty5IL9No/rvk13n97dx3b3sRs3btzPMAAAJkUOBgDMxP4WhM5OcuJ4/8Qkr1uw/lHjlS7uluRLC7o1AwCwPHIwAGAm9jqHUFW9LMm9khxaVZckeXqSZyd5ZVU9NsnFSR467v76JA9MclGSryV5zArEDACw7snBAICVtNeCUHefsIdN99nNvp3kicsNCgBg6uRgAMBK2t+rjAEAAFN10MEuGw+wxikIAQAA++bKy11SHmCNW/Zl5wG4pk2bt6SqFr0BAADMkx5CADO2fdtWZ00BAIBVTQ8hAAAAgInRQwiAgQlCAQBgMhSEABgsYYLQxHA3AABYDwwZAwAApmHsDbvYbdPmLfOOEuCA0EMIAACYhiX0htUTFpgKPYQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJkZBCAAAAGBiFIQAAAAAJmZZBaGqenJVXVBVH66ql1XVd1TVMVX1zqq6qKpeUVXXnlWwAADIwQCA5dvvglBVHZHkl5Mc2923S3JQkocnOS3Jn3T3dyf5QpLHziJQAADkYADAbCx3yNiGJNetqg1Jrpfk0iT3TvLqcftZSR68zOcAAOCa5GAAwLLsd0Gou7cl+cMkn8qQhHwpyXuSfLG7rxh3uyTJEcsNEgCAgRwMAJiF5QwZu3GS45Mck+TwJN+Z5P77cPxJVXV+VZ2/Y8eO/Q0DAGBS5GAAwCwsZ8jYjyb5r+7e0d2XJ3lNkh9KcsjYfTlJNifZtruDu/v07j62u4/duHHjMsIAAJgUORgAsGzLKQh9Ksndqup6VVVJ7pPkwiRvTvKQcZ8Tk7xueSECALCAHAwAWLblzCH0zgwTF743yYfGxzo9yclJfq2qLkpy0yRnzCBOAAAiBwMAZmPD3nfZs+5+epKn77L6E0nuspzHBQBgz+RgAMByLfey8wCwfhx0cKpq0dumzVvmHSUwZ5s2b9nrZ8Uwmg8AVq9l9RACgHXlystz1MnnLLrLxacdd4CCAVar7du27vWzIvF5AcDqpocQAAAAwMQoCAEAAABMjIIQAAAAwMQoCAEAAABMjIIQAAAAwMQoCAEAAABMjIIQAAAAwMQoCAEAAABMjIIQAAAAwMQoCAEAAABMjIIQwD7YtHlLqmrRGwCwhh108F7/12/avGXeUQIs24Z5BwCwlmzftjVHnXzOovtcfNpxBygaAGDmrrzc/3pgEvQQAgAAAJgYBSEAAACAiVEQAgAAAJgYBSEAAACAiVEQAgAAAJgYBSEAAACAiVEQAgAAAJgYBSEAAACAiVEQAgAAAJgYBSEAAACAiVEQAgAAAJgYBSEAAACAiVlWQaiqDqmqV1fVR6vqI1X1g1V1k6o6t6o+Pv688ayCBQBADgYALN9yewg9N8n/7e7vTXL7JB9JckqSN3b3LZO8cVwGAGB25GAAwLLsd0Goqm6U5B5JzkiS7v6f7v5ikuOTnDXudlaSBy83SAAABnIwAGAWltND6JgkO5L8VVW9r6peVFXfmeTm3X3puM/2JDdfbpAAAFxFDgYALNtyCkIbktwpyQu7+45JvppduiZ3dyfp3R1cVSdV1flVdf6OHTuWEQYAwKTIwQCAZVtOQeiSJJd09zvH5VdnSE4+U1WbkmT8ednuDu7u07v72O4+duPGjcsIAwBgUuRgAMCy7XdBqLu3J9laVd8zrrpPkguTnJ3kxHHdiUlet6wIAQC4ihwMAJiFDcs8/peSvLSqrp3kE0kek6HI9MqqemySi5M8dJnPAQDANcnBAIBlWVZBqLvfn+TY3Wy6z3IeFwCAPZODAQDLtZw5hAAAAABYgxSEAAAAACZGQQgAAABgYhSEAAAAACZGQQgAAABgYhSEAAAAACZGQQgAAABgYhSEAAAAACZGQQgAAABgYhSEAAAAACZGQQgAgEnYtHlLqmrR26bNW+YdJgAcEBvmHQAAABwI27dtzVEnn7PoPhefdtwBigYA5ksPIQCYsaX0QtATAQCAedJDCABmbCm9EBI9EQAAmB89hAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGKWXRCqqoOq6n1Vdc64fExVvbOqLqqqV1TVtZcfJgAAC8nBAIDlmEUPoV9J8pEFy6cl+ZPu/u4kX0jy2Bk8BwAA1yQHAwD227IKQlW1OcmDkrxoXK4k907y6nGXs5I8eDnPAUzXps1bUlWL3jZt3jLvMGH/HXSwv3H2ixwMpkVOBKyEDcs8/k+TPDXJDcblmyb5YndfMS5fkuSIZT4HMFHbt23NUSefs+g+F5923AGKBlbAlZf7G2d/ycFgQuREwErY7x5CVXVcksu6+z37efxJVXV+VZ2/Y8eO/Q0DYEmcWQPWCznY7i3lcx4ONPkHsJotp4fQDyX5iap6YJLvSHLDJM9NckhVbRjPUG1Osm13B3f36UlOT5Jjjz22lxEHwF45swasI3Kw3fA5z2rk7xJYzfa7h1B3n9rdm7v76CQPT/Km7n5Ekjcneci424lJXrfsKAEASCIHAwBmYxZXGdvVyUl+raouyjCe/YwVeA4AAK5JDgYALNlyJ5VOknT3W5K8Zbz/iSR3mcXjAgCwZ3IwAGB/rUQPIQAAAABWMQUhAAAAgIlREAKANc5ljQEA2FczmUMIAJgflzVmrdq0eUu2b9u66D6HHXFkLr3kUwcoIgCYDgUhAADmQjETAObHkDEAAACAiVEQAgAAAJgYBSEAAACAiVEQAgAAAJgYBSEAAACAiXGVMWBtO+jgVNW8owAApmSW+YdcBpgTBSFgbbvy8r1esjhx2WIAYIZmmX8s4bHkMcBKUBAC2MkZOgAAYCIUhAB2coYOAACYCJNKAwAAAEyMghAAAADAxBgyBgD7wlxTcGAd6Pec9zgAE6EgBAD74kDPNeXLKVN3oK8maT45ACZCQQgAVjNfTgEAWAHmEAIAAACYGAUhAAAAgIlREAIAAACYGAUhAAAAgIlREAIAAACYGAUhAAAAgIlREAIAAACYmP0uCFXVkVX15qq6sKouqKpfGdffpKrOraqPjz9vPLtwAQCmTQ4GAMzCcnoIXZHkKd19myR3S/LEqrpNklOSvLG7b5nkjeMyAACzIQcDAJZtvwtC3X1pd793vP+VJB9JckSS45OcNe52VpIHLzdIAAAGcjAAYBZmModQVR2d5I5J3pnk5t196bhpe5Kbz+I5AAC4JjkYALC/ll0QqqrrJ/m7JL/a3V9euK27O0nv4biTqur8qjp/x44dyw0DAGBS5GAAwHIsqyBUVQdnSERe2t2vGVd/pqo2jds3Jblsd8d29+ndfWx3H7tx48blhAEAMClyMABguZZzlbFKckaSj3T3Hy/YdHaSE8f7JyZ53f6HBwDAQnIwAGAWNizj2B9K8sgkH6qq94/rfiPJs5O8sqoem+TiJA9dXogAACwgBwMAlm2/C0Ld/W9Jag+b77O/jwsAwJ7JwQCAWZjJVcYAAAAAWDsUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhAAAAAAmRkEIAAAAYGIUhJJs2rwlVbXX26bNW2byWEt5HAAAAGC2Zvmdfa1//98w7wBWg+3btuaok8/Z634Xn3bcTB5rKY8DAAAAzNYsv7Ov9e//egitUrPstXQgrdW4AQAApmCt9mpZq3GvZnoIrVKz7LV0IK3VuAEAAKZgrfZqWatxr2brvofQUqqIAAAAADN30MGrtmfTuu8hpIoIAAAAzMWVl6/amsS67yE0U0uo7B3Qx1niY7k6GgAAALN0oK/WtSQH8Dv7ehhttO57CM3UrCp7s6wQzuix9KQCWOfGxGYxhx1xZC695FMHKCDWu02bt2T7tq3zDgOAFbIqr9Z1AL+zL/mxVjEFIQCYglXcXZn1yckmAFjdVmzIWFXdv6o+VlUXVdUpK/U8LMEsh6gBAKuW/AuAVWUiQ6/WqhXpIVRVByV5QZIfS3JJkndX1dndfeFKPB974awwAKx78i8AVp2JDL1aq1aqh9BdklzU3Z/o7v9J8vIkx6/Qc3Gg6GkEAKuZ/AuA2fDdbxJWag6hI5IsnEXwkiR3XaHn4kDR0wgAVjP5FwCz4bvfJFR3z/5Bqx6S5P7d/fPj8iOT3LW7n7Rgn5OSnDQufk+Sj804jEOTfHbGj8nyaJPVR5usPtpkddIuy3dUd2+cdxDr2VLyr3H9Sudgy+X9tvpok9VHm6xO2mX10SaL5GAr1UNoW1vQSfwAAAx/SURBVJIjFyxvHtddpbtPT3L6Cj1/qur87j52pR6ffadNVh9tsvpok9VJu7BG7DX/SlY+B1su77fVR5usPtpkddIuq482WdxKzSH07iS3rKpjquraSR6e5OwVei4AAORfAMA+WJEeQt19RVU9KckbkhyU5MXdfcFKPBcAAPIvAGDfrNSQsXT365O8fqUefwlWbVfoCdMmq482WX20yeqkXVgTVkH+NQveb6uPNll9tMnqpF1WH22yiBWZVBoAAACA1Wul5hACAAAAYJVaUwWhqnpxVV1WVR9esO4OVXVeVb2/qs6vqruM66uqnldVF1XVB6vqTguOObGqPj7eTpzH77Je7GOb3KuqvjSuf39VPW3BMfevqo+N7XXKPH6X9WIPbXL7qnpHVX2oqv6hqm64YNup4+v+saq634L12mRG9qVNquroqvr6gvfJXyw45s7j/heNn281j99nPaiqI6vqzVV1YVVdUFW/Mq6/SVWdO/5/OLeqbjyu9z8FDoCqWrHpDACAa1pTBaEkZya5/y7r/iDJb3f3HZI8bVxOkgckueV4OynJC5Mh2U/y9CR3TXKXJE/fmfCzX87M0tskSf61u+8w3p6ZJFV1UJIXZGiz2yQ5oapus+KRr19n5tvb5EVJTunu70vy2iS/niTj6/zwJLcdj/nzqjpIm8zcmVlim4z+c8H75HEL1r8wyS/k6s+2XR+TpbsiyVO6+zZJ7pbkiePf+ClJ3tjdt0zyxnE58T8FVlRVbaiqP0zyR1X1o/OOh2uqqkdV1T2r6kbj8lr7DrGuVdXmBfe1zSpQVd9VVdcb72uTVaCqHlpVv1ZVd5t3LKvJmvrj7O63Jfn8rquT7OztcKMknx7vH5/kJT04L8khVbUpyf2SnNvdn+/uLyQ5N75U7bd9bJM9uUuSi7r7E939P0lenqH92A97aJNbJXnbeP/cJD813j8+ycu7+5vd/V9JLsrQHtpkhvaxTXZr/Py6YXef18Pkby9J8uBZxzoV3X1pd793vP+VJB9JckSGv/Ozxt3OytWvsf8psELG3o7PS7IpybuSnFxVT6yq68w3smmrqmtV1eFV9eYkJyb5mSQvrKpDu/tbeqnOX1Vtqao3Jfnbqjqrqo7p7m/NO64pq6pNVfW2JH+T5HVVdVttMl/jye6nJTl5XPWXVfWT84xpNVlTBaE9+NUkz6mqrUn+MMmp4/ojkmxdsN8l47o9rWd29tQmSfKDVfWBqvqnqrrtuE6brLwLcnVB56eTHDne9z6Znz21SZIcU1Xvq6q3VtWPjOuOyNAOO2mTGamqo5PcMck7k9y8uy8dN21PcvPxvvcKrJwbJLlDksd190sz5A63yvDZyBxU1c3GL7E3SLKtu++T5AlJPhtX7JmrXQpxj09yXnffI8mlSZ5bVYfMJzJGD0vy7u6+e4aexqdW1Z3nHNOkdfeVSb4nQ8/wP87Qs/tJVXXr+Ua2OqyHgtDjkzy5u49M8uQkZ8w5HvbcJu9NclR33z7JnyX5+znFN0U/l+QJVfWeDMnd/8w5HvbcJpcm2dLdd0zyaxnO+t1wD4/BMlXV9ZP8XZJf7e4vL9w29sRyKU5YYeN775NJHj2uenuS9yW5e1UdNqewJmk8k/7MJG+vqsMzfIlKctWXql/N0C737O42FGYurrvgfmc4eZHuPiXJt5I8rKoOnkdgUzXO8XijcfHaSQ5Oku5+dpLLkvyYz7IDa8Ew150F0s8kuXFVbeju1yS5MMN7ZfKfYevhBTgxyWvG+6/KMNQlSbblmmfcN4/r9rSe2dltm3T3l7v7v8f7r09ycFUdGm2y4rr7o9193+6+c5KXJfnPcZP3yZzsqU3G4XufG++/Z1x/qwyv/+YFD6FNlmlMmP8uyUvH5CBJPjMOBds5TO+ycb33Cqys1ya5Q1VtGnOFDyX5ZoZhZBwAY4/Uj2c4SXHP7v50hmGwP1LjBULGXkPPGG87lzkAquo+VfVvSV5QVY8YV38lybcWnDh6QZKH5OqpG1hBY5v8a5KfzzA3YTJMlbGjqraMy6/IMB/nUXMIcVLGC4BsWjDM9REZ3i/Xz9C78fuSXH/c/c+S/H+5uif4ZK2HgtCnk9xzvH/vDP/IkuTsJI8a/zDuluRL4zCANyS5b1XdeJz4877jOmZnt21SVYft7OY6JhbXSvK5JO9OcsuqOqaqrp1hkuOzD3jU61hV3Wz8ea0kv5Vk55Wrzk7y8Kq6TlUdk2HC3HdFm6y4PbVJVW0cJ/VOVX1Xhjb5xPj59eWqutv4PnpUktfNJfh1YHwNz0jykbH78E5nZ0giMv583YL1/qfAyvm3DAn7o5OrCuI/kGv2hmBlfTnJDbr7yd396aq6VXd/PckfZfjytPN/1t9n+MLrC+4BUsMFDH43yZ9mmEPwYVX1pAyF1PsmObKqqrvPzdBL6GfH48zzNGNjHnBQVT0hwzxBL+jux3f3V8ddLkqyJcn3j23yziRXJvnRncfPJfB1rqoOGnt2Lxzm+vgkX8owR92fJ7l7hna5Xnd/LMP8kZMfmrymLu1ZVS9Lcq8kh1bVJRnG//1ChvGyG5J8I8PVX5Lk9UkemOFN+bUkj0mS7v58Vf1Ohi+8SfLM7t51sleWaB/b5CFJHl9VVyT5epKHj2/cK8Z/am9IclCSF3f3BQf2N1k/9tAm16+qJ467vCbJXyVJd19QVa/M0G3yiiRPHLuER5vMzr60SZJ7JHlmVV2eIal73ILPqCdkuGLZdZP803hj//xQkkcm+VBVvX9c9xtJnp3klVX12CQXJ3nouM3/FFhB3X1pVb0uybOr6qIM76lv5Oqz7qyw7v5AVb12zAu+kOTWVfXfSZ6bZGNV/UKGK2RuTnJFd188x3DXvZ1DWcZeWIdn6DX32u6+sqq2JTkvw8UPLsiQY78ywxfcV2W4qMvOoc/MyII2ubKqvpqhh/ebx20PSvL27j6vqn4gyQ8n+e8kb0nyjxnyDm0yY+NJ1N9JclBVvT5D77grk6va6ZcyTMfwR0n+NsNJ7k0Zem5dmWH+yEkrf5MAAKwGVfWADGds757k+d39/DmHNCnjPCjvTPKy7v7tsQh0s3HzYUmOyTCM+S+7+zljDwhfJmasqh6T5FkZTsj9Vg0XP/iHJD/a3Z8Z93lhhvlqfjnDEL5bZSg+PDnJz3X3vxzwwNexBW1yZnf/RlVtTPKkJHdKcosk/5GhgP3uJM9J8sQMPR5fNe73xO7Ws3uGquqeGQrW78gwV+0jk/xehonvH9Ld7xr3e2KSB3X3A6vqwRk6Lxyeoc0e091fm0f8q4WCEAAAq0YN83t1d+sdNAdVdVh3b1+w/E9J/ri7z62q/5XkP7rbXGkrZJzv5G8y9Dw5MckJ3f2xqjorybW7+4RxvxtmuIrVT2aYMPenMhRSX97db59L8OvUbtrk4d39H1V13yQ/nuRFYw+770vy0iQP6+6PVNUDkxyb5E3d/W/zin+9qmHes6O7+6/H5T/P0JPu60l+qbvvPPbqulmS52e46NHWGib4vl53f2Jesa8mCkIAAMC3qapbZJjj7mnd/Y55xzMVVbWluz9VVc9Ockx3P6yqvjPD1fh+orvfMU7N8MIkv9Pdn5pnvFOwS5sc1d0njMWG6++8SulYzP7LJH/S3R+YZ7xTUFXXyzDs64pxeNgjktyuu08dpwM4o7v/rKqOzXDJ+RPmGvAqtR4mlQYAAGZgnDT3plX1kgzzbLxKMejAWlDg+dMkx1TVg8ZJi5+R5LfG4Uu/meGqSV+ZT5TTskub3KKq7jfO7/TVBbs9NcMcWwp0B0B3f228Ou+V46ofS7JjvP+YDPOgnZNhrqf3ziPGtUAPIQAA4CrjEJlHZJgv5ZvzjmfKquoXk/xsd//IuPyAJP8ryRFJTunurfOMb4rGNvmZ7r7nuPygDMWgbUl+3ZDKA2ucWLozTN79S919UVV9d4YrV94uyX9pkz1TEAIAAFhlqupa3f2tqnp1ku0Zrj76oiQfMpn3fOzSJpdmuJLY+5N8vLv1QpmDqqoME6y/KMlrk/xcks9lKA59eZ6xrQWGjAEAAKwyY+HhehkmxX1Ykou6+4OKQfOzS5uckOTS7n6FYtD8jO+HO2bo1fhrSV7b3ScqBi3NhnkHAAAAwG49IcP8Jz9m+N6qoU1Wn0syzKv1x9pk3xgyBgAAsArtHKI07zi4mjZhPVEQAgAAAJgYcwgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATIyCEAAAAMDEKAgBAAAATMz/A+hLSax3Ev5lAAAAAElFTkSuQmCC\n" }, "metadata": { "needs_background": "light" } } ] }, { "cell_type": "markdown", "source": [ "Now it's time to start the calculation of standard deviation to know how values are distributed around all dataset, and detect possible outliers: candidates to be outliers will be multiple standard-deviations far from the mean.\n", "As we can see with means and standard deviations calculated, it's easier to work with dates transformed to ages than to simple number." ], "metadata": { "id": "gcxvOeIM3SLo" } }, { "cell_type": "code", "source": [ "std_birthdates_num = df['birthdates_num'].std()\n", "mean_birthdates_num = df['birthdates_num'].mean()\n", "print(\"Mean of birthdates_num: \" + str(mean_birthdates_num))\n", "print(\"Standard deviation of birthdates_num: \" + str(std_birthdates_num))\n", "\n", "std_ages = df['ages'].std()\n", "mean_ages = df['ages'].mean()\n", "print(\"\\nMean deviation of ages: \" + str(mean_ages))\n", "print(\"Standard deviation of ages: \" + str(std_ages))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4RLNZ9x03tN1", "outputId": "f991e995-d80d-434e-8330-33e2ba9f6fd8" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Mean of birthdates_num: 19681984.715186547\n", "Standard deviation of birthdates_num: 336693.98125531274\n", "\n", "Mean deviation of ages: 53.895953757225435\n", "Standard deviation of ages: 33.67419358794017\n" ] } ] }, { "cell_type": "markdown", "source": [ "Now, selecting ages transformed like esay way, let's calculate z-score:\n", "\n", "z = (x – μ) / σ\n", "\n", "where Z is the score, x is the value to calculate the score, μ is the mean and σ is the standard deviation.\n", "The z-score is a calculation that measure how many standard deviations a value is far away from the mean, and the probability of data to be unusual in a distribution.\n", "It's recomended to use z-score with a normal distribution, because in a normal distribution over 99% of values fall within 3 standard deviations from the mean. For that, we can assume:\n", "- if a z-score returned is lower than 1 shoud be a normal data value\n", "- if a z-score returned is larger than 1 and lower than 3, could be an error\n", "- if a z-score returned is larger than 3 should be an error" ], "metadata": { "id": "8rFVJx6n-WhT" } }, { "cell_type": "code", "source": [ "df['zscore'] = [(a-mean_ages)/std_ages for a in df['ages']] \n", "\n", "f = plt.figure()\n", "f.set_figwidth(15)\n", "f.set_figheight(5)\n", "\n", "good_ages = df.loc[(df['zscore'] <= 1) & (df['zscore'] >= -1)]['ages']\n", "regular_ages = df.loc[((df['zscore'] > 1) & (df['zscore'] <= 3)) | ((df['zscore'] < -1) & (df['zscore'] >= -3))]['ages']\n", "bad_ages = df.loc[(df['zscore'] > 3) | (df['zscore'] < -3)]['ages']\n", "\n", "plt.hist([good_ages, regular_ages ,bad_ages], color=['Green', 'Orange', 'Red'], label=['good', 'regular', 'bad'], edgecolor='black', bins=60, histtype='barstacked')\n", "\n", "#add vertical line at mean value of x\n", "plt.axvline(x=mean_ages, color='blue', linewidth=3, label='mean')\n", "\n", "plt.title(\"Ages\")\n", "plt.legend()\n", "plt.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 425 }, "id": "39KesNitEt0r", "outputId": "e333bd54-b8e4-4c32-c7c3-d634c2849c78" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py:3208: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n", " return asarray(a).size\n", "/usr/local/lib/python3.8/dist-packages/matplotlib/cbook/__init__.py:1376: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n", " X = np.atleast_1d(X.T if isinstance(X, np.ndarray) else np.asarray(X))\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "
" ], "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAE/CAYAAADVKysfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAf0UlEQVR4nO3df7BedX0v+veH/CCWCEGhCZLUpD0UBEIh2URtLFJ/gLUKdq6X0ZtiSriTtsi9nnsP06POcN2OTKc6nPb0cm2n3CMQBWot2OHHbU9FKyq2CgkEC0QL1xMwXIiRYyI/DCTwvX/sB42YQPaPZz8re79eM8/sZ63nu9b6rJ01K3nn+13fp1prAQAAoJsOGnQBAAAA7JvQBgAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0ATElVdWtV/bCqDh50LQAwHkIbAFNOVS1O8htJWpKzBloMAIyT0AbAVPS+JN9IclWS1c+vrKpXVtVNVfWjqrqjqi6pqtv2+Py4qrqlqv57VX2nqs7Z47O3V9V9VfV4VT1cVRdN5gkBMH3NHHQBANAH70vyp0m+meQbVTW/tbY1ySeTPJlkQZLFSf4xyYNJUlWHJLklyf+R5LeSLE1yS1Xd01q7L8mnkpzTWvtaVR2eZMnknhIA05WeNgCmlKp6Q5JXJ/lca21Dkv83yf9UVTOS/A9JPtJae6oXxNbtsek7kmxurV3ZWtvdWrsryfVJ/sfe57uSHF9Vh7bWfthau3PSTgqAaU1oA2CqWZ3kC621H/SWr+2tOzIjI0y+t0fbPd+/Oslrq2r7868kqzLSK5eMBL63J3mwqr5SVa/v50kAwPMMjwRgyqiqlyU5J8mMqnq0t/rgJPOSzE+yO8nCJP/W+2zRHpt/L8lXWmtv3du+W2t3JDm7qmYluTDJ516wPQD0hZ42AKaSdyV5NsnxSU7uvV6T5GsZec7t80mGq+oXquq43rrn3ZzkV6vq3Kqa1XudWlWvqarZVbWqqg5rre1K8qMkz03miQEwfQltAEwlq5Nc2Vp7qLX26POvJP9XRoY6XpjksCSPJvlMkr9O8nSStNYeT3JGkvck+f96bT6ekZ66JDk3yeaq+lGSP+jtDwD6rlprg64BAAaiqj6eZEFrbfVLNgaAAdHTBsC00fsetpNqxIok5yf5u0HXBQAvxkQkAEwnL8/IkMhXJdma5D8luWGgFQHASzA8EgAAoMMMjwQAAOgwoQ0AAKDDOvFM2xFHHNEWL1486DKg8+68886MdkhzVWXZsmV9qijZsOGn75cv79thAACmtA0bNvygtXbk3j7rxDNtQ0NDbf369YMuAzqvqpLhUW40nFEHvdGo+un7DtxOAAAOSFW1obU2tLfPDI8EAADoMKENAACgw4Q2AACADuvERCTQNbMPnp1dz+wa1TazZs/KM08/06eKAAAOHLt27cqWLVuyc+fOQZfSOXPmzMnChQsza9as/d5GaIO92PXMrlFP+LFreHQhDwBgqtqyZUte/vKXZ/HixSMTqZFkZHK4xx57LFu2bMmSJUv2ezvDIwEAgAm1c+fOvPKVrxTYXqCq8spXvnLUPZBCGwAAMOEEtr0by+9FaAMAAOgwoQ0AAKDDhDYAAGDK2bx5c4477rj83u/9Xn71V381q1atyhe/+MWsXLkyxxxzTG6//fY8+eSTWbNmTVasWJFTTjklN9xww0+2/Y3f+I0sW7Ysy5Ytyz//8z8nSW699dacfvrpefe7353jjjsuq1atSmut7+di9kgAAGBKeuCBB/K3f/u3ueKKK3Lqqafm2muvzW233ZYbb7wxf/zHf5zjjz8+b3rTm3LFFVdk+/btWbFiRd7ylrfkF3/xF3PLLbdkzpw5uf/++/Pe974369evT5Lcdddduffee/OqV70qK1euzNe//vW84Q1v6Ot5CG0AAEDf9HM+kpfq5FqyZEmWLl2aJDnhhBPy5je/OVWVpUuXZvPmzdmyZUtuvPHGXHrppUlGZr186KGH8qpXvSoXXnhhNm7cmBkzZuTf/u3ffrLPFStWZOHChUmSk08+OZs3bxbaAAAAxuLggw/+yfuDDjroJ8sHHXRQdu/enRkzZuT666/Pscce+zPbDQ8PZ/78+bn77rvz3HPPZc6cOXvd54wZM7J79+4+n4Vn2gAAgGnqzDPPzGWXXfaT59LuuuuuJMmOHTty1FFH5aCDDspnPvOZPPvss4MsU2gDAAD6p7X+vcbr4osvzq5du3LSSSflhBNOyMUXX5wkueCCC7Ju3br82q/9Wr797W/nkEMOGf/BxqEmY7aTlzI0NNSef7APuqCqkuFRbjScvs8e1MW69hyn3oHbCQDQAZs2bcprXvOaQZfRWXv7/VTVhtba0N7a62kDAADoMKENAACgw4Q2AACADhPaAAAAOkxoAwAA6DChDQAAoMOENgAAgP2wefPmnHjiiZN+XKENAADoqwULF6SqJuy1YOGCQZ/SpJo56AIAAICpbevDW5PhCdzf8Nb9avexj30sV199dY488sgsWrQoy5cvz1ve8pb8wR/8QZ566qn8yq/8Sq644oocfvjh2bhx417Xb9iwIWvWrEmSnHHGGRN3EqOgpw0AAJhy7rjjjlx//fW5++678w//8A9Zv359kuR973tfPv7xj+db3/pWli5dmo9+9KMvuv68887LZZddlrvvvntg5yK0wQQaTbf+7INnD7pcAIAp6+tf/3rOPvvszJkzJy9/+cvzzne+M08++WS2b9+eN77xjUmS1atX56tf/Wp27Nix1/Xbt2/P9u3bc9pppyVJzj333IGci+GRMJGG97/pruFdfSsDAICpQ08bAAAw5axcuTI33XRTdu7cmSeeeCI333xzDjnkkBx++OH52te+liT5zGc+kze+8Y057LDD9rp+3rx5mTdvXm677bYkyTXXXDOQc9HTBgAATDmnnnpqzjrrrJx00kmZP39+li5dmsMOOyzr1q37yYQjv/zLv5wrr7wySfa5/sorr8yaNWtSVQObiERo44Az++DZ2fXM/g8tnDV7Vp55+pk+VgQAwIuZf/T8/Z7xcX/3tz8uuuiiDA8P56mnnsppp52W5cuX5+STT843vvGNn2u7r/XLly//mUlIPvGJT4y98DES2jjg7Hpml2fHAAAOII9ueXQgx127dm3uu+++7Ny5M6tXr86yZcsGUsd4CW0AAMCUdO211w66hAlhIhIAAIAOE9oAAAA6zPBI4IAw2gloEpPQAABTw0uGtqq6Isk7kny/tXZib90rkvxNksVJNic5p7X2w6qqJH+e5O1Jnkrye621O/tTOjCdjHYCmsQkNADA1LA/wyOvSvK2F6z7YJIvtdaOSfKl3nKS/FaSY3qvtUn+cmLKhKmpqkb1AgBgsG699da84x3vmNRjvmRPW2vtq1W1+AWrz05yeu/9uiS3JvmPvfWfbq21JN+oqnlVdVRr7ZGJKhimlOE+twcA6IDFixbkwS0T9z1tr144P5u/t/9fI9BaS2stBx00+VN67N69OzNnju+ptLFuPX+PIPZokue/3e7oJN/bo92W3rqfC21VtTYjvXH5pV/6pTGWAUy00T871vpWCwAwNTy4ZWvaNRO3v1r10gFw8+bNOfPMM/Pa1742GzZsyDnnnJObb745Tz/9dH7nd34nH/3oR5MkH/vYx3L11VfnyCOPzKJFi7J8+fJcdNFFOf3003PppZdmaGgoP/jBDzI0NJTNmzf/zDFuv/32fOADH8jOnTvzspe9LFdeeWWOPfbYXHXVVfn85z+fJ554Is8++2y+8pWvjOt8xz0RSWutVdWo/9XWWrs8yeVJMjQ05F990BGjfnZsNG0BACbR/fffn3Xr1uVHP/pRrrvuutx+++1preWss87KV7/61bzsZS/L9ddfn7vvvju7du3KsmXLsnz58v3e/3HHHZevfe1rmTlzZr74xS/mwx/+cK6//vokyZ133plvfetbecUrXjHu8xhraNv6/LDHqjoqyfd76x9OsmiPdgt76wAAACbVq1/96rzuda/LRRddlC984Qs55ZRTkiRPPPFE7r///jz++OM5++yzM2fOnMyZMyfvfOc7R7X/HTt2ZPXq1bn//vtTVdm166ejld761rdOSGBLxv49bTcmWd17vzrJDXusf1+NeF2SHZ5nAw4ksw+eParJYWYfPHvQJQMA+3DIIYckGXmm7UMf+lA2btyYjRs35oEHHsj555//otvOnDkzzz33XJJk586de21z8cUX5zd/8zdzzz335KabbvqZds8feyLsz5T/f52RSUeOqKotST6S5E+SfK6qzk/yYJJzes3/PiPT/T+QkSn/z5uwSgEmwWiHh/paAQDovjPPPDMXX3xxVq1alblz5+bhhx/OrFmzsnLlyvz+7/9+PvShD2X37t25+eabs3bt2iTJ4sWLs2HDhqxYsSLXXXfdXve7Y8eOHH300UmSq666qm/178/ske/dx0dv3kvbluT94y0KAABgopxxxhnZtGlTXv/61ydJ5s6dm6uvvjqnnnpqzjrrrJx00kmZP39+li5dmsMOOyxJctFFF+Wcc87J5Zdfnt/+7d/e637/6I/+KKtXr84ll1yyzzYToUZy1mANDQ219evXD7oMDhBVNeqJMkZ7nY/6GL3jjHoCj34fo7fNaM5/9L/fn+67n7eTsf6Z9P3PfgzHAICpbtOmTXnNa17zk+VBT/n/Yp544onMnTs3Tz31VE477bRcfvnlWbZs2YTse19e+PtJkqra0Fob2lv7cc8eCQAA8GImKmD1w9q1a3Pfffdl586dWb16dd8D21gIbQAAwLR17bXXDrqElzT5XwkOAADAfhPaAAAAOkxoAwAA6DChDQAAoMOENgAAYMrZvHlzTjzxxEnfth+ENmDCVNV+v2YfPHvQ5QIAk2TxggWj+nfCS70WL1gw6FOaVKb8BybO8P433TW8q29lAADd8uDWrWkTuL/aun9f1L179+6sWrUqd955Z0444YR8+tOfzqWXXpqbbropP/7xj/Prv/7r+au/+qtUVTZs2JA1a9YkSc4444wJrHb89LQBAABT0ne+851ccMEF2bRpUw499ND8xV/8RS688MLccccdueeee/LjH/84N998c5LkvPPOy2WXXZa77757wFX/PKENYJxGO6TD0FAAmByLFi3KypUrkyS/+7u/m9tuuy1f/vKX89rXvjZLly7NP/3TP+Xee+/N9u3bs3379px22mlJknPPPXeQZf8cwyMBxmt4dM0NDQWAyVFVP7d8wQUXZP369Vm0aFGGh4ezc+fOAVW3//S0AQAAU9JDDz2Uf/mXf0mSXHvttXnDG96QJDniiCPyxBNP5LrrrkuSzJs3L/Pmzcttt92WJLnmmmsGU/A+6GkDAACmpGOPPTaf/OQns2bNmhx//PH5wz/8w/zwhz/MiSeemAULFuTUU0/9Sdsrr7wya9asSVV1biISoQ0AAOirV8+fv98zPu7v/l7K4sWL8+1vf/vn1l9yySW55JJLfm798uXLf2YSkk984hPjK3ICCW0AAEBfbX700UGXcEDzTBsAAECHCW0AAAAdJrQBAAATrrU26BI6aSy/F6ENAACYUHPmzMljjz0muL1Aay2PPfZY5syZM6rtTEQCAABMqIULF2bLli3Ztm3boEvpnDlz5mThwoWj2kZoAwAAJtSsWbOyZMmSQZcxZRgeCQAA0GF62gAGoKpG1X7W7Fl55uln+lQNANBlQhvAIAyPrvmu4V19KQMA6D7DIwEAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADhPaAAAAOkxoAwAA6DChDQAAoMPGFdqq6n+rqnur6p6q+uuqmlNVS6rqm1X1QFX9TVXNnqhiAQAAppsxh7aqOjrJ/5pkqLV2YpIZSd6T5ONJ/qy19u+S/DDJ+RNRKAAAwHQ03uGRM5O8rKpmJvmFJI8keVOS63qfr0vyrnEeAwAAYNoac2hrrT2c5NIkD2UkrO1IsiHJ9tba7l6zLUmOHm+RAAAA09V4hkcenuTsJEuSvCrJIUneNort11bV+qpav23btrGWAfulqkb1AgCArpg5jm3fkuS/tda2JUlVfT7JyiTzqmpmr7dtYZKH97Zxa+3yJJcnydDQUBtHHfDShvvcHgAA+mQ8z7Q9lOR1VfULNdI18eYk9yX5cpJ399qsTnLD+EoEpiq9nwAAL23MPW2ttW9W1XVJ7kyyO8ldGek5+3+SfLaqLumt+9REFApMQcN9agsAMIWMZ3hkWmsfSfKRF6z+bpIV49kvAAAAI8Y75T8AAAB9NK6eNuDA4JkwAIADl9AG08Fwn9oCANB3hkcCAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdNnPQBQD0U1UNugQAgHER2oCpbbjP7QEA+szwSAAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw8YV2qpqXlVdV1XfrqpNVfX6qnpFVd1SVff3fh4+UcUCAABMN+PtafvzJP+1tXZckl9LsinJB5N8qbV2TJIv9ZYBAAAYgzGHtqo6LMlpST6VJK21Z1pr25OcnWRdr9m6JO8ab5EAAADT1Xh62pYk2Zbkyqq6q6r+S1UdkmR+a+2RXptHk8wfb5EAAADT1XhC28wky5L8ZWvtlCRP5gVDIVtrLUnb28ZVtbaq1lfV+m3bto2jDAAAgKlrPKFtS5ItrbVv9pavy0iI21pVRyVJ7+f397Zxa+3y1tpQa23oyCOPHEcZAAAAU9eYQ1tr7dEk36uqY3ur3pzkviQ3JlndW7c6yQ3jqhAAAGAamznO7f+XJNdU1ewk301yXkaC4Oeq6vwkDyY5Z5zHAAAAmLbGFdpaaxuTDO3lozePZ78AAACMGO/3tAEAANBHQhsAAECHCW0M1LxD56aqRvWa4aoFAGAaGe9EJDAuOx5/Mu2a0W1Tq/pTCwAAdJE+CwAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMN/TxkD9wsEHpVY9N6ptDp6VPN2negAAoGuENgbqqaefS4ZHt83To2wPAAAHMsMjAQAAOkxoAwAA6DChjX2ad+jcVNV+v+YdOnfQJQMAwJTjmTb2acfjT6Zds//ta9WT/SsGAACmKT1tAAAAHaanDeAAUVX73XbW7Fl55uln+lgNADBZhDaAA8Xw/jfdNbyrb2UAAJPL8EgAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADhPaAAAAOkxoAwAA6DChDQAAoMOENgAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADht3aKuqGVV1V1Xd3FteUlXfrKoHqupvqmr2+MsEAACYniaip+0DSTbtsfzxJH/WWvt3SX6Y5PwJOAYAAMC0NK7QVlULk/x2kv/SW64kb0pyXa/JuiTvGs8xAAAAprPx9rT95yR/lOS53vIrk2xvre3uLW9JcvQ4jwEAADBtjTm0VdU7kny/tbZhjNuvrar1VbV+27ZtYy0DAABgShtPT9vKJGdV1eYkn83IsMg/TzKvqmb22ixM8vDeNm6tXd5aG2qtDR155JHjKAMAAGDqGnNoa619qLW2sLW2OMl7kvxTa21Vki8neXev2eokN4y7SgAAgGmqH9/T9h+T/O9V9UBGnnH7VB+OAQAAMC3MfOkmL621dmuSW3vvv5tkxUTsFwAAYLrrR08bAAAAE0RoAwAA6DChDQAAoMOENgAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADhPaAAAAOkxoG7B5h85NVY3qNe/QuYMuGwAAmCQzB13AdLfj8SfTrhndNrXqyf4UAwAAdI6eNgDGbDJGCxiRAMB0p6cNgDGbjNECRiQAMN3paQMAAOgwoQ0AAKDDhDYAAIAOE9oASDK2CT8AgP4zEQkAScY64Ud/agEAfkpoA5ii9IQBwNQgtAFMVcN9bg8ATArPtAEAAHSY0AYAANBhQhsAAECHCW0HoBkHZdTTcs87dO6gywYAAMbARCQHoGefyxim5X6yP8UAAAB9JbRNE8/3zgEAAAcWoW2aGFvvXH9qAQAA9p9n2gAAADpMTxsTZixDMGcclDzbp3qAbjJcGwBGR2hjwhiCCeyP0d4r3CcAmO4MjwQAAOgwoQ0AAKDDhDYAAIAOG3Noq6pFVfXlqrqvqu6tqg/01r+iqm6pqvt7Pw+fuHIBAACml/H0tO1O8h9aa8cneV2S91fV8Uk+mORLrbVjknyptwwAAMAYjDm0tdYeaa3d2Xv/eJJNSY5OcnaSdb1m65K8a7xFAgAATFcT8kxbVS1OckqSbyaZ31p7pPfRo0nmT8QxAAAApqNxh7aqmpvk+iT/vrX2oz0/a621JG0f262tqvVVtX7btm3jLaMz5h06N1W13y8AAIAXM64v166qWRkJbNe01j7fW721qo5qrT1SVUcl+f7etm2tXZ7k8iQZGhraa7A7EO14/ElfGgsAAEyY8cweWUk+lWRTa+1P9/joxiSre+9XJ7lh7OUBAABMb+PpaVuZ5Nwk/1pVG3vrPpzkT5J8rqrOT/JgknPGVyIAAMD0NebQ1lq7Lcm+Hsp681j3CwAAwE9NyOyRAAAA9IfQBgAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAAdMLiBQtSVfv9WrxgwaBLnhQzB10AAABAkjy4dWvaKNrX1q19q6VL9LQBANA3o+05mU69J3szWb8vPVoHFj1tAAD0zWh7TpLp03uyN5P1+9KjdWDR0wYAANBhQhsAAECHCW0AAAAdJrQBAHDAm84TaxycjHrykskwlklVRmss534g/tmbiAQAgAPedJ5Y4+lk9JOX9KOQFxjTpCqjbD+mcz8A/+z1tL2IeYfO7eT/WgAAMPnG0nN0yIwZ/i3JuOlpexE7Hn8y7ZrRbVOr+lMLAACDNaaeo+eeG10P4Cj3z/QgtAEAAC/q+WfHGIy+DI+sqrdV1Xeq6oGq+mA/jgEAAEyO558dG82LiTPhPW1VNSPJJ5O8NcmWJHdU1Y2ttfsm+lgAAEw9enXgZ/Wjp21Fkgdaa99trT2T5LNJzu7DcQAAmIL06sDP6kdoOzrJ9/ZY3tJbBwAAwChVaxP7fxNV9e4kb2ut/c+95XOTvLa1duEL2q1Nsra3eGyS70xoId13RJIfDLoI2AvXJl3l2qSrXJt0lWvzwPLq1tqRe/ugH7NHPpxk0R7LC3vrfkZr7fIkl/fh+AeEqlrfWhsadB3wQq5Nusq1SVe5Nukq1+bU0Y/hkXckOaaqllTV7CTvSXJjH44DAAAw5U14T1trbXdVXZjkH5PMSHJFa+3eiT4OAADAdNCXL9durf19kr/vx76nkGk7NJTOc23SVa5Nusq1SVe5NqeICZ+IBAAAgInTj2faAAAAmCBC2wBU1duq6jtV9UBVfXDQ9TC9VdXmqvrXqtpYVet7615RVbdU1f29n4cPuk6mvqq6oqq+X1X37LFur9dijfg/e/fRb1XVssFVzlS3j2tzuKoe7t07N1bV2/f47EO9a/M7VXXmYKpmqquqRVX15aq6r6ruraoP9Na7b05BQtskq6oZST6Z5LeSHJ/kvVV1/GCrgvxma+3kPaYF/mCSL7XWjknypd4y9NtVSd72gnX7uhZ/K8kxvdfaJH85STUyPV2Vn782k+TPevfOk3vP86f3d/p7kpzQ2+Yven/3w0TbneQ/tNaOT/K6JO/vXX/um1OQ0Db5ViR5oLX23dbaM0k+m+TsAdcEL3R2knW99+uSvGuAtTBNtNa+muS/v2D1vq7Fs5N8uo34RpJ5VXXU5FTKdLOPa3Nfzk7y2dba0621/5bkgYz83Q8TqrX2SGvtzt77x5NsSnJ03DenJKFt8h2d5Ht7LG/prYNBaUm+UFUbqmptb9381tojvfePJpk/mNJgn9eieyldcGFvmNkVewwjd20y6apqcZJTknwz7ptTktAGvKG1tiwjwybeX1Wn7flhG5li1jSzDJxrkY75yyS/kuTkJI8k+U+DLYfpqqrmJrk+yb9vrf1oz8/cN6cOoW3yPZxk0R7LC3vrYCBaaw/3fn4/yd9lZBjP1ueHTPR+fn9wFTLN7etadC9loFprW1trz7bWnkvyf+enQyBdm0yaqpqVkcB2TWvt873V7ptTkNA2+e5IckxVLamq2Rl5WPnGAdfENFVVh1TVy59/n+SMJPdk5Jpc3Wu2OskNg6kQ9nkt3pjkfb3Z0F6XZMcew4Gg717wLNDvZOTemYxcm++pqoOraklGJn24fbLrY+qrqkryqSSbWmt/usdH7ptT0MxBFzDdtNZ2V9WFSf4xyYwkV7TW7h1wWUxf85P83ch9PzOTXNta+69VdUeSz1XV+UkeTHLOAGtkmqiqv05yepIjqmpLko8k+ZPs/Vr8+yRvz8gkD08lOW/SC2ba2Me1eXpVnZyRoWebk/x+krTW7q2qzyW5LyOz+72/tfbsIOpmyluZ5Nwk/1pVG3vrPhz3zSmpRoa6AgAA0EWGRwIAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHfb/A57ATcGlppf3AAAAAElFTkSuQmCC\n" }, "metadata": { "needs_background": "light" } } ] }, { "cell_type": "markdown", "source": [ "Another way to calculate z-score is with scipy.stats funtion:\n", "import scipy.stats as stats\n", "df['zscore'] = stats.zscore(df['ages'])\n", "\n", "Now, we can print to ten ouliers detected with a z-score by both ends." ], "metadata": { "id": "SkNYd0H_i6RP" } }, { "cell_type": "code", "source": [ "sorted_df = df.sort_values('zscore')\n", "print(\"Top 10 left: \")\n", "print(sorted_df.head(10))\n", "\n", "print(\"\\nTop 10 right: \")\n", "print(sorted_df.tail(10))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "KdnDPj3BhRgS", "outputId": "d50041be-c488-410c-fb26-7f2241d6a8d5" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Top 10 left: \n", " ids birthdates birthdates_num ages zscore\n", "1902 C0000001903 2029-10-15 20291015 -7 -1.808386\n", "1901 C0000001902 2029-01-21 20290121 -7 -1.808386\n", "1900 C0000001901 2028-03-10 20280310 -6 -1.778690\n", "1540 C0000001541 2024-09-23 20240923 -2 -1.659905\n", "1284 C0000001285 2024-10-27 20241027 -2 -1.659905\n", "1028 C0000001029 2023-04-13 20230413 -1 -1.630208\n", "1564 C0000001565 2023-03-03 20230303 -1 -1.630208\n", "1675 C0000001676 2023-03-22 20230322 -1 -1.630208\n", "1703 C0000001704 2023-05-27 20230527 -1 -1.630208\n", "1667 C0000001668 2023-11-21 20231121 -1 -1.630208\n", "\n", "Top 10 right: \n", " ids birthdates birthdates_num ages zscore\n", "1894 C0000001895 1808-10-03 18081003 214 4.754503\n", "1859 C0000001860 1808-04-01 18080401 214 4.754503\n", "1856 C0000001857 1807-01-19 18070119 215 4.784199\n", "1857 C0000001858 1806-01-10 18060110 216 4.813895\n", "1893 C0000001894 1806-12-10 18061210 216 4.813895\n", "1891 C0000001892 1805-01-28 18050128 217 4.843592\n", "1851 C0000001852 1803-04-27 18030427 219 4.902984\n", "1873 C0000001874 1802-11-22 18021122 220 4.932681\n", "1860 C0000001861 1801-12-29 18011229 221 4.962377\n", "1898 C0000001899 1801-09-03 18010903 221 4.962377\n" ] } ] }, { "cell_type": "markdown", "source": [ "**Isolation Forest**\n", "\n", "IsolationForest is an unsupervised learning algorithm that identifies possible anomalies by isolating outliers in a dataset. Its calculation is inspired by the Random Forest classification and regression algorithm.\n", "\n", "Firstly, we are going to define and fit the model. We have to instance IsolationForest with the next three parameters:\n", "- n_estimators: number of base estimators or trees in the ensemble. It's optional and the default value is 100.\n", "- max_samples: number of samples used to train each base estimator. The default value of max_samples is 'auto', max_samples=min(256, n_samples).\n", "- contamination: expected proportion of outliers in the dataset. The default value is 'auto', determined as in the original paper of Isolation Forest.\n" ], "metadata": { "id": "Hqkstc-movRW" } }, { "cell_type": "code", "source": [ "from sklearn.ensemble import IsolationForest\n", "import numpy as np\n", "\n", "model = IsolationForest(n_estimators = 1000, max_samples = 'auto', contamination=float(0.1))\n", "print(model.get_params())\n", "\n", "model.fit(df[['ages']])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "OzaRC64uXT-0", "outputId": "a6c4980f-55fa-47ac-c636-4913a4adc89d" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'bootstrap': False, 'contamination': 0.1, 'max_features': 1.0, 'max_samples': 'auto', 'n_estimators': 1000, 'n_jobs': None, 'random_state': None, 'verbose': 0, 'warm_start': False}\n" ] }, { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.8/dist-packages/sklearn/base.py:450: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names\n", " warnings.warn(\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "IsolationForest(contamination=0.1, n_estimators=1000)" ] }, "metadata": {}, "execution_count": 32 } ] }, { "cell_type": "markdown", "source": [ "After the model is defined and fitted, it will show the IsolationForest instance result as shown in the output.\n", "\n", "Now, we will create two new columns with decision function and predict information:\n", "\n", "- decision_function(). Average anomaly score of X of the base classifiers.\n", "- predict(). Predict if a particular sample is an outlier or not.\n", "\n", "To show results in a plot we have to split the ages with anomaly_score criteria." ], "metadata": { "id": "QCtBeEhxwWoj" } }, { "cell_type": "code", "source": [ "df['scores'] = model.decision_function(df[['ages']])\n", "df['anomaly_score'] = model.predict(df[['ages']])\n", "\n", "ok = df[df['anomaly_score']==1]\n", "ko = df[df['anomaly_score']==-1]\n", "\n", "f = plt.figure()\n", "f.set_figwidth(15)\n", "f.set_figheight(5)\n", "\n", "plt.hist([ok['ages'], ko['ages']], color=['Green', 'Red'], label=['oks', 'kos'], edgecolor='black', bins=60, histtype='barstacked')\n", "\n", "plt.title(\"Ages\")\n", "plt.legend()\n", "plt.show()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 425 }, "id": "reWg5tpWwcoX", "outputId": "e1f44a58-4817-4aa2-e881-6606c99052dd" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py:3208: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n", " return asarray(a).size\n", "/usr/local/lib/python3.8/dist-packages/matplotlib/cbook/__init__.py:1376: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n", " X = np.atleast_1d(X.T if isinstance(X, np.ndarray) else np.asarray(X))\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "
" ], "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAE/CAYAAADVKysfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAab0lEQVR4nO3dfZBlZX0n8O9PZnB0mjeRmjaM2uOGTZxFFGsWsQwhxiWrbCoD4hIi0dHCYrdKxexmsyH+oZOqWAVVG7OxktViI6++hVIQBCuJhRqYP2QZxEKEZcOSITQLzYTwMjalCPPsH/eO6Uzm7fbt7nu67+dTderec85z7vndmTO359vPc55brbUAAADQTS8adQEAAADsn9AGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQCsSFX17ap6sqpePOpaAGAYQhsAK05VTSU5LUlL8msjLQYAhiS0AbASvTfJd5JcmWTLno1VdWxVfa2qnqmqO6rqD6pq25z9P19V36iqf6iq+6vq3Dn7zqyqe6tqV1U9UlX/ZSnfEADja9WoCwCARfDeJJ9McnuS71TVutbaTJI/TTKbZDLJVJK/TPJQklTV2iTfSPKxJO9I8rok36iqe1pr9yb5bJJzW2u3VdUxSTYs7VsCYFzpaQNgRamqX0jy6iTXttbuTPJ/k7y7qg5Lck6Sj7fWnu0HsavmHPqrSXa01q5orT3fWrsryVeS/Pv+/p8k2VhVR7bWnmytfXfJ3hQAY01oA2Cl2ZLkr1prf99f/0J/23HpjTB5eE7buc9fneRNVfXUniXJ+en1yiW9wHdmkoeq6q+r6s2L+SYAYA/DIwFYMarqJUnOTXJYVT3W3/ziJEcnWZfk+STrk/yf/r5Xzjn84SR/3Vo7Y1+v3Vq7I8nmqlqd5ENJrt3reABYFHraAFhJzkryQpKNSd7QX16b5Lb07nO7LsnWqnppVf18f9seNyX5l1X1nqpa3V/+dVW9tqoOr6rzq+qo1tpPkjyTZPdSvjEAxpfQBsBKsiXJFa21v2utPbZnSfIn6Q11/FCSo5I8luSaJF9M8uMkaa3tSvIrSc5L8v/6bS5Nr6cuSd6TZEdVPZPkP/ZfDwAWXbXWRl0DAIxEVV2aZLK1tuWgjQFgRPS0ATA2+t/DdlL1nJLkgiTXj7ouADgQE5EAME6OSG9I5M8kmUnyh0luGGlFAHAQhkcCAAB0mOGRAAAAHSa0AQAAdFgn7ml7+ctf3qampkZdBnTeXXfdld27B/tqqBe96EU5+eSTF6kiAAAWwp133vn3rbXj9rWvE/e0bdq0qW3fvn3UZUDnVVWydcCDtiZd+HcOAMD+VdWdrbVN+9pneCQAAECHCW0AAAAdJrQBAAB0WCcmIoGumThyIrO7Zgc6Zu0Ra/PDZ364SBUBAIynn/zkJ5mens6PfvSjUZeyINasWZP169dn9erVh3yM0Ab7MLtrduAJP2a3DhbyAAA4uOnp6RxxxBGZmprqTcq2jLXW8sQTT2R6ejobNmw45OMMjwQAADrrRz/6UY499thlH9iS3kzgxx577MC9hkIbAADQaSshsO0xn/citAEAAAxox44dOfHEE5fkXEIbAACwbEyun0xVLdgyuX5y1G/poExEAgAALBszj8wMPGHcAV9v68whtfvkJz+Zyy+/PEnygQ98IGedddZP9z344IM555xzctlll+WlL31p3v/+9+e5557L7t2785WvfCUnnHDCUDUKbQAAAAdw55135oorrsjtt9+e1lre9KY35fTTT0+S3H///TnvvPNy5ZVX5vWvf30+/OEP5yMf+UjOP//8PPfcc3nhhReGPr/QBgAAcADbtm3L2WefnbVr1yZJ3vnOd+a2227Lzp07s3nz5lx33XXZuHFjkuTNb35zPvGJT2R6ejrvfOc7h+5lS9zTBgAAMC9HHXVUXvWqV2Xbtm0/3fbud787N954Y17ykpfkzDPPzDe/+c2hzyO0AQAAHMBpp52Wr371q3n22WczOzub66+/PqeddloOP/zwXH/99bn66qvzhS98IUnv/rbXvOY1ueiii7J58+bcfffdQ5/f8EgAAIADeOMb35j3ve99OeWUU5L0JiI55phjkiRr167NTTfdlDPOOCMTExO59957c80112T16tWZnJzMRz/60aHPX621oV9kWJs2bWrbt28fdRnwU1U1+KxEW5PF/vfU1boAABbLfffdl9e+9rU/XZ9cP9mbQXKBrDt+XR6bfmzBXu9Q7P2ekqSq7mytbdpXez1tAADAsrHUAasL3NMGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAwAHs2LEjJ5544sjOL7QBAADLxtTkZKpqwZapyclRv6WDEtoAAIBl46GZmbRkwZaHZgb7ou4HH3wwJ598cu64446ceuqpOemkk3L22WfnySefTJJ86lOfysaNG3PSSSflvPPOG/r9JkIbAADAIbn//vtzzjnn5Morr8wFF1yQSy+9NHfffXde97rX5fd///eTJJdccknuuuuu3H333fnMZz6zIOcV2mChVAbqip84cmLUFQMAcIh27tyZzZs35/Of/3ympqby1FNP5fTTT0+SbNmyJbfeemuS5KSTTsr555+fz33uc1m1atWCnHthXgXo9a9vPfTms1tnF6sSAAAW2FFHHZVXvepV2bZtW3791399v+1uvvnm3Hrrrfna176WT3ziE/n+978/dHjT0wYAAHAQhx9+eK6//vpcffXVufnmm3PMMcfktttuS5Jcc801Of3007N79+48/PDDeetb35pLL700Tz/9dH74wx8OfW49bQAAAIdg7dq1uemmm3LGGWfknHPOye/8zu/k2WefzWte85pcccUVeeGFF/Kbv/mbefrpp9Nay0UXXZSjjz566PMKbSw7E0dOZHbXoQ8tXHvE2vzwmeF/wwEAwOi9et261IAzPh7s9Q5mamoq99xzT5Lk6KOPzh133JEk+djHPvbP2m7btm3BattDaGPZmd01694xAIAxteOxx0ZdwpJzTxsAAECHCW0AAAAdJrQBAACd1lobdQkLZj7vRWgDloWJIycG+vJyX2AOACvDmjVr8sQTT6yI4NZayxNPPJE1a9YMdNxBJyKpqsuT/GqSx1trJ/a3vSzJnyeZSrIjybmttSerqpL8cZIzkzyb5H2tte8OVBHAPgw6AU1iEhoAWAnWr1+f6enp7Ny5c9SlLIg1a9Zk/fr1Ax1zKLNHXpnkT5JcPWfbxUluaa1dUlUX99d/N8k7kpzQX96U5NP9R2BvlfR+zwEAwP6sXr06GzZsGHUZI3XQ0NZau7WqpvbavDnJL/WfX5Xk2+mFts1Jrm69vsvvVNXRVfWK1tqjC1UwrBgtA/ccDdweAIBlb773tK2bE8QeS7LnG+mOT/LwnHbT/W3/TFVdWFXbq2r7SunqhJXAvWMAAN0y9Jdrt9ZaVQ18V2Br7bIklyXJpk2blv9dhbBCuHcMAKBb5tvTNlNVr0iS/uPj/e2PJHnlnHbr+9sAAACYh/mGthuTbOk/35Lkhjnb31s9pyZ52v1swHIy6PBQQ0MBgMV2KFP+fzG9SUdeXlXTST6e5JIk11bVBUkeSnJuv/nX05vu/4H0pvx//yLUDLBoBh0eamgoALDYDmX2yN/Yz6637aNtS/LBYYsCAACgZ77DIwEAAFgCQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgyvkqoaaJk4cmLUVQMALAurRl0AsAK0JFsHO2R26+xiVAIAsOLoaQMAAOgwoQ1gGIaGAgCLzPBIgGEYGgoALDI9bQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCG8BSq6SqBlomjpwYddUAwIisGnUBAGOnJdk62CGzW2cXoxIAYBnQ0wYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQAAAB02VGirqv9UVT+oqnuq6otVtaaqNlTV7VX1QFX9eVUdvlDFAgAAjJt5h7aqOj7JRUk2tdZOTHJYkvOSXJrkj1prP5vkySQXLEShAAAA42jY4ZGrkrykqlYleWmSR5P8cpIv9/dfleSsIc8BAAAwtuYd2lprjyT5b0n+Lr2w9nSSO5M81Vp7vt9sOsnxwxYJAAAwroYZHnlMks1JNiT5mSRrk7x9gOMvrKrtVbV9586d8y0DDq6SqhpoAQCArlg1xLH/JsnfttZ2JklVXZfkLUmOrqpV/d629Uke2dfBrbXLklyWJJs2bWpD1AEH1pJsHfCYQdsDAMAiGeaetr9LcmpVvbR6XRNvS3Jvkm8leVe/zZYkNwxXIrAiDdgDCgAwrubd09Zau72qvpzku0meT3JXej1nNyf5UlX9QX/bZxeiUGCFGbQHdJC2AAAryDDDI9Na+3iSj++1+cEkpwzzugAAAPQMO+U/AAAAi2ionjZgGejfOwYAwPIktMFK594xAIBlzfBIAACADhPaAAAAOkxoAwAA6DChDQAAoMOENgAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADhPaAAAAOkxoAwAA6DChDQAAoMOENgAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw1aNugCARVNJVY26CgCAoQhtwMrVkmwd8JhB2wMALDLDIwEAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADhsqtFXV0VX15ar631V1X1W9uapeVlXfqKq/6T8es1DFAgAAjJthe9r+OMlftNZ+Psnrk9yX5OIkt7TWTkhyS38dAACAeZh3aKuqo5L8YpLPJklr7bnW2lNJNie5qt/sqiRnDVskAADAuBqmp21Dkp1Jrqiqu6rqz6pqbZJ1rbVH+20eS7Ju2CIBAADG1TChbVWSNyb5dGvt5CSz2WsoZGutJWn7OriqLqyq7VW1fefOnUOUAQAAsHINE9qmk0y31m7vr385vRA3U1WvSJL+4+P7Ori1dllrbVNrbdNxxx03RBkAAAAr17xDW2vtsSQPV9XP9Te9Lcm9SW5MsqW/bUuSG4aqEAAAYIytGvL4Dyf5fFUdnuTBJO9PLwheW1UXJHkoyblDngMAAGBsDRXaWmvfS7JpH7veNszrAgAA0DPs97QBAACwiIQ2AACADhv2njYYysSRE5ndNTvqMgAAoLOENkZqdtdssnXAgwZtDwAAy5jhkQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCG/s1ceREquqQl4kjJ0ZdMgAArDirRl0A3TW7azbZOkD7rbOLVgsAAIwrPW0AAAAdJrQBLAcVw5UBYEwZHgmwHLQYrgwAY0pPGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHDR3aquqwqrqrqm7qr2+oqtur6oGq+vOqOnz4MgEAAMbTQvS0fSTJfXPWL03yR621n03yZJILFuAcAAAAY2mo0FZV65P8uyR/1l+vJL+c5Mv9JlclOWuYcwAAAIyzYXva/nuS/5pkd3/92CRPtdae769PJzl+yHMAAACMrXmHtqr61SSPt9bunOfxF1bV9qravnPnzvmWAQAAsKIN09P2liS/VlU7knwpvWGRf5zk6Kpa1W+zPskj+zq4tXZZa21Ta23TcccdN0QZAAAAK9e8Q1tr7fdaa+tba1NJzkvyzdba+Um+leRd/WZbktwwdJUAAABjajG+p+13k/znqnogvXvcPrsI5wAAABgLqw7e5OBaa99O8u3+8weTnLIQrwsAADDuFqOnDQAAgAUitAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB0mNAGAADQYUIbAABAhwltAAAAHSa0AQAAdJjQBgAA0GFCGwAAQIcJbQAAAB0mtI3Y1ORkqmqgZWpyctRlAwAAS2TVqAsYdw/NzKQNeEzNzCxKLQAAQPfoaQNg3iaOnBh4tMDEkRMDncOIBADGnZ42AOZtdtdssnXAY7bODtTeiAQAxp2eNgAAgA4T2gAAADpMaAMAAOgw97QBkKQ3qcjsrsHuNwMAFp/QBkCS+U0qMnB7AGBgQhvASlRJVY26CgBgAQhtACtRi14zAFghTEQCAADQYUIbAABAhwltAAAAHSa0LUMvTm+CgUGWqcnJUZcNAADMg4lIlqEfpzfHwCBqZmYxSgEAABaZ0DZGTP8NAADLj9A2TrYucnsAAGDBuacNAACgw/S0sXDKEEzg4PZMpgQAHBqhjYXTYggmcFCDTqYk3gEw7gyPBAAA6DChDQAAoMOENgAAgA6bd2irqldW1beq6t6q+kFVfaS//WVV9Y2q+pv+4zELVy4AAMB4Gaan7fkkv91a25jk1CQfrKqNSS5Ocktr7YQkt/TXAQAAmId5h7bW2qOtte/2n+9Kcl+S45NsTnJVv9lVSc4atkgAAIBxtSD3tFXVVJKTk9yeZF1r7dH+rseSrFuIcwAAAIyjoUNbVU0k+UqS32qtPTN3X2utZT9fx1NVF1bV9qravnPnzmHL6IypyclU1SEvAAAABzJUaKuq1ekFts+31q7rb56pqlf0978iyeP7Ora1dllrbVNrbdNxxx03TBmd8tDMTPYk1UNZAAAADmSY2SMryWeT3Nda++ScXTcm2dJ/viXJDfMvDwAAYLytGuLYtyR5T5LvV9X3+ts+muSSJNdW1QVJHkpy7nAlAgAAjK95h7bW2rYk+7sp623zfV0AAAD+0YLMHgkAAMDiENoAAAA6TGgDAADoMKENAACgw4Q2AACADhPaAAAAOkxoAwAA6DChDQAAoMOENgAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAAA6TGgDAADoMKENAACgw4Q2AACADhPaAAAAOkxoAwAA6DChDQAAoMOENgAAgA4T2gAAADpMaAMAAOgwoQ0AAKDDhDYAAIAOE9oAAIBOmJqcTFUd8jI1OTnqkpfEqlEXAAAAkCQPzcykDdC+ZmYWrZYu0dMGAMCiGbTnZJx6T/Zlqf689GgtL3raAABYNIP2nCTj03uyL0v156VHa3nR0wYAANBhQhsAAECHCW0AAAAdJrQBALDsjfPEGi9OBp68ZCnMZ1KVQc3nvS/Hv3sTkQAAsOyN88QaP04Gn7xkMQrZy7wmVRmw/bze+zL8u9fTdgBL8dsBAACWh/n833DtYYf5vyRD09N2AEvx2wEAAJaHef3fcPfuwXoAB3x9xoPQBgAAHNCee8cYjUUZHllVb6+q+6vqgaq6eDHOAQAALI09944NsrBwFrynraoOS/KnSc5IMp3kjqq6sbV270KfCwCAlUevDvxTi9HTdkqSB1prD7bWnkvypSSbF+E8AACsQHp14J9ajNB2fJKH56xP97cBAAAwoGptYX83UVXvSvL21toH+uvvSfKm1tqH9mp3YZIL+6s/l+T+BS2k+16e5O9HXQTsg2uTrnJt0lWuTbrKtbm8vLq1dty+dizG7JGPJHnlnPX1/W3/RGvtsiSXLcL5l4Wq2t5a2zTqOmBvrk26yrVJV7k26SrX5sqxGMMj70hyQlVtqKrDk5yX5MZFOA8AAMCKt+A9ba2156vqQ0n+MslhSS5vrf1goc8DAAAwDhbly7Vba19P8vXFeO0VZGyHhtJ5rk26yrVJV7k26SrX5gqx4BORAAAAsHAW4542AAAAFojQNgJV9faqur+qHqiqi0ddD+OtqnZU1fer6ntVtb2/7WVV9Y2q+pv+4zGjrpOVr6our6rHq+qeOdv2eS1Wz6f6n6N3V9UbR1c5K91+rs2tVfVI/7Pze1V15px9v9e/Nu+vqn87mqpZ6arqlVX1raq6t6p+UFUf6W/3ubkCCW1LrKoOS/KnSd6RZGOS36iqjaOtCvLW1tob5kwLfHGSW1prJyS5pb8Oi+3KJG/fa9v+rsV3JDmhv1yY5NNLVCPj6cr882szSf6o/9n5hv79/On/TD8vyb/qH/M/+j/7YaE9n+S3W2sbk5ya5IP968/n5goktC29U5I80Fp7sLX2XJIvJdk84ppgb5uTXNV/flWSs0ZYC2OitXZrkn/Ya/P+rsXNSa5uPd9JcnRVvWJpKmXc7Ofa3J/NSb7UWvtxa+1vkzyQ3s9+WFCttUdba9/tP9+V5L4kx8fn5ooktC2945M8PGd9ur8NRqUl+auqurOqLuxvW9dae7T//LEk60ZTGuz3WvRZShd8qD/M7PI5w8hdmyy5qppKcnKS2+Nzc0US2oBfaK29Mb1hEx+sql+cu7P1ppg1zSwj51qkYz6d5F8keUOSR5P84WjLYVxV1USSryT5rdbaM3P3+dxcOYS2pfdIklfOWV/f3wYj0Vp7pP/4eJLr0xvGM7NnyET/8fHRVciY29+16LOUkWqtzbTWXmit7U7yP/OPQyBdmyyZqlqdXmD7fGvtuv5mn5srkNC29O5IckJVbaiqw9O7WfnGEdfEmKqqtVV1xJ7nSX4lyT3pXZNb+s22JLlhNBXCfq/FG5O8tz8b2qlJnp4zHAgW3V73Ap2d3mdn0rs2z6uqF1fVhvQmffhfS10fK19VVZLPJrmvtfbJObt8bq5Aq0ZdwLhprT1fVR9K8pdJDktyeWvtByMui/G1Lsn1vc/9rEryhdbaX1TVHUmuraoLkjyU5NwR1siYqKovJvmlJC+vqukkH09ySfZ9LX49yZnpTfLwbJL3L3nBjI39XJu/VFVvSG/o2Y4k/yFJWms/qKprk9yb3ux+H2ytvTCKulnx3pLkPUm+X1Xf62/7aHxurkjVG+oKAABAFxkeCQAA0GFCGwAAQIcJbQAAAB0mtAEAAHSY0AYAANBhQhsAAECHCW0AAAAdJrQBAAB02P8HGLpHiQn27d8AAAAASUVORK5CYII=\n" }, "metadata": { "needs_background": "light" } } ] }, { "cell_type": "markdown", "source": [ "Now, we can print top ten ouliers detected with a Isolation Forest algorithm, agreed with the maximum obtained with z-score." ], "metadata": { "id": "Il4JJfpb0Y8w" } }, { "cell_type": "code", "source": [ "sorted_df = df.sort_values('scores')\n", "print(\"Top 10 left: \")\n", "print(sorted_df.head(10))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "eX1bBGOD0mi2", "outputId": "30f07fd0-6ac9-412d-f36a-6b2ee1090f1f" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Top 10 left: \n", " ids birthdates birthdates_num ages zscore scores \\\n", "1898 C0000001899 1801-09-03 18010903 221 4.962377 -0.206648 \n", "1860 C0000001861 1801-12-29 18011229 221 4.962377 -0.206648 \n", "1873 C0000001874 1802-11-22 18021122 220 4.932681 -0.204303 \n", "1851 C0000001852 1803-04-27 18030427 219 4.902984 -0.201246 \n", "1891 C0000001892 1805-01-28 18050128 217 4.843592 -0.195308 \n", "1857 C0000001858 1806-01-10 18060110 216 4.813895 -0.192998 \n", "1893 C0000001894 1806-12-10 18061210 216 4.813895 -0.192998 \n", "1856 C0000001857 1807-01-19 18070119 215 4.784199 -0.189079 \n", "1877 C0000001878 1849-05-16 18490516 173 3.536953 -0.186404 \n", "1865 C0000001866 1849-08-14 18490814 173 3.536953 -0.186404 \n", "\n", " anomaly_score \n", "1898 -1 \n", "1860 -1 \n", "1873 -1 \n", "1851 -1 \n", "1891 -1 \n", "1857 -1 \n", "1893 -1 \n", "1856 -1 \n", "1877 -1 \n", "1865 -1 \n" ] } ] }, { "cell_type": "markdown", "source": [ "**Conclusion**\n", "\n", "With z-score and Isolation Forest algorithm are two easy ways to identify possible anomalies in a dataset, through the use of scores for each of the data and through its visualization." ], "metadata": { "id": "98F0fR_Nv7lk" } } ] }