{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Getting Started" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/whylogs/blob/1.0.x/python/examples/basic/Getting_Started.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "whylogs provides a standard to log any kind of data.\n", "\n", "With whylogs, we will show how to log data, generating statistical summaries called *profiles*. These profiles can be used in a number of ways, like:\n", "\n", "* Data Visualization\n", "* Data Validation\n", "* Tracking changes in your datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Table of Content" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, we'll explore the basics of logging data with whylogs:\n", "- Installing whylogs\n", "- Profiling data\n", "- Interacting with the profile\n", "- Writing/Reading profiles to/from disk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installing whylogs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "whylogs is made available as a Python package. You can get the latest version from PyPI with `pip install whylogs`:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "!pip install -q whylogs --pre" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading a Pandas DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before showing how we can log data, we first need the data itself. Let's create a simple Pandas DataFrame:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "data = {\n", " \"animal\": [\"cat\", \"hawk\", \"snake\", \"cat\"],\n", " \"legs\": [4, 2, 0, 4],\n", " \"weight\": [4.3, 1.8, 1.3, 4.1],\n", "}\n", "\n", "df = pd.DataFrame(data)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Profiling with whylogs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To obtain a profile of your data, you can simply use whylogs' `log` call, and navigate through the result to a specific profile with `get_profile`:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import whylogs as why\n", "\n", "results = why.log(df)\n", "profile = results.profile()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analyzing Profiles" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once you're done logging the data, you can generate a `Profile View` and inspect it in a Pandas Dataframe format:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
counts/ncounts/nulltypes/integraltypes/fractionaltypes/booleantypes/stringtypes/objectcardinality/estcardinality/upper_1cardinality/lower_1...distribution/ndistribution/maxdistribution/mindistribution/q_10distribution/q_25distribution/mediandistribution/q_75distribution/q_90ints/maxints/min
column
animal80000806.06.000306.0...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
weight80080007.07.000357.0...8.030.11.31.34.14.314.330.1NaNNaN
legs80800003.03.000153.0...8.04.00.00.02.04.04.04.04.00.0
\n", "

3 rows × 24 columns

\n", "
" ], "text/plain": [ " counts/n counts/null types/integral types/fractional \\\n", "column \n", "animal 8 0 0 0 \n", "weight 8 0 0 8 \n", "legs 8 0 8 0 \n", "\n", " types/boolean types/string types/object cardinality/est \\\n", "column \n", "animal 0 8 0 6.0 \n", "weight 0 0 0 7.0 \n", "legs 0 0 0 3.0 \n", "\n", " cardinality/upper_1 cardinality/lower_1 ... distribution/n \\\n", "column ... \n", "animal 6.00030 6.0 ... NaN \n", "weight 7.00035 7.0 ... 8.0 \n", "legs 3.00015 3.0 ... 8.0 \n", "\n", " distribution/max distribution/min distribution/q_10 \\\n", "column \n", "animal NaN NaN NaN \n", "weight 30.1 1.3 1.3 \n", "legs 4.0 0.0 0.0 \n", "\n", " distribution/q_25 distribution/median distribution/q_75 \\\n", "column \n", "animal NaN NaN NaN \n", "weight 4.1 4.3 14.3 \n", "legs 2.0 4.0 4.0 \n", "\n", " distribution/q_90 ints/max ints/min \n", "column \n", "animal NaN NaN NaN \n", "weight 30.1 NaN NaN \n", "legs 4.0 4.0 0.0 \n", "\n", "[3 rows x 24 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "prof_view = profile.view()\n", "prof_df = prof_view.to_pandas()\n", "\n", "prof_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will provide you with valuable statistics on a column (feature) basis, such as:\n", "\n", "- Counters, such as number of samples and null values\n", "- Inferred types, such as integral, fractional and boolean\n", "- Estimated Cardinality\n", "- Frequent Items\n", "- Distribution Metrics: min,max, median, quantile values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Writing to Disk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also store your profile in disk for further inspection:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "why.write(profile,\"profile.bin\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will create a profile binary file in your local filesystem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading from Disk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can read the profile back into memory with:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "n_prof = why.read(\"profile.bin\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note: `write` expects a profile as parameter, while `read` returns a `Profile View`. That means that you can use the loaded profile for visualization purposes and merging, but not for further tracking and updates." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What's Next?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There's a lot you can do with the profiles you just created. Keep getting your hands dirty with the following examples!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Basic\n", " - [Visualizing Profiles](https://whylogs-v1-doc-dev.netlify.app/examples/basic/notebook_profile_visualizer) - Compare profiles to detect distribution shifts, visualize histograms and bar charts and explore your data\n", " - [Schema Configuration for Tracking Metrics](https://whylogs-v1-doc-dev.netlify.app/examples/basic/schema_configuration) - Configure tracking metrics according to data type or column features\n", " - More to Come!" ] } ], "metadata": { "interpreter": { "hash": "f76ec28949fecf16b926a3fc5a03c1aa6468ee82fa5da4ce6fd607df021af5b5" }, "kernelspec": { "display_name": "Python 3.8.13 ('v1.x')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }