{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*
\n",
">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Drift_Configuration)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Drift_Configuration) to leverage the power of whylogs and WhyLabs together!*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Drift Algorithm Configuration"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/whylabs/whylogs/blob/mainline/python/examples/advanced/Drift_Algorithm_Configuration.ipynb)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In whylogs, you can calculate drift scores and generate a summary drift report between two profiles, as shown in the [Notebook Profile Visualizer example](-).\n",
"\n",
"In this example, we will show you how to apply drift calculation with the default algorithm selection, and also to customize the drift calculations in two ways: by choosing the algorithm of your choosing and by changing the algorithm's internal parameters and thresholds for drift detection. We will also show you how to calculate drifts in a standalone manner, without the need to generate a visualization with the summary report."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"> Currently, whylogs supports the following drift algorithms: __Kolmogorov-Smirnov Test__, __ChiSquare Test__, and __Hellinger distance__ - Stay tuned for more algorithms to be added in the future!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing whylogs\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Note: you may need to restart the kernel to use updated packages.\n",
"%pip install whylogs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generating the Target and Reference Profiles"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we will generate two profiles, one as the target and one as the reference.\n",
"\n",
"We will use those profiles in order to calculate drift scores for each column in both profiles."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import whylogs as why\n",
"import pandas as pd\n",
"\n",
"data = {\n",
" \"animal\": [\"cat\", \"hawk\", \"snake\", \"cat\"],\n",
" \"legs\": [4, 2, 0, 4],\n",
" \"weight\": [4.3, 1.8, None, 4.1],\n",
"}\n",
"\n",
"df = pd.DataFrame(data)\n",
"\n",
"data2 = {\n",
" \"animal\": [\"cat\", \"hawk\", \"snake\", \"cat\"],\n",
" \"legs\": [13, 34, 99, 123],\n",
" \"weight\": [4.9, 13.3, None, 232.3],\n",
"}\n",
"\n",
"df2 = pd.DataFrame(data2)\n",
"\n",
"\n",
"target_view = why.log(df).profile().view()\n",
"ref_view = why.log(df2).profile().view()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculating Drift - Default Behavior\n",
"\n",
"You can calculate drift scores between your profiles in two ways. The first is to use `calculate_drift_scores`, which whill give you a dictionary of drift scores for each feature.\n",
"\n",
"The second is to view it integrated into the Notebook Profile Visualizer by calling `summary_drift_report`. This will give you a drift summary report in the format of an in-notebook visualization or a downloadable HTML file.\n",
"\n",
"Let's see both cases for the default behavior scenario - we won't specify any drift algorithms or parameters.\n",
"\n",
"To get a dictionary with the drift scores, you can use the `calculate_drift_scores` method:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'animal': {'algorithm': 'chi-square',\n",
" 'pvalue': 1.0,\n",
" 'statistic': 0.0,\n",
" 'thresholds': {'NO_DRIFT': (0.15, 1),\n",
" 'POSSIBLE_DRIFT': (0.05, 0.15),\n",
" 'DRIFT': (0, 0.05)},\n",
" 'drift_category': 'NO_DRIFT'},\n",
" 'legs': {'algorithm': 'ks',\n",
" 'pvalue': 0.0,\n",
" 'statistic': 1.0,\n",
" 'thresholds': {'NO_DRIFT': (0.15, 1),\n",
" 'POSSIBLE_DRIFT': (0.05, 0.15),\n",
" 'DRIFT': (0, 0.05)},\n",
" 'drift_category': 'DRIFT'},\n",
" 'weight': {'algorithm': 'ks',\n",
" 'pvalue': 0.0,\n",
" 'statistic': 1.0,\n",
" 'thresholds': {'NO_DRIFT': (0.15, 1),\n",
" 'POSSIBLE_DRIFT': (0.05, 0.15),\n",
" 'DRIFT': (0, 0.05)},\n",
" 'drift_category': 'DRIFT'}}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from whylogs.viz.drift.column_drift_algorithms import calculate_drift_scores\n",
"\n",
"scores = calculate_drift_scores(target_view=target_view, reference_view=ref_view, with_thresholds = True)\n",
"scores"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The `scores` object is a dictionary with the drift scores for each feature with additional metadata.\n",
"\n",
"## Algorithm Selection\n",
"\n",
"We can see that the KS test was applied for both `weight` and `animal`, and `chi-squared` was applied for `animal`. The default behavior for choosing which drift algorithm to use is the following: KS is calculated if distribution metrics exists for said column. If not, Chi2 is calculated if frequent items, cardinality and count metric exists. If not, then no drift value is associated to the column.\n",
"\n",
"## Thresholds\n",
"\n",
"We can also see the thresholds defined by default for each algorithm. Each drift category contains a tuple defining a range: if the measure falls within the range, then the drift category is assigned to the column. For each range, the lower bound is inclusive, while the upper bound is exclusive, except for the maximum upper bound, which is inclusive.\n",
"\n",
"The drift categorization will use either the `pvalue` or `statistic` value, depending on the algorithm. Both KS and Chi Square tests compare the pvalue against the thresholds, while the Hellinger distance compares the `statistic` value against the thresholds."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also visualize this information integrated with the NotebookProfileVisualizer `summary_drift_report`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from whylogs.viz import NotebookProfileVisualizer\n",
"\n",
"visualization = NotebookProfileVisualizer()\n",
"visualization.set_profiles(target_profile_view=target_view, reference_profile_view=ref_view)\n",
"\n",
"visualization.summary_drift_report()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Feel free to explore the dashboard! You can search by column names and filter by drift categorization. You can also drift on each column's drift category to check the thresholds that were used for the categorization."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculating Drift - Choosing a Drift Algorithm"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can overwrite the default algorithm selection logic by explicitly stating which algorithms you want to be run.\n",
"\n",
"Suppose now we want __hellinger__ to be used for the `weight` column, and __chi-squared__ to be used for the `legs` column. We can do this by passing a dictionary with the column names as keys and the drift algorithm as values."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'animal': {'algorithm': 'chi-square',\n",
" 'pvalue': 1.0,\n",
" 'statistic': 0.0,\n",
" 'thresholds': {'NO_DRIFT': (0.15, 1),\n",
" 'POSSIBLE_DRIFT': (0.05, 0.15),\n",
" 'DRIFT': (0, 0.05)},\n",
" 'drift_category': 'NO_DRIFT'},\n",
" 'legs': {'algorithm': 'chi-square',\n",
" 'pvalue': 0.0,\n",
" 'statistic': inf,\n",
" 'thresholds': {'NO_DRIFT': (0.15, 1),\n",
" 'POSSIBLE_DRIFT': (0.05, 0.15),\n",
" 'DRIFT': (0, 0.05)},\n",
" 'drift_category': 'DRIFT'},\n",
" 'weight': {'algorithm': 'hellinger',\n",
" 'pvalue': None,\n",
" 'statistic': 0.4283729905961321,\n",
" 'thresholds': {'NO_DRIFT': (0, 0.15),\n",
" 'POSSIBLE_DRIFT': (0.15, 0.4),\n",
" 'DRIFT': (0.4, 1)},\n",
" 'drift_category': 'DRIFT'}}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from whylogs.viz.drift.column_drift_algorithms import Hellinger, ChiSquare\n",
"\n",
"drift_map = {\"weight\": Hellinger(),\"legs\": ChiSquare()}\n",
"\n",
"scores = calculate_drift_scores(target_view=target_view, reference_view=ref_view, with_thresholds = True, drift_map=drift_map)\n",
"\n",
"scores"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that we didn't specify an algorithm for `animal`, and we got a drift score nonetheless. If you don't specify an algorithm for a column, the default algorithm selection logic will be used."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In the Visualizer's case, you can choose the algorithms by using the `add_drift_config` method.\n",
"\n",
"The cell below will also define `hellinger` for `weight` and `chi-squared` for `legs`:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"visualization.add_drift_config(column_names=[\"weight\"], algorithm=Hellinger())\n",
"visualization.add_drift_config(column_names=[\"legs\"], algorithm=ChiSquare())\n",
"\n",
"visualization.summary_drift_report()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculating Drift - Customizing Internal Parameters"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In addition to selecting which algorithms to use, you can also customize the internal parameters of each algorithm.\n",
"\n",
"For example, suppose we want to change the thresholds for the `hellinger` algorithm, making it less sensitive to drift. We can do this by passing a `parameter_config` object when instantiating the Hellinger algorithm. One of the parameters in the `parameter_config` object is a DriftThresholds object, which contains the thresholds for the drift categorization.\n",
"\n",
"We might also want to change the KS algorithm. In whylogs, the quantiles are split into 100 bins by default. If you want to use another number, you can create a `KSTestConfig` object with your own value for `quantiles` and pass it to the `KSTest` algorithm.\n",
"\n",
"Finally, suppose we don't want the Chi Square algorithm to categorize into 3 different classes. We want only a binary categorization, where the column is either drifted or not. We can do this by passing a `ChiSquareConfig` object with a `DriftThresholds` object with only two thresholds.\n",
"\n",
"Let's see how to create those config objects and how to pass them to either `calculate_drift_score` or `summary_drift_report`:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from whylogs.viz.drift.configs import KSTestConfig, HellingerConfig, ChiSquareConfig, DriftThresholds\n",
"\n",
"hellingerconfig = HellingerConfig(thresholds=DriftThresholds(NO_DRIFT=(0, 0.15), POSSIBLE_DRIFT=(0.15,0.5), DRIFT=(0.5, 1)))\n",
"\n",
"quantiles = [0.0, 0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99, 1.0]\n",
"ksconfig = KSTestConfig(quantiles=quantiles)\n",
"\n",
"chisquareconfig = ChiSquareConfig(thresholds=DriftThresholds(DRIFT=(0, 0.1), NO_DRIFT=(0.1, 1)))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With the configs at hand, we can now pass them to the Drift Algorithms:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'animal': {'algorithm': 'chi-square',\n",
" 'pvalue': 1.0,\n",
" 'statistic': 0.0,\n",
" 'thresholds': {'NO_DRIFT': (0.1, 1), 'DRIFT': (0, 0.1)},\n",
" 'drift_category': 'NO_DRIFT'},\n",
" 'legs': {'algorithm': 'ks',\n",
" 'pvalue': 0.0,\n",
" 'statistic': 1.0,\n",
" 'thresholds': {'NO_DRIFT': (0.15, 1),\n",
" 'POSSIBLE_DRIFT': (0.05, 0.15),\n",
" 'DRIFT': (0, 0.05)},\n",
" 'drift_category': 'DRIFT'},\n",
" 'weight': {'algorithm': 'hellinger',\n",
" 'pvalue': None,\n",
" 'statistic': 0.4283729905961321,\n",
" 'thresholds': {'NO_DRIFT': (0, 0.15),\n",
" 'POSSIBLE_DRIFT': (0.15, 0.5),\n",
" 'DRIFT': (0.5, 1)},\n",
" 'drift_category': 'POSSIBLE_DRIFT'}}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from whylogs.viz.drift.column_drift_algorithms import Hellinger, ChiSquare, KS\n",
"\n",
"drift_map = {\n",
" \"weight\": Hellinger(hellingerconfig),\n",
" \"animal\": ChiSquare(chisquareconfig),\n",
" \"legs\": KS(ksconfig),\n",
"}\n",
"\n",
"scores = calculate_drift_scores(target_view=target_view, reference_view=ref_view, with_thresholds = True, drift_map=drift_map)\n",
"scores"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Overwriting existing drift configuration for column weight.\n",
"Overwriting existing drift configuration for column legs.\n"
]
},
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"visualization.add_drift_config(column_names=[\"weight\"], algorithm=Hellinger(hellingerconfig))\n",
"visualization.add_drift_config(column_names=[\"animal\"], algorithm=ChiSquare(chisquareconfig))\n",
"visualization.add_drift_config(column_names=[\"legs\"], algorithm=KS(ksconfig))\n",
"\n",
"visualization.summary_drift_report()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can check on either the scores object or the drift report that the changes were indeed applied.\n",
"\n",
"Stay tuned for more Drift Algorithms to be added to whylogs!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "5dd5901cadfd4b29c2aaf95ecd29c0c3b10829ad94dcfe59437dbee391154aea"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}