{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Creating Metric Constraints on Condition Count Metrics" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "whylogs profiles contain summarized information about our data. This means that it's a __lossy__ process, and once we get the profiles, we don't have access anymore to the complete set of data.\n", "\n", "This makes some types of constraints impossible to be created from standard metrics itself. For example, suppose you need to check every row of a column to check that there are no textual information that matches a credit card number or email information. Or maybe you're interested in ensuring that there are no even numbers in a certain column. How do we do that if we don't have access to the complete data?\n", "\n", "The answer is that you need to define a __Condition Count Metric__ to be tracked __before__ logging your data. This metric will count the number of times the values of a given column meets a user-defined condition. When the profile is generated, you'll have that information to check against the constraints you'll create.\n", "\n", "In this example, you'll learn how to:\n", "- Define additional Condition Count Metrics\n", "- Define actions to be triggered whenever those conditions are met during the logging process.\n", "- Use the Condition Count Metrics to create constraints against said conditions\n", "\n", "If you want more information on Condition Count Metrics, you can see [this example](https://nbviewer.org/github/whylabs/whylogs/blob/mainline/python/examples/advanced/Condition_Count_Metrics.ipynb) and also the documentation for [Data Validation](https://whylogs.readthedocs.io/en/stable/features/data_validation.html)\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Installing whylogs\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Note: you may need to restart the kernel to use updated packages.\n", "%pip install whylogs" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Context\n", "\n", "Let's assume we have a DataFrame for which we wish to log standard metrics through whylogs' default logging process. But additionally, we want specific information on two columns:\n", "\n", "- `url`: Regex pattern validation: the values in this column should always start with `https:://www.mydomain.com/profile`\n", "- `subscription_date`: Date Format validation: the values in this column should be a string with a date format of `%Y-%m-%d`\n", "\n", "In addition, we consider these cases to be critical, so we wish to make certain actions whenever the condition fails. In this example we will:\n", "\n", "- Send an alert in Slack whenever `subscription_date` fails the condition\n", "- Send an alert in Slack and pull a symbolic Andon Cord whenever `url` is not from the domain we expect\n", "\n", "Let's first create a simple DataFrame to demonstrate:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "data = {\n", " \"name\": [\"Alice\", \"Bob\", \"Charles\"],\n", " \"age\": [31,0,25],\n", " \"url\": [\"https://www.mydomain.com/profile/123\", \"www.wrongdomain.com\", \"http://mydomain.com/unsecure\"],\n", " \"subscription_date\": [\"2021-12-28\",\"2019-29-11\",\"04/08/2021\"],\n", " }\n", "\n", "df = pd.DataFrame(data)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "In this case, both `url` and `subscription_date` has 2 values out of 3 that are not what we expect." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Defining the Relations\n", "\n", "Let's first define the relations that will actually check whether the value passes our constraint. For the date format validation, we'll use the __datetime__ module in a user defined function. As for the Regex pattern matching, we will use whylogs' `Predicates` along with regular expressions, which allows us to build simple relations intuitively." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import datetime\n", "from typing import Any\n", "from whylogs.core.relations import Predicate\n", "\n", "\n", "def date_format(x: Any) -> bool:\n", " date_format = '%Y-%m-%d'\n", " try:\n", " datetime.datetime.strptime(x, date_format)\n", " return True\n", " except ValueError:\n", " return False\n", "\n", "# matches accept a regex expression\n", "matches_domain_url = Predicate().matches(\"^https:\\/\\/www.mydomain.com\\/profile\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Defining the Actions\n", "\n", "Next, we need to define the actions that will be triggered whenever the conditions fail.\n", "\n", "We will define two placeholder functions that, in a real scenario, would execute the defined actions." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from typing import Any\n", "\n", "def pull_andon_cord(validator_name, condition_name: str, value: Any):\n", " print(\"Validator: {}\\n Condition name {} failed for value {}\".format(validator_name, condition_name, value))\n", " print(\" Pulling andon cord....\")\n", " # Do something here to respond to the constraint violation\n", " return\n", "\n", "def send_slack_alert(validator_name, condition_name: str, value: Any):\n", " print(\"Validator: {}\\n Condition name {} failed for value {}\".format(validator_name, condition_name, value))\n", " print(\" Sending slack alert....\")\n", " # Do something here to respond to the constraint violation\n", " return" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Conditions = Relations + Actions\n", "\n", "Conditions are defined by the combination of a relation and a set of actions. Now that we have both relations and actions, we can create two sets of conditions - in this example, each set contain a single condition, but we could have multiple." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from whylogs.core.metrics.condition_count_metric import Condition\n", "\n", "has_date_format = {\n", " \"Y-m-d format\": Condition(date_format, actions=[send_slack_alert]),\n", "}\n", "\n", "regex_conditions = {\"url_matches_domain\": Condition(matches_domain_url, actions=[pull_andon_cord,send_slack_alert])}\n", "\n", "ints_conditions = {\n", " \"integer_zeros\": Condition(Predicate().equals(0)),\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Passing the conditions to the Logger\n", "\n", "Now, we need to let the logger aware of our Conditions. This can be done by creating a custom schema object that will be passed to `why.log()`.\n", "\n", "To create the schema object, we will use the __Declarative Schema__, which is an auxiliary class that will enable us to create a schema in a simple way.\n", "\n", "In this case, we want our schema to start with the default behavior (standard metrics for the default datatypes). Then, we want to add two condition count metrics based on the conditions we defined earlier and the name of the column we want to bind those conditions to. We can do so by calling the schema's `add_condition_count_metric` method:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from whylogs.core.resolvers import STANDARD_RESOLVER\n", "from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n", "from whylogs.core.schema import DeclarativeSchema\n", "\n", "schema = DeclarativeSchema(STANDARD_RESOLVER)\n", "\n", "schema.add_resolver_spec(column_name=\"subscription_date\", metrics=[ConditionCountMetricSpec(has_date_format)])\n", "schema.add_resolver_spec(column_name=\"url\", metrics=[ConditionCountMetricSpec(regex_conditions)])\n", "schema.add_resolver_spec(column_name=\"age\", metrics=[ConditionCountMetricSpec(ints_conditions)])\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's pass the schema to why.log() and start logging our data:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Validator: condition_count\n", " Condition name url_matches_domain failed for value www.wrongdomain.com\n", " Pulling andon cord....\n", "Validator: condition_count\n", " Condition name url_matches_domain failed for value www.wrongdomain.com\n", " Sending slack alert....\n", "Validator: condition_count\n", " Condition name url_matches_domain failed for value http://mydomain.com/unsecure\n", " Pulling andon cord....\n", "Validator: condition_count\n", " Condition name url_matches_domain failed for value http://mydomain.com/unsecure\n", " Sending slack alert....\n", "Validator: condition_count\n", " Condition name Y-m-d format failed for value 2019-29-11\n", " Sending slack alert....\n", "Validator: condition_count\n", " Condition name Y-m-d format failed for value 04/08/2021\n", " Sending slack alert....\n" ] } ], "source": [ "import whylogs as why\n", "profile_view = why.log(df, schema=schema).profile().view()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "You can see that during the logging process, our actions were triggered whenever the condition failed. We can see the name of the failed condition and the specific value that triggered it.\n", "\n", "We see the actions were triggered, but we also expect the Condition Count Metrics to be generated. Let's see if this is the case:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cardinality/estcardinality/lower_1cardinality/upper_1condition_count/integer_zeroscondition_count/totalcounts/infcounts/ncounts/nancounts/nulldistribution/maxdistribution/meandistribution/mediandistribution/mindistribution/ndistribution/q_01distribution/q_05distribution/q_10distribution/q_25distribution/q_75distribution/q_90distribution/q_95distribution/q_99distribution/stddevfrequent_items/frequent_stringsints/maxints/mintypetypes/booleantypes/fractionaltypes/integraltypes/objecttypes/stringtypes/tensorcondition_count/Y-m-d formatcondition_count/url_matches_domain
column
age3.03.03.000151.03.0030031.018.66666725.00.030.00.00.00.031.031.031.031.016.441817[FrequentItem(value='25', est=1, upper=1, lowe...31.00.0SummaryType.COLUMN003000NaNNaN
name3.03.03.00015NaNNaN0300NaN0.000000NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaN0.000000[FrequentItem(value='Alice', est=1, upper=1, l...NaNNaNSummaryType.COLUMN000030NaNNaN
subscription_date3.03.03.00015NaN3.00300NaN0.000000NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaN0.000000[FrequentItem(value='2019-29-11', est=1, upper...NaNNaNSummaryType.COLUMN0000301.0NaN
url3.03.03.00015NaN3.00300NaN0.000000NaNNaN0NaNNaNNaNNaNNaNNaNNaNNaN0.000000[FrequentItem(value='www.wrongdomain.com', est...NaNNaNSummaryType.COLUMN000030NaN1.0
\n", "
" ], "text/plain": [ " cardinality/est ... condition_count/url_matches_domain\n", "column ... \n", "age 3.0 ... NaN\n", "name 3.0 ... NaN\n", "subscription_date 3.0 ... NaN\n", "url 3.0 ... 1.0\n", "\n", "[4 rows x 35 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "profile_view.to_pandas()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "At the far right of our summary dataframe, you can find the Condition Count Metrics: the `Y-m-d format` condition was met only once of a total of 3. The same happens for the `url_matches_domain`. Note that for columns where the condition was not defined, a `NaN` is displayed." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Creating Metric Constraints based on Condition Count Metrics" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "So far, we created Condition Count Metrics for both of the desired conditions. During the logging process, the set of actions defined for each of the conditions were triggered whenever the conditions failed to be met.\n", "\n", "Now, we wish to create Metric Constraints on top of the Condition Count Metrics, so we can generate a Constraints Report. This can be done by using the `condition_meets` helper constraint. You only need to specify the column name and the name of the condition you want to check:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[ReportResult(name='subscription_date meets condition Y-m-d format', passed=0, failed=1, summary=None),\n", " ReportResult(name='url never meets condition url_matches_domain', passed=0, failed=1, summary=None),\n", " ReportResult(name='age.integer_zeros lower than or equal to 1', passed=1, failed=0, summary=None)]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from whylogs.core.constraints.factories import condition_meets, condition_never_meets, condition_count_below\n", "from whylogs.core.constraints import ConstraintsBuilder\n", "\n", "builder = ConstraintsBuilder(dataset_profile_view=profile_view)\n", "\n", "builder.add_constraint(condition_meets(column_name=\"subscription_date\", condition_name=\"Y-m-d format\"))\n", "builder.add_constraint(condition_never_meets(column_name=\"url\", condition_name=\"url_matches_domain\"))\n", "builder.add_constraint(condition_count_below(column_name=\"age\", condition_name=\"integer_zeros\", max_count=1))\n", "\n", "constraints = builder.build()\n", "constraints.generate_constraints_report()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The `condition_meets` constraint will fail if the said condition is not met at least once. In other words, if `condition_count/condition_name` is smaller than `condition_count/total`.\n", "\n", "The `condition_never_meets` constraint will fail if the said condition is met at least once. In other words, if `condition_count/condition_name` is greater than 0.\n", "\n", "The `condition_count_below` constraint will fail if the said condition is met more than a specified number of times." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Visualizing the Report" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "You can visualize the Constraints Report as usual by calling `NotebookProfileVisualizer`'s `constraints_report`:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from whylogs.viz import NotebookProfileVisualizer\n", "\n", "visualization = NotebookProfileVisualizer()\n", "visualization.constraints_report(constraints, cell_height=300)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "By hovering on the status, you can view the number of times the condition failed, and the total number of times the condition was checked." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "5dd5901cadfd4b29c2aaf95ecd29c0c3b10829ad94dcfe59437dbee391154aea" } } }, "nbformat": 4, "nbformat_minor": 2 }