{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Splink data linking demo (link only)\n", "\n", "In this demo we link two small datasets. \n", "\n", "The larger table contains duplicates, but in this notebook we use the `link_only` setting, so `splink` makes no attempt to deduplicate these records. \n", "\n", "Note it is possible to simultaneously link and dedupe using the `link_and_dedupe` setting.\n", "\n", "**Important** Where deduplication is not required, `link_only` can provide an important performance boost by dramatically reducing the number of records which need to be compared.\n", "\n", "For example, if you wanted to link 10 records to 1,000, then the maximum number of comparisons that need to be made (i.e. with no blocking rules) is 10,000. If you need to dedupe as well, that number would be n(n-1)/2 = 509,545.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Imports and setup\n", "\n", "The following is just boilerplate code that sets up the Spark session and sets some other non-essential configuration options" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "RendererRegistry.enable('mimetype')" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd \n", "pd.options.display.max_columns = 500\n", "pd.options.display.max_rows = 100\n", "import altair as alt\n", "alt.renderers.enable('mimetype')" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import logging \n", "logging.basicConfig() # Means logs will print in Jupyter Lab\n", "\n", "# Set to DEBUG if you want splink to log the SQL statements it's executing under the hood\n", "logging.getLogger(\"splink\").setLevel(logging.INFO)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "22/01/11 05:40:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n", "Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\n", "Setting default log level to \"WARN\".\n", "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n", "22/01/11 05:40:57 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.\n", "22/01/11 05:40:57 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.\n" ] } ], "source": [ "from utility_functions.demo_utils import get_spark\n", "spark = get_spark()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Read in the data\n", "\n", "In this example, we link two datasets, but you can link as many as you like.\n", "\n", "⚠️ Note that `splink` makes the following assumptions about your data:\n", "\n", "- There is a field containing a unique record identifier in each dataset. By default, this should be called `unique_id`, but you can change this in the settings\n", "- There is a field containing a dataset name in each dataset, to disambiguate the `unique_id` column if the same id values occur in more than one dataset. By default, this column is called `source_dataset`, but you can change this in the settings.\n", "- The two datasets being linked have common column names - e.g. date of birth is represented in both datasets in a field of the same name. In many cases, this means that the user needs to rename columns prior to using `splink`\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The count of rows in `df_1` is 181\n", "+---------+----------+-------+----------+------------+--------------------+-----+--------------+\n", "|unique_id|first_name|surname| dob| city| email|group|source_dataset|\n", "+---------+----------+-------+----------+------------+--------------------+-----+--------------+\n", "| 0| Julia | null|2015-10-29| London| hannah88@powers.com| 0| df_1|\n", "| 4| oNah| Watson|2008-03-23| Bolton|matthew78@ballard...| 1| df_1|\n", "| 13| Molly | Bell|2002-01-05|Peterborough| null| 2| df_1|\n", "| 15| Alexander|Amelia |1983-05-19| Glasgow|ic-mpbell@alleale...| 3| df_1|\n", "| 20| Ol vri|ynnollC|1972-03-08| Plymouth|derekwilliams@nor...| 4| df_1|\n", "+---------+----------+-------+----------+------------+--------------------+-----+--------------+\n", "only showing top 5 rows\n", "\n", "The count of rows in `df_2` is 819\n", "+---------+----------+-------+----------+------+--------------------+-----+--------------+\n", "|unique_id|first_name|surname| dob| city| email|group|source_dataset|\n", "+---------+----------+-------+----------+------+--------------------+-----+--------------+\n", "| 1| Julia | Taylor|2015-07-31|London| hannah88@powers.com| 0| df_2|\n", "| 2| Julia | Taylor|2016-01-27|London| hannah88@powers.com| 0| df_2|\n", "| 3| Julia | Taylor|2015-10-29| null| hannah88opowersc@m| 0| df_2|\n", "| 5| Noah | Watson|2008-03-23|Bolton|matthew78@ballard...| 1| df_2|\n", "| 6| Watson| Noah |2008-03-23| null|matthew78@ballard...| 1| df_2|\n", "+---------+----------+-------+----------+------+--------------------+-----+--------------+\n", "only showing top 5 rows\n", "\n" ] } ], "source": [ "from pyspark.sql.functions import lit \n", "df_1 = spark.read.parquet(\"data/fake_df_l.parquet\")\n", "df_1 = df_1.withColumn(\"source_dataset\", lit(\"df_1\"))\n", "df_2 = spark.read.parquet(\"data/fake_df_r.parquet\")\n", "df_2 = df_2.withColumn(\"source_dataset\", lit(\"df_2\"))\n", "print(f\"The count of rows in `df_1` is {df_1.count()}\")\n", "df_1.show(5)\n", "print(f\"The count of rows in `df_2` is {df_2.count()}\")\n", "df_2.show(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3: Configure splink using the `settings` object\n", "\n", "Most of `splink` configuration options are stored in a settings dictionary. This dictionary allows significant customisation, and can therefore get quite complex. \n", "\n", "💥 We provide an tool for helping to author valid settings dictionaries, which includes tooltips and autocomplete, which you can find [here](http://robinlinacre.com/splink_settings_editor/).\n", "\n", "Customisation overrides default values built into splink. For the purposes of this demo, we will specify a simple settings dictionary, which means we will be relying on these sensible defaults.\n", "\n", "To help with authoring and validation of the settings dictionary, we have written a [json schema](https://json-schema.org/), which can be found [here](https://github.com/moj-analytical-services/splink/blob/master/splink/files/settings_jsonschema.json). \n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# The comparison expression allows for the case where a first name and surname have been inverted \n", "sql_case_expression = \"\"\"\n", "CASE \n", "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4 \n", "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3\n", "WHEN first_name_l = first_name_r THEN 2\n", "WHEN surname_l = surname_r THEN 1\n", "ELSE 0 \n", "END\n", "\"\"\"\n", "\n", "settings = {\n", " \"link_type\": \"link_only\", \n", " \"max_iterations\": 20,\n", " \"blocking_rules\": [\n", " ],\n", " \"comparison_columns\": [\n", " {\n", " \"custom_name\": \"name_inversion\",\n", " \"custom_columns_used\": [\"first_name\", \"surname\"],\n", " \"case_expression\": sql_case_expression,\n", " \"num_levels\": 5\n", " },\n", " {\n", " \"col_name\": \"city\",\n", " \"num_levels\": 3\n", " },\n", " {\n", " \"col_name\": \"email\",\n", " \"num_levels\": 3\n", " },\n", " {\n", " \"col_name\": \"dob\"\n", " }\n", " ],\n", " \"additional_columns_to_retain\": [\"group\"],\n", " \"em_convergence\": 0.01,\n", " \"max_iterations\": 4,\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In words, this setting dictionary says:\n", "\n", "- We are performing a data linking task (the other options are `dedupe_only`, or `link_and_dedupe`)\n", "- Since the input datasets are so small, we do not specify any blocking rules and instead generate all possible comparisons.\n", "- When comparing records, we will use information from the `first_name`, `surname`, `city` and `email` columns to compute a match score.\n", "- For the comparisons on the `first_name` and `surname` column we allow the possibility that the names have been inputted in the wrong order. \n", " - The highest level of similarity is that both `first_name` and `surname` both match.\n", " - There are other levels of similarity for the names being inverted, and just first name, or just surname matching.\n", "- We will retain the `group` column in the results even though this is not used as part of comparisons. This is a labelled dataset and `group` contains the true match - i.e. where group matches, the records pertain to the same person" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 4: Estimate match scores using the Expectation Maximisation algorithm" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/robinlinacre/anaconda3/lib/python3.8/site-packages/splink/default_settings.py:262: UserWarning: You have not specified any blocking rules, meaning all comparisons between the input dataset(s) will be generated and blocking will not be used.For large input datasets, this will generally be computationally intractable because it will generate comparisons equal to the number of rows squared.\n", " warnings.warn(\n", "/Users/robinlinacre/anaconda3/lib/python3.8/site-packages/splink/default_settings.py:185: UserWarning: No -1 level found in case statement. You usually want to use -1 as the level for the null value. e.g. WHEN col_l is null or col_r is null then -1 Case statement is:\n", " \n", "CASE \n", "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4 \n", "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3\n", "WHEN first_name_l = first_name_r THEN 2\n", "WHEN surname_l = surname_r THEN 1\n", "ELSE 0 \n", "END\n", ".\n", " warnings.warn(\n", "INFO:splink.iterate:Iteration 0 complete \n", "INFO:splink.model:The maximum change in parameters was 0.40458029469636825 for key name_inversion, level 4\n", "INFO:splink.iterate:Iteration 1 complete\n", "INFO:splink.model:The maximum change in parameters was 0.07434341748102571 for key email, level 1\n", "INFO:splink.iterate:Iteration 2 complete\n", "INFO:splink.model:The maximum change in parameters was 0.025150011310513642 for key dob, level 1\n", "INFO:splink.iterate:Iteration 3 complete\n", "INFO:splink.model:The maximum change in parameters was 0.009595088165527288 for key name_inversion, level 0\n", "INFO:splink.iterate:EM algorithm has converged\n" ] } ], "source": [ "from splink import Splink\n", "\n", "linker = Splink(settings, [df_1, df_2], spark)\n", "df_e = linker.get_scored_comparisons()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 5: Inspect results \n", "\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
match_weightmatch_probabilitysource_dataset_lunique_id_lsource_dataset_runique_id_rsurname_lsurname_rfirst_name_lfirst_name_rgamma_name_inversioncity_lcity_rgamma_cityemail_lemail_rgamma_emaildob_ldob_rgamma_dobgroup_lgroup_r
9310137.8163461.0df_1664df_2668TaylorIvyIvyTaylor3LononLondon1jonesjennmfer@pitt.coijonesjennifer@pitts.com11980-01-131980-01-131113113
9310637.8163461.0df_1664df_2673TaylorIvyIvyTaylor3LononLondon1jonesjennmfer@pitt.coijonesjennifer@pitts.com11980-01-131980-01-131113113
7993037.8163461.0df_1581df_2585ShawEleanorEleanorShaw3BirminghamBirmingha1stephaniewebbhart.netstephaniewebb@hart.net11979-03-311979-03-3119797
7332737.3730571.0df_1517df_2526MarthaBrownBrownMartha3Southend-on-SeaSouthend-on-Sea2watsonthomas@jones-stuart.bizwatsonthomas@jones-s.urttbiz12002-09-012002-09-0118989
7910537.3730571.0df_1574df_2578WilliamsGeorgeGeorgeWilliams3LondonLondon2desek58gibbr.bizderek58@gibbs.biz11981-08-061981-08-0619696
\n", "
" ], "text/plain": [ " match_weight match_probability source_dataset_l unique_id_l \\\n", "93101 37.816346 1.0 df_1 664 \n", "93106 37.816346 1.0 df_1 664 \n", "79930 37.816346 1.0 df_1 581 \n", "73327 37.373057 1.0 df_1 517 \n", "79105 37.373057 1.0 df_1 574 \n", "\n", " source_dataset_r unique_id_r surname_l surname_r first_name_l \\\n", "93101 df_2 668 Taylor Ivy Ivy \n", "93106 df_2 673 Taylor Ivy Ivy \n", "79930 df_2 585 Shaw Eleanor Eleanor \n", "73327 df_2 526 Martha Brown Brown \n", "79105 df_2 578 Williams George George \n", "\n", " first_name_r gamma_name_inversion city_l city_r \\\n", "93101 Taylor 3 Lonon London \n", "93106 Taylor 3 Lonon London \n", "79930 Shaw 3 Birmingham Birmingha \n", "73327 Martha 3 Southend-on-Sea Southend-on-Sea \n", "79105 Williams 3 London London \n", "\n", " gamma_city email_l \\\n", "93101 1 jonesjennmfer@pitt.coi \n", "93106 1 jonesjennmfer@pitt.coi \n", "79930 1 stephaniewebbhart.net \n", "73327 2 watsonthomas@jones-stuart.biz \n", "79105 2 desek58gibbr.biz \n", "\n", " email_r gamma_email dob_l dob_r \\\n", "93101 jonesjennifer@pitts.com 1 1980-01-13 1980-01-13 \n", "93106 jonesjennifer@pitts.com 1 1980-01-13 1980-01-13 \n", "79930 stephaniewebb@hart.net 1 1979-03-31 1979-03-31 \n", "73327 watsonthomas@jones-s.urttbiz 1 2002-09-01 2002-09-01 \n", "79105 derek58@gibbs.biz 1 1981-08-06 1981-08-06 \n", "\n", " gamma_dob group_l group_r \n", "93101 1 113 113 \n", "93106 1 113 113 \n", "79930 1 97 97 \n", "73327 1 89 89 \n", "79105 1 96 96 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Inspect main dataframe that contains the match scores\n", "df_e.toPandas().sort_values(\"match_probability\", ascending=False).head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `params` property of the `linker` is an object that contains a lot of diagnostic information about how the match probability was computed. The following cells demonstrate some of its functionality" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/vnd.vegalite.v4+json": { "$schema": "https://vega.github.io/schema/vega-lite/v4.json", "config": { "header": { "title": null }, "title": { "anchor": "middle", "offset": 10 }, "view": { "continuousHeight": 300, "continuousWidth": 400, "height": 300, "width": 400 } }, "data": { "name": "data-f306f12e00d3f5f049f79d2780a7ab73" }, "datasets": { "data-f306f12e00d3f5f049f79d2780a7ab73": [ { "bayes_factor": 7866.369234599884, "column_name": "name_inversion", "comparison_vector_value": "4", "gamma_column_name": "gamma_name_inversion", "gamma_index": 4, "is_null": false, "label": "level_4", "level_name": "level_4", "level_proportion": 0.0009781501494208627, "log2_bayes_factor": 12.941482190295073, "m_probability": 0.1735820750664971, "num_levels": 5, "sql_expr": "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4", "u_probability": 0.000022066352327196117 }, { "bayes_factor": 10816.87507346276, "column_name": "name_inversion", "comparison_vector_value": "3", "gamma_column_name": "gamma_name_inversion", "gamma_index": 3, "is_null": false, "label": "level_3", "level_name": "level_3", "level_proportion": 0.000802757708835054, "log2_bayes_factor": 13.400996153467629, "m_probability": 0.14333415629686924, "num_levels": 5, "sql_expr": "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3", "u_probability": 0.00001325097639784281 }, { "bayes_factor": 84.21275991056915, "column_name": "name_inversion", "comparison_vector_value": "2", "gamma_column_name": "gamma_name_inversion", "gamma_index": 2, "is_null": false, "label": "level_2", "level_name": "level_2", "level_proportion": 0.003082859436450116, "log2_bayes_factor": 6.3959669417774965, "m_probability": 0.178015412463166, "num_levels": 5, "sql_expr": "WHEN first_name_l = first_name_r THEN 2", "u_probability": 0.0021138769546587928 }, { "bayes_factor": 43.14154400089717, "column_name": "name_inversion", "comparison_vector_value": "1", "gamma_column_name": "gamma_name_inversion", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.005140347681783208, "log2_bayes_factor": 5.43100590519603, "m_probability": 0.1799810780566816, "num_levels": 5, "sql_expr": "WHEN surname_l = surname_r THEN 1", "u_probability": 0.004171873821969346 }, { "bayes_factor": 0.32715524872500057, "column_name": "name_inversion", "comparison_vector_value": "0", "gamma_column_name": "gamma_name_inversion", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.9899958850235109, "log2_bayes_factor": -1.6119526780809075, "m_probability": 0.3250872781167861, "num_levels": 5, "sql_expr": "ELSE 0", "u_probability": 0.9936789318946468 }, { "bayes_factor": 7.5754215135378, "column_name": "city", "comparison_vector_value": "2", "gamma_column_name": "gamma_city", "gamma_index": 2, "is_null": false, "label": "level_2", "level_name": "level_2", "level_proportion": 0.09247482158475576, "log2_bayes_factor": 2.9213261656247806, "m_probability": 0.6760480820908364, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 1.0 THEN 2", "u_probability": 0.0892423056436783 }, { "bayes_factor": 10.300307763779484, "column_name": "city", "comparison_vector_value": "1", "gamma_column_name": "gamma_city", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.02173640315205869, "log2_bayes_factor": 3.364615539348775, "m_probability": 0.2129801834705903, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 0.88 THEN 1", "u_probability": 0.020677069885185802 }, { "bayes_factor": 0.12467604775074131, "column_name": "city", "comparison_vector_value": "0", "gamma_column_name": "gamma_city", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.8857887752631854, "log2_bayes_factor": -3.003743767733968, "m_probability": 0.11097173443857297, "num_levels": 3, "sql_expr": "ELSE 0", "u_probability": 0.8900806244711358 }, { "bayes_factor": 8537.588804915724, "column_name": "email", "comparison_vector_value": "2", "gamma_column_name": "gamma_email", "gamma_index": 2, "is_null": false, "label": "level_2", "level_name": "level_2", "level_proportion": 0.0036458147411618713, "log2_bayes_factor": 13.059612964535702, "m_probability": 0.6481277887828767, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 1.0 THEN 2", "u_probability": 0.00007591461753343069 }, { "bayes_factor": 26193.64119339461, "column_name": "email", "comparison_vector_value": "1", "gamma_column_name": "gamma_email", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.0018015059731803925, "log2_bayes_factor": 14.676929003060637, "m_probability": 0.32479282624986167, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 0.88 THEN 1", "u_probability": 0.000012399682192018664 }, { "bayes_factor": 0.02708177667540402, "column_name": "email", "comparison_vector_value": "0", "gamma_column_name": "gamma_email", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.9945526792856578, "log2_bayes_factor": -5.2065338010569935, "m_probability": 0.02707938496726161, "num_levels": 3, "sql_expr": "ELSE 0", "u_probability": 0.9999116857002746 }, { "bayes_factor": 14971.363071299604, "column_name": "dob", "comparison_vector_value": "1", "gamma_column_name": "gamma_dob", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.00333920223422985, "log2_bayes_factor": 13.869917957437533, "m_probability": 0.5989503315946443, "num_levels": 2, "sql_expr": "WHEN dob_l = dob_r THEN 1", "u_probability": 0.00004000639946691586 }, { "bayes_factor": 0.4010657136005066, "column_name": "dob", "comparison_vector_value": "0", "gamma_column_name": "gamma_dob", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.9966607977657703, "log2_bayes_factor": -1.318089456914419, "m_probability": 0.40104966840535583, "num_levels": 2, "sql_expr": "ELSE 0", "u_probability": 0.9999599936005332 } ] }, "hconcat": [ { "encoding": { "color": { "value": "red" }, "row": { "field": "column_name", "header": { "labelAlign": "left", "labelAnchor": "middle", "labelAngle": 0 }, "sort": { "field": "gamma_index" }, "type": "nominal" }, "tooltip": [ { "field": "column_name", "type": "nominal" }, { "field": "level_name", "type": "ordinal" }, { "field": "u_probability", "format": ".4f", "type": "quantitative" }, { "field": "bayes_factor", "format": ".4f", "type": "quantitative" }, { "field": "level_proportion", "format": ".2%", "title": "Percentage of record comparisons in this level", "type": "nominal" }, { "field": "log2_bayes_factor", "format": ".4f", "type": "quantitative" } ], "x": { "axis": { "title": "proportion" }, "field": "u_probability", "type": "quantitative" }, "y": { "axis": { "title": null }, "field": "label", "sort": { "field": "gamma_index" }, "type": "nominal" } }, "height": 50, "mark": "bar", "resolve": { "scale": { "y": "independent" } }, "title": { "fontWeight": "normal", "text": "Non-matches" }, "transform": [ { "filter": "(datum.bayes_factor != 'unnecessary filter2 due to vega lite issue 4680')" } ], "width": 150 }, { "encoding": { "color": { "value": "green" }, "row": { "field": "column_name", "header": { "labels": false }, "sort": { "field": "gamma_index" }, "type": "nominal" }, "tooltip": [ { "field": "column_name", "type": "nominal" }, { "field": "level_name", "type": "ordinal" }, { "field": "m_probability", "format": ".4f", "type": "quantitative" }, { "field": "bayes_factor", "format": ".4f", "type": "quantitative" }, { "field": "level_proportion", "format": ".2%", "title": "Percentage of record comparisons in this level", "type": "nominal" }, { "field": "log2_bayes_factor", "format": ".4f", "type": "quantitative" } ], "x": { "axis": { "title": "proportion" }, "field": "m_probability", "type": "quantitative" }, "y": { "axis": { "title": null }, "field": "label", "sort": { "field": "gamma_index" }, "type": "nominal" } }, "height": 50, "mark": "bar", "resolve": { "scale": { "y": "independent" } }, "title": { "fontWeight": "normal", "text": "Matches" }, "transform": [ { "filter": "(datum.bayes_factor != 'unnecessary filter due to vega lite issue 4680')" } ], "width": 150 } ], "title": { "subtitle": "Estimated proportion of matches λ = 0.00551", "text": "Probability distributions of non-matches and matches " }, "transform": [] }, "text/plain": [ "\n", "\n", "If you see this message, it means the renderer has not been properly enabled\n", "for the frontend that you are using. For more information, see\n", "https://altair-viz.github.io/user_guide/troubleshooting.html\n" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = linker.model\n", "model.probability_distribution_chart()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An alternative representation of the parameters displays them in terms of the effect different values in the comparison vectors have on the match probability:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/vnd.vegalite.v4+json": { "$schema": "https://vega.github.io/schema/vega-lite/v4.json", "config": { "header": { "title": null }, "mark": { "tooltip": null }, "title": { "anchor": "middle" }, "view": { "continuousHeight": 300, "continuousWidth": 400, "height": 300, "width": 400 } }, "data": { "name": "data-f306f12e00d3f5f049f79d2780a7ab73" }, "datasets": { "data-f306f12e00d3f5f049f79d2780a7ab73": [ { "bayes_factor": 7866.369234599884, "column_name": "name_inversion", "comparison_vector_value": "4", "gamma_column_name": "gamma_name_inversion", "gamma_index": 4, "is_null": false, "label": "level_4", "level_name": "level_4", "level_proportion": 0.0009781501494208627, "log2_bayes_factor": 12.941482190295073, "m_probability": 0.1735820750664971, "num_levels": 5, "sql_expr": "WHEN first_name_l = first_name_r AND surname_l = surname_r THEN 4", "u_probability": 0.000022066352327196117 }, { "bayes_factor": 10816.87507346276, "column_name": "name_inversion", "comparison_vector_value": "3", "gamma_column_name": "gamma_name_inversion", "gamma_index": 3, "is_null": false, "label": "level_3", "level_name": "level_3", "level_proportion": 0.000802757708835054, "log2_bayes_factor": 13.400996153467629, "m_probability": 0.14333415629686924, "num_levels": 5, "sql_expr": "WHEN first_name_l = surname_r AND surname_l = first_name_r THEN 3", "u_probability": 0.00001325097639784281 }, { "bayes_factor": 84.21275991056915, "column_name": "name_inversion", "comparison_vector_value": "2", "gamma_column_name": "gamma_name_inversion", "gamma_index": 2, "is_null": false, "label": "level_2", "level_name": "level_2", "level_proportion": 0.003082859436450116, "log2_bayes_factor": 6.3959669417774965, "m_probability": 0.178015412463166, "num_levels": 5, "sql_expr": "WHEN first_name_l = first_name_r THEN 2", "u_probability": 0.0021138769546587928 }, { "bayes_factor": 43.14154400089717, "column_name": "name_inversion", "comparison_vector_value": "1", "gamma_column_name": "gamma_name_inversion", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.005140347681783208, "log2_bayes_factor": 5.43100590519603, "m_probability": 0.1799810780566816, "num_levels": 5, "sql_expr": "WHEN surname_l = surname_r THEN 1", "u_probability": 0.004171873821969346 }, { "bayes_factor": 0.32715524872500057, "column_name": "name_inversion", "comparison_vector_value": "0", "gamma_column_name": "gamma_name_inversion", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.9899958850235109, "log2_bayes_factor": -1.6119526780809075, "m_probability": 0.3250872781167861, "num_levels": 5, "sql_expr": "ELSE 0", "u_probability": 0.9936789318946468 }, { "bayes_factor": 7.5754215135378, "column_name": "city", "comparison_vector_value": "2", "gamma_column_name": "gamma_city", "gamma_index": 2, "is_null": false, "label": "level_2", "level_name": "level_2", "level_proportion": 0.09247482158475576, "log2_bayes_factor": 2.9213261656247806, "m_probability": 0.6760480820908364, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 1.0 THEN 2", "u_probability": 0.0892423056436783 }, { "bayes_factor": 10.300307763779484, "column_name": "city", "comparison_vector_value": "1", "gamma_column_name": "gamma_city", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.02173640315205869, "log2_bayes_factor": 3.364615539348775, "m_probability": 0.2129801834705903, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(city_l, city_r) >= 0.88 THEN 1", "u_probability": 0.020677069885185802 }, { "bayes_factor": 0.12467604775074131, "column_name": "city", "comparison_vector_value": "0", "gamma_column_name": "gamma_city", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.8857887752631854, "log2_bayes_factor": -3.003743767733968, "m_probability": 0.11097173443857297, "num_levels": 3, "sql_expr": "ELSE 0", "u_probability": 0.8900806244711358 }, { "bayes_factor": 8537.588804915724, "column_name": "email", "comparison_vector_value": "2", "gamma_column_name": "gamma_email", "gamma_index": 2, "is_null": false, "label": "level_2", "level_name": "level_2", "level_proportion": 0.0036458147411618713, "log2_bayes_factor": 13.059612964535702, "m_probability": 0.6481277887828767, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 1.0 THEN 2", "u_probability": 0.00007591461753343069 }, { "bayes_factor": 26193.64119339461, "column_name": "email", "comparison_vector_value": "1", "gamma_column_name": "gamma_email", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.0018015059731803925, "log2_bayes_factor": 14.676929003060637, "m_probability": 0.32479282624986167, "num_levels": 3, "sql_expr": "WHEN JARO_WINKLER_SIM(email_l, email_r) >= 0.88 THEN 1", "u_probability": 0.000012399682192018664 }, { "bayes_factor": 0.02708177667540402, "column_name": "email", "comparison_vector_value": "0", "gamma_column_name": "gamma_email", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.9945526792856578, "log2_bayes_factor": -5.2065338010569935, "m_probability": 0.02707938496726161, "num_levels": 3, "sql_expr": "ELSE 0", "u_probability": 0.9999116857002746 }, { "bayes_factor": 14971.363071299604, "column_name": "dob", "comparison_vector_value": "1", "gamma_column_name": "gamma_dob", "gamma_index": 1, "is_null": false, "label": "level_1", "level_name": "level_1", "level_proportion": 0.00333920223422985, "log2_bayes_factor": 13.869917957437533, "m_probability": 0.5989503315946443, "num_levels": 2, "sql_expr": "WHEN dob_l = dob_r THEN 1", "u_probability": 0.00004000639946691586 }, { "bayes_factor": 0.4010657136005066, "column_name": "dob", "comparison_vector_value": "0", "gamma_column_name": "gamma_dob", "gamma_index": 0, "is_null": false, "label": "level_0", "level_name": "level_0", "level_proportion": 0.9966607977657703, "log2_bayes_factor": -1.318089456914419, "m_probability": 0.40104966840535583, "num_levels": 2, "sql_expr": "ELSE 0", "u_probability": 0.9999599936005332 } ] }, "encoding": { "color": { "field": "log2_bayes_factor", "scale": { "domain": [ -10, 0, 10 ], "range": [ "red", "orange", "green" ] }, "title": "Match weight", "type": "quantitative" }, "row": { "field": "column_name", "header": { "labelAlign": "left", "labelAnchor": "middle", "labelAngle": 0 }, "sort": { "field": "gamma_index" }, "type": "nominal" }, "tooltip": [ { "field": "column_name", "type": "nominal" }, { "field": "label", "title": "level label", "type": "ordinal" }, { "field": "sql_expr", "type": "nominal" }, { "field": "m_probability", "format": ".4f", "type": "quantitative" }, { "field": "u_probability", "format": ".4f", "type": "quantitative" }, { "field": "bayes_factor", "format": ".4f", "title": "Bayes factor = m/u", "type": "quantitative" }, { "field": "log2_bayes_factor", "format": ".4f", "title": "Match weight = log2(m/u)", "type": "quantitative" }, { "field": "level_proportion", "format": ".2%", "title": "Percentage of non-null comparisons in this level", "type": "nominal" } ], "x": { "axis": { "title": "Match weight = log2(m/u)", "values": [ -10, -5, 0, 5, 10 ] }, "field": "log2_bayes_factor", "scale": { "domain": [ -10, 10 ] }, "type": "quantitative" }, "y": { "axis": { "title": null }, "field": "label", "sort": { "field": "gamma_index" }, "type": "nominal" } }, "height": 50, "mark": { "clip": true, "type": "bar" }, "resolve": { "scale": { "y": "independent" } }, "selection": { "selector076": { "bind": "scales", "encodings": [ "x" ], "type": "interval" } }, "title": { "subtitle": "Use mousewheeel to zoom", "text": "Influence of comparison vector values on match probability" } }, "text/plain": [ "\n", "\n", "If you see this message, it means the renderer has not been properly enabled\n", "for the frontend that you are using. For more information, see\n", "https://altair-viz.github.io/user_guide/troubleshooting.html\n" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.bayes_factor_chart()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# If charts aren't displaying correctly in your notebook, you can write them to a file (by default splink_charts.html)\n", "model.all_charts_write_html_file(\"splink_charts.html\", overwrite=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also generate a report which explains how the match probability was computed for an individual comparison row. \n", "\n", "Note that you need to convert the row to a dictionary for this to work" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Initial probability of match (prior) = λ = 0.005509\n", "------\n", "Comparison of name_inversion. Values are: \n", "name_inversion_l: Thomas, J cob\n", "name_inversion_r: Bond, Harry \n", "Comparison has: 5 levels\n", "Level for this comparison: gamma_name_inversion = 0\n", "m probability = P(level|match): 0.3251\n", "u probability = P(level|non-match): 0.9937\n", "Bayes factor = m/u: 0.3272\n", "New probability of match (updated belief): 0.001809\n", "\n", "------\n", "Comparison of city. Values are: \n", "city_l: Sudlreand\n", "city_r: None\n", "Comparison has: 3 levels\n", "Level for this comparison: gamma_city = -1\n", "m probability = P(level|match): 1\n", "u probability = P(level|non-match): 1\n", "Bayes factor = m/u: 1\n", "New probability of match (updated belief): 0.001809\n", "\n", "------\n", "Comparison of email. Values are: \n", "email_l: hknapp@davis-allen.info\n", "email_r: alexanderthomas@reyes.com\n", "Comparison has: 3 levels\n", "Level for this comparison: gamma_email = 0\n", "m probability = P(level|match): 0.02708\n", "u probability = P(level|non-match): 0.9999\n", "Bayes factor = m/u: 0.02708\n", "New probability of match (updated belief): 4.907e-05\n", "\n", "------\n", "Comparison of dob. Values are: \n", "dob_l: 2007-05-28\n", "dob_r: 1979-12-08\n", "Comparison has: 2 levels\n", "Level for this comparison: gamma_dob = 0\n", "m probability = P(level|match): 0.401\n", "u probability = P(level|non-match): 1\n", "Bayes factor = m/u: 0.4011\n", "New probability of match (updated belief): 1.968e-05\n", "\n", "\n", "Final probability of match = 1.968e-05\n", "\n", "Reminder:\n", "\n", "The m probability for a given level is the proportion of matches which are in this level.\n", "We would generally expect the highest similarity level to have the largest proportion of matches.\n", "For example, we would expect first name field to match exactly amongst most matching records, except where nicknames, aliases or typos have occurred.\n", "For a comparison column that changes through time, like address, we may expect a lower proportion of comparisons to be in the highest similarity level.\n", "\n", "The u probability for a given level is the proportion of non-matches which are in this level.\n", "We would generally expect the lowest similarity level to have the highest proportion of non-matches, but the magnitude depends on the cardinality of the field.\n", "For example, we would expect that in the vast majority of non-matching records, the date of birth field would not match. However, we would expect it to be common for gender to match amongst non-matches.\n", "\n" ] } ], "source": [ "from splink.intuition import intuition_report\n", "row_dict = df_e.toPandas().sample(1).to_dict(orient=\"records\")[0]\n", "print(intuition_report(row_dict, model))" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "data": { "application/vnd.vegalite.v4+json": { "$schema": "https://vega.github.io/schema/vega-lite/v4.8.1.json", "config": { "title": { "fontSize": 14 }, "view": { "continuousHeight": 300, "continuousWidth": 400 } }, "data": { "name": "data-b6f9d8ee7ef48a0713fc85e66a024fff" }, "datasets": { "data-b6f9d8ee7ef48a0713fc85e66a024fff": [ { "binwidth": 0.564527783571009, "count_rows": 81038, "freqdensity": 143550.06495407797, "normalised": 0.5466712538535742, "splink_score_bin_high": -18.071904405390065, "splink_score_bin_low": -18.636432188961074 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -17.507376621819056, "splink_score_bin_low": -18.071904405390065 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -16.942848838248047, "splink_score_bin_low": -17.507376621819056 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -16.378321054677038, "splink_score_bin_low": -16.942848838248047 }, { "binwidth": 0.5645277835710072, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -15.81379327110603, "splink_score_bin_low": -16.378321054677038 }, { "binwidth": 0.564527783571009, "count_rows": 23697, "freqdensity": 41976.67624098306, "normalised": 0.15985671786776753, "splink_score_bin_high": -15.249265487535022, "splink_score_bin_low": -15.81379327110603 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -14.684737703964013, "splink_score_bin_low": -15.249265487535022 }, { "binwidth": 0.5645277835710072, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -14.120209920393005, "splink_score_bin_low": -14.684737703964013 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -13.555682136821996, "splink_score_bin_low": -14.120209920393005 }, { "binwidth": 0.564527783571009, "count_rows": 23868, "freqdensity": 42279.58427310561, "normalised": 0.1610102604577742, "splink_score_bin_high": -12.991154353250987, "splink_score_bin_low": -13.555682136821996 }, { "binwidth": 0.564527783571009, "count_rows": 8852, "freqdensity": 15680.361990344012, "normalised": 0.05971438015636908, "splink_score_bin_high": -12.426626569679978, "splink_score_bin_low": -12.991154353250987 }, { "binwidth": 0.564527783571009, "count_rows": 2047, "freqdensity": 3626.039425467035, "normalised": 0.013808781764582865, "splink_score_bin_high": -11.86209878610897, "splink_score_bin_low": -12.426626569679978 }, { "binwidth": 0.564527783571009, "count_rows": 305, "freqdensity": 540.2745602185861, "normalised": 0.0020574882453335487, "splink_score_bin_high": -11.29757100253796, "splink_score_bin_low": -11.86209878610897 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -10.733043218966952, "splink_score_bin_low": -11.29757100253796 }, { "binwidth": 0.5645277835710072, "count_rows": 5171, "freqdensity": 9159.868035705958, "normalised": 0.034882858087278076, "splink_score_bin_high": -10.168515435395944, "splink_score_bin_low": -10.733043218966952 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -9.603987651824935, "splink_score_bin_low": -10.168515435395944 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -9.039459868253926, "splink_score_bin_low": -9.603987651824935 }, { "binwidth": 0.5645277835710072, "count_rows": 92, "freqdensity": 162.96806406593467, "normalised": 0.0006206194051497936, "splink_score_bin_high": -8.47493208468292, "splink_score_bin_low": -9.039459868253926 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -7.91040430111191, "splink_score_bin_low": -8.47493208468292 }, { "binwidth": 0.564527783571009, "count_rows": 1683, "freqdensity": 2981.252737206165, "normalised": 0.011353287596381515, "splink_score_bin_high": -7.345876517540901, "splink_score_bin_low": -7.91040430111191 }, { "binwidth": 0.564527783571009, "count_rows": 357, "freqdensity": 632.3869442558532, "normalised": 0.00240827312650517, "splink_score_bin_high": -6.781348733969892, "splink_score_bin_low": -7.345876517540901 }, { "binwidth": 0.564527783571009, "count_rows": 74, "freqdensity": 131.083008053034, "normalised": 0.0004991938693596151, "splink_score_bin_high": -6.216820950398883, "splink_score_bin_low": -6.781348733969892 }, { "binwidth": 0.564527783571009, "count_rows": 64, "freqdensity": 113.36908804586724, "normalised": 0.0004317352383650725, "splink_score_bin_high": -5.652293166827874, "splink_score_bin_low": -6.216820950398883 }, { "binwidth": 0.5645277835710072, "count_rows": 101, "freqdensity": 178.9105920723848, "normalised": 0.0006813321730448821, "splink_score_bin_high": -5.087765383256867, "splink_score_bin_low": -5.652293166827874 }, { "binwidth": 0.564527783571009, "count_rows": 24, "freqdensity": 42.51340801720021, "normalised": 0.00016190071438690218, "splink_score_bin_high": -4.523237599685858, "splink_score_bin_low": -5.087765383256867 }, { "binwidth": 0.564527783571009, "count_rows": 2, "freqdensity": 3.5427840014333514, "normalised": 0.000013491726198908516, "splink_score_bin_high": -3.9587098161148493, "splink_score_bin_low": -4.523237599685858 }, { "binwidth": 0.564527783571009, "count_rows": 3, "freqdensity": 5.314176002150027, "normalised": 0.000020237589298362772, "splink_score_bin_high": -3.3941820325438403, "splink_score_bin_low": -3.9587098161148493 }, { "binwidth": 0.564527783571009, "count_rows": 27, "freqdensity": 47.82758401935024, "normalised": 0.00018213830368526493, "splink_score_bin_high": -2.8296542489728314, "splink_score_bin_low": -3.3941820325438403 }, { "binwidth": 0.5645277835710072, "count_rows": 11, "freqdensity": 19.485312007883493, "normalised": 0.00007420449409399706, "splink_score_bin_high": -2.265126465401824, "splink_score_bin_low": -2.8296542489728314 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -1.7005986818308152, "splink_score_bin_low": -2.265126465401824 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": -1.1360708982598062, "splink_score_bin_low": -1.7005986818308152 }, { "binwidth": 0.564527783571009, "count_rows": 1, "freqdensity": 1.7713920007166757, "normalised": 0.000006745863099454258, "splink_score_bin_high": -0.5715431146887973, "splink_score_bin_low": -1.1360708982598062 }, { "binwidth": 0.564527783571009, "count_rows": 29, "freqdensity": 51.370368020783594, "normalised": 0.00019563002988417347, "splink_score_bin_high": -0.007015331117788293, "splink_score_bin_low": -0.5715431146887973 }, { "binwidth": 0.564527783571009, "count_rows": 12, "freqdensity": 21.256704008600106, "normalised": 0.00008095035719345109, "splink_score_bin_high": 0.5575124524532207, "splink_score_bin_low": -0.007015331117788293 }, { "binwidth": 0.5645277835710054, "count_rows": 2, "freqdensity": 3.5427840014333736, "normalised": 0.000013491726198908599, "splink_score_bin_high": 1.122040236024226, "splink_score_bin_low": 0.5575124524532207 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 1.686568019595235, "splink_score_bin_low": 1.122040236024226 }, { "binwidth": 0.564527783571009, "count_rows": 7, "freqdensity": 12.399744005016728, "normalised": 0.0000472210416961798, "splink_score_bin_high": 2.251095803166244, "splink_score_bin_low": 1.686568019595235 }, { "binwidth": 0.564527783571009, "count_rows": 8, "freqdensity": 14.171136005733405, "normalised": 0.000053966904795634064, "splink_score_bin_high": 2.815623586737253, "splink_score_bin_low": 2.251095803166244 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 3.380151370308262, "splink_score_bin_low": 2.815623586737253 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 3.944679153879271, "splink_score_bin_low": 3.380151370308262 }, { "binwidth": 0.564527783571009, "count_rows": 7, "freqdensity": 12.399744005016728, "normalised": 0.0000472210416961798, "splink_score_bin_high": 4.50920693745028, "splink_score_bin_low": 3.944679153879271 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 5.073734721021289, "splink_score_bin_low": 4.50920693745028 }, { "binwidth": 0.564527783571009, "count_rows": 42, "freqdensity": 74.39846403010037, "normalised": 0.0002833262501770788, "splink_score_bin_high": 5.638262504592298, "splink_score_bin_low": 5.073734721021289 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 6.202790288163307, "splink_score_bin_low": 5.638262504592298 }, { "binwidth": 0.564527783571009, "count_rows": 5, "freqdensity": 8.856960003583378, "normalised": 0.00003372931549727129, "splink_score_bin_high": 6.767318071734316, "splink_score_bin_low": 6.202790288163307 }, { "binwidth": 0.564527783571009, "count_rows": 21, "freqdensity": 37.199232015050185, "normalised": 0.0001416631250885394, "splink_score_bin_high": 7.331845855305325, "splink_score_bin_low": 6.767318071734316 }, { "binwidth": 0.5645277835710054, "count_rows": 31, "freqdensity": 54.91315202221729, "normalised": 0.0002091217560830833, "splink_score_bin_high": 7.89637363887633, "splink_score_bin_low": 7.331845855305325 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 8.46090142244734, "splink_score_bin_low": 7.89637363887633 }, { "binwidth": 0.564527783571009, "count_rows": 1, "freqdensity": 1.7713920007166757, "normalised": 0.000006745863099454258, "splink_score_bin_high": 9.025429206018348, "splink_score_bin_low": 8.46090142244734 }, { "binwidth": 0.564527783571009, "count_rows": 3, "freqdensity": 5.314176002150027, "normalised": 0.000020237589298362772, "splink_score_bin_high": 9.589956989589357, "splink_score_bin_low": 9.025429206018348 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 10.154484773160366, "splink_score_bin_low": 9.589956989589357 }, { "binwidth": 0.564527783571009, "count_rows": 8, "freqdensity": 14.171136005733405, "normalised": 0.000053966904795634064, "splink_score_bin_high": 10.719012556731375, "splink_score_bin_low": 10.154484773160366 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 11.283540340302384, "splink_score_bin_low": 10.719012556731375 }, { "binwidth": 0.564527783571009, "count_rows": 2, "freqdensity": 3.5427840014333514, "normalised": 0.000013491726198908516, "splink_score_bin_high": 11.848068123873393, "splink_score_bin_low": 11.283540340302384 }, { "binwidth": 0.564527783571009, "count_rows": 7, "freqdensity": 12.399744005016728, "normalised": 0.0000472210416961798, "splink_score_bin_high": 12.412595907444402, "splink_score_bin_low": 11.848068123873393 }, { "binwidth": 0.564527783571009, "count_rows": 18, "freqdensity": 31.88505601290016, "normalised": 0.00012142553579017663, "splink_score_bin_high": 12.97712369101541, "splink_score_bin_low": 12.412595907444402 }, { "binwidth": 0.5645277835710054, "count_rows": 5, "freqdensity": 8.856960003583433, "normalised": 0.0000337293154972715, "splink_score_bin_high": 13.541651474586416, "splink_score_bin_low": 12.97712369101541 }, { "binwidth": 0.564527783571009, "count_rows": 27, "freqdensity": 47.82758401935024, "normalised": 0.00018213830368526493, "splink_score_bin_high": 14.106179258157425, "splink_score_bin_low": 13.541651474586416 }, { "binwidth": 0.564527783571009, "count_rows": 11, "freqdensity": 19.485312007883433, "normalised": 0.00007420449409399683, "splink_score_bin_high": 14.670707041728434, "splink_score_bin_low": 14.106179258157425 }, { "binwidth": 0.564527783571009, "count_rows": 27, "freqdensity": 47.82758401935024, "normalised": 0.00018213830368526493, "splink_score_bin_high": 15.235234825299443, "splink_score_bin_low": 14.670707041728434 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 15.799762608870452, "splink_score_bin_low": 15.235234825299443 }, { "binwidth": 0.564527783571009, "count_rows": 8, "freqdensity": 14.171136005733405, "normalised": 0.000053966904795634064, "splink_score_bin_high": 16.36429039244146, "splink_score_bin_low": 15.799762608870452 }, { "binwidth": 0.564527783571009, "count_rows": 3, "freqdensity": 5.314176002150027, "normalised": 0.000020237589298362772, "splink_score_bin_high": 16.92881817601247, "splink_score_bin_low": 16.36429039244146 }, { "binwidth": 0.564527783571009, "count_rows": 14, "freqdensity": 24.799488010033457, "normalised": 0.0000944420833923596, "splink_score_bin_high": 17.49334595958348, "splink_score_bin_low": 16.92881817601247 }, { "binwidth": 0.564527783571009, "count_rows": 17, "freqdensity": 30.113664012183484, "normalised": 0.00011467967269072237, "splink_score_bin_high": 18.057873743154488, "splink_score_bin_low": 17.49334595958348 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 18.622401526725497, "splink_score_bin_low": 18.057873743154488 }, { "binwidth": 0.564527783571009, "count_rows": 10, "freqdensity": 17.713920007166756, "normalised": 0.00006745863099454258, "splink_score_bin_high": 19.186929310296506, "splink_score_bin_low": 18.622401526725497 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 19.751457093867515, "splink_score_bin_low": 19.186929310296506 }, { "binwidth": 0.5645277835710019, "count_rows": 20, "freqdensity": 35.42784001433396, "normalised": 0.00013491726198908686, "splink_score_bin_high": 20.315984877438517, "splink_score_bin_low": 19.751457093867515 }, { "binwidth": 0.564527783571009, "count_rows": 69, "freqdensity": 122.22604804945061, "normalised": 0.00046546455386234375, "splink_score_bin_high": 20.880512661009526, "splink_score_bin_low": 20.315984877438517 }, { "binwidth": 0.564527783571009, "count_rows": 18, "freqdensity": 31.88505601290016, "normalised": 0.00012142553579017663, "splink_score_bin_high": 21.445040444580535, "splink_score_bin_low": 20.880512661009526 }, { "binwidth": 0.564527783571009, "count_rows": 12, "freqdensity": 21.256704008600106, "normalised": 0.00008095035719345109, "splink_score_bin_high": 22.009568228151544, "splink_score_bin_low": 21.445040444580535 }, { "binwidth": 0.564527783571009, "count_rows": 38, "freqdensity": 67.31289602723368, "normalised": 0.00025634279777926177, "splink_score_bin_high": 22.574096011722553, "splink_score_bin_low": 22.009568228151544 }, { "binwidth": 0.564527783571009, "count_rows": 34, "freqdensity": 60.22732802436697, "normalised": 0.00022935934538144474, "splink_score_bin_high": 23.13862379529356, "splink_score_bin_low": 22.574096011722553 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 23.70315157886457, "splink_score_bin_low": 23.13862379529356 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 24.26767936243558, "splink_score_bin_low": 23.70315157886457 }, { "binwidth": 0.564527783571009, "count_rows": 1, "freqdensity": 1.7713920007166757, "normalised": 0.000006745863099454258, "splink_score_bin_high": 24.83220714600659, "splink_score_bin_low": 24.26767936243558 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 25.396734929577597, "splink_score_bin_low": 24.83220714600659 }, { "binwidth": 0.564527783571009, "count_rows": 6, "freqdensity": 10.628352004300053, "normalised": 0.000040475178596725545, "splink_score_bin_high": 25.961262713148606, "splink_score_bin_low": 25.396734929577597 }, { "binwidth": 0.564527783571009, "count_rows": 8, "freqdensity": 14.171136005733405, "normalised": 0.000053966904795634064, "splink_score_bin_high": 26.525790496719615, "splink_score_bin_low": 25.961262713148606 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 27.090318280290624, "splink_score_bin_low": 26.525790496719615 }, { "binwidth": 0.564527783571009, "count_rows": 3, "freqdensity": 5.314176002150027, "normalised": 0.000020237589298362772, "splink_score_bin_high": 27.654846063861633, "splink_score_bin_low": 27.090318280290624 }, { "binwidth": 0.564527783571009, "count_rows": 31, "freqdensity": 54.91315202221694, "normalised": 0.00020912175608308196, "splink_score_bin_high": 28.219373847432642, "splink_score_bin_low": 27.654846063861633 }, { "binwidth": 0.564527783571009, "count_rows": 22, "freqdensity": 38.970624015766866, "normalised": 0.00014840898818799367, "splink_score_bin_high": 28.78390163100365, "splink_score_bin_low": 28.219373847432642 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 29.34842941457466, "splink_score_bin_low": 28.78390163100365 }, { "binwidth": 0.564527783571009, "count_rows": 21, "freqdensity": 37.199232015050185, "normalised": 0.0001416631250885394, "splink_score_bin_high": 29.91295719814567, "splink_score_bin_low": 29.34842941457466 }, { "binwidth": 0.564527783571009, "count_rows": 13, "freqdensity": 23.028096009316783, "normalised": 0.00008769622029290535, "splink_score_bin_high": 30.477484981716678, "splink_score_bin_low": 29.91295719814567 }, { "binwidth": 0.564527783571009, "count_rows": 12, "freqdensity": 21.256704008600106, "normalised": 0.00008095035719345109, "splink_score_bin_high": 31.042012765287687, "splink_score_bin_low": 30.477484981716678 }, { "binwidth": 0.564527783571009, "count_rows": 1, "freqdensity": 1.7713920007166757, "normalised": 0.000006745863099454258, "splink_score_bin_high": 31.606540548858696, "splink_score_bin_low": 31.042012765287687 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 32.171068332429705, "splink_score_bin_low": 31.606540548858696 }, { "binwidth": 0.564527783571009, "count_rows": 12, "freqdensity": 21.256704008600106, "normalised": 0.00008095035719345109, "splink_score_bin_high": 32.735596116000714, "splink_score_bin_low": 32.171068332429705 }, { "binwidth": 0.564527783571009, "count_rows": 6, "freqdensity": 10.628352004300053, "normalised": 0.000040475178596725545, "splink_score_bin_high": 33.30012389957172, "splink_score_bin_low": 32.735596116000714 }, { "binwidth": 0.564527783571009, "count_rows": 0, "freqdensity": 0, "normalised": 0, "splink_score_bin_high": 33.86465168314273, "splink_score_bin_low": 33.30012389957172 }, { "binwidth": 0.5645277835710019, "count_rows": 3, "freqdensity": 5.314176002150093, "normalised": 0.000020237589298363026, "splink_score_bin_high": 34.429179466713734, "splink_score_bin_low": 33.86465168314273 }, { "binwidth": 0.564527783571009, "count_rows": 2, "freqdensity": 3.5427840014333514, "normalised": 0.000013491726198908516, "splink_score_bin_high": 34.99370725028474, "splink_score_bin_low": 34.429179466713734 }, { "binwidth": 0.564527783571009, "count_rows": 9, "freqdensity": 15.94252800645008, "normalised": 0.000060712767895088313, "splink_score_bin_high": 35.55823503385575, "splink_score_bin_low": 34.99370725028474 }, { "binwidth": 0.564527783571009, "count_rows": 33, "freqdensity": 58.455936023650295, "normalised": 0.0002226134822819905, "splink_score_bin_high": 36.12276281742676, "splink_score_bin_low": 35.55823503385575 }, { "binwidth": 0.564527783571009, "count_rows": 6, "freqdensity": 10.628352004300053, "normalised": 0.000040475178596725545, "splink_score_bin_high": 36.68729060099777, "splink_score_bin_low": 36.12276281742676 }, { "binwidth": 0.564527783571009, "count_rows": 11, "freqdensity": 19.485312007883433, "normalised": 0.00007420449409399683, "splink_score_bin_high": 37.25181838456878, "splink_score_bin_low": 36.68729060099777 }, { "binwidth": 0.564527783571009, "count_rows": 16, "freqdensity": 28.34227201146681, "normalised": 0.00010793380959126813, "splink_score_bin_high": 37.81634616813979, "splink_score_bin_low": 37.25181838456878 } ] }, "encoding": { "tooltip": [ { "field": "count_rows", "title": "count", "type": "quantitative" } ], "x": { "axis": { "title": "splink score" }, "bin": "binned", "field": "splink_score_bin_low", "type": "quantitative" }, "x2": { "field": "splink_score_bin_high" }, "y": { "axis": { "title": "probability density" }, "field": "normalised", "type": "quantitative" } }, "height": 200, "mark": "bar", "selection": { "selector076": { "bind": "scales", "encodings": [ "y" ], "type": "interval" } }, "title": { "subtitle": "Use mousewheeel to zoom", "text": "Histogram of splink scores" }, "width": 700 }, "text/plain": [ "\n", "\n", "If you see this message, it means the renderer has not been properly enabled\n", "for the frontend that you are using. For more information, see\n", "https://altair-viz.github.io/user_guide/troubleshooting.html\n" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from splink.diagnostics import splink_score_histogram\n", "splink_score_histogram(df_e, spark, 100, \"match_weight\", symmetric=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }