{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "96ec678e-b20c-4213-8616-542010f46342", "metadata": {}, "source": [ "# Dirty ER\n", "\n", "---\n", "\n", "In this notebook we present the pyJedAI approach in the well-known ABT-BUY dataset. Dirty ER, is the process of dedeplication of one set." ] }, { "cell_type": "markdown", "id": "5274855c-ba95-49b1-ba68-4f50ca2bbd89", "metadata": {}, "source": [ "# Instalation\n", "\n", "pyJedAI is an open-source library that can be installed from PyPI.\n", "\n", "For more: [pypi.org/project/pyjedai/](https://pypi.org/project/pyjedai/)" ] }, { "cell_type": "code", "execution_count": null, "id": "4697d149-c1a4-4767-9ed1-14444485e409", "metadata": {}, "outputs": [], "source": [ "!python --version" ] }, { "cell_type": "code", "execution_count": null, "id": "776843a2-570d-4d87-bb1d-b5b61f07b1da", "metadata": {}, "outputs": [], "source": [ "!pip install pyjedai -U" ] }, { "cell_type": "code", "execution_count": 2, "id": "6d2e5cf7-ff2e-4271-9242-fe3d638263e9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: pyjedai\n", "Version: 0.0.5\n", "Summary: An open-source library that builds powerful end-to-end Entity Resolution workflows.\n", "Home-page: \n", "Author: \n", "Author-email: Konstantinos Nikoletos , George Papadakis \n", "License: Apache Software License 2.0\n", "Location: c:\\users\\nikol\\anaconda3\\lib\\site-packages\n", "Requires: strsim, seaborn, matplotlib, optuna, networkx, faiss-cpu, scipy, rdflib, strsimpy, rdfpandas, regex, sentence-transformers, tqdm, nltk, pandas, pandocfilters, matplotlib-inline, numpy, pandas-profiling, PyYAML, transformers, tomli, gensim\n", "Required-by: \n" ] } ], "source": [ "!pip show pyjedai" ] }, { "cell_type": "markdown", "id": "15d28272-269a-4e87-bb03-0a45a5492a06", "metadata": {}, "source": [ "Imports" ] }, { "cell_type": "code", "execution_count": 5, "id": "a0890ce6-3a10-4e66-913f-78095bd786a1", "metadata": {}, "outputs": [], "source": [ "import os\n", "import sys\n", "import pandas as pd\n", "import networkx\n", "from networkx import draw, Graph\n", "\n", "from pyjedai.utils import print_clusters, print_blocks, print_candidate_pairs\n", "from pyjedai.evaluation import Evaluation" ] }, { "cell_type": "markdown", "id": "af77914f-5e76-4da8-a0ad-1c53e0111a0f", "metadata": { "tags": [] }, "source": [ "# Reading the dataset - Dirty ER example\n", "\n", "pyJedAI in order to perfrom needs only the tranformation of the initial data into a pandas DataFrame. Hence, pyJedAI can function in every structured or semi-structured data. In this case Abt-Buy dataset is provided as .csv files. \n", "\n", "
\n", " \n", "
\n", "\n", "\n", "## pyjedai module\n", "\n", "Data module offers a numpber of options\n", "- Selecting the parameters (columns) of the dataframe, in D1 (and in D2)\n", "- Prints a detailed text analysis\n", "- Stores a hidden mapping of the ids, and creates it if not exists." ] }, { "cell_type": "code", "execution_count": 7, "id": "3d3feb89-1406-4c90-a1aa-dc2cf4707739", "metadata": {}, "outputs": [], "source": [ "from pyjedai.datamodel import Data\n", "\n", "d1 = pd.read_csv(\"./../data/der/cora/cora.csv\", sep='|')\n", "gt = pd.read_csv(\"./../data/der/cora/cora_gt.csv\", sep='|', header=None)\n", "attr = ['Entity Id','author', 'title']" ] }, { "cell_type": "markdown", "id": "fda32323-c74d-4374-b322-5c11a175c3ea", "metadata": {}, "source": [ "Data is the connecting module of all steps of the workflow" ] }, { "cell_type": "code", "execution_count": 9, "id": "e257597d-ea77-4090-ba34-e1038d8f9a0d", "metadata": {}, "outputs": [], "source": [ "data = Data(\n", " dataset_1=d1,\n", " id_column_name_1='Entity Id',\n", " ground_truth=gt,\n", " attributes_1=attr\n", ")" ] }, { "cell_type": "markdown", "id": "93464edd-b88a-40d9-aa4d-7fe1523db662", "metadata": {}, "source": [ "# Workflow with Block Cleaning Methods\n", "\n", "In this notebook we created the bellow architecture:\n", "\n", "![workflow1-cora.png](https://github.com/AI-team-UoA/pyJedAI/blob/main/documentation/workflow1-cora.png?raw=true)\n", "\n" ] }, { "cell_type": "markdown", "id": "9c068252-4a69-405a-a320-c2875ec08ea5", "metadata": {}, "source": [ "## Block Building\n", "\n", "It clusters entities into overlapping blocks in a lazy manner that relies on unsupervised blocking keys: every token in an attribute value forms a key. Blocks are then extracted, possibly using a transformation, based on its equality or on its similarity with other keys.\n", "\n", "The following methods are currently supported:\n", "\n", "- Standard/Token Blocking\n", "- Sorted Neighborhood\n", "- Extended Sorted Neighborhood\n", "- Q-Grams Blocking\n", "- Extended Q-Grams Blocking\n", "- Suffix Arrays Blocking\n", "- Extended Suffix Arrays Blocking" ] }, { "cell_type": "code", "execution_count": 10, "id": "9c1b6213-a218-40cf-bc72-801b77d28da9", "metadata": {}, "outputs": [], "source": [ "from pyjedai.block_building import (\n", " StandardBlocking,\n", " QGramsBlocking,\n", " SuffixArraysBlocking,\n", " ExtendedSuffixArraysBlocking,\n", " ExtendedQGramsBlocking\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "id": "7ee34038-1352-440e-8c34-98c5cf036523", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1b3d9aac0ca94663b04832c034e8c18a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Suffix Arrays Blocking: 0%| | 0/1295 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw(pairs_graph)" ] }, { "cell_type": "code", "execution_count": 34, "id": "00bc2e82-9bc1-4119-b8cb-4a1c18afee19", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "***************************************************************************************************************************\n", " Μethod: Entity Matching\n", "***************************************************************************************************************************\n", "Method name: Entity Matching\n", "Parameters: \n", "\tTokenizer: white_space_tokenizer\n", "\tMetric: jaccard\n", "\tSimilarity Threshold: 0.5\n", "Runtime: 0.0726 seconds\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n", "Performance:\n", "\tPrecision: 97.67% \n", "\tRecall: 0.24%\n", "\tF1-score: 0.49%\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n" ] } ], "source": [ "_ = EM.evaluate(pairs_graph)" ] }, { "cell_type": "markdown", "id": "93b72120-4578-4d5c-a408-a24ee78bf6cb", "metadata": {}, "source": [ "## Entity Clustering\n", "\n", "It takes as input the similarity graph produced by Entity Matching and partitions it into a set of equivalence clusters, with every cluster corresponding to a distinct real-world object." ] }, { "cell_type": "code", "execution_count": 39, "id": "500d2ef7-7017-4dba-bbea-acdba8abf5b7", "metadata": {}, "outputs": [], "source": [ "from pyjedai.clustering import ConnectedComponentsClustering" ] }, { "cell_type": "code", "execution_count": 40, "id": "aebd9329-3a4b-48c9-bd05-c7bd4aed3ca9", "metadata": {}, "outputs": [], "source": [ "ec = ConnectedComponentsClustering()\n", "clusters = ec.process(pairs_graph, data)" ] }, { "cell_type": "code", "execution_count": 43, "id": "3d2aa574", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "***************************************************************************************************************************\n", " Μethod: Connected Components Clustering\n", "***************************************************************************************************************************\n", "Method name: Connected Components Clustering\n", "Parameters: \n", "Runtime: 0.0000 seconds\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n", "Performance:\n", "\tPrecision: 97.92% \n", "\tRecall: 0.27%\n", "\tF1-score: 0.55%\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n" ] } ], "source": [ "_ = ec.evaluate(clusters)" ] }, { "cell_type": "markdown", "id": "6e44642b-b0d9-4f8d-9fe4-4fca2ad716aa", "metadata": {}, "source": [ "# Workflow with Similarity Joins\n", "\n", "In this notebook we created the bellow archtecture:\n", "\n", "![workflow2-cora.png](https://github.com/AI-team-UoA/pyJedAI/blob/main/documentation/workflow2-cora.png?raw=true)\n", "\n" ] }, { "cell_type": "markdown", "id": "2b07ea13-58cb-498a-949a-4702ea9ee4ce", "metadata": { "tags": [] }, "source": [ "## Data Reading" ] }, { "cell_type": "markdown", "id": "a0b7acf5-32b3-45b1-8347-197dfa869fb9", "metadata": {}, "source": [ "Data is the connecting module of all steps of the workflow" ] }, { "cell_type": "code", "execution_count": 44, "id": "3006b051-8348-4922-a627-56441a1db7b7", "metadata": { "tags": [] }, "outputs": [], "source": [ "from pyjedai.datamodel import Data\n", "d1 = pd.read_csv(\"./../data/der/cora/cora.csv\", sep='|')\n", "gt = pd.read_csv(\"./../data/der/cora/cora_gt.csv\", sep='|', header=None)\n", "attr = ['Entity Id','author', 'title']\n", "data = Data(\n", " dataset_1=d1,\n", " id_column_name_1='Entity Id',\n", " ground_truth=gt,\n", " attributes_1=attr\n", ")" ] }, { "cell_type": "markdown", "id": "b3eedb4c-b86f-4f98-abf2-9d6f7d0271c5", "metadata": {}, "source": [ "## Similarity Joins" ] }, { "cell_type": "code", "execution_count": 45, "id": "afd97b7e-4bf8-4256-b9fc-a301e413e834", "metadata": {}, "outputs": [], "source": [ "from pyjedai.joins import ΕJoin, TopKJoin" ] }, { "cell_type": "code", "execution_count": 47, "id": "7bc8ba43-b059-4839-8958-0b31fab95e46", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d86c40bfd439456a8cce134c648a89f2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "EJoin (jaccard): 0%| | 0/2590 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw(g)" ] }, { "cell_type": "code", "execution_count": 51, "id": "a4c57244-0c98-4598-8ae2-06cdd0c48385", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "***************************************************************************************************************************\n", " Μethod: Top-K Join\n", "***************************************************************************************************************************\n", "Method name: Top-K Join\n", "Parameters: \n", "\tsimilarity_threshold: 0.25547445255474455\n", "\tK: 20\n", "\tmetric: jaccard\n", "\ttokenization: qgrams\n", "\tqgrams: 3\n", "Runtime: 41.0031 seconds\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n", "Performance:\n", "\tPrecision: 58.34% \n", "\tRecall: 63.75%\n", "\tF1-score: 60.92%\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n" ] }, { "data": { "text/plain": [ "{'Precision %': 58.340434597358325,\n", " 'Recall %': 63.74534450651769,\n", " 'F1 %': 60.923248053392655,\n", " 'True Positives': 10954,\n", " 'False Positives': 7822,\n", " 'True Negatives': 823813.0,\n", " 'False Negatives': 6230}" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "topk_join.evaluate(g)" ] }, { "cell_type": "markdown", "id": "e686ba55-cbbe-4133-8256-fa5339e70720", "metadata": {}, "source": [ "## Entity Clustering" ] }, { "cell_type": "code", "execution_count": 52, "id": "e48627ef-0be6-45dd-b3ba-863818eadbfb", "metadata": {}, "outputs": [], "source": [ "from pyjedai.clustering import ConnectedComponentsClustering" ] }, { "cell_type": "code", "execution_count": 54, "id": "a9d5a28d-79f0-479f-b154-f5f7e3661897", "metadata": {}, "outputs": [], "source": [ "ccc = ConnectedComponentsClustering()\n", "\n", "clusters = ccc.process(g, data)" ] }, { "cell_type": "code", "execution_count": 56, "id": "f95539fe-2569-4569-8929-2f4220f14157", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "***************************************************************************************************************************\n", " Μethod: Connected Components Clustering\n", "***************************************************************************************************************************\n", "Method name: Connected Components Clustering\n", "Parameters: \n", "Runtime: 0.0630 seconds\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n", "Performance:\n", "\tPrecision: 2.05% \n", "\tRecall: 100.00%\n", "\tF1-score: 4.02%\n", "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n" ] } ], "source": [ "_ = ccc.evaluate(clusters)" ] }, { "cell_type": "markdown", "id": "778b9b19-0964-4095-a293-a8336fe0607a", "metadata": {}, "source": [ "
\n", "
\n", "K. Nikoletos, G. Papadakis & M. Koubarakis\n", "
\n", "
\n", "Apache License 2.0\n", "
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }