{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "96ec678e-b20c-4213-8616-542010f46342",
   "metadata": {},
   "source": [
    "<div align=\"center\"> \n",
    "    <br><br>\n",
    "    <img src=\"https://raw.githubusercontent.com/AI-team-UoA/pyJedAI/main/documentation/pyjedai.logo.drawio.png?raw=true\" alt=\"drawing\" width=\"400\"/>\n",
    "    <br>\n",
    "    <hr>\n",
    "    <font size=\"3\">Clean-Clean Entity Resolution Tutorial</font>\n",
    "    <br>\n",
    "</div>\n",
    "<hr>\n",
    "\n",
    "\n",
    "In this notebook we present the pyJedAI approach in the well-known ABT-BUY dataset. Clean-Clean ER in the link discovery/deduplication between two sets of entities."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c49d2b7-11b5-40b3-9341-de98608dde13",
   "metadata": {},
   "source": [
    "Dataset: __Abt-Buy dataset__\n",
    "\n",
    "The Abt-Buy dataset for entity resolution derives from the online retailers Abt.com and Buy.com. The dataset contains 1076 entities from abt.com and 1076 entities from buy.com as well as a gold standard (perfect mapping) with 1076 matching record pairs between the two data sources. The common attributes between the two data sources are: product name, product description and product price."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744b3017-9a5c-4d3c-8e0a-fe39b069b647",
   "metadata": {},
   "source": [
    "# Instalation\n",
    "\n",
    "pyJedAI is an open-source library that can be installed from PyPI.\n",
    "\n",
    "For more: [pypi.org/project/pyjedai/](https://pypi.org/project/pyjedai/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "029a5825-799d-4c3f-a6cd-a75e257cadcc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyjedai in c:\\users\\nikol\\anaconda3\\lib\\site-packages (0.0.5)\n",
      "Requirement already satisfied: strsim>=0.0.3 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (0.0.3)\n",
      "Requirement already satisfied: regex>=2022.6.2 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (2022.6.2)\n",
      "Requirement already satisfied: pandocfilters>=1.5 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (1.5.0)\n",
      "Requirement already satisfied: strsimpy>=0.2.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (0.2.1)\n",
      "Requirement already satisfied: seaborn>=0.11 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (0.11.2)\n",
      "Requirement already satisfied: gensim>=4.2.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (4.2.0)\n",
      "Requirement already satisfied: matplotlib>=3.1.3 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (3.5.3)\n",
      "Requirement already satisfied: matplotlib-inline>=0.1.3 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (0.1.6)\n",
      "Requirement already satisfied: numpy>=1.21 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (1.21.2)\n",
      "Requirement already satisfied: faiss-cpu>=1.7 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (1.7.2)\n",
      "Requirement already satisfied: transformers>=4.21 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (4.21.3)\n",
      "Requirement already satisfied: nltk>=3.7 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (3.7)\n",
      "Requirement already satisfied: scipy>=1.7 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (1.7.1)\n",
      "Requirement already satisfied: pandas>=0.25.3 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (1.3.4)\n",
      "Requirement already satisfied: tomli in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (2.0.1)\n",
      "Requirement already satisfied: tqdm>=4.64 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (4.64.0)\n",
      "Requirement already satisfied: optuna>=3.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (3.0.1)\n",
      "Requirement already satisfied: rdfpandas>=1.1.5 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (1.1.5)\n",
      "Requirement already satisfied: networkx>=2.3 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (2.6.3)\n",
      "Requirement already satisfied: PyYAML>=6.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (6.0)\n",
      "Requirement already satisfied: pandas-profiling>=3.2 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (3.2.0)\n",
      "Requirement already satisfied: rdflib>=6.1.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (6.1.1)\n",
      "Requirement already satisfied: sentence-transformers>=2.2 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pyjedai) (2.2.2)\n",
      "Requirement already satisfied: Cython==0.29.28 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from gensim>=4.2.0->pyjedai) (0.29.28)\n",
      "Requirement already satisfied: smart-open>=1.8.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from gensim>=4.2.0->pyjedai) (5.1.0)\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (3.0.4)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (2.8.2)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (21.3)\n",
      "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (8.4.0)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (0.10.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (4.25.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib>=3.1.3->pyjedai) (1.3.1)\n",
      "Requirement already satisfied: six in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cycler>=0.10->matplotlib>=3.1.3->pyjedai) (1.16.0)\n",
      "Requirement already satisfied: traitlets in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from matplotlib-inline>=0.1.3->pyjedai) (5.1.1)\n",
      "Requirement already satisfied: click in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from nltk>=3.7->pyjedai) (8.0.3)\n",
      "Requirement already satisfied: joblib in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from nltk>=3.7->pyjedai) (1.1.0)\n",
      "Requirement already satisfied: typing-extensions>=3.10.0.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from optuna>=3.0->pyjedai) (4.3.0)\n",
      "Requirement already satisfied: colorlog in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from optuna>=3.0->pyjedai) (6.6.0)\n",
      "Requirement already satisfied: cmaes>=0.8.2 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from optuna>=3.0->pyjedai) (0.8.2)\n",
      "Requirement already satisfied: sqlalchemy>=1.1.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from optuna>=3.0->pyjedai) (1.4.22)\n",
      "Requirement already satisfied: cliff in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from optuna>=3.0->pyjedai) (3.10.0)\n",
      "Requirement already satisfied: alembic in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from optuna>=3.0->pyjedai) (1.7.5)\n",
      "Requirement already satisfied: pytz>=2017.3 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas>=0.25.3->pyjedai) (2021.3)\n",
      "Requirement already satisfied: pydantic>=1.8.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (1.10.2)\n",
      "Requirement already satisfied: htmlmin>=0.1.12 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (0.1.12)\n",
      "Requirement already satisfied: visions[type_image_path]==0.7.4 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (0.7.4)\n",
      "Requirement already satisfied: tangled-up-in-unicode==0.2.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (0.2.0)\n",
      "Requirement already satisfied: jinja2>=2.11.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (3.0.2)\n",
      "Requirement already satisfied: requests>=2.24.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (2.26.0)\n",
      "Requirement already satisfied: multimethod>=1.4 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (1.9)\n",
      "Requirement already satisfied: missingno>=0.4.2 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (0.5.1)\n",
      "Requirement already satisfied: phik>=0.11.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (0.12.2)\n",
      "Requirement already satisfied: markupsafe~=2.1.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from pandas-profiling>=3.2->pyjedai) (2.1.1)\n",
      "Requirement already satisfied: attrs>=19.3.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from visions[type_image_path]==0.7.4->pandas-profiling>=3.2->pyjedai) (21.2.0)\n",
      "Requirement already satisfied: imagehash in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from visions[type_image_path]==0.7.4->pandas-profiling>=3.2->pyjedai) (4.3.1)\n",
      "Requirement already satisfied: importlib-metadata in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from rdflib>=6.1.1->pyjedai) (4.8.1)\n",
      "Requirement already satisfied: setuptools in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from rdflib>=6.1.1->pyjedai) (58.0.4)\n",
      "Requirement already satisfied: isodate in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from rdflib>=6.1.1->pyjedai) (0.6.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from requests>=2.24.0->pandas-profiling>=3.2->pyjedai) (2022.9.14)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from requests>=2.24.0->pandas-profiling>=3.2->pyjedai) (1.26.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from requests>=2.24.0->pandas-profiling>=3.2->pyjedai) (3.3)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from requests>=2.24.0->pandas-profiling>=3.2->pyjedai) (2.0.4)\n",
      "Requirement already satisfied: torch>=1.6.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from sentence-transformers>=2.2->pyjedai) (1.12.1)\n",
      "Requirement already satisfied: scikit-learn in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from sentence-transformers>=2.2->pyjedai) (1.0.1)\n",
      "Requirement already satisfied: sentencepiece in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from sentence-transformers>=2.2->pyjedai) (0.1.97)\n",
      "Requirement already satisfied: torchvision in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from sentence-transformers>=2.2->pyjedai) (0.13.1)\n",
      "Requirement already satisfied: huggingface-hub>=0.4.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from sentence-transformers>=2.2->pyjedai) (0.10.0)\n",
      "Requirement already satisfied: filelock in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from huggingface-hub>=0.4.0->sentence-transformers>=2.2->pyjedai) (3.3.1)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from sqlalchemy>=1.1.0->optuna>=3.0->pyjedai) (1.1.1)\n",
      "Requirement already satisfied: colorama in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from tqdm>=4.64->pyjedai) (0.4.4)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from transformers>=4.21->pyjedai) (0.12.1)\n",
      "Requirement already satisfied: Mako in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from alembic->optuna>=3.0->pyjedai) (1.1.4)\n",
      "Requirement already satisfied: importlib-resources in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from alembic->optuna>=3.0->pyjedai) (5.4.0)\n",
      "Requirement already satisfied: stevedore>=2.0.1 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cliff->optuna>=3.0->pyjedai) (3.5.0)\n",
      "Requirement already satisfied: autopage>=0.4.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cliff->optuna>=3.0->pyjedai) (0.4.0)\n",
      "Requirement already satisfied: cmd2>=1.0.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cliff->optuna>=3.0->pyjedai) (2.3.2)\n",
      "Requirement already satisfied: PrettyTable>=0.7.2 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cliff->optuna>=3.0->pyjedai) (2.4.0)\n",
      "Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cliff->optuna>=3.0->pyjedai) (5.6.0)\n",
      "Requirement already satisfied: wcwidth>=0.1.7 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cmd2>=1.0.0->cliff->optuna>=3.0->pyjedai) (0.2.5)\n",
      "Requirement already satisfied: pyreadline in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cmd2>=1.0.0->cliff->optuna>=3.0->pyjedai) (2.1)\n",
      "Requirement already satisfied: pyperclip>=1.6 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from cmd2>=1.0.0->cliff->optuna>=3.0->pyjedai) (1.8.2)\n",
      "Requirement already satisfied: zipp>=0.5 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from importlib-metadata->rdflib>=6.1.1->pyjedai) (3.6.0)\n",
      "Requirement already satisfied: PyWavelets in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from imagehash->visions[type_image_path]==0.7.4->pandas-profiling>=3.2->pyjedai) (1.1.1)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\users\\nikol\\anaconda3\\lib\\site-packages (from scikit-learn->sentence-transformers>=2.2->pyjedai) (2.2.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install pyjedai -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "462695ec-3af1-4048-9971-9ed0bce0f07b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: pyjedai\n",
      "Version: 0.0.5\n",
      "Summary: An open-source library that builds powerful end-to-end Entity Resolution workflows.\n",
      "Home-page: \n",
      "Author: \n",
      "Author-email: Konstantinos Nikoletos <nikoletos.kon@gmail.com>, George Papadakis <gpapadis84@gmail.com>\n",
      "License: Apache Software License 2.0\n",
      "Location: c:\\users\\nikol\\anaconda3\\lib\\site-packages\n",
      "Requires: PyYAML, optuna, scipy, gensim, pandocfilters, numpy, rdflib, pandas, transformers, regex, strsim, tqdm, networkx, seaborn, rdfpandas, strsimpy, matplotlib-inline, matplotlib, pandas-profiling, tomli, nltk, faiss-cpu, sentence-transformers\n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip show pyjedai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b4c62c5-6581-4d2e-9d44-c7c02f43d441",
   "metadata": {},
   "source": [
    "Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6db50d83-51d8-4c95-9f27-30ef867338f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import pandas as pd\n",
    "import networkx\n",
    "from networkx import draw, Graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4d4e6a90-9fd8-4f7a-bf4f-a5b994e0adfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyjedai\n",
    "from pyjedai.utils import (\n",
    "    text_cleaning_method,\n",
    "    print_clusters,\n",
    "    print_blocks,\n",
    "    print_candidate_pairs\n",
    ")\n",
    "from pyjedai.evaluation import Evaluation, write"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "451bf970-4425-487b-8756-776abb9536ea",
   "metadata": {},
   "source": [
    "# Workflow Architecture\n",
    "\n",
    "![workflow-example.png](https://github.com/AI-team-UoA/pyJedAI/blob/main/documentation/workflow-example.png?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af77914f-5e76-4da8-a0ad-1c53e0111a0f",
   "metadata": {},
   "source": [
    "# Data Reading\n",
    "\n",
    "pyJedAI in order to perfrom needs only the tranformation of the initial data into a pandas DataFrame. Hence, pyJedAI can function in every structured or semi-structured data. In this case Abt-Buy dataset is provided as .csv files. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e6aabec4-ef4f-4267-8c1e-377054e669d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyjedai.datamodel import Data\n",
    "from pyjedai.evaluation import Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3d3feb89-1406-4c90-a1aa-dc2cf4707739",
   "metadata": {},
   "outputs": [],
   "source": [
    "d1 = pd.read_csv(\"./../data/ccer/D2/abt.csv\", sep='|', engine='python', na_filter=False).astype(str)\n",
    "d2 = pd.read_csv(\"./../data/ccer/D2/buy.csv\", sep='|', engine='python', na_filter=False).astype(str)\n",
    "gt = pd.read_csv(\"./../data/ccer/D2/gt.csv\", sep='|', engine='python').astype(str)\n",
    "\n",
    "data = Data(dataset_1=d1,\n",
    "            id_column_name_1='id',\n",
    "            dataset_2=d2,\n",
    "            id_column_name_2='id',\n",
    "            ground_truth=gt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d8a8a78-858e-4c79-90fe-197a68e95e11",
   "metadata": {},
   "source": [
    "pyJedAI offers also dataset analysis methods (more will be developed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7cb87af2-adda-49e0-82cc-b1a5f7a595ef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------- Data -------------------------\n",
      "Type of Entity Resolution:  Clean-Clean\n",
      "Dataset-1:\n",
      "\tNumber of entities:  1076\n",
      "\tNumber of NaN values:  0\n",
      "\tAttributes: \n",
      "\t\t ['name', 'description', 'price']\n",
      "Dataset-2:\n",
      "\tNumber of entities:  1076\n",
      "\tNumber of NaN values:  0\n",
      "\tAttributes: \n",
      "\t\t ['name', 'description', 'price']\n",
      "\n",
      "Total number of entities:  2152\n",
      "Number of matching pairs in ground-truth:  1076\n",
      "-------------------------------------------------------- \n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.print_specs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b822d7c0-19a2-4050-9554-c35a208bb848",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Sony Turntable - PSLX350H</td>\n",
       "      <td>Sony Turntable - PSLX350H/ Belt Drive System/ ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Bose Acoustimass 5 Series III Speaker System -...</td>\n",
       "      <td>Bose Acoustimass 5 Series III Speaker System -...</td>\n",
       "      <td>399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Sony Switcher - SBV40S</td>\n",
       "      <td>Sony Switcher - SBV40S/ Eliminates Disconnecti...</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Sony 5 Disc CD Player - CDPCE375</td>\n",
       "      <td>Sony 5 Disc CD Player- CDPCE375/ 5 Disc Change...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Bose 27028 161 Bookshelf Pair Speakers In Whit...</td>\n",
       "      <td>Bose 161 Bookshelf Speakers In White - 161WH/ ...</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  id                                               name  \\\n",
       "0  0                          Sony Turntable - PSLX350H   \n",
       "1  1  Bose Acoustimass 5 Series III Speaker System -...   \n",
       "2  2                             Sony Switcher - SBV40S   \n",
       "3  3                   Sony 5 Disc CD Player - CDPCE375   \n",
       "4  4  Bose 27028 161 Bookshelf Pair Speakers In Whit...   \n",
       "\n",
       "                                         description price  \n",
       "0  Sony Turntable - PSLX350H/ Belt Drive System/ ...        \n",
       "1  Bose Acoustimass 5 Series III Speaker System -...   399  \n",
       "2  Sony Switcher - SBV40S/ Eliminates Disconnecti...    49  \n",
       "3  Sony 5 Disc CD Player- CDPCE375/ 5 Disc Change...        \n",
       "4  Bose 161 Bookshelf Speakers In White - 161WH/ ...   158  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dataset_1.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5c26b595-5e02-4bfc-8e79-e476ab2830ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Linksys EtherFast EZXS88W Ethernet Switch - EZ...</td>\n",
       "      <td>Linksys EtherFast 8-Port 10/100 Switch (New/Wo...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Linksys EtherFast EZXS55W Ethernet Switch</td>\n",
       "      <td>5 x 10/100Base-TX LAN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Netgear ProSafe FS105 Ethernet Switch - FS105NA</td>\n",
       "      <td>NETGEAR FS105 Prosafe 5 Port 10/100 Desktop Sw...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Belkin Pro Series High Integrity VGA/SVGA Moni...</td>\n",
       "      <td>1 x HD-15 - 1 x HD-15 - 10ft - Beige</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Netgear ProSafe JFS516 Ethernet Switch</td>\n",
       "      <td>Netgear ProSafe 16 Port 10/100 Rackmount Switc...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  id                                               name  \\\n",
       "0  0  Linksys EtherFast EZXS88W Ethernet Switch - EZ...   \n",
       "1  1          Linksys EtherFast EZXS55W Ethernet Switch   \n",
       "2  2    Netgear ProSafe FS105 Ethernet Switch - FS105NA   \n",
       "3  3  Belkin Pro Series High Integrity VGA/SVGA Moni...   \n",
       "4  4             Netgear ProSafe JFS516 Ethernet Switch   \n",
       "\n",
       "                                         description price  \n",
       "0  Linksys EtherFast 8-Port 10/100 Switch (New/Wo...        \n",
       "1                              5 x 10/100Base-TX LAN        \n",
       "2  NETGEAR FS105 Prosafe 5 Port 10/100 Desktop Sw...        \n",
       "3               1 x HD-15 - 1 x HD-15 - 10ft - Beige        \n",
       "4  Netgear ProSafe 16 Port 10/100 Rackmount Switc...        "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dataset_2.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b3c9827e-a08a-47b2-a7f2-6f3f72184a17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>D1</th>\n",
       "      <th>D2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>206</td>\n",
       "      <td>216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>182</td>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    D1   D2\n",
       "0  206  216\n",
       "1   60   46\n",
       "2  182  160"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.ground_truth.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c068252-4a69-405a-a320-c2875ec08ea5",
   "metadata": {},
   "source": [
    "# Block Building\n",
    "\n",
    "It clusters entities into overlapping blocks in a lazy manner that relies on unsupervised blocking keys: every token in an attribute value forms a key. Blocks are then extracted, possibly using a transformation, based on its equality or on its similarity with other keys.\n",
    "\n",
    "The following methods are currently supported:\n",
    "\n",
    "- Standard/Token Blocking\n",
    "- Sorted Neighborhood\n",
    "- Extended Sorted Neighborhood\n",
    "- Q-Grams Blocking\n",
    "- Extended Q-Grams Blocking\n",
    "- Suffix Arrays Blocking\n",
    "- Extended Suffix Arrays Blocking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9c1b6213-a218-40cf-bc72-801b77d28da9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created embeddings directory at: C:\\Users\\nikol\\Desktop\\test\\tutorials\\.embeddings\n"
     ]
    }
   ],
   "source": [
    "from pyjedai.block_building import (\n",
    "    StandardBlocking,\n",
    "    QGramsBlocking,\n",
    "    ExtendedQGramsBlocking,\n",
    "    SuffixArraysBlocking,\n",
    "    ExtendedSuffixArraysBlocking,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9741f0c4-6250-455f-9c88-b8dc61ab7d4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8f45ed2068a488d8987344463a830bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Suffix Arrays Blocking:   0%|          | 0/2152 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qgb = SuffixArraysBlocking()\n",
    "blocks = qgb.build_blocks(data, attributes_1=['name'], attributes_2=['name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d2d9ae46-28fa-4438-87b7-ba901c75bd99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Method name: Suffix Arrays Blocking\n",
      "Method info: Creates one block for every suffix that appears in the attribute value tokens of at least two entities.\n",
      "Parameters: \n",
      "\tSuffix length: 6\n",
      "\tMaximum Block Size: 53\n",
      "Attributes from D1:\n",
      "\tname\n",
      "Attributes from D2:\n",
      "\tname\n",
      "Runtime: 0.2220 seconds\n"
     ]
    }
   ],
   "source": [
    "qgb.report()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b0ac846d-0f13-4b90-b4c8-688054ed7ffe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************************************************************************************************************************\n",
      "                                         Μethod:  Suffix Arrays Blocking\n",
      "***************************************************************************************************************************\n",
      "Method name: Suffix Arrays Blocking\n",
      "Parameters: \n",
      "\tSuffix length: 6\n",
      "\tMaximum Block Size: 53\n",
      "Runtime: 0.2220 seconds\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Performance:\n",
      "\tPrecision:      1.41% \n",
      "\tRecall:        97.03%\n",
      "\tF1-score:       2.78%\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Classification report:\n",
      "\tTrue positives: 1044\n",
      "\tFalse positives: 73021\n",
      "\tTrue negatives: 1084723\n",
      "\tFalse negatives: 32\n",
      "\tTotal comparisons: 74065\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Precision %': 1.4095726726523998,\n",
       " 'Recall %': 97.02602230483272,\n",
       " 'F1 %': 2.7787759013055453,\n",
       " 'True Positives': 1044,\n",
       " 'False Positives': 73021,\n",
       " 'True Negatives': 1084723,\n",
       " 'False Negatives': 32}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_ = qgb.evaluate(blocks, with_classification_report=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f9e77d5-c906-431a-bdc7-68dc9c00cc31",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Block Cleaning\n",
    "\n",
    "___Optional step___\n",
    "\n",
    "Its goal is to clean a set of overlapping blocks from unnecessary comparisons, which can be either redundant (i.e., repeated comparisons that have already been executed in a previously examined block) or superfluous (i.e., comparisons that involve non-matching entities). Its methods operate on the coarse level of individual blocks or entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9c2c0e42-485a-444e-9161-975f30d21a02",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyjedai.block_cleaning import BlockFiltering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bf5c20ac-b16a-484d-82b0-61ecb9e7f3ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8f34559708e407ab3588e7e5d89f8ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Block Filtering:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bf = BlockFiltering(ratio=0.8)\n",
    "filtered_blocks = bf.process(blocks, data, tqdm_disable=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "25fd0be0-91c3-4d0b-b596-c66dccba3c79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************************************************************************************************************************\n",
      "                                         Μethod:  Block Filtering\n",
      "***************************************************************************************************************************\n",
      "Method name: Block Filtering\n",
      "Parameters: \n",
      "\tRatio: 0.8\n",
      "Runtime: 0.0600 seconds\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Performance:\n",
      "\tPrecision:      1.90% \n",
      "\tRecall:        94.42%\n",
      "\tF1-score:       3.72%\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Precision %': 1.8975048558195131,\n",
       " 'Recall %': 94.42379182156134,\n",
       " 'F1 %': 3.7202489930428415,\n",
       " 'True Positives': 1016,\n",
       " 'False Positives': 52528,\n",
       " 'True Negatives': 1105188,\n",
       " 'False Negatives': 60}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_ = bf.evaluate(filtered_blocks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cd12048-bd0c-4571-ba70-488d46afcdd6",
   "metadata": {},
   "source": [
    "# Comparison Cleaning\n",
    "\n",
    "___Optional step___\n",
    "\n",
    "Similar to Block Cleaning, this step aims to clean a set of blocks from both redundant and superfluous comparisons. Unlike Block Cleaning, its methods operate on the finer granularity of individual comparisons.\n",
    "\n",
    "The following methods are currently supported:\n",
    "\n",
    "- Comparison Propagation\n",
    "- Cardinality Edge Pruning (CEP)\n",
    "- Cardinality Node Pruning (CNP)\n",
    "- Weighed Edge Pruning (WEP)\n",
    "- Weighed Node Pruning (WNP)\n",
    "- Reciprocal Cardinality Node Pruning (ReCNP)\n",
    "- Reciprocal Weighed Node Pruning (ReWNP)\n",
    "- BLAST\n",
    "\n",
    "Most of these methods are Meta-blocking techniques. All methods are optional, but competive, in the sense that only one of them can part of an ER workflow. For more details on the functionality of these methods, see here. They can be combined with one of the following weighting schemes:\n",
    "\n",
    "- Aggregate Reciprocal Comparisons Scheme (ARCS)\n",
    "- Common Blocks Scheme (CBS)\n",
    "- Enhanced Common Blocks Scheme (ECBS)\n",
    "- Jaccard Scheme (JS)\n",
    "- Enhanced Jaccard Scheme (EJS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "725426e2-0af8-4295-baff-92653c841fdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyjedai.block_cleaning import BlockPurging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7997b2b6-9629-44f0-a66d-5bc4fea28fb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e64c79e8404844d391e7e1cda359fe50",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Block Purging:   0%|          | 0/4680 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cbbp = BlockPurging()\n",
    "cleaned_blocks = cbbp.process(filtered_blocks, data, tqdm_disable=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d8842b00-8765-449f-bdb7-f9b2206e91c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Method name: Block Purging\n",
      "Method info: Discards the blocks exceeding a certain number of comparisons.\n",
      "Parameters: \n",
      "\tSmoothing factor: 1.025\n",
      "\tMax Comparisons per Block: 600.0\n",
      "Runtime: 0.0630 seconds\n"
     ]
    }
   ],
   "source": [
    "cbbp.report()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "bfbef308-2ae0-4a2b-aec4-1bac09e426a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************************************************************************************************************************\n",
      "                                         Μethod:  Block Purging\n",
      "***************************************************************************************************************************\n",
      "Method name: Block Purging\n",
      "Parameters: \n",
      "\tSmoothing factor: 1.025\n",
      "\tMax Comparisons per Block: 600.0\n",
      "Runtime: 0.0630 seconds\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Performance:\n",
      "\tPrecision:      1.90% \n",
      "\tRecall:        94.42%\n",
      "\tF1-score:       3.72%\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
     ]
    }
   ],
   "source": [
    "_ = cbbp.evaluate(cleaned_blocks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4920ae72-7ad6-42aa-932b-aaae20ace85a",
   "metadata": {},
   "source": [
    "## Meta Blocking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1f7d75f3-6bed-482d-a572-c3b4927236a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyjedai.comparison_cleaning import (\n",
    "    WeightedEdgePruning,\n",
    "    WeightedNodePruning,\n",
    "    CardinalityEdgePruning,\n",
    "    CardinalityNodePruning,\n",
    "    BLAST,\n",
    "    ReciprocalCardinalityNodePruning,\n",
    "    ReciprocalWeightedNodePruning,\n",
    "    ComparisonPropagation\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c92e0ca3-5591-4620-b3f4-012a23637416",
   "metadata": {},
   "outputs": [],
   "source": [
    "mb = CardinalityEdgePruning(weighting_scheme='X2')\n",
    "candidate_pairs_blocks = mb.process(filtered_blocks, data, tqdm_disable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f469e387-e135-4945-b97f-da14d391c6b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************************************************************************************************************************\n",
      "                                         Μethod:  Cardinality Edge Pruning\n",
      "***************************************************************************************************************************\n",
      "Method name: Cardinality Edge Pruning\n",
      "Parameters: \n",
      "\tNode centric: False\n",
      "\tWeighting scheme: X2\n",
      "Runtime: 1.4746 seconds\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Performance:\n",
      "\tPrecision:     13.19% \n",
      "\tRecall:        85.97%\n",
      "\tF1-score:      22.86%\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
     ]
    }
   ],
   "source": [
    "_ = mb.evaluate(candidate_pairs_blocks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aeff39a-b51b-4166-a55b-f8452ec258a7",
   "metadata": {},
   "source": [
    "# Entity Matching\n",
    "\n",
    "It compares pairs of entity profiles, associating every pair with a similarity in [0,1]. Its output comprises the similarity graph, i.e., an undirected, weighted graph where the nodes correspond to entities and the edges connect pairs of compared entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f479d967-8bac-4870-99bd-68c01e75747b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyjedai.matching import EntityMatching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ae7b1e6a-e937-44fe-bfe5-34696ea1156c",
   "metadata": {},
   "outputs": [],
   "source": [
    "EM = EntityMatching(\n",
    "    metric='dice',\n",
    "    similarity_threshold=0.5,\n",
    "    attributes = ['description', 'name']\n",
    ")\n",
    "\n",
    "pairs_graph = EM.predict(candidate_pairs_blocks, data, tqdm_disable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4d606bfc-3265-4042-93f3-22a1117c4886",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb4AAAEuCAYAAADx63eqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAn/ElEQVR4nO3df2xc15Uf8O8bDsNhKtHEyrSsmnbUhBYpeUO7kmLQMGpTdlwlKpo2AINlUbqRjYJGpRqwiwLGQsUmLSq0RoFV0VSCUnYTea0WIEBgtynAbSVVpOytTf+QvFQciaQor1wyKzGk3DHFLmeWQ07/GD9rRJEz89499829734/f8rSzDOH885995x7jpfP5/MgIiJyRKLaF0BERBQlBj4iInIKAx8RETmFgY+IiJzCwEdERE5h4CMiIqcw8BERkVMY+IiIyCkMfERE5BQGPiIicgoDHxEROYWBj4iInMLAR0RETmHgIyIipzDwERGRUxj4iIjIKQx8RETkFAY+IiJyCgMfERE5hYGPiIicwsBHREROYeAjIiKnJKt9AUQq5hayGDg/jbEb85jP5NCQSqLt/gb8YFczNm2oq/jvEJE7vHw+n6/2RRAFNTqVxtHhSZybmAUAZHMrX/63VDKBPIC/+VAj8nngz6bS6/6dztYmHHi6BY8+2BjdxRNRVTHwkXVOjlzD4cExZHLLUP3t9TwglazBoX1t6OnYKnJ9RGQ2bnWSVQpB7zIWl1bK/+UK5PPA4tIyDg9eBgAGPyIH8ImPrDE6lUZ33wgWl5a1vH59bQ36ezvQ3tyo5fV1YP6SKDgGPrJG75sf4vTlGeXtzfV4HrB3x2Yc79mt5w0EVZLjZP6SaG0MfGSFuYUsnnz97B03eB3qkgm889ozRj8tVZrjZP6SaG08x0dWGDg/Hcn7eAAGLkTzXmHcznGWL+wpzl+eHLkWyfUR2YCBj6wwdmNe+9MeAGRyKxi7fkv7+4QxOpXG4cGxwIU9i0srODw4hovTaT0XRmQZBj6ywnwmF+F7LUX2XkEcHZ5EJheusCeTW8ax4UnhKyKyEwMfWaEhFd3Jm4ZUbWTvVam5hSzOTcyGLuzJ54Gh8VncXMjKXhiRhRj4yApt9zegLqn/1zWVTKBty0bt7xOURI7T9PwlUVQY+MgKXbuaI3mfPICundG8VxASOU6T85dEUWLnFhKh+yD1vRvq8PS2Ju3n+Pa0Nhl5lEEqx2lq/pIoSgx8pKT0QeobOHJmQuwg9cHOFrx9ZU5b55ZUsgYHOlu0vLYqqRyniflLoqgx8FFo5Q5SZ74IgqcuzeCtiTnlg9SPPtiIQ/vaRHt1+uprEzi0r83YdmVt9zegxvsLLCs87ZqavwTYeo2ixcBHoQRpFi3ZCNr/t+5NZ8grBb3CK5iXv4xyx4DIx5ZlFJhKs2ipRtAXp9M4NjyJofFZeLj9dAnc7lW586HCe1z4P+l1/86e1iYc6Gwx9kkPkGnObWIfUrZeo2rhEx8FJnGQWvUG3N7ciOM9u3FzIYuBC9MYu34L85klNKRq0bZlI7p23t4iq+TvmEzl5+0zLX9ZrR0DIoBPfBSQRLNoGxpBm0KqOffvfqcVLz1tRuAzYceA3MZzfBQID1JHS+LnnUx4hf1CQ6g8wS4uLePImQnhKyLXcKuTAnHxIHU1Kw4lft65lbwxP2/V1mtAofXaCyfexyvPbmPBC4XCwEeBSB2knrOgZ6QJFYdxO7guNV5qaHwWI598xoIXCoVbnRSI1EHq/311Di+d/BCjU2mR15N2cuQauvtGcPryDLK5lbueujJf/NmpSzPo7hvRNu8ubgfXJcdLcdYghcXAR4FINYteyUN70AjLpGGvEj9vkw6uS4+X4qxBCoOBjwKRbBZt4oRw04a9Svy8TTq4rmO8FGcNUlAMfBSI3yxaskjQpFW7acNeVX/euhpvzy1kcfzcVbzS/xFefOMDvNL/EY6fu1p23p+O8VKcNUhBsbiFAtPRLFrqYLsKyWGvkoFG5ectfXBdteCna1ezluMI/hGZl576hvhrVxN7mOrBwEeB6WgWrStoBCF5RlHyBhz25y3deFuiKbmu8VK2HZEpx4SK4jjjVieF0tOxFYf2bUd9bY3Ytme1D7abfEYxyM/b8wodTg7t2y5W6i9Z8HOwswWpZI3IdRUz5ciGKlMqiuOMgY9C6+nYiv7eDuzdsRkJgeBX7VW76Wfmin/edckEUqtyZalkAnXJBPbu2Iz+3g6xoCdd8OM/wdbXyt5+TDmyocKkiuI441YnKfGbRff8wXv408k55der5qrdhjNzQZpzS9HRlLx4vJRErtikIxthqS4w2psb2cO0Qgx8jtCdJL93w1cErrK6q/ZCxeEN5e3Oe76q/2u1aUNdJIUcOgt+ejq2or25EUfOTGBofFbpOk06shGWCVNPXMHAF3NRJcklgka1V+1SFYef3vxLgasxg+6Cn/bmRvx8/+PY//P3MTwRLvjpOrIRJVMriuOKgU+zapYjS1ThVUoiaFR71X7vhjp0fH3Tl4uEsN65ejM2N6CoCn5e/fY2vPfnnxlxZCMKq+8LNz7PILes9nOO65EOHRj4NKl2OXLUgz5Vy9RNWbV/7be+qvwacboBRVXwY8qRDd1K3RdUVbs4zCas6tSg2uXI1Wq7pVKmbsqqXaK4Jk43oCgLfqp9ZEO3cvcFCXE50qEbA58wE8qRq9V2K2yZukmrdtOPNETtnnr1YqMgudtqHdnQLch9QUUcjnREgVudgkwoR652kry4TH293KLP8wpPeibNVLPhSEOUPv1MvVAnaO62Gkc2fDpy8mHvC0FVuzjMJgx8gkwoR5aowlteyeMP372GV59rDfXv/TL1Y8OTGBqfhYfbhTRA4QuaRyGnd6CzxYgnPV8cqlOlzC1k8e7Vm8qv8+Q3NoUKGlEd2QD05uRV7gtBVLs4zCYMfEIknrROXZrB758exw+f2Bp6dSlRhZdbyeMnQ5MYm7kVuvimmqt2FXGoTpUiNS39IYGCIZ10Vj+r3hcqZUpxmC0Y+IRI3CRW8sDRoUn89K1PQq8upXJU/qBY1WMOUa7aJcSlOlWC1LT0zxdlh89K0l39LLV4KMeU4jBbsLhFiNRNYjkPpYpPyUGfrvYCjEN1qoS4F/pEUf0sdV8oxaTiMFsw8AmRukn4wgYdHYM+TRoUG4U4VKdKiHuhj0rubXGpsupn6ftCMRuPdJiCgU+I5JNWsaBBp2uXntySjuniJov7mbJKSCyiTC30kci9FVIBvyn5d3TcF2w90mESBj4hOp60fEGCjp+jkpqR5ys+5uCKuJ4pq5TEIsrUQh+pnPyLb3xYckdG4r6QTHh4ZEsDnm27D99/7AG8+tw2vPPaMzjeszs2uwtRY3GLEKkGx2sJerbuYGcL3r4yJzLupVicWnFVytbqVAlxLvSRyr3lVvIli10k7gs1CQ9/+OLjRv4cbcXAJ0T1JlFOkKATtu9hOXFqxRWUbdWpUlQWUSYX+kjm3ko1oIjz4sFm3OoUpFINWE7QoBMkRxWEqRV6pEdcC32kc2+l0hGsEjYPA5+gsDeJSgUNOsU5qoRQ8DO1Qo/0iWOhj3ROvlQOPK6LB5sx8AnT9aQFhAs6fo7qn+5pQY3i9ZhaoUf6xa3QR0f1s5+OWEscFw82Y45Pg9W9KpdX8sitqCX+VIPOP3piK3761idYVkjom1qhR9GIU6GPjpx8uXSEzT1s48bL53V3kXPbzYUs3nj3Gv7j0CRUYl9dMoF3XntG6cbS++aHSkn2vTs2KzfRJjLF6FQa3X0jotXPz7bdhz/44bfK/r04LB5sxic+zTZtqMM/e64VYzduVb2yK64VekRh6Kh+rjQd4WqVsCmcC3w65m1VwoSgE/aLziQ7xZWfQ/vxf79U9XQERceZrc7S87YKe+thJyJUKkgneF8h6MgmucuNYfGZOCiWSIe3Jn6D/Sc+qHo6gqLhROAz6UZvyrVcnE4zyU5UpPfND3H60gzC3BCZA7dL7AOfKU9ZxUwKOkyyExWoFLvU19agv7eDi0RLxDrwmf6LzKBDZBYTF8qVqFbtgq1iHfgkyvf/9d//Jn+hiBxiSjqiEibULtgotoFvbiGLJ18/q9SBPeEVRoJ4nsdfKCKHmJSOWI9NAdo0sQ18x89dxZEzEyKjR9bDXyiieDM1HWHrlqwpYnuOT2reVin5PLC4tFxyHhcR2cvEg+ajU2kcHhwLfOi+1Pgk18S2SbXkvK1y/F+oi9PpyN6TiNx0dHgSmVy4NmuLS8v495oGZtsktoFPet5WOaXmcRERSZhbyOLcxKxSY+2z47N44cT7GJ1Ki12XbWIb+KTnbZVTah4XEZGEgfNrjz0Kamh8Ft19Izg5ck3k9WwT28CnY95WOaXmcRERqZKsXfDrE1wMfrENfP68LelhsKWUm8dFRKRCunbB1fqE2AY+oDARIZWsifQ95zNLkb4fEblDR+2Ci/UJsQ58/hie+tro/jcrncdFRBSUjtoFF+sTYh34gMLZukP7tqO+tkb7tifncRGRTrpqF1yrT4h94AMKwa+/twN7d2xGXTKB1KoVUyqZwFdqEkgoBsY8gK6d0RfVEJEbdNUuuFafENvOLau1NzfieM/uki2IfvePfqnU1HpPaxMbVxORVgc7W/D2lblQU2dKcak+wZnA5yvVgkjlFyqVrMGBzhbVyyMiKsmvXQjaq7Mcl+oTnNjqrFTYYphC89c25/vfEVE0imsXJLhWn8DAt0qQYhjPKwysZcdzIoqaX7uwp7VJ+bVcq0+I7VgiVTbM4yIiAoD9J97H8PhsqH/rD90+3rNb+KrMxcBXhqnzuIiIfKNTaXT3jYSqT6ivrUF/b4dTC3gGPiKiGOBw2so5V9VJRBRHfvA6PDiGTG655LEszytUoh/a1+Zc0AP4xEdEFCusTyiPgY+IKIZYn7A+Bj4iInIKc3xERJaaW8hi4Pw0xm7MYz6TQ0Mqibb7G/CDXXyqK8X6Jz5+8ETkmtGpNI4OT+LcROHsXnaNPF5naxMOPN2CRx9srM5FGszawMcPnojippKFfOHYAis3VVgZ+PjBE1GcVLqQ/xub/hreePcaz+opsi7w8ZAmEcVJpQt5FS52ZynFmibVcwtZ/OgXH+NHv/hV4FEci0srODw4hovTaT0XR0QUwu2FvL6gBwCZ3DKODU/qewPLGF/VWbwFsLS8gpWQvxz+By/ZiJWFNUQU1uhUGocHx0Rn6q0nnweGxmdxcyHLexMMD3ySWwCSH3zp/fgbOHJmgoU1RFTS0eFJZHKyU9RL8QAMXJhedxC3S4zd6tSxBeB/8KrX1d03gtOXZ5DNrdwR9IBCe6BsbgWnLs2gu28EJ0euKb0fEcXP3EIW5yZmtW5vrpbJrWDs+q3o3tBgRj7x6doCUP3ggxTW5PPA4tIyDg9eBgAW1hDRlwbOqy3Aw5rPLFXlfU1j5BOfzi2AsB982GDMwhoiWm3sxvxdu0VRaEjVRv6eJjIu8OneAgj7wasEY1ZUEVGx+Uwu8vdMJRNo27Ix8vc1kXGBT+cWQNgPXjUYFxfWEBE1pKLPMuUBdO1sjvx9TWRc4NO5BRD2g5cIxhKFNUQUD233N6AuGd3t1/MKM/h4lKHAuMCnawtA5YOXCMasqCIiX9euaJ+8UskaHOhsifQ9TWZcVaeuLQCVD14qGLOiiogA4N4NdXh6WxNOX57RfqSh0LKxrWS7MteacRgX+ApbADdEtzsr+eBLkQrGy2HbzhBR7BzsbMHbV+awuBSuaM5DIX2z7n+voEm/q804jNvqlNwC8LxCc1bVBtVS+/HXP19Ufg0iiodHH2zEoX1tqK8Ndm+pr03gnzz1dex9ZDPqkgmkVt2bUskE6pIJ7N2xGf29Heve+1xuxmHcE5/EFkDCA2prEtjT2oQDnS3KHcm7djXj909PKL0GAPz53P9jrzwi+pIflMKOWbu5kMXAhWmMXb+F+cwSGlK1aNuyEV07S29Rut6Mw8ixRKNTaXT3jYTaAqjxgOc7voaXn3lYNMA8d+QcrvxmQek1UskEXn1uG3vlEdEdLk6ncWx4EkPjs/BQeNry+fP4pBbyKvfXuIw3Mu6JD7i9BWDS3L2/3phSDnys7CTTuFbUYKr25kYc79kd+gkuCIlmHJJTbqrByMAHqG8BSKtJyKRDWdlJJnC1qMF0mzbUad0RkmzGYfPCyLjilmI9HVvR39uBvTvUkrgSpCo72SuPqs3logbXDZyfxopidisOzTiMfeLzRbkFUIrEMQv2yqNqc72owXWDH1/H0rJa4ItDysb4wOeT3AIIk9fo2tWMI2fUKjvZK4+qSXXCSHtzo/VFDS47OXINH//6c5HXsj1lY03gq1SpoDb9fxdD5zVUj1mwVx5Vm1JRw1I8ihpc5S96pHpo2J6yiU3gK5es/3f/cwx5FLZv1vrs/fLhU5dm8NbE3JqFMiqdFtgrj6pJuagB/ndjFk9taxK9NtJPcsZpHFI2Rhe3VKqSZP1yHlhZJ+gVK85rrE7qq3RaUGmZRqRKYsLISh548Y0PWOxiGekZp3FI2Vgf+G4n60sfeQhqvcnpPR1bcWjfdtTX1sDzSr+GVMs0IlVS475yK/k1F4VkLskZp3FJ2Vgd+MIm6yu13uR0k45ZEFVCctzXeotCMpPkjNO4pGyszvFJ7luvpdRhTVOOWRBVQnrcV1w6eLhAatGT8BCblI21gU9633o9/mHN9Y5S6O60QCRBetxXXDp4uEBq0dP+wD2x2b2ydqtTct+6lDgc1iTSMfE7Dh08XCAxVu0rNR6++80tQldUfdYGPsl963JsP6xJ5J9DLVeQFQQXhXaQWPR4nmd9JWcxawOfZLK+HNsPaxIBhXOoqWSN6GtyUWg+1UVPXCo5i1kb+KST9euJw2FNIiD8OdRSuCi0g8qiJy6VnMWsDXwS+9aViMNhTSKffw41mVDf8+Si0B5svnEnawOfjmT9anF8xCfq6diKn/3wW1CNfVwU2oXNN26zNvDpSNavFsdHfCIAeGpbE769fXPof89FoZ3YfKPAy+d1n4TTZ3Qqje6+kVBNo8spPOLHc7VDBKh9f+pra9Df2xG7LTCXuNx8w+rABwQbrFkJzys86a01nYEobsJ8f7goJNtZ27nF53/5Dg+OIZMr36g64QEJz0My4X05iggoPOLnUdi+OdDZwpUsOSHI94eLQooL65/4fBen0zg2PImh8Vl4QMmg9kBjvbOP+ERrCfL94aKQbBebwOdzed+aSBW/P+SC2AU+cs/cQhYD56cxdmMe85kcGlJJtN3fgB/s4s2aiO7GwEfWGp1K4+jwJM5NzALAHb1b/e25ztYmHHi6BY8+2FidiySKkbgsMhn4yEqFakQWZBBFIW6LTAY+sg5L8ImiE8dFprWdW8hNo1NpHB4cC3xuc3FpBYcHx3BxOq3nwohi6PYis/xRsXweWFxaxuHByzg5ci2S6wuLgY+scnR4EplcuE49mdwyjg1PCl8RUTzFeZHJwEfWmFvI4tzEbNmV53ryeeDs2CxuLmRlL4wohuK8yGTgI2sMnJ9Wfo2/Wl7BP/jPIxidSqtfEFFMSSwyh8bNXWQy8JE1xm7M31FNFtbEzAK6+0aMz0MQVYvEIjOfz2Pggvrr6MDAR9aYz+TEXsuWJDxRNUgsMv9qOY/BX14XuiJZDHxkjYaUbE91G5LwRNUgtcj85a8/N3JxycBH1mi7vwF1SdlfWdOT8ETVILXIXMnDyMUlAx9Zo2tXs/hrmp6EJ6oGyUWmiYtLBj6yxr0b6vD0tiZ4nuzreoCxSXiiapBcZJq4uLR+EG1QcWmy6qqDnS14+8ocFpfCnS9aSya3grHrt8Rej8h2/iLz9OWZ0EcaivmLy5ee+ob6iwlwJvCVbrJ6A0fOTFjVZNVVjz7YiEP72gL36ixnPrMk9lpEUdC9iJdcZJq2uLQ28AX50Ms1WfWnTZ+6NIO3JuasaLLqMv+zKbRTknnya0jVirwOkW5RLeL9Rebv/eJXWBF46jNpcWnddIag4zHYyT++Lk6n8c8HRjExs6D0OqlkAq8+t82YbRii9VRjUsLfO/qnGJ3+XOk1AOD7jz2AI7/zmPLrSLCquOXkyDV0943g9OUZZHMrdx2wzHzxZ6cuzaC7bwT/9k8ux7bJKgHtzY34r/+4A1+pUat2yQPo2ilfMUokqVqTEr7721uUv2OpZAJtWzYqvYYkawJfmA/9P739CTIht8JMLMGlu927oQ6drfeFrvT0PGBPaxMLm8hoUpMS5hayOH7uKl7p/wgvvvEBXun/CMfPXS1Zcdm1qxmeYim1aYtLK3J8YT90lX3p4hJc3hTNppKETyVrcKCzRcNVEclRnZTwb/7kMjamakPlBVUrPE1cXFrxxKfyoavg+S47+En4+tpgv86FXG4b2psb9VwYkQCJSQnvfvIZTl+qLEW01tbowc4WpJI1od7fxMWl8YFP9UNXYVoJLq2vp2MrDu3bjvramrLbnp4H1NfWsICJrCAxKQEobDeW/O8l8oJxW1wav9Up9aGHZVIJLpXW07EV7c2NODY8iaHxWXi4fVQFuF31u6e1CQc6W4z7MhKtRWocV6X8vGB7c+Md35HiY0RRVpXqYHzgi/pDX43nu+zS3tyI4z27cXMhi4EL0xi7fgvzmSU0pGrRtmUjunayQw/ZRXIcV6X84r7jPbvv+PO4LC6ND3zV+NB9ppXgUuU2bajjuTyKBelxXJUoVdwXh8Wl8YGvGh+6z7QSXCJyT2FSwo3Id77K9de0eXFpfHGLjhlslTCxBJeI3KNjHFcl4lzcZ3zgq9aHbmIJLhG5R9c4rkrEtbjP+MAn8aEnAv5bU0twichNKufoVMS1uM/4wAeofej1tTV46amv83wXEVkr7Dk6FXEu7rMi8KkennztO9vR39uBvTs2oy6ZQGpVzjCVTKAumcDeHZvR39vBoEdExgnSpEFCnIv7rBpLJDGSw+YSXCKyn+oA2YvT6YrO0aX/cgnvXfssdH/NvTs233WOLy6sCnxA5R+6yYcno6Z7UjMRlRd0lmg55Rbxo1NpdPeNhGreXl9bg/7ejtjeQ60LfD4+uZUn/UUjonCqMUD29vtyEPdq1gY+Kq1aXzQiulO1gw/vBXdj4Iuhan/RiKjAlO1GpojuxMAXM6Z80YgI6H3zw9ADXAHgmdYm/Gz/42LXwxRRAQNfzJj2RSNy1dxCFk++fla5x+ae1ia88uw25uEFWXGOjyojMbT37PgsXjjxPkan0mLXReQiqVmiQ+Oz605Gp3AY+GKEXzQic0jOEl1vMjqFw8AXI/yiEZlDepaoPxn94nRa9HVdxMAXI/yiEZlDxyxRfzI6qWHgixF+0YjMoWOWaPFkdArP+AnsOsWtlZeOSc3FXzQbfyZE1dK1qxlHzkyIv265yehUnpOBr3Qrrxs4cmbCylZe/KIRmcOfJapyvGgtcZ6MHhXntjpPjlxDd98ITl+eQTa3ctfTUeaLPzt1aca6ykZdk5r5RSMKR9cA2bhORo+KU4Hvdiuv0j3rgMIWn42VjfyiEZlD1wDZuE5Gj4ozgW90Ko3Dg2OB+lcC9lU28otGZJbiAbIS4jwZPSrOBL6jw5PI5IL3rwQKT342VTbyi0Zklp6Orejv7cCe1ibl14rzZPSoONGrU6JnXsIDTuz/Fp7adp/glRXoqi69OJ3GkTMTGBqfVbq+umQC77z2DKs6iQTsP/E+hkN+J+M+GT0qTgS+4+eu4siZCeUy/2TCw4//7g6x0T1RDYrlF43IHJygUn1ObHVKtfLKreTFil2irC599dltobc9U8kaHOhsCf3eRHSnsHn4wszMNgY9AU4EPslWXhLFLlFXl/KLRmSW4jx8ueNHnld40uOgaDlOBD7pVl4qbbyqVV3KLxqRWXo6tuKnPTvxcNMGJLxCHUGxVDKBumQCe3dsRn9vB7+Lgpzo3CLdykuljZdKdakfcMPm23o6tqK9uRHHhicxND4LD4UtVZ+fV9zT2oQDnS180iPSZHV+f6Vo58cPgA/91lfxL/7Odi0Fda5zorhFahJysVQygVef21ZRGy+/anN0Oo3/8asbSu2LpCosby5kMXBhGmPXb2E+s4SGVC3atmxE1047+5QS2aKQ6hhDJlc61eF5hRz7oX1tfNoT5sQTn46eeZW08SpVtRmWVN/MTRvq2HuTKGK38/vl7wXF+X0ADH6CnMjxAXpaeZVq41WuajMs9s0kspMr3aNs4Ezg09HKa702XkGqNsNg30wi+0jk90mGM4EPuF3ZmFxdPhXCem28wq7qgmDfTCK7zC1kcW5iNvRCmANoZTkV+IBC8PvZD3ffVToc1Hr98lRWdZVg30wi+wycn0ZuWW0x7Of3SZ1zgQ8Antp2H769fTPCxj7PK5T8r65+VF3VVYINaonsM/jxdSwr3heY35fjZOADvih2EW7jNXBe72psvYBLROYanUrj419/LvJazO/LcDbw6WjjJdUTdD3sm0lkn6PDk3ccUFfB/L4MJ87xrcc/FyN1mFSyJ+hq7JtJZB8//SGB+X05Tgc+QLaNl3RPUIDdG4hsJpn+YH5fjvOBDwDamxtxvGe3chsvyZ6g7JtJZD/J9Afz+3IY+IqotvHq2tWMI2cmlK4h4QHfeeR+PPpgI/tmEllOKv2R8MD8viAGPkGqPUE9D/jbOzbj2D/cJX9xRBQ5qfRH+wP3cNdHkLNVnbqo9ARl1SZRvBTSH2q32WTCw3e/uUXoighg4BPHaedE5OvapV6MUpPwWNQijIFPA047JyLgdvqj3H1gPWxaoYcTg2ir5eJ0mtPOiRw3OpVGd98IFpeC9/Ctr61Bf28H7w/CGPgiwGnnRG4LMoDWV0h/cCdIBwY+IqIIFIKfTJcoUsPAR0QUEaY/zMDAZ5m5hSwGzk9j7MY85jM5NKSSaLu/AT/YxW1Toqiofg+Z/qguBj5LjE6lcXR48suGt9k1VoqdrU048HQLHn2wsToXSRRz/B7GAwOfBZgbIKo+fg/jg4HPcKwGIxeZtqXP72G8MPAZjOd/yDWmbSXOLWTxk7NXcHLkUyyHuFPye2gmBj6D9b75YeiG1wDwTGsTfrb/cdmLItLEpK3E4gC8tLwSeoK65wF7d2zG8Z7dshdIStiyzFD+5GaVZcnZ8Vm8cOJ9jE6lxa6LSIfbW4mlgx4A5PPA4tIyDg9exsmRa1qupbtvBKcvzyCbCx/0gMK1Do3P4uZCVu4CSRkDn6GkJjcPjc+iu29Eyw2CSMLoVBqHB8cC5c8AYHFpBYcHx3BxOi12LUECcKU8AAMX5CaxkzoGPkNJTm7WuTomUnV0eBKZXPA8NlD43VYd/uwLG4DLyeRWMHb9luhrkhoGPkNJTW726VgdE6mS2NIfEtrSVwnA5cxnlrS8LoXDwGcoqcnNxTK5ZRwbnhR/XaKwTNnSlwjApTSkavW8MIXCwGcoicnNqzHRTqYxZUtfKgCvJZVMoG3LRm2vT8Ex8BlKYnLzWphoJ5OYsqUvGYBXywOcoG4YBj5DqU5uXg8T7WQSU7b0pQOwjxPUzST/W0diDna24O0rc6E6t5Rie6LdtHZWFF5hS/+G6NNW8ZZ+pb8POgIwUDhof6CzRctrU3gMfAZ79MFGHNrXFrhHYDm2JtpLt7O6gSNnJtgZ3zJdu5rFjiMU87f0X3rqGxX9fR0BuNCrs43tygzErU7D9XRsxaF921FfWyPyerYm2ld301h9g8p88WenLs3wwL5FTNnSl8ype16hRycbVJuLgc8CPR1b0d/bgT2tTcqvZWOi3aR2ViTvYGcLUkmZhV2xIFv6EgE44QF1yQT27tiM/t4OBj2DsUm1ZfafeB/D47Oh/q10w9wocm0qEyoSHvC3Hr4XT3z9Xub/DBdm7E8533/sARz5nccq/vsqv2s1HvB8x9fw8jMP8/fMAgx8ljFhVFGUo2NUJ1TouCbSw5/OIFHMlUom8Opz2yrO8d15DZy7F3cMfBaq5pczytExcwtZPPn6WbGCA07GNt/F6TSOnJnAUMhdDV9dMoF3Xnsm1NOXSeORSA8GPktV48sZdcA9fu4qjpyZED9YzBW6PlLb3/t//j6GJ6q3pX9xOo1jw5MYGp+Fh0KxjM/fQdjT2oQDnS2s2rQQA5/FovxyVmOL9ZX+j/DHf/YXgd9P5zXR2qS3v03Y0geAmwtZDFyYxtj1W5jPLKEhVYu2LRvRtZM5Y5sx8MVAFF9OlVxb2BX4i298gLNjvwn+hhqvie6ma/eB+TbShQfYY2DThrrASfwgVDvXh+mkAejrpqFyTXSnIMGp+KgJgLLByf/vzLeRNJ7jo7L+w/+6gqVltTxbmObYOiZUFGPDbjVRTE73z7Du3bEZdckEUqt+H1LJBM/OUWB84qOSTo5cw39571OsKG6Ih2mOraudlU+qYbervUNVBrf6jaQr2Wpub27E8Z7dzLeRGAY+Wpe/jbUslAUO2hzb76aheo5P8pqKudw7tBrb37q39Mkd3OqkNYXdxiolTHNsXe2sfGEbdrveO1RicCu3mqlaGPhoTSrbWGsJ2xzbn1BRXyv/qxr2mtg7VGZwK2dDUrVwq5PuorqNtRaV5thBqvt0XtPcQhY/OXsFJ0c+Dbz96xd0tDc3ajk7GHWeUWpwq+2zIclODHx0F4ltrGISU6h7Oraivblx3QP7Oq+pOJe3tLwSutAnSEFHpaqVZ5Q6anLp+jxGp9Kxy4GS2Rj46C4S21jFpKZQr67ue/fqTbx1ZTZUIKr0mio9nF0J6bOD5a7NXxicujSDtybmRM+4SQ1uvf55Bt19Izx/R5Fijo/uIrWNBeiZQu1X95144XH8q+89Ejj/V+k1BcnlVUqqoKPaeUbJwa1xzIGS2Rj46C5S21g1nqe9fVTxhPpyQ0SDTMbWUdUKqBd0zC1k8Xv/7WP86Be/0npwvBzpyemS10ZUDgMf3UWiY0rCA55/4qFItq90dPeQrmotFqagY3Qqjd43P8STr5/FyfeCF9f4/DyjBOmjJpLXRlQKc3x0F4mOKbU1Cby852GhKypPsruHjqrWYkHPDpqaZ/SPmkhNTmf/VIoKAx/dRbVjikQVZ1gS3T2kq1qLBT07GGZCQTl+nlGiC0rxUROJyemS10a0Hm510ppUtrGkqjirRbqqtViQs4Om5hlX87eat9yjvtDhoXaKAgMfrSlsxxQdVZxRk6xqLRb0Sdi0PGMp7c2N2L7lHpHX4qF20o1bnbQuV+eh6ZoDGORJ2LQ8Y2WvKfNz03FtRMUY+Kikch1TUskE8ig8yRzobLH6Sc8ndTi7WNAnYZPyjJWS+LnpujaiYgx8VJZr89Ak5wCGfRI2Jc8YhMTPTde1ERVj4KOKuTIPTWIOYMIrHOkI+yRsSp4xCJurgcktDHxEazjY2YK3r8yFKtGv8YDnO76Gl595OPRN3IQ8YxgqPzfbq4HJHqzqJFqDSlXrv/zeI/jx935b6clFonvOalFU3LpcDUz2YOAjWoeuPqCVkGwCLX1t5VTz50ZUCS+f11UwTRQPF6fTValq7X3zw6rmGVVV6+dGVA4DH1GFoq5qHZ1Ko7tvJGSe0cPzTzyEl/eEzzNKcaUamOzBwEdksDC9Ogv5Mm4dEq2HVZ1EBnO1ew6RTnziI7IA82VEchj4iCzCfBmROgY+IiJyCs/xERGRUxj4iIjIKQx8RETkFAY+IiJyCgMfERE5hYGPiIicwsBHREROYeAjIiKnMPAREZFTGPiIiMgpDHxEROQUBj4iInIKAx8RETmFgY+IiJzCwEdERE5h4CMiIqcw8BERkVMY+IiIyCkMfERE5BQGPiIicgoDHxEROeX/AwTVtbKTioycAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw(pairs_graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "4a2a5f4a-6ffa-4c16-ae49-ff4fec4c467d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************************************************************************************************************************\n",
      "                                         Μethod:  Entity Matching\n",
      "***************************************************************************************************************************\n",
      "Method name: Entity Matching\n",
      "Parameters: \n",
      "\tTokenizer: white_space_tokenizer\n",
      "\tMetric: dice\n",
      "\tSimilarity Threshold: 0.5\n",
      "Runtime: 2.2469 seconds\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Performance:\n",
      "\tPrecision:     97.14% \n",
      "\tRecall:         3.16%\n",
      "\tF1-score:       6.12%\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
     ]
    }
   ],
   "source": [
    "_ = EM.evaluate(pairs_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93b72120-4578-4d5c-a408-a24ee78bf6cb",
   "metadata": {},
   "source": [
    "# Entity Clustering\n",
    "\n",
    "It takes as input the similarity graph produced by Entity Matching and partitions it into a set of equivalence clusters, with every cluster corresponding to a distinct real-world object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "500d2ef7-7017-4dba-bbea-acdba8abf5b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyjedai.clustering import ConnectedComponentsClustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "aebd9329-3a4b-48c9-bd05-c7bd4aed3ca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "ccc = ConnectedComponentsClustering()\n",
    "clusters = ccc.process(pairs_graph, data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "5b52a534-691a-48be-b5e9-c073dc04b154",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Method name: Connected Components Clustering\n",
      "Method info: Gets equivalence clusters from the transitive closure of the similarity graph.\n",
      "Parameters:  None\n",
      "Runtime: 0.0010 seconds\n"
     ]
    }
   ],
   "source": [
    "ccc.report()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "00bc2e82-9bc1-4119-b8cb-4a1c18afee19",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************************************************************************************************************************\n",
      "                                         Μethod:  Connected Components Clustering\n",
      "***************************************************************************************************************************\n",
      "Method name: Connected Components Clustering\n",
      "Parameters: \n",
      "Runtime: 0.0010 seconds\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
      "Performance:\n",
      "\tPrecision:     97.14% \n",
      "\tRecall:         3.16%\n",
      "\tF1-score:       6.12%\n",
      "───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
     ]
    }
   ],
   "source": [
    "_ = ccc.evaluate(clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "315369d8-6564-44d4-aea0-14034b54cf16",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<div align=\"right\">\n",
    "K. Nikoletos, G. Papadakis & M. Koubarakis\n",
    "</div>\n",
    "<div align=\"right\">\n",
    "<a href=\"https://github.com/Nikoletos-K/pyJedAI/blob/main/LICENSE\">Apache License 2.0</a>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "824e5f4123a1a5b690f910010b2896a5dc6379151ca1c56e0c0465c15ebbd094"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}