"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"em.plot_distribution_of_scores()"
]
},
{
"cell_type": "markdown",
"id": "93b72120-4578-4d5c-a408-a24ee78bf6cb",
"metadata": {},
"source": [
"## Entity Clustering\n",
"\n",
"It takes as input the similarity graph produced by Entity Matching and partitions it into a set of equivalence clusters, with every cluster corresponding to a distinct real-world object."
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "500d2ef7-7017-4dba-bbea-acdba8abf5b7",
"metadata": {},
"outputs": [],
"source": [
"from pyjedai.clustering import ConnectedComponentsClustering"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "aebd9329-3a4b-48c9-bd05-c7bd4aed3ca9",
"metadata": {},
"outputs": [],
"source": [
"ec = ConnectedComponentsClustering()\n",
"clusters = ec.process(pairs_graph, data, similarity_threshold=0.3)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "3d2aa574",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"***************************************************************************************************************************\n",
" Method: Connected Components Clustering\n",
"***************************************************************************************************************************\n",
"Method name: Connected Components Clustering\n",
"Parameters: \n",
"\tSimilarity Threshold: 0.3\n",
"Runtime: 0.0396 seconds\n",
"───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
"Performance:\n",
"\tPrecision: 76.45% \n",
"\tRecall: 43.99%\n",
"\tF1-score: 55.85%\n",
"───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
]
}
],
"source": [
"_ = ec.evaluate(clusters)"
]
},
{
"cell_type": "markdown",
"id": "6e44642b-b0d9-4f8d-9fe4-4fca2ad716aa",
"metadata": {},
"source": [
"# Workflow with Similarity Joins\n",
"\n",
"In this notebook we created the bellow archtecture:\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "2b07ea13-58cb-498a-949a-4702ea9ee4ce",
"metadata": {
"tags": []
},
"source": [
"## Data Reading"
]
},
{
"cell_type": "markdown",
"id": "a0b7acf5-32b3-45b1-8347-197dfa869fb9",
"metadata": {},
"source": [
"Data is the connecting module of all steps of the workflow"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "3006b051-8348-4922-a627-56441a1db7b7",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from pyjedai.datamodel import Data\n",
"d1 = pd.read_csv(\"./../data/der/cora/cora.csv\", sep='|')\n",
"gt = pd.read_csv(\"./../data/der/cora/cora_gt.csv\", sep='|', header=None)\n",
"attr = ['Entity Id','author', 'title']\n",
"data = Data(\n",
" dataset_1=d1,\n",
" id_column_name_1='Entity Id',\n",
" ground_truth=gt,\n",
" attributes_1=attr\n",
")"
]
},
{
"cell_type": "markdown",
"id": "b3eedb4c-b86f-4f98-abf2-9d6f7d0271c5",
"metadata": {},
"source": [
"## Similarity Joins"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "afd97b7e-4bf8-4256-b9fc-a301e413e834",
"metadata": {},
"outputs": [],
"source": [
"from pyjedai.joins import EJoin, TopKJoin"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "7bc8ba43-b059-4839-8958-0b31fab95e46",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"EJoin (jaccard): 0%| | 0/1295 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"draw(g)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "a4c57244-0c98-4598-8ae2-06cdd0c48385",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"***************************************************************************************************************************\n",
" Method: Top-K Join\n",
"***************************************************************************************************************************\n",
"Method name: Top-K Join\n",
"Parameters: \n",
"\tsimilarity_threshold: 0.25547445255474455\n",
"\tK: 20\n",
"\tmetric: jaccard\n",
"\ttokenization: qgrams\n",
"\tqgrams: 3\n",
"Runtime: 15.0497 seconds\n",
"───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
"Performance:\n",
"\tPrecision: 58.34% \n",
"\tRecall: 63.75%\n",
"\tF1-score: 60.92%\n",
"───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
]
},
{
"data": {
"text/plain": [
"{'Precision %': 58.340434597358325,\n",
" 'Recall %': 63.74534450651769,\n",
" 'F1 %': 60.923248053392655,\n",
" 'True Positives': 10954,\n",
" 'False Positives': 7822,\n",
" 'True Negatives': 814451.0,\n",
" 'False Negatives': 6230}"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"topk_join.evaluate(g)"
]
},
{
"cell_type": "markdown",
"id": "e686ba55-cbbe-4133-8256-fa5339e70720",
"metadata": {},
"source": [
"## Entity Clustering"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "e48627ef-0be6-45dd-b3ba-863818eadbfb",
"metadata": {},
"outputs": [],
"source": [
"from pyjedai.clustering import ConnectedComponentsClustering"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "a9d5a28d-79f0-479f-b154-f5f7e3661897",
"metadata": {},
"outputs": [],
"source": [
"ccc = ConnectedComponentsClustering()\n",
"\n",
"clusters = ccc.process(g, data)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "f95539fe-2569-4569-8929-2f4220f14157",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"***************************************************************************************************************************\n",
" Method: Connected Components Clustering\n",
"***************************************************************************************************************************\n",
"Method name: Connected Components Clustering\n",
"Parameters: \n",
"\tSimilarity Threshold: None\n",
"Runtime: 0.1237 seconds\n",
"───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
"Performance:\n",
"\tPrecision: 2.05% \n",
"\tRecall: 100.00%\n",
"\tF1-score: 4.02%\n",
"───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
]
}
],
"source": [
"_ = ccc.evaluate(clusters)"
]
},
{
"cell_type": "markdown",
"id": "778b9b19-0964-4095-a293-a8336fe0607a",
"metadata": {},
"source": [
"
\n",
"\n",
"K. Nikoletos, J. Maciejewski, G. Papadakis & M. Koubarakis\n",
"
\n",
""
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
![](\"https://github.com/AI-team-UoA/pyJedAI/blob/main/docs/img/reading-process.png?raw=true\")
\n",
"