{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Down sampling is typically done when the input tables are large (e.g. each containing more than 100K tuples). This IPython notebook illustrates how to down sample two large tables that are loaded in memory."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we need to import *py_entitymatching* package as follows:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import py_entitymatching as em"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we need to read in the input tables. For the purposes of this notebook we will use two large datasets: Citeseer and DBLP. You can download Citeseer dataset from http://pages.cs.wisc.edu/~anhai/data/falcon_data/citations/citeseer.csv and DBLP dataset from http://pages.cs.wisc.edu/~anhai/data/falcon_data/citations/dblp.csv. Once downloaded, save these datasets as 'citeseer.csv' and 'dblp.csv' in the current directory."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Read the CSV files\n",
"A = em.read_csv_metadata('./citeseer.csv',low_memory=False) # setting the parameter low_memory to False to speed up loading.\n",
"B = em.read_csv_metadata('./dblp.csv', low_memory=False)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(1823978, 2512927)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(A), len(B)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" id | \n",
" title | \n",
" authors | \n",
" journal | \n",
" month | \n",
" year | \n",
" publication_type | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 1 | \n",
" An Arithmetic Analogue of Bezouts Theorem | \n",
" David Mckinnon | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 1 | \n",
" 2 | \n",
" Thompsons Group F is Not Minimally Almost Convex | \n",
" James Belk, Kai-uwe Bux | \n",
" NaN | \n",
" NaN | \n",
" 2002.0 | \n",
" NaN | \n",
"
\n",
" \n",
" 2 | \n",
" 3 | \n",
" Cognitive Dimensions Tradeoffs in Tangible User Interface Design | \n",
" Darren Edge, Alan Blackwell | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 3 | \n",
" 4 | \n",
" ACTIVITY NOUNS, UNACCUSATIVITY, AND ARGUMENT MARKING IN YUKATEKAN SSILA meeting; Special Session... | \n",
" J. Bohnemeyer, Max Planck, I. Introduction | \n",
" NaN | \n",
" NaN | \n",
" 2002.0 | \n",
" NaN | \n",
"
\n",
" \n",
" 4 | \n",
" 5 | \n",
" PS1-6 A6 ULTRASOUND-GUIDED HIFU NEUROLYSIS OF PERIPHERAL NERVES TO TREAT SPASTICITY AND | \n",
" J. L. Foley, J. W. Little, F. L. Starr Iii, C. Frantz | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" id \\\n",
"0 1 \n",
"1 2 \n",
"2 3 \n",
"3 4 \n",
"4 5 \n",
"\n",
" title \\\n",
"0 An Arithmetic Analogue of Bezouts Theorem \n",
"1 Thompsons Group F is Not Minimally Almost Convex \n",
"2 Cognitive Dimensions Tradeoffs in Tangible User Interface Design \n",
"3 ACTIVITY NOUNS, UNACCUSATIVITY, AND ARGUMENT MARKING IN YUKATEKAN SSILA meeting; Special Session... \n",
"4 PS1-6 A6 ULTRASOUND-GUIDED HIFU NEUROLYSIS OF PERIPHERAL NERVES TO TREAT SPASTICITY AND \n",
"\n",
" authors journal month \\\n",
"0 David Mckinnon NaN NaN \n",
"1 James Belk, Kai-uwe Bux NaN NaN \n",
"2 Darren Edge, Alan Blackwell NaN NaN \n",
"3 J. Bohnemeyer, Max Planck, I. Introduction NaN NaN \n",
"4 J. L. Foley, J. W. Little, F. L. Starr Iii, C. Frantz NaN NaN \n",
"\n",
" year publication_type \n",
"0 NaN NaN \n",
"1 2002.0 NaN \n",
"2 NaN NaN \n",
"3 2002.0 NaN \n",
"4 NaN NaN "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A.head()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" id | \n",
" title | \n",
" authors | \n",
" journal | \n",
" month | \n",
" year | \n",
" publication_type | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 1 | \n",
" Klaus Tschira Stiftung gemeinntzige GmbH, KTS | \n",
" Klaus Tschira | \n",
" NaN | \n",
" NaN | \n",
" 2012 | \n",
" www | \n",
"
\n",
" \n",
" 1 | \n",
" 2 | \n",
" The SGML/XML Web Page | \n",
" Robin Cover | \n",
" NaN | \n",
" NaN | \n",
" 2006 | \n",
" www | \n",
"
\n",
" \n",
" 2 | \n",
" 3 | \n",
" The Future of Classic Data Administration: Objects + Databases + CASE | \n",
" Arnon Rosenthal | \n",
" NaN | \n",
" NaN | \n",
" 1998 | \n",
" www | \n",
"
\n",
" \n",
" 3 | \n",
" 4 | \n",
" XML Query Data Model | \n",
" Mary F. Fernandez, Jonathan Robie | \n",
" NaN | \n",
" NaN | \n",
" 2001 | \n",
" www | \n",
"
\n",
" \n",
" 4 | \n",
" 5 | \n",
" The XML Query Algebra | \n",
" Peter Fankhauser, Mary F. Fernndez, Ashok Malhotra, Michael Rys, Jrme Simon, Philip Wadler | \n",
" NaN | \n",
" NaN | \n",
" 2001 | \n",
" www | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" id title \\\n",
"0 1 Klaus Tschira Stiftung gemeinntzige GmbH, KTS \n",
"1 2 The SGML/XML Web Page \n",
"2 3 The Future of Classic Data Administration: Objects + Databases + CASE \n",
"3 4 XML Query Data Model \n",
"4 5 The XML Query Algebra \n",
"\n",
" authors \\\n",
"0 Klaus Tschira \n",
"1 Robin Cover \n",
"2 Arnon Rosenthal \n",
"3 Mary F. Fernandez, Jonathan Robie \n",
"4 Peter Fankhauser, Mary F. Fernndez, Ashok Malhotra, Michael Rys, Jrme Simon, Philip Wadler \n",
"\n",
" journal month year publication_type \n",
"0 NaN NaN 2012 www \n",
"1 NaN NaN 2006 www \n",
"2 NaN NaN 1998 www \n",
"3 NaN NaN 2001 www \n",
"4 NaN NaN 2001 www "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"B.head()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Set 'id' as the keys to the input tables\n",
"em.set_key(A, 'id')\n",
"em.set_key(B, 'id')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"('id', 'id')"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Display the keys\n",
"em.get_key(A), em.get_key(B)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Downsample the Input Tables"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, the input tables can be down sampled as shown below:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Downsample the datasets \n",
"sample_A, sample_B = em.down_sample(A, B, size=1000, y_param=1, show_progress=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the down_sample command, set the `size` to the number of tuples that should be sampled from B (this would be the size of sampled B table) and set the `y_param` to be the number of tuples to be picked from A (for each tuple in sampled B table). In the above, we set the number of tuples to be sampled from B to be 1000. We set the `y_param` to 1."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Display the number of tuples in the sampled datasets\n",
"len(sample_A), len(sample_B)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, the input tables `A` and `B` have been down sampled to smaller tables `sample_A` and `sample_B` ."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Show the metadata of sample_A, sample_B\n",
"em.show_properties(sample_A)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"em.show_properties(sample_B)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the sampled tables retain the same properies as the input tables."
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 0
}