{
"cells": [
{
"cell_type": "markdown",
"id": "cardiac-movement",
"metadata": {},
"source": [
"## Step - 3: A Framework for Unlocking and Linking WWII Japanese American Incarceration Biographical Data - Data Visualization"
]
},
{
"cell_type": "markdown",
"id": "iraqi-grade",
"metadata": {},
"source": [
"In the previous two modules we explored ways to manipulate and structure data to prepare the data for modeling and visualization.\n",
"\n",
"As seen below there are several tools (i.e., libraries) that will need to be imported into your jupyter notebook to model the data. Once these libraries have been imported the next step is to import your csv files into your notebook. These files can be ones created separately in an Excel spreadsheet that were then saved as a comma separated value (csv) file, or they can be datasets manipulated in a jupyter notebooks (as we did in part 2), or even a combination of the both. Keep in mind, your datasets can still be manipulated in your jupyter notebook as we are modeling since both pandas and numpy are being imported into the notebook. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "armed-undergraduate",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# import the following libraries for visualization purposes\n",
"import pandas as pd\n",
"import networkx\n",
"import numpy as np\n",
"import geopandas as gpd\n",
"import shapely as shp\n",
"from plotly.offline import init_notebook_mode, iplot\n",
"init_notebook_mode(connected=True)\n",
"import plotly.graph_objs as go\n",
"import plotly.graph_objects as go\n",
"import plotly.express as px\n",
"import plotly\n",
"\n",
"import plotly.graph_objs as go\n",
"from plotly.offline import iplot, init_notebook_mode\n",
"# # Using plotly + cufflinks in offline mode\n",
"# import cufflinks\n",
"# cufflinks.go_offline(connected=True)\n",
"init_notebook_mode(connected=True)\n",
"\n",
"# import matplotlib library\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# import bokeh library for network analysis\n",
"from bokeh.io import output_notebook, show, save\n",
"\n",
"# the libraries below are needed for Network Visualization\n",
"from bokeh.io import output_notebook, show, save\n",
"from bokeh.models import Range1d, Circle, ColumnDataSource, MultiLine\n",
"from bokeh.plotting import figure\n",
"from bokeh.plotting import from_networkx\n",
"\n",
"# these are needed for Geo Map visualization below\n",
"import plotly.figure_factory as ff\n",
"from urllib.request import urlopen\n",
"import json\n",
"with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response:\n",
" counties = json.load(response) "
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "consecutive-cloud",
"metadata": {},
"outputs": [],
"source": [
"# Import the csv data files\n",
"fullstackeddf = pd.read_csv('python-fullmovements-stacked.csv',dtype=object,na_values=[],keep_default_na=False)\n",
"pathsdf = pd.read_csv('python-paths.csv',dtype=object,na_values=[],keep_default_na=False)\n",
"totalsdf = pd.read_csv('python-totals.csv',dtype=object,na_values=[],keep_default_na=False)\n",
"#totalsdf['total'] = totalsdf['total'].astype(np.int64) #have to change the type for total to integer\n"
]
},
{
"cell_type": "markdown",
"id": "heard-tyler",
"metadata": {},
"source": [
"If you find that you are not able to import a library into your notebook then it is likely you will need to install the library. A python library can be installed by using !pip install (and the name of the library). Once the library has been installed then the library will need to be imported into the notebook, as shown below. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "occasional-laser",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: cufflinks in /opt/conda/lib/python3.8/site-packages (0.17.3)\n",
"Requirement already satisfied: colorlover>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (0.3.0)\n",
"Requirement already satisfied: ipython>=5.3.0 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (7.20.0)\n",
"Requirement already satisfied: pandas>=0.19.2 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (1.2.2)\n",
"Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (1.15.0)\n",
"Requirement already satisfied: plotly>=4.1.1 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (5.9.0)\n",
"Requirement already satisfied: setuptools>=34.4.1 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (49.6.0.post20210108)\n",
"Requirement already satisfied: ipywidgets>=7.0.0 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (7.6.3)\n",
"Requirement already satisfied: numpy>=1.9.2 in /opt/conda/lib/python3.8/site-packages (from cufflinks) (1.19.5)\n",
"Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (5.0.5)\n",
"Requirement already satisfied: decorator in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (4.4.2)\n",
"Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (3.0.16)\n",
"Requirement already satisfied: backcall in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (0.2.0)\n",
"Requirement already satisfied: pygments in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (2.7.4)\n",
"Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (0.18.0)\n",
"Requirement already satisfied: pickleshare in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (0.7.5)\n",
"Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.8/site-packages (from ipython>=5.3.0->cufflinks) (4.8.0)\n",
"Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.8/site-packages (from ipywidgets>=7.0.0->cufflinks) (5.4.3)\n",
"Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets>=7.0.0->cufflinks) (3.5.1)\n",
"Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets>=7.0.0->cufflinks) (5.1.2)\n",
"Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets>=7.0.0->cufflinks) (1.0.0)\n",
"Requirement already satisfied: tornado>=4.2 in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->cufflinks) (6.1)\n",
"Requirement already satisfied: jupyter-client in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.0.0->cufflinks) (6.1.11)\n",
"Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from jedi>=0.16->ipython>=5.3.0->cufflinks) (0.8.1)\n",
"Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets>=7.0.0->cufflinks) (3.2.0)\n",
"Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets>=7.0.0->cufflinks) (0.2.0)\n",
"Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets>=7.0.0->cufflinks) (4.7.1)\n",
"Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.0.0->cufflinks) (0.17.3)\n",
"Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.0.0->cufflinks) (20.3.0)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.8/site-packages (from pandas>=0.19.2->cufflinks) (2.8.1)\n",
"Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas>=0.19.2->cufflinks) (2021.1)\n",
"Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.8/site-packages (from pexpect>4.3->ipython>=5.3.0->cufflinks) (0.7.0)\n",
"Requirement already satisfied: tenacity>=6.2.0 in /opt/conda/lib/python3.8/site-packages (from plotly>=4.1.1->cufflinks) (8.0.1)\n",
"Requirement already satisfied: wcwidth in /opt/conda/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.3.0->cufflinks) (0.2.5)\n",
"Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.8/site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (6.2.0)\n",
"Requirement already satisfied: Send2Trash>=1.5.0 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (1.5.0)\n",
"Requirement already satisfied: pyzmq>=17 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (20.0.0)\n",
"Requirement already satisfied: nbconvert in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (6.0.7)\n",
"Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.9.0)\n",
"Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (20.1.0)\n",
"Requirement already satisfied: jinja2 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (2.11.3)\n",
"Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.9.2)\n",
"Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (1.14.4)\n",
"Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (2.20)\n",
"Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.8/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (1.1.1)\n",
"Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.3)\n",
"Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.1.2)\n",
"Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (1.4.2)\n",
"Requirement already satisfied: testpath in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.4.4)\n",
"Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.8.4)\n",
"Requirement already satisfied: defusedxml in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.6.0)\n",
"Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.5.2)\n",
"Requirement already satisfied: bleach in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (3.3.0)\n",
"Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.8/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (1.4.3)\n",
"Requirement already satisfied: async-generator in /opt/conda/lib/python3.8/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (1.10)\n",
"Requirement already satisfied: webencodings in /opt/conda/lib/python3.8/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (0.5.1)\n",
"Requirement already satisfied: packaging in /opt/conda/lib/python3.8/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (20.9)\n",
"Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging->bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->cufflinks) (2.4.7)\n"
]
},
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# The cufflinks package handles temp objects which is a pandas.series dataframe 'df' created above which does not have a iplot method when not linked to plotly. \n",
"# We need cufflinks to link plotly to pandas and add the iplot method:\n",
"# import cufflinks now\n",
"import cufflinks as cf\n",
"cf.go_offline()\n",
"cf.set_config_file(offline=False, world_readable=True)"
]
},
{
"cell_type": "markdown",
"id": "emotional-airline",
"metadata": {},
"source": [
"### Plotly Maps"
]
},
{
"cell_type": "markdown",
"id": "diagnostic-crossing",
"metadata": {},
"source": [
"Plotly offers a variety of tools to model geographical data in unique ways. For example, the first three maps use Plotly's scatter_geo feature to plot points. Whereas, the fourth visualization still harnesses Plotly's scatter_geo but maps count data so that we can view relative sizes at each location. Other features of Plotly are scattergeo and .add_trace. These two features can be used together, as shown in the fifth, sixth, and seventh map, to model and visualize paths. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "subject-factor",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"geo": "geo",
"hovertemplate": "%{hovertext}
city=san diego
order=origin
lat=%{lat}
long=%{lon}
city=san diego
order=origin
lat=%{lat}
long=%{lon}
city=santa anita
order=assembly
lat=%{lat}
long=%{lon}
city=jerome
order=first camp
lat=%{lat}
long=%{lon}
city=tule lake
order=second camp
lat=%{lat}
long=%{lon}
city=pennsylvania
order=final departure
lat=%{lat}
long=%{lon}