{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pathfinding\n",
"\n",
"Different ontologies exhibit different degrees of latticeyness. Highly latticed ontologies will have a combinatorial expolosion of paths to a root node.\n",
"\n",
"This notebook has an analysis of path counts for the HPO\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"## We use a Factory object in the ontobio library\n",
"from ontobio import OntologyFactory"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"## Get the HPO using default method (currently OntoBee SPARQL)\n",
"## This may take 5-10s the first time you run it; afterwards it is cached\n",
"ofa = OntologyFactory()\n",
"ont = ofa.create('hp')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"## The OWL version of HPO (used here) has many interesting relationship types;\n",
"## for now we just care about is-a (subClassOf between named classes)\n",
"ont = ont.subontology(relations='subClassOf')"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'HP:0000118'"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## Get the root of the abnormality subset\n",
"[root] = ont.search('Phenotypic abnormality')\n",
"root"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'HP:0040024'"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## Arbitrary term\n",
"[t] = ont.search('Clinodactyly of the 3rd finger')\n",
"t"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<networkx.classes.multidigraph.MultiDiGraph at 0x10d7292b0>"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## We use the standard python networkx library for pathfinding here\n",
"## This is easily extracted from an ontology object\n",
"from networkx import nx\n",
"G = ont.get_graph()\n",
"G"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use networkx to find all paths from an arbitrary term\n",
"\n",
"See https://networkx.github.io/documentation/development/reference/generated/networkx.algorithms.simple_paths.all_simple_paths.html"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"17"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## number of paths\n",
"## (for the mapping of networkx to an ontology, source is root, and descendant is target)\n",
"len(list(nx.all_simple_paths(G, root, t)))"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[['HP:0000118',\n",
" 'HP:0000924',\n",
" 'HP:0040068',\n",
" 'HP:0002813',\n",
" 'HP:0011297',\n",
" 'HP:0030084',\n",
" 'HP:0040019',\n",
" 'HP:0040024'],\n",
" ['HP:0000118',\n",
" 'HP:0000924',\n",
" 'HP:0040068',\n",
" 'HP:0002813',\n",
" 'HP:0011297',\n",
" 'HP:0001167',\n",
" 'HP:0004097',\n",
" 'HP:0009317',\n",
" 'HP:0040024']]"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## nx returns a list of lists, each list is a path\n",
"## Examine the first 2\n",
"list(nx.all_simple_paths(G, root, t))[0:2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## We (heart) pandas\n",
"\n",
"Pandas are cute.\n",
"\n",
"We use a DataFrame object, which we will construct by making a table of terms plus their pathstats"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'id': 'HP:0005237',\n",
" 'label': 'Degenerative liver disease',\n",
" 'longest': 5,\n",
" 'pathcount': 1},\n",
" {'id': 'HP:0002251',\n",
" 'label': 'Aganglionic megacolon',\n",
" 'longest': 8,\n",
" 'pathcount': 3},\n",
" {'id': 'HP:0005102',\n",
" 'label': 'Cochlear degeneration',\n",
" 'longest': 6,\n",
" 'pathcount': 1}]"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"def get_pathstats(nodes):\n",
" \"\"\"\n",
" for any given node, return a table row with stats\n",
" \"\"\"\n",
" items = []\n",
" for n in nodes:\n",
" paths = list(nx.all_simple_paths(G, root, n))\n",
" longest = len(max(paths, key=lambda p: len(p)))\n",
" items.append({'id':n, \n",
" 'label': ont.label(n),\n",
" 'pathcount': len(paths),\n",
" 'longest': longest})\n",
" return items\n",
"\n",
"## Test it out\n",
"sample = list(ont.descendants(root))[0:20]\n",
"items = get_pathstats(sample)\n",
"items[0:3]"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>label</th>\n",
" <th>longest</th>\n",
" <th>pathcount</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>HP:0005237</td>\n",
" <td>Degenerative liver disease</td>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>HP:0002251</td>\n",
" <td>Aganglionic megacolon</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>HP:0005102</td>\n",
" <td>Cochlear degeneration</td>\n",
" <td>6</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>HP:0006466</td>\n",
" <td>Ankle contracture</td>\n",
" <td>9</td>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>HP:0004292</td>\n",
" <td>Undermodelled hand bones</td>\n",
" <td>6</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>HP:0004839</td>\n",
" <td>Pyropoikilocytosis</td>\n",
" <td>7</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>HP:0008970</td>\n",
" <td>Scapulohumeral muscular dystrophy</td>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>HP:0008573</td>\n",
" <td>Low-frequency sensorineural hearing impairment</td>\n",
" <td>6</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>HP:0005435</td>\n",
" <td>Impaired T cell function</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>HP:0009218</td>\n",
" <td>Fragmentation of the epiphysis of the middle p...</td>\n",
" <td>13</td>\n",
" <td>96</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>HP:0005021</td>\n",
" <td>Bilateral elbow dislocations</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>HP:0010964</td>\n",
" <td>Abnormality of long-chain fatty-acid metabolism</td>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>HP:0008019</td>\n",
" <td>Superior lens subluxation</td>\n",
" <td>9</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>HP:0030883</td>\n",
" <td>Femoroacetabular Impingement</td>\n",
" <td>8</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>HP:0005303</td>\n",
" <td>Aortic arch calcification</td>\n",
" <td>9</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>HP:0000741</td>\n",
" <td>Apathy</td>\n",
" <td>7</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>HP:0040208</td>\n",
" <td>Elevated CSF biopterin level</td>\n",
" <td>7</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>HP:0030031</td>\n",
" <td>Small toe</td>\n",
" <td>10</td>\n",
" <td>13</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>HP:0025348</td>\n",
" <td>Abnormality of the corneal limbus</td>\n",
" <td>7</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>HP:0100720</td>\n",
" <td>Hypoplasia of the ear cartilage</td>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id label longest \\\n",
"0 HP:0005237 Degenerative liver disease 5 \n",
"1 HP:0002251 Aganglionic megacolon 8 \n",
"2 HP:0005102 Cochlear degeneration 6 \n",
"3 HP:0006466 Ankle contracture 9 \n",
"4 HP:0004292 Undermodelled hand bones 6 \n",
"5 HP:0004839 Pyropoikilocytosis 7 \n",
"6 HP:0008970 Scapulohumeral muscular dystrophy 5 \n",
"7 HP:0008573 Low-frequency sensorineural hearing impairment 6 \n",
"8 HP:0005435 Impaired T cell function 8 \n",
"9 HP:0009218 Fragmentation of the epiphysis of the middle p... 13 \n",
"10 HP:0005021 Bilateral elbow dislocations 8 \n",
"11 HP:0010964 Abnormality of long-chain fatty-acid metabolism 5 \n",
"12 HP:0008019 Superior lens subluxation 9 \n",
"13 HP:0030883 Femoroacetabular Impingement 8 \n",
"14 HP:0005303 Aortic arch calcification 9 \n",
"15 HP:0000741 Apathy 7 \n",
"16 HP:0040208 Elevated CSF biopterin level 7 \n",
"17 HP:0030031 Small toe 10 \n",
"18 HP:0025348 Abnormality of the corneal limbus 7 \n",
"19 HP:0100720 Hypoplasia of the ear cartilage 5 \n",
"\n",
" pathcount \n",
"0 1 \n",
"1 3 \n",
"2 1 \n",
"3 6 \n",
"4 1 \n",
"5 1 \n",
"6 1 \n",
"7 2 \n",
"8 3 \n",
"9 96 \n",
"10 3 \n",
"11 1 \n",
"12 1 \n",
"13 4 \n",
"14 5 \n",
"15 1 \n",
"16 2 \n",
"17 13 \n",
"18 1 \n",
"19 1 "
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## Look at same table in pandas\n",
"import pandas as pd\n",
"df = pd.DataFrame(items)\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"7.3499999999999996"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## Basic aggregate stats (over our small sample, which may not be representative)\n",
"df['pathcount'].mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plotting with plotly\n",
"\n",
"Let's do a simple barchart showing distribution of pathcounts for our sample"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"\n",
"import plotly.plotly as py\n",
"import plotly.graph_objs as go\n"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~cmungall/13.embed\" height=\"525px\" width=\"100%\"></iframe>"
],
"text/plain": [
"<plotly.tools.PlotlyDisplay object>"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = [\n",
" go.Bar(\n",
" x=df['label'], # assign x as the dataframe column 'x'\n",
" y=df['pathcount']\n",
" )\n",
"]\n",
"\n",
"# IPython notebook\n",
"py.iplot(data, filename='pandas-bar-chart')\n",
"\n",
"# use this in non-notebook context\n",
"# url = py.plot(data, filename='pandas-bar-chart')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summarizing over whole ontology\n",
"\n",
"__warning__ this can take over an hour, if running interactively, be patient!\n",
"\n",
"__help wanted__ is there a way to make Jupyter show a progress bar for cases like this?\n"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'id': 'HP:0005237',\n",
" 'label': 'Degenerative liver disease',\n",
" 'longest': 5,\n",
" 'pathcount': 1},\n",
" {'id': 'HP:0002251',\n",
" 'label': 'Aganglionic megacolon',\n",
" 'longest': 8,\n",
" 'pathcount': 3},\n",
" {'id': 'HP:0005102',\n",
" 'label': 'Cochlear degeneration',\n",
" 'longest': 6,\n",
" 'pathcount': 1}]"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sample = list(ont.descendants(root))\n",
"items = get_pathstats(sample)\n",
"items[0:3]"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"12066"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(items)"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"df = pd.DataFrame(items)\n"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"6.6176031824962704"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['pathcount'].mean()"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"200"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['pathcount'].max()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plotting all HP terms\n"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~cmungall/15.embed\" height=\"525px\" width=\"100%\"></iframe>"
],
"text/plain": [
"<plotly.tools.PlotlyDisplay object>"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = [\n",
" go.Bar(\n",
" x=df['label'], # assign x as the dataframe column 'x'\n",
" y=df['pathcount']\n",
" )\n",
"]\n",
"\n",
"# IPython notebook\n",
"py.iplot(data, filename='pandas-bar-chart-all')\n"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~cmungall/17.embed\" height=\"525px\" width=\"100%\"></iframe>"
],
"text/plain": [
"<plotly.tools.PlotlyDisplay object>"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"data = [\n",
" go.Scatter(\n",
" x=df['longest'], # assign x as the dataframe column 'x'\n",
" y=df['pathcount'],\n",
" mode = 'markers'\n",
" )\n",
"]\n",
"\n",
"# IPython notebook\n",
"py.iplot(data, filename='pandas-longest-vs-numpaths')\n"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['HP:0100379', 'HP:0010432', 'HP:0010102', 'HP:0100378']"
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"max_num_paths = df['pathcount'].max()\n",
"nodes_with_max = [x['id'] for x in items if x['pathcount'] == max_num_paths]\n",
"nodes_with_max"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['Aplasia of the distal phalanx of the 4th toe',\n",
" 'Absent distal phalanx of the 2nd toe',\n",
" 'Aplasia of the distal phalanx of the hallux',\n",
" 'Absent distal phalanx of the 3rd toe']"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[ont.label(n) for n in nodes_with_max]"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4"
]
},
"execution_count": 70,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(nodes_with_max)"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [],
"source": [
"## Pick an arbitrary term from list\n",
"t = nodes_with_max[0]"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"36"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ancs = ont.ancestors(t, reflexive=True)\n",
"ancs = [a for a in ancs if a.startswith('HP:')]\n",
"len(ancs)"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [],
"source": [
"## Make a sub-ontology with just term and ancestors\n",
"subont = ont.subontology(ancs)"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['HP:0000118',\n",
" 'HP:0000924',\n",
" 'HP:0040068',\n",
" 'HP:0040069',\n",
" 'HP:0006493',\n",
" 'HP:0006494',\n",
" 'HP:0001991',\n",
" 'HP:0010760',\n",
" 'HP:0010185',\n",
" 'HP:0100370',\n",
" 'HP:0100379']"
]
},
"execution_count": 83,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sample_path = list(nx.all_simple_paths(G, root, t))[0]\n",
"sample_path"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [],
"source": [
"## Render the sub-ontology,\n",
"## highlighting a sample path\n",
"from ontobio.io.ontol_renderers import GraphRenderer\n",
"w = GraphRenderer.create('png')\n",
"w.outfile = 'output/multipath.png'\n",
"w.write(subont,query_ids=sample_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}