{
"metadata": {
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
},
"orig_nbformat": 2,
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"metadata": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 2,
"cells": [
{
"source": [
"# Causal Discovery example\n",
"\n",
"The goal of this notebook is to show how causal discovery methods can work with DoWhy. We use discovery methods from [Causal Discovery Tool (CDT)](https://github.com/FenTechSolutions/CausalDiscoveryToolbox) repo. As we will see, causal discovery methods are not fool-proof and there is no guarantee that they will recover the correct causal graph. Even for the simple examples below, there is a large variance in results. These methods, however, may be combined usefully with domain knowledge to construct the final causal graph."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"The rpy2.ipython extension is already loaded. To reload it, use:\n %reload_ext rpy2.ipython\n"
]
}
],
"source": [
"import dowhy\n",
"from dowhy import CausalModel\n",
"\n",
"from rpy2.robjects import r as R\n",
"%load_ext rpy2.ipython\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"import graphviz\n",
"import networkx as nx \n",
"\n",
"np.set_printoptions(precision=3, suppress=True)\n",
"np.random.seed(0)"
]
},
{
"source": [
"## Utility function\n",
"We define a utility function to draw the directed acyclic graph."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"def make_graph(adjacency_matrix, labels=None):\n",
" idx = np.abs(adjacency_matrix) > 0.01\n",
" dirs = np.where(idx)\n",
" d = graphviz.Digraph(engine='dot')\n",
" names = labels if labels else [f'x{i}' for i in range(len(adjacency_matrix))]\n",
" for name in names:\n",
" d.node(name)\n",
" for to, from_, coef in zip(dirs[0], dirs[1], adjacency_matrix[idx]):\n",
" d.edge(names[from_], names[to], label=str(coef))\n",
" return d\n",
"\n",
"def str_to_dot(string):\n",
" '''\n",
" Converts input string from graphviz library to valid DOT graph format.\n",
" '''\n",
" graph = string.replace('\\n', ';').replace('\\t','')\n",
" graph = graph[:9] + graph[10:-2] + graph[-1] # Removing unnecessary characters from string\n",
" return graph"
]
},
{
"source": [
"# Experiments on the Auto-MPG dataset\n",
"\n",
"In this section, we will use a dataset on the technical specification of cars. The dataset is downloaded from UCI Machine Learning Repository. The dataset contains 9 attributes and 398 instances. We do not know the true causal graph for the dataset and will use CDT to discover it. The causal graph obtained will then be used to estimate the causal effect.\n"
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"## 1. Load the data"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"(392, 6)\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
" mpg cylinders displacement horsepower weight acceleration\n",
"0 18.0 8.0 307.0 130.0 3504.0 12.0\n",
"1 15.0 8.0 350.0 165.0 3693.0 11.5\n",
"2 18.0 8.0 318.0 150.0 3436.0 11.0\n",
"3 16.0 8.0 304.0 150.0 3433.0 12.0\n",
"4 17.0 8.0 302.0 140.0 3449.0 10.5"
],
"text/html": "
\n\n
\n \n
\n
\n
mpg
\n
cylinders
\n
displacement
\n
horsepower
\n
weight
\n
acceleration
\n
\n \n \n
\n
0
\n
18.0
\n
8.0
\n
307.0
\n
130.0
\n
3504.0
\n
12.0
\n
\n
\n
1
\n
15.0
\n
8.0
\n
350.0
\n
165.0
\n
3693.0
\n
11.5
\n
\n
\n
2
\n
18.0
\n
8.0
\n
318.0
\n
150.0
\n
3436.0
\n
11.0
\n
\n
\n
3
\n
16.0
\n
8.0
\n
304.0
\n
150.0
\n
3433.0
\n
12.0
\n
\n
\n
4
\n
17.0
\n
8.0
\n
302.0
\n
140.0
\n
3449.0
\n
10.5
\n
\n \n
\n
"
},
"metadata": {},
"execution_count": 14
}
],
"source": [
"data_mpg = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data-original',\n",
" delim_whitespace=True, header=None,\n",
" names = ['mpg', 'cylinders', 'displacement',\n",
" 'horsepower', 'weight', 'acceleration',\n",
" 'model year', 'origin', 'car name'])\n",
"data_mpg.dropna(inplace=True)\n",
"data_mpg.drop(['model year', 'origin', 'car name'], axis=1, inplace=True)\n",
"print(data_mpg.shape)\n",
"data_mpg.head()"
]
},
{
"source": [
"# Causal Discovery with Causal Discovery Tool (CDT)\n",
"\n",
"We use the CDT library to perform causal discovery on the Auto-MPG dataset. We use three methods for causal discovery here -LiNGAM, PC and GES. These methods are widely used and do not take much time to run. Hence, these are ideal for an introduction to the topic. Other neural network based methods are also available in CDT and the users are encouraged to try them out by themselves. \n",
"\n",
"The documentation for the methods used are as follows:\n",
"- LiNGAM [[link]](https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/_modules/cdt/causality/graph/LiNGAM.html)\n",
"- PC [[link]](https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/_modules/cdt/causality/graph/PC.html)\n",
"- GES [[link]](https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/_modules/cdt/causality/graph/GES.html)"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Method : LiNGAM\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Method : PC\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Method : GES\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
}
],
"source": [
"from cdt.causality.graph import LiNGAM, PC, GES\n",
"\n",
"graphs = {}\n",
"labels = [f'{col}' for i, col in enumerate(data_mpg.columns)]\n",
"functions = {\n",
" 'LiNGAM' : LiNGAM,\n",
" 'PC' : PC,\n",
" 'GES' : GES,\n",
"}\n",
"\n",
"for method, lib in functions.items():\n",
" obj = lib()\n",
" output = obj.predict(data_mpg)\n",
" adj_matrix = nx.to_numpy_matrix(output)\n",
" adj_matrix = np.asarray(adj_matrix)\n",
" graph_dot = make_graph(adj_matrix, labels)\n",
" graphs[method] = graph_dot\n",
"\n",
"# Visualize graphs\n",
"for method, graph in graphs.items():\n",
" print(\"Method : %s\"%(method))\n",
" display(graph)"
]
},
{
"source": [
"As you can see, no two methods agree on the graphs. PC and GES effectively produce an undirected graph whereas LiNGAM produces a directed graph. In the absence of directed edges, the causal effect can be obtained for PC and GES for the Auto-MPG dataset. This can be observed in the next section."
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"## Estimate causal effects using Linear Regression\n",
"\n",
"Now let us see whether these differences in the graphs also lead to signficant differences in the causal estimate of effect of *mpg* on *weight*."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"*****************************************************************************\n",
"\n",
"Causal Discovery Method : LiNGAM\n",
"Estimand type: nonparametric-ate\n",
"\n",
"### Estimand : 1\n",
"Estimand name: backdoor\n",
"Estimand expression:\n",
" d \n",
"──────(Expectation(weight|cylinders,acceleration))\n",
"d[mpg] \n",
"Estimand assumption 1, Unconfoundedness: If U→{mpg} and U→weight then P(weight|mpg,cylinders,acceleration,U) = P(weight|mpg,cylinders,acceleration)\n",
"\n",
"### Estimand : 2\n",
"Estimand name: iv\n",
"No such variable found!\n",
"\n",
"### Estimand : 3\n",
"Estimand name: frontdoor\n",
"No such variable found!\n",
"\n",
"Causal Estimate is -39.56010817288552\n",
"\n",
"*****************************************************************************\n",
"\n",
"Causal Discovery Method : PC\n",
"Estimand type: nonparametric-ate\n",
"\n",
"### Estimand : 1\n",
"Estimand name: backdoor\n",
"No such variable found!\n",
"\n",
"### Estimand : 2\n",
"Estimand name: iv\n",
"No such variable found!\n",
"\n",
"### Estimand : 3\n",
"Estimand name: frontdoor\n",
"No such variable found!\n",
"\n",
"Causal Estimate is None\n",
"\n",
"*****************************************************************************\n",
"\n",
"Causal Discovery Method : GES\n",
"Estimand type: nonparametric-ate\n",
"\n",
"### Estimand : 1\n",
"Estimand name: backdoor\n",
"No such variable found!\n",
"\n",
"### Estimand : 2\n",
"Estimand name: iv\n",
"No such variable found!\n",
"\n",
"### Estimand : 3\n",
"Estimand name: frontdoor\n",
"No such variable found!\n",
"\n",
"Causal Estimate is None\n"
]
}
],
"source": [
"for method, graph in graphs.items():\n",
" print('\\n*****************************************************************************\\n')\n",
" print(\"Causal Discovery Method : %s\"%(method))\n",
"\n",
" # Obtain valid dot format\n",
" graph_dot = str_to_dot(graph.source)\n",
"\n",
" # Define Causal Model\n",
" model=CausalModel(\n",
" data = data_mpg,\n",
" treatment='mpg',\n",
" outcome='weight',\n",
" graph=graph_dot)\n",
"\n",
" # Identification\n",
" identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)\n",
" print(identified_estimand)\n",
" \n",
" # Estimation\n",
" estimate = model.estimate_effect(identified_estimand,\n",
" method_name=\"backdoor.linear_regression\",\n",
" control_value=0,\n",
" treatment_value=1,\n",
" confidence_intervals=True,\n",
" test_significance=True)\n",
" print(\"Causal Estimate is \" + str(estimate.value))"
]
},
{
"source": [
"As mentioned earlier, due to the absence of directed edges, no backdoor, instrmental or frontdoor variables can be found out for PC and GES. Thus, causal effect estimation is not possible for these methods. However, LiNGAM does discover a DAG and hence, its possible to output a causal estimate for LiNGAM. The estimate is still pretty far from the original estimate of -70.466 (which can be calculated from the graph)."
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"# Experiments on the Sachs dataset\n",
"\n",
"The dataset consists of the simultaneous measurements of 11 phosphorylated proteins and phospholipids derived from thousands of individual primary immune system cells, subjected to both general and specific molecular interventions (Sachs et al., 2005).\n",
"\n",
"The specifications of the dataset are as follows - \n",
"- Number of nodes: 11\n",
"- Number of arcs: 17\n",
"- Number of parameters: 178\n",
"- Average Markov blanket size: 3.09\n",
"- Average degree: 3.09\n",
"- Maximum in-degree: 3\n",
"- Number of instances: 7466\n",
"\n",
"The original causal graph is known for the Sachs dataset and we compare the original graph with the ones discovered using CDT in this section."
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"## 1. Load the data"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"(7466, 11)\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
" praf pmek plcg PIP2 PIP3 p44/42 pakts473 PKA PKC P38 pjnk\n",
"0 26.4 13.2 8.82 18.30 58.80 6.61 17.0 414.0 17.00 44.9 40.0\n",
"1 35.9 16.5 12.30 16.80 8.13 18.60 32.5 352.0 3.37 16.5 61.5\n",
"2 59.4 44.1 14.60 10.20 13.00 14.90 32.5 403.0 11.40 31.9 19.5\n",
"3 73.0 82.8 23.10 13.50 1.29 5.83 11.8 528.0 13.70 28.6 23.1\n",
"4 33.7 19.8 5.19 9.73 24.80 21.10 46.1 305.0 4.66 25.7 81.3"
],
"text/html": "
\n\n
\n \n
\n
\n
praf
\n
pmek
\n
plcg
\n
PIP2
\n
PIP3
\n
p44/42
\n
pakts473
\n
PKA
\n
PKC
\n
P38
\n
pjnk
\n
\n \n \n
\n
0
\n
26.4
\n
13.2
\n
8.82
\n
18.30
\n
58.80
\n
6.61
\n
17.0
\n
414.0
\n
17.00
\n
44.9
\n
40.0
\n
\n
\n
1
\n
35.9
\n
16.5
\n
12.30
\n
16.80
\n
8.13
\n
18.60
\n
32.5
\n
352.0
\n
3.37
\n
16.5
\n
61.5
\n
\n
\n
2
\n
59.4
\n
44.1
\n
14.60
\n
10.20
\n
13.00
\n
14.90
\n
32.5
\n
403.0
\n
11.40
\n
31.9
\n
19.5
\n
\n
\n
3
\n
73.0
\n
82.8
\n
23.10
\n
13.50
\n
1.29
\n
5.83
\n
11.8
\n
528.0
\n
13.70
\n
28.6
\n
23.1
\n
\n
\n
4
\n
33.7
\n
19.8
\n
5.19
\n
9.73
\n
24.80
\n
21.10
\n
46.1
\n
305.0
\n
4.66
\n
25.7
\n
81.3
\n
\n \n
\n
"
},
"metadata": {},
"execution_count": 17
}
],
"source": [
"from cdt.data import load_dataset\n",
"data_sachs, graph_sachs = load_dataset(\"sachs\")\n",
"\n",
"data_sachs.dropna(inplace=True)\n",
"print(data_sachs.shape)\n",
"data_sachs.head()"
]
},
{
"source": [
"## Ground truth of the causal graph"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
}
],
"source": [
"labels = [f'{col}' for i, col in enumerate(data_sachs.columns)]\n",
"adj_matrix = nx.to_numpy_matrix(graph_sachs)\n",
"adj_matrix = np.asarray(adj_matrix)\n",
"graph_dot = make_graph(adj_matrix, labels)\n",
"display(graph_dot)"
]
},
{
"source": [
"# Causal Discovery with Causal Discovery Tool (CDT)\n",
"\n",
"We use the CDT library to perform causal discovery on the Auto-MPG dataset. We use three methods for causal discovery here -LiNGAM, PC and GES. These methods are widely used and do not take much time to run. Hence, these are ideal for an introduction to the topic. Other neural network based methods are also available in CDT and the users the encourages to try them out by themselves. \n",
"\n",
"The documentation for the methods used in as follows:\n",
"- LiNGAM [[link]](https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/_modules/cdt/causality/graph/LiNGAM.html)\n",
"- PC [[link]](https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/_modules/cdt/causality/graph/PC.html)\n",
"- GES [[link]](https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/_modules/cdt/causality/graph/GES.html)"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Method : LiNGAM\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Method : PC\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Method : GES\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": "\n\n\n\n\n"
},
"metadata": {}
}
],
"source": [
"from cdt.causality.graph import LiNGAM, PC, GES\n",
"\n",
"graphs = {}\n",
"graphs_nx = {}\n",
"labels = [f'{col}' for i, col in enumerate(data_sachs.columns)]\n",
"functions = {\n",
" 'LiNGAM' : LiNGAM,\n",
" 'PC' : PC,\n",
" 'GES' : GES,\n",
"}\n",
"\n",
"for method, lib in functions.items():\n",
" obj = lib()\n",
" output = obj.predict(data_sachs)\n",
" graphs_nx[method] = output\n",
" adj_matrix = nx.to_numpy_matrix(output)\n",
" adj_matrix = np.asarray(adj_matrix)\n",
" graph_dot = make_graph(adj_matrix, labels)\n",
" graphs[method] = graph_dot\n",
"\n",
"# Visualize graphs\n",
"for method, graph in graphs.items():\n",
" print(\"Method : %s\"%(method))\n",
" display(graph)"
]
},
{
"source": [
"As you can see, no two methods agree on the graphs. Next we study the causal effects of these different graphs"
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"## Estimate effects using Linear Regression\n",
"\n",
"Now let us see whether these differences in the graphs also lead to signficant differences in the causal estimate of effect of *PIP2* on *PKC*."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"*****************************************************************************\n",
"\n",
"Causal Discovery Method : LiNGAM\n",
"Estimand type: nonparametric-ate\n",
"\n",
"### Estimand : 1\n",
"Estimand name: backdoor\n",
"Estimand expression:\n",
" d \n",
"───────(Expectation(PKC))\n",
"d[PIP₂] \n",
"Estimand assumption 1, Unconfoundedness: If U→{PIP2} and U→PKC then P(PKC|PIP2,,U) = P(PKC|PIP2,)\n",
"\n",
"### Estimand : 2\n",
"Estimand name: iv\n",
"Estimand expression:\n",
"Expectation(Derivative(PKC, [PIP3, plcg])*Derivative([PIP2], [PIP3, plcg])**(-\n",
"1))\n",
"Estimand assumption 1, As-if-random: If U→→PKC then ¬(U →→{PIP3,plcg})\n",
"Estimand assumption 2, Exclusion: If we remove {PIP3,plcg}→{PIP2}, then ¬({PIP3,plcg}→PKC)\n",
"\n",
"### Estimand : 3\n",
"Estimand name: frontdoor\n",
"No such variable found!\n",
"\n",
"Causal Estimate is 0.10456597412424884\n",
"\n",
"*****************************************************************************\n",
"\n",
"Causal Discovery Method : PC\n",
"Estimand type: nonparametric-ate\n",
"\n",
"### Estimand : 1\n",
"Estimand name: backdoor\n",
"Estimand expression:\n",
" d \n",
"───────(Expectation(PKC))\n",
"d[PIP₂] \n",
"Estimand assumption 1, Unconfoundedness: If U→{PIP2} and U→PKC then P(PKC|PIP2,,U) = P(PKC|PIP2,)\n",
"\n",
"### Estimand : 2\n",
"Estimand name: iv\n",
"No such variable found!\n",
"\n",
"### Estimand : 3\n",
"Estimand name: frontdoor\n",
"Estimand expression:\n",
"Expectation(Derivative(PKC, [plcg])*Derivative([plcg], [PIP2]))\n",
"Estimand assumption 1, Full-mediation: plcg intercepts (blocks) all directed paths from PIP2 to P,K,C.\n",
"Estimand assumption 2, First-stage-unconfoundedness: If U→{PIP2} and U→{plcg} then P(plcg|PIP2,U) = P(plcg|PIP2)\n",
"Estimand assumption 3, Second-stage-unconfoundedness: If U→{plcg} and U→PKC then P(PKC|plcg, PIP2, U) = P(PKC|plcg, PIP2)\n",
"\n",
"Causal Estimate is 0.10456597412424884\n",
"\n",
"*****************************************************************************\n",
"\n",
"Causal Discovery Method : GES\n",
"Estimand type: nonparametric-ate\n",
"\n",
"### Estimand : 1\n",
"Estimand name: backdoor\n",
"Estimand expression:\n",
" d \n",
"───────(Expectation(PKC|PKA,P38))\n",
"d[PIP₂] \n",
"Estimand assumption 1, Unconfoundedness: If U→{PIP2} and U→PKC then P(PKC|PIP2,PKA,P38,U) = P(PKC|PIP2,PKA,P38)\n",
"\n",
"### Estimand : 2\n",
"Estimand name: iv\n",
"No such variable found!\n",
"\n",
"### Estimand : 3\n",
"Estimand name: frontdoor\n",
"No such variable found!\n",
"\n",
"Causal Estimate is -0.013018300750143652\n"
]
}
],
"source": [
"for method, graph in graphs.items():\n",
" print('\\n*****************************************************************************\\n')\n",
" print(\"Causal Discovery Method : %s\"%(method))\n",
"\n",
" # Obtain valid dot format\n",
" graph_dot = str_to_dot(graph.source)\n",
"\n",
" # Define Causal Model\n",
" model=CausalModel(\n",
" data = data_sachs,\n",
" treatment='PIP2',\n",
" outcome='PKC',\n",
" graph=graph_dot)\n",
"\n",
" # Identification\n",
" identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)\n",
" print(identified_estimand)\n",
"\n",
" # Estimation\n",
" estimate = model.estimate_effect(identified_estimand,\n",
" method_name=\"backdoor.linear_regression\",\n",
" control_value=0,\n",
" treatment_value=1,\n",
" confidence_intervals=True,\n",
" test_significance=True)\n",
" print(\"Causal Estimate is \" + str(estimate.value))"
]
},
{
"source": [
"From the causal estimates obtained, it can be seen that the three estimates differ in different aspects. The graph obtained using LiNGAM contains a backdoor path and instrumental variables. On the other hand, the graph obtained using PC contains a backdoor path and a frontdoor path. However, despite these differences, both obtain the same mean causal estimate.\n",
"\n",
"The graph obtained using GES contains only a backdoor path with different backdoor variables and obtains a different causal estimate than the first two cases. "
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"## Graph Validation\n",
"\n",
"We compare the graphs obtained with the true causal graph using the causal discovery methods using 2 graph distance metrics - Structural Hamming Distance (SHD) and Structural Intervention Distance (SID). SHD between two graphs is, in simple terms, the number of edge insertions, deletions or flips in order to transform one graph to another graph. SID, on the other hand, is based on a graphical criterion only and quantifies the closeness between two DAGs in terms of their corresponding causal inference statements."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"***********************************************************\n",
"Method: LiNGAM\n",
"SHD_CPDAG = 17.000000\n",
"SHD = 18.000000\n",
"SID_CPDAG = [80.000000, 80.000000]\n",
"SID = 80.000000\n",
"***********************************************************\n",
"Method: PC\n",
"SHD_CPDAG = 29.000000\n",
"SHD = 27.000000\n",
"SID_CPDAG = [82.000000, 82.000000]\n",
"SID = 82.000000\n",
"***********************************************************\n",
"Method: GES\n",
"SHD_CPDAG = 34.000000\n",
"SHD = 31.000000\n",
"SID_CPDAG = [82.000000, 88.000000]\n",
"SID = 91.000000\n"
]
}
],
"source": [
"from cdt.metrics import SHD, SHD_CPDAG, SID, SID_CPDAG\n",
"from numpy.random import randint\n",
"\n",
"for method, graph in graphs_nx.items():\n",
" print(\"***********************************************************\")\n",
" print(\"Method: %s\"%(method))\n",
" tar, pred = graph_sachs, graph\n",
" print(\"SHD_CPDAG = %f\"%(SHD_CPDAG(tar, pred)))\n",
" print(\"SHD = %f\"%(SHD(tar, pred, double_for_anticausal=False)))\n",
" print(\"SID_CPDAG = [%f, %f]\"%(SID_CPDAG(tar, pred)))\n",
" print(\"SID = %f\"%(SID(tar, pred)))"
]
},
{
"source": [
"The graph similarity metrics show that the scores are the lowest for the LiNGAM method of graph extraction. Hence, of the three methods used, LiNGAM provides the graph that is most similar to the original graph."
],
"cell_type": "markdown",
"metadata": {}
},
{
"source": [
"## Graph Refutation\n",
"\n",
"Here, we use the same SHD and SID metric to find out how different the discovered graph are from each other."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"***********************************************************\n",
"Methods: LiNGAM and PC\n",
"SHD_CPDAG = 25.000000\n",
"SHD = 24.000000\n",
"SID_CPDAG = [13.000000, 13.000000]\n",
"SID = 13.000000\n",
"***********************************************************\n",
"Methods: LiNGAM and GES\n",
"SHD_CPDAG = 33.000000\n",
"SHD = 33.000000\n",
"SID_CPDAG = [10.000000, 13.000000]\n",
"SID = 13.000000\n",
"***********************************************************\n",
"Methods: PC and GES\n",
"SHD_CPDAG = 22.000000\n",
"SHD = 22.000000\n",
"SID_CPDAG = [70.000000, 78.000000]\n",
"SID = 78.000000\n"
]
}
],
"source": [
"import itertools\n",
"from numpy.random import randint\n",
"from cdt.metrics import SHD, SHD_CPDAG, SID, SID_CPDAG\n",
"\n",
"# Find combinations of pair of methods to compare\n",
"combinations = list(itertools.combinations(graphs_nx, 2))\n",
"\n",
"for pair in combinations:\n",
" print(\"***********************************************************\")\n",
" graph1 = graphs_nx[pair[0]]\n",
" graph2 = graphs_nx[pair[1]]\n",
" print(\"Methods: %s and %s\"%(pair[0], pair[1]))\n",
" print(\"SHD_CPDAG = %f\"%(SHD_CPDAG(graph1, graph2)))\n",
" print(\"SHD = %f\"%(SHD(graph1, graph2, double_for_anticausal=False)))\n",
" print(\"SID_CPDAG = [%f, %f]\"%(SID_CPDAG(graph1, graph2)))\n",
" print(\"SID = %f\"%(SID(graph1, graph2)))"
]
},
{
"source": [
"The values for the metrics show how different the graphs are from each other. A higher distance value implies that the difference between the graphs is more."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
]
}