{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Google colab link:\n",
"\n",
"[https://colab.research.google.com/github/oegedijk/explainerdashboard/blob/master/custom_examples.ipynb](https://colab.research.google.com/github/oegedijk/explainerdashboard/blob/master/custom_examples.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"uncomment to install explainerdashboard:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:17:23.055000Z",
"start_time": "2021-01-20T16:17:23.050915Z"
}
},
"outputs": [],
"source": [
"#!pip install explainerdashboard"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# notebook properties:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:17:24.297328Z",
"start_time": "2021-01-20T16:17:24.292798Z"
}
},
"outputs": [],
"source": [
"from IPython.core.interactiveshell import InteractiveShell\n",
"\n",
"InteractiveShell.ast_node_interactivity = \"all\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:17:56.270605Z",
"start_time": "2021-01-20T16:17:52.395582Z"
}
},
"outputs": [],
"source": [
"from sklearn.ensemble import RandomForestClassifier\n",
"\n",
"from jupyter_dash import JupyterDash\n",
"\n",
"from explainerdashboard import *\n",
"from explainerdashboard.datasets import *\n",
"from explainerdashboard.custom import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# train model and build explainer"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:18:40.518834Z",
"start_time": "2021-01-20T16:18:39.584750Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Note: shap=='guess' so guessing for RandomForestClassifier shap='tree'...\n",
"Detected RandomForestClassifier model: Changing class type to RandomForestClassifierExplainer...\n",
"Note: model_output=='probability', so assuming that raw shap output of RandomForestClassifier is in probability space...\n",
"Generating self.shap_explainer = shap.TreeExplainer(model)\n",
"Calculating shap values...\n"
]
},
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"hoverinfo": "skip",
"marker": {
"color": "rgba(1,1,1, 0.0)"
},
"name": "",
"type": "bar",
"x": [
"Population
average",
"Sex",
"Fare",
"PassengerClass",
"Deck",
"Embarked",
"No_of_parents_plus_children_on_board",
"No_of_siblings_plus_spouses_on_board",
"Age",
"Other features combined",
"Final Prediction"
],
"y": [
0,
39.26,
66.81,
75.81,
84.24,
91.12,
95.89,
97.92,
99.79,
98,
0
]
},
{
"hoverinfo": "text",
"marker": {
"color": [
"rgba(230, 230, 30, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(50, 200, 50, 1.0)",
"rgba(219, 64, 82, 0.7)",
"rgba(219, 64, 82, 0.7)",
"rgba(55, 128, 191, 0.7)"
],
"line": {
"color": [
"rgba(190, 190, 30, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(40, 160, 50, 1.0)",
"rgba(219, 64, 82, 1.0)",
"rgba(219, 64, 82, 1.0)",
"rgba(55, 128, 191, 1.0)"
],
"width": 2
}
},
"name": "contribution",
"text": [
"Population
average=
+39.26 ",
"Sex=female
+27.56 ",
"Fare=71.2833
+9.0 ",
"PassengerClass=1
+8.42 ",
"Deck=C
+6.88 ",
"Embarked=Cherbourg
+4.77 ",
"No_of_parents_plus_children_on_board=0
+2.03 ",
"No_of_siblings_plus_spouses_on_board=1
+1.87 ",
"Age=38.0
-1.79 ",
"Other features combined=
0.0 ",
"Final Prediction=
+98.0 "
],
"type": "bar",
"x": [
"Population
average",
"Sex",
"Fare",
"PassengerClass",
"Deck",
"Embarked",
"No_of_parents_plus_children_on_board",
"No_of_siblings_plus_spouses_on_board",
"Age",
"Other features combined",
"Final Prediction"
],
"y": [
39.26,
27.56,
9,
8.42,
6.88,
4.77,
2.03,
1.87,
-1.79,
0,
98
]
}
],
"layout": {
"barmode": "stack",
"height": 600,
"margin": {
"b": 216,
"l": 50,
"pad": 4,
"r": 100,
"t": 50
},
"plot_bgcolor": "#fff",
"showlegend": false,
"template": {
"data": {
"scatter": [
{
"type": "scatter"
}
]
}
},
"title": {
"text": "Contribution to prediction probability = 98.0%",
"x": 0.5
},
"yaxis": {
"range": [
0,
100
],
"title": {
"text": "Predicted %"
}
}
}
},
"text/html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"\n",
"X_train, y_train, X_test, y_test = titanic_survive()\n",
"model = RandomForestClassifier(n_estimators=50, max_depth=15).fit(X_train, y_train)\n",
"explainer = ClassifierExplainer(\n",
" model,\n",
" X_test,\n",
" y_test,\n",
" cats=[\"Sex\", \"Deck\", \"Embarked\"],\n",
" descriptions=feature_descriptions,\n",
" labels=[\"Not survived\", \"Survived\"],\n",
")\n",
"\n",
"explainer.plot_contributions(0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Design Pattern I: Custom `ExplainerComponent` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Constructing the ExplainerComponent\n",
"\n",
"By constructing a custom `ExplainerComponent` a lot of the boilerplate stuff gets taken care of for you. \n",
"This includes:\n",
" - by registering the callbacks for all your subcomponents\n",
" - by making it easy to pre calculate all lazily calculated dependencies such as shap values up front. (otherwise these may get calculated multiple times by each component individually)\n",
" - Makes it easy to run and start the dashboard by passing it on to an `ExplainerDashboard`\n",
" \n",
"\n",
"Important elements:\n",
"- call `super().__init__(explainer, title, name) at beginning of ``__init__``\n",
"- define your callbacks in `.component_callbacks(self, app)`. This way the ExplainerComponent automatically register all calbacks of register components first, and then call `component_callbacks(app)`. Alternatively you just have to make sure that you register all callbacks of subcomponents in the `register_callbacks(app)` method. \n",
"\n",
"By calling `register_components()` your component callbacks are automatically registered, and you can\n",
"can calculate all dependencies of your Component at once by calling `.calculate_dependencies()`\n",
"\n",
"Finally you can pass the CustomComponent to `ExplainerDashboard`, without having to write the dash boilerplate yourself."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building the layout\n",
"\n",
"Each ExplainerComponent has a `.layout()` method that you can use inside your layout definition. \n",
"All the configuration of the component is done when you instantiate the component. \n",
"You can hide the different toggles(`hide_cats=True`) and titles, and set default values of \n",
"elements (`col='Fare'`)\n",
"of the components.\n",
"\n",
"For example we set 'Fare' and 'PassengerClass' as default col en color_col for the ShapDependenceComponent.\n",
"\n",
"\n",
"The rest of the layout is done with standard dash bootstrap components (dbc). https://dash-bootstrap-components.opensource.faculty.ai/docs/"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:19:13.613406Z",
"start_time": "2021-01-20T16:19:13.608777Z"
}
},
"outputs": [],
"source": [
"import dash_html_components as html\n",
"import dash_bootstrap_components as dbc\n",
"\n",
"from explainerdashboard.custom import *"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:20:49.390495Z",
"start_time": "2021-01-20T16:20:49.376706Z"
}
},
"outputs": [],
"source": [
"class CustomDashboard(ExplainerComponent):\n",
" def __init__(self, explainer, name=None):\n",
" super().__init__(explainer, title=\"Titanic Explainer\")\n",
"\n",
" self.precision = PrecisionComponent(\n",
" explainer,\n",
" name=self.name + \"precision\",\n",
" hide_cutoff=True,\n",
" hide_binsize=True,\n",
" hide_binmethod=True,\n",
" hide_multiclass=True,\n",
" hide_selector=True,\n",
" hide_title=True,\n",
" hide_footer=True,\n",
" cutoff=None,\n",
" )\n",
" self.shap_summary = ShapSummaryComponent(\n",
" explainer,\n",
" name=self.name + \"summary\",\n",
" hide_title=True,\n",
" hide_selector=True,\n",
" hide_depth=True,\n",
" depth=8,\n",
" hide_cats=True,\n",
" cats=True,\n",
" )\n",
" self.shap_dependence = ShapDependenceComponent(\n",
" explainer,\n",
" name=self.name + \"dependence\",\n",
" hide_title=True,\n",
" hide_selector=True,\n",
" hide_cats=True,\n",
" cats=True,\n",
" hide_index=True,\n",
" col=\"Fare\",\n",
" color_col=\"PassengerClass\",\n",
" )\n",
" self.connector = ShapSummaryDependenceConnector(\n",
" self.shap_summary, self.shap_dependence\n",
" )\n",
"\n",
" def layout(self):\n",
" return dbc.Container(\n",
" [\n",
" html.H1(\"Titanic Explainer\"),\n",
" dbc.Row(\n",
" [\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Model Performance\"),\n",
" html.Div(\n",
" \"As you can see on the right, the model performs quite well.\"\n",
" ),\n",
" html.Div(\n",
" \"The higher the predicted probability of survival predicted by\"\n",
" \"the model on the basis of learning from examples in the training set\"\n",
" \", the higher is the actual percentage for a person surviving in \"\n",
" \"the test set\"\n",
" ),\n",
" ],\n",
" width=4,\n",
" ),\n",
" dbc.Col(\n",
" [html.H3(\"Model Precision Plot\"), self.precision.layout()]\n",
" ),\n",
" ]\n",
" ),\n",
" dbc.Row(\n",
" [\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Feature Importances Plot\"),\n",
" self.shap_summary.layout(),\n",
" ]\n",
" ),\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Feature importances\"),\n",
" html.Div(\n",
" \"On the left you can check out for yourself which parameters were the most important.\"\n",
" ),\n",
" html.Div(\n",
" f\"{self.explainer.columns_ranked_by_shap()[0]} was the most important\"\n",
" f\", followed by {self.explainer.columns_ranked_by_shap()[1]}\"\n",
" f\" and {self.explainer.columns_ranked_by_shap()[2]}.\"\n",
" ),\n",
" html.Div(\n",
" \"If you select 'detailed' you can see the impact of that variable on \"\n",
" \"each individual prediction. With 'aggregate' you see the average impact size \"\n",
" \"of that variable on the finale prediction.\"\n",
" ),\n",
" html.Div(\n",
" \"With the detailed view you can clearly see that the the large impact from Sex \"\n",
" \"stems both from males having a much lower chance of survival and females a much \"\n",
" \"higher chance.\"\n",
" ),\n",
" ],\n",
" width=4,\n",
" ),\n",
" ]\n",
" ),\n",
" dbc.Row(\n",
" [\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Relations between features and model output\"),\n",
" html.Div(\n",
" \"In the plot to the right you can see that the higher the priace\"\n",
" \"of the Fare that people paid, the higher the chance of survival. \"\n",
" \"Probably the people with more expensive tickets were in higher up cabins, \"\n",
" \"and were more likely to make it to a lifeboat.\"\n",
" ),\n",
" html.Div(\n",
" \"When you color the impacts by the PassengerClass, you can clearly see that \"\n",
" \"the more expensive tickets were mostly 1st class, and the cheaper tickets \"\n",
" \"mostly 3rd class.\"\n",
" ),\n",
" html.Div(\n",
" \"On the right you can check out for yourself how different features impact \"\n",
" \"the model output.\"\n",
" ),\n",
" ],\n",
" width=4,\n",
" ),\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Feature impact plot\"),\n",
" self.shap_dependence.layout(),\n",
" ]\n",
" ),\n",
" ]\n",
" ),\n",
" ]\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:23:28.387006Z",
"start_time": "2021-01-20T16:23:19.788958Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building ExplainerDashboard..\n",
"Detected notebook environment, consider setting mode='external', mode='inline' or mode='jupyterlab' to keep the notebook interactive while the dashboard is running...\n",
"Generating layout...\n",
"ExplainerComponent does not accept a name parameter, so cannot assign name='1': CustomDashboard.__init__(self, explainer). Make sure to set super().__init__(name=...) explicitly yourself inside the __init__ if you want to deploy across multiple workers or a cluster, as otherwise each instance in the cluster will generate its own random uuid name!\n",
"Calculating dependencies...\n",
"Reminder: you can store the explainer (including calculated dependencies) with explainer.dump('explainer.joblib') and reload with e.g. ClassifierExplainer.from_file('explainer.joblib')\n",
"Registering callbacks...\n",
"Starting ExplainerDashboard on http://localhost:8050\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n",
"Dash is running on http://0.0.0.0:8050/\n",
"\n"
]
}
],
"source": [
"from explainerdashboard import ExplainerDashboard\n",
"\n",
"ExplainerDashboard(\n",
" explainer, CustomDashboard, boostrap=dbc.themes.FLATLY, hide_header=True\n",
").run()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Design pattern II: traditional flat dash definition"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:22:50.650331Z",
"start_time": "2021-01-20T16:22:50.632097Z"
}
},
"outputs": [],
"source": [
"precision = PrecisionComponent(\n",
" explainer,\n",
" hide_cutoff=True,\n",
" hide_binsize=True,\n",
" hide_selector=True,\n",
" hide_binmethod=True,\n",
" hide_multiclass=True,\n",
" hide_title=True,\n",
" hide_footer=True,\n",
" cutoff=None,\n",
")\n",
"\n",
"shap_summary = ShapSummaryComponent(\n",
" explainer,\n",
" hide_title=True,\n",
" hide_selector=True,\n",
" hide_depth=True,\n",
" depth=8,\n",
" hide_cats=True,\n",
" cats=True,\n",
")\n",
"\n",
"shap_dependence = ShapDependenceComponent(\n",
" explainer,\n",
" hide_title=True,\n",
" hide_selector=True,\n",
" hide_cats=True,\n",
" cats=True,\n",
" hide_index=True,\n",
" col=\"Fare\",\n",
" color_col=\"PassengerClass\",\n",
")\n",
"\n",
"connector = ShapSummaryDependenceConnector(shap_summary, shap_dependence)\n",
"\n",
"layout = dbc.Container(\n",
" [\n",
" html.H1(\"Titanic Explainer\"),\n",
" dbc.Row(\n",
" [\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Model Performance\"),\n",
" html.Div(\n",
" \"As you can see on the right, the model performs quite well.\"\n",
" ),\n",
" html.Div(\n",
" \"The higher the predicted probability of survival predicted by\"\n",
" \"the model on the basis of learning from examples in the training set\"\n",
" \", the higher is the actual percentage for a person surviving in \"\n",
" \"the test set\"\n",
" ),\n",
" ],\n",
" width=4,\n",
" ),\n",
" dbc.Col([html.H3(\"Model Precision Plot\"), precision.layout()]),\n",
" ]\n",
" ),\n",
" dbc.Row(\n",
" [\n",
" dbc.Col([html.H3(\"Feature Importances Plot\"), shap_summary.layout()]),\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Feature importances\"),\n",
" html.Div(\n",
" \"On the left you can check out for yourself which parameters were the most important.\"\n",
" ),\n",
" html.Div(\n",
" f\"{explainer.columns_ranked_by_shap()[0]} was the most important\"\n",
" f\", followed by {explainer.columns_ranked_by_shap()[1]}\"\n",
" f\" and {explainer.columns_ranked_by_shap()[2]}.\"\n",
" ),\n",
" html.Div(\n",
" \"If you select 'detailed' you can see the impact of that variable on \"\n",
" \"each individual prediction. With 'aggregate' you see the average impact size \"\n",
" \"of that variable on the finale prediction.\"\n",
" ),\n",
" html.Div(\n",
" \"With the detailed view you can clearly see that the the large impact from Sex \"\n",
" \"stems both from males having a much lower chance of survival and females a much \"\n",
" \"higher chance.\"\n",
" ),\n",
" ],\n",
" width=4,\n",
" ),\n",
" ]\n",
" ),\n",
" dbc.Row(\n",
" [\n",
" dbc.Col(\n",
" [\n",
" html.H3(\"Relations between features and model output\"),\n",
" html.Div(\n",
" \"In the plot to the right you can see that the higher the priace\"\n",
" \"of the Fare that people paid, the higher the chance of survival. \"\n",
" \"Probably the people with more expensive tickets were in higher up cabins, \"\n",
" \"and were more likely to make it to a lifeboat.\"\n",
" ),\n",
" html.Div(\n",
" \"When you color the impacts by the PassengerClass, you can clearly see that \"\n",
" \"the more expensive tickets were mostly 1st class, and the cheaper tickets \"\n",
" \"mostly 3rd class.\"\n",
" ),\n",
" html.Div(\n",
" \"On the right you can check out for yourself how different features impact \"\n",
" \"the model output.\"\n",
" ),\n",
" ],\n",
" width=4,\n",
" ),\n",
" dbc.Col([html.H3(\"Feature impact plot\"), shap_dependence.layout()]),\n",
" ]\n",
" ),\n",
" ]\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:22:51.300470Z",
"start_time": "2021-01-20T16:22:51.296548Z"
}
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"ExecuteTime": {
"end_time": "2021-01-20T16:22:52.083237Z",
"start_time": "2021-01-20T16:22:51.711469Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dash app running on http://127.0.0.1:8050/\n"
]
}
],
"source": [
"app = JupyterDash(\n",
" __name__, external_stylesheets=[dbc.themes.FLATLY], assets_url_path=\"\"\n",
")\n",
"app.title = \"Titanic Explainer\"\n",
"app.layout = layout\n",
"\n",
"precision.register_callbacks(app)\n",
"shap_summary.register_callbacks(app)\n",
"shap_dependence.register_callbacks(app)\n",
"connector.register_callbacks(app)\n",
"\n",
"app.run_server(port=8050)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 4
}