![\"logo\"](\"../docs/img/lale_logo.jpg\"){kind=logo}
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook is the basis for a\n",
"[talk](https://pydata.org/nyc2019/schedule/presentation/29/type-driven-automated-learning-with-lale/)\n",
"about [Lale](https://github.com/ibm/lale). Lale is an open-source\n",
"Python library for semi-automated data science. Lale is compatible\n",
"with scikit-learn, adding a simple interface to existing\n",
"machine-learning automation tools. Lale lets you search over possible\n",
"pipelines in just a few lines of code while remaining in control of\n",
"your work."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Value Proposition\n",
"\n",
"When writing machine-learning pipelines, you have a lot of decisions\n",
"to make, such as picking transformers, estimators, and\n",
"hyperparameters. Since some of these decisions are tricky, you will\n",
"likely find yourself searching over many possible pipelines.\n",
"Machine-learning automation tools help with this search.\n",
"Unfortunately, each of these tools has its own API, and the search\n",
"spaces are not necessarily consistent nor even correct. We have\n",
"discovered that types (such as enum, float, or dictionary) can both\n",
"check the correctness of, and help automatically search over,\n",
"hyperparameters and pipeline configurations.\n",
"\n",
"To address this issue, we have open-sourced Lale, an open-source\n",
"Python library for semi-automated data science. Lale is compatible\n",
"with scikit-learn, adding a simple interface to existing\n",
"machine-learning automation tools.\n",
"Lale is designed to augment, but not replace, the data scientist.\n",
"\n",
"The **target user** of Lale is the working data scientist.\n",
"The **scope** of Lale includes machine learning (both deep learning\n",
"and non-DL) and data preparation. The **value** of Lale encompasses:\n",
"\n",
"![](\"img/2019-1105-three-values.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Categorical + Continuous Dataset\n",
"\n",
"To demonstrate automated machine learning (AutoML), we first need\n",
"a dataset. We will use a tabular dataset that has two kinds of\n",
"features: categorical features (columns that can contain one of a\n",
"small set of strings) and continuous features (numerical columns). In\n",
"particular, we use the `credit-g` dataset from OpenML. After fetching\n",
"the data, we display a few rows to get a better understanding of its\n",
"labels and features."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: liac-arff>=2.4.0 in /usr/local/lib/python3.7/site-packages (2.4.0)\n",
"\u001b[33mWARNING: You are using pip version 19.3.1; however, version 21.0.1 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
"source": [
"!pip install 'liac-arff>=2.4.0'"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"![](\"img/2019-1105-bindings.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The interested reader can explore Lale's concept of bindings as\n",
"lifecycle in more detail in our\n",
"[paper](https://arxiv.org/pdf/1906.03957.pdf).\n",
"\n",
"The above diagram is actually a Venn diagram. Each lifecycle state is\n",
"a subset of its predecessor. For instance, a trained operator has all\n",
"the bindings of a planned operator and in addition also binds learned\n",
"coefficients. Since a trained operator has all bindings required for\n",
"being a trainable operator, it can be used where a trainable operator\n",
"is expected. Thus, the set of trained operators is a subset of the set\n",
"of trainable operators. This relationship between the lifecycle states is\n",
"essential for offering a flexible semi-automated data science\n",
"experience."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Semi-Automated Data Science\n",
"\n",
"Why would data scientists prefer semi-automation over full automation?\n",
"The main reason is to excert control over what kind of pipelines they\n",
"find. Below are several scenarios motivating semi-automated data\n",
"science.\n",
"\n",
"| Manual control over automation | Examples |\n",
"| ----------------------------------- | -------- |\n",
"| Restrict available operator choices | Interpretable, or based on licenses, or based on GPU requirements, ... |\n",
"| Tweak graph topology | Custom preprocessing, or multi-modal data, or fairness mitigation, ... |\n",
"| Tweak hyperparametrer schemas | Adjust range for continuous, or restrict choices for categorical, ... |\n",
"| Expand available operator choices | Wrap existing library, or write your own operators, ... |\n",
"\n",
"In fact, we envision that this will often happen via trial-and-error,\n",
"where you as the data scientist specify a first planned pipeline, let\n",
"AutoML do its search, then inspect the results, edit your code, and\n",
"try again. You already saw some of the Lale features for inspecting\n",
"the results of automation above. Lale further supports this\n",
"workflow by letting you specify *partial bindings* of some\n",
"hyperparameters while allowing other to remain free, and by letting\n",
"you *freeze* an operator at the trainable or trained state.\n",
"\n",
"![](\"img/2019-1105-semi-automated.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 20 Newsgroups Dataset\n",
"\n",
"For the next examples in this notebook, we will need a different\n",
"dataset. The following code fetches the 20 newsgroups data, using a\n",
"function that comes with scikit-learn. Then, it prints a\n",
"representative sample of the labels and features."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" y X\n",
"0 rec.autos From: lerxst@wam.umd.edu (where's my thing)\\nS...\n",
"1 comp.sys.mac.hardware From: guykuo@carson.u.washington.edu (Guy Kuo)...\n",
"2 comp.sys.mac.hardware From: twillis@ec.ecn.purdue.edu (Thomas E Will...\n",
"3 comp.graphics From: jgreen@amber (Joe Green)\\nSubject: Re: W...\n",
"4 sci.space From: jcm@head-cfa.harvard.edu (Jonathan McDow..."
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sklearn.datasets\n",
"news = sklearn.datasets.fetch_20newsgroups()\n",
"news_X, news_y = news.data, news.target\n",
"pd.DataFrame({'y': [news.target_names[i] for i in news_y], 'X': news_X}).head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The labels `y` are strings taken from a set of 20 different newsgroup\n",
"names. That means that this is a 20-way classification dataset. The\n",
"features `X` consist of a single string with a message posted to one\n",
"of the newsgroups. The task is to use the message to predict which\n",
"group it was posted on."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Constraints in Scikit-learn\n",
"\n",
"As implied by the title of this notebook and the corresponding talk,\n",
"Lale is built around the concept of types. In programming languages, a\n",
"*type* specifies a set of valid values for a variable (e.g., for a\n",
"hyperparameter). While types are in the foreground of some\n",
"programming languages, Python keeps types more in the background.\n",
"Python is dynamically typed, and libraries such as\n",
"scikit-learn rely more on exceptions than on types for their error\n",
"checking. To demonstrate this, we first import some scikit-learn\n",
"modules."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"import sklearn.feature_extraction.text\n",
"import sklearn.pipeline\n",
"import sklearn.linear_model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we create a scikit-learn pipeline, using fully qualified names\n",
"to ensure and clarify that we are not using any Lale facilities. The\n",
"pipeline consists of just two operators, a TFIDF transformer for\n",
"extracting features from the message text followed by a\n",
"logistic regression for classifying the message by newsgroup."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"no error detected yet\n"
]
}
],
"source": [
"sklearn_misconfigured = sklearn.pipeline.make_pipeline(\n",
" sklearn.feature_extraction.text.TfidfVectorizer(),\n",
" sklearn.linear_model.LogisticRegression(solver='sag', penalty='l1'))\n",
"print('no error detected yet')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above code actually contains a mistake in the hyperparameters for\n",
"`LogisticRegression`. Unfortunately, scikit-learn does not detect this\n",
"mistake when the hyperparameters are configured in the code. Instead,\n",
"it delays its error checking until we attempt to fit the pipeline to\n",
"data."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 3.45 s, sys: 92.9 ms, total: 3.54 s\n",
"Wall time: 3.59 s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Solver sag supports only 'l2' or 'none' penalties, got l1 penalty.\n"
]
}
],
"source": [
"%%time\n",
"import sys\n",
"try:\n",
" sklearn_misconfigured.fit(news_X, news_y)\n",
"except ValueError as e:\n",
" print(e, file=sys.stderr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above output demonstrates that scikit-learn takes a few seconds to\n",
"report the error. This is because it first trains the TFIDF\n",
"transformer on the data, and only when that is done, it attempts to\n",
"train the `LogisticRegression`, triggering the exception. While a few\n",
"seconds are no big deal, the 20 newsgroup dataset is small, and for larger\n",
"datasets, the delay is larger. Furthermore, while in this case, the\n",
"erroneous code is not far away from the code that triggers the error\n",
"report, for larger code bases, that distance is also further. The\n",
"error message is nice and clear about the erroneous hyperpareters, but\n",
"does not mention which operator was misconfigured (here,\n",
"`LogisticRegression`)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Constraints in AutoML\n",
"\n",
"The above example illustrates an exception caused by manually\n",
"misconfigured hyperparameters. Tools for automated machine learning\n",
"usually run many trials with different hyperparameter configurations.\n",
"Just as in the manual case, in general, some of these automated trials\n",
"may raise exceptions.\n",
"\n",
"**Solution 1:** Unconstrained search space\n",
"\n",
"- {solver: \\[linear, sag, lbfgs\\], penalty: \\[l1, l2\\]}\n",
"- catch exception (after some time)\n",
"- return made-up loss `np.float.max`\n",
"\n",
"This first solution has the benefit of being simple, but the drawback\n",
"that it may waste computational resources on training upstream\n",
"operators before detecting erroneous downstream operators. It also\n",
"leads to a larger-than-necessary search space. While we can use a high\n",
"loss to steer the AutoML tool away from poorly performing points in\n",
"the search space, in our experiments, we have encountered cases where\n",
"this adversely affects convergence.\n",
"\n",
"**Solution 2:** Constrained search space\n",
"\n",
"- {solver: \\[linear, sag, lbfgs\\], penalty: \\[l1, l2]\\} **and** (**if** solver: [sag, lbfgs] **then** penalty: [l2])}\n",
"- no exceptions (no time wasted)\n",
"- no made-up loss\n",
"\n",
"The second solution is the one we advocate in Lale. The Lale library\n",
"contains hyperparameter schemas for a large number of operators. These\n",
"schemas are type specifications that encode not just the valid values\n",
"for each hyperparameter in isolation, but also constraints cutting\n",
"across multiple hyperparameters such as `solver` and `penalty` in the\n",
"example."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Constraints in Lale\n",
"\n",
"To demonstrate how Lale handles constraints, we first import Lale's\n",
"wrapper for TFIDF. We need not import LogisticRegression here since we\n",
"already imported it earlier in the notebook."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"from lale.lib.sklearn import TfidfVectorizer as Tfidf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lale uses JSON Schema to express types. JSON Schema is a widely\n",
"supported and adopted standard proposal for specifying the valid\n",
"structure of JSON documents. By using JSON Schema for hyperparameters,\n",
"we avoided the need to implement any custom schema validation code.\n",
"Instead, we simply put the concrete hyperparameters into a JSON\n",
"document and then use an off-the-shelf validator to check that against\n",
"the schema."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 23 ms, sys: 2.6 ms, total: 25.6 ms\n",
"Wall time: 25.5 ms\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Invalid configuration for LR(solver='sag', penalty='l1') due to constraint the newton-cg, sag, and lbfgs solvers support only l2 or no penalties.\n",
"Schema of constraint 1: {\n",
" \"description\": \"The newton-cg, sag, and lbfgs solvers support only l2 or no penalties.\",\n",
" \"anyOf\": [\n",
" {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"solver\": {\"not\": {\"enum\": [\"newton-cg\", \"sag\", \"lbfgs\"]}}\n",
" },\n",
" },\n",
" {\n",
" \"type\": \"object\",\n",
" \"properties\": {\"penalty\": {\"enum\": [\"l2\", \"none\"]}},\n",
" },\n",
" ],\n",
"}\n",
"Value: {'solver': 'sag', 'penalty': 'l1', 'dual': False, 'C': 1.0, 'tol': 0.0001, 'fit_intercept': True, 'intercept_scaling': 1.0, 'class_weight': None, 'random_state': None, 'max_iter': 100, 'multi_class': 'auto', 'verbose': 0, 'warm_start': False, 'n_jobs': None, 'l1_ratio': None}\n"
]
}
],
"source": [
"%%time\n",
"import jsonschema\n",
"try:\n",
" lale_misconfigured = Tfidf >> LR(LR.enum.solver.sag, LR.enum.penalty.l1)\n",
"except jsonschema.ValidationError as e:\n",
" print(e.message, file=sys.stderr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output demonstrates that the error check happened more than 100\n",
"times faster than in the previous example. This is because the code\n",
"did not need to train TFIDF. The error check also happened closer to\n",
"the root cause of the error, so the code dinstance is smaller.\n",
"Furthermore, the error message indicates\n",
"not just the wrong hyperparameters but also which operator was\n",
"misconfigured (here, `LR`)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Schemas as Documentation\n",
"\n",
"The above code example demonstrates that Lale uses JSON Schema for\n",
"error checking. But we can do better than just using schemas for one\n",
"purpose. Since all Lale operators carry hyperparameter schemas,\n",
"we can also use those same schemas for interactive documentation. The\n",
"following code illustrates that by inspecting the schema of a\n",
"continuous hyperparameter."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'description': 'Number of trees to fit.',\n",
" 'type': 'integer',\n",
" 'default': 100,\n",
" 'minimumForOptimizer': 50,\n",
" 'maximumForOptimizer': 1000}"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"XGBoost.hyperparam_schema('n_estimators')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Of course, the same interactive documentation approach also works for\n",
"categorical hyperparameters."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'description': 'Specify which booster to use.',\n",
" 'enum': ['gbtree', 'gblinear', 'dart', None],\n",
" 'default': None}"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"XGBoost.hyperparam_schema('booster')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the same schemas for two purposes (validation and documentation)\n",
"makes them stronger. As a user, you can be confident that the\n",
"documentation you read is in sync with the error checks in the code.\n",
"Furthermore, while schemas check hyperparameters, the reverse is also\n",
"true: when tests reveal mistakes in the schemas themselves, they can\n",
"be corrected in a single location, thus improving both validation and\n",
"documentation. However, we can do even better than using the same\n",
"schemas for two purposes: we can also use them for a third purpose\n",
"(automated search)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Types as Search Spaces\n",
"\n",
"The following picture illustrates how Lale uses types for AutoML.\n",
"The data scientist is shown on the left. They interact with a pipeline\n",
"at either the planned or trainable lifecycle state, or at some mixed\n",
"state as discussed above under semi-automated AutoML. Every\n",
"individual operator in Lale carries types in the form of a schema.\n",
"Users rarely have to write these schemas by hand, since the Lale\n",
"library already includes schemas for, at last count, 133 individual\n",
"operators. Given a planned pipeline, when the data scientist kicks\n",
"off a search, Lale automatically generates a search space appropriate\n",
"for the given combined algorithm selection and hyperparameter\n",
"optimization tool, e.g., hyperopt. For each trial, the\n",
"CASH tool then acquires a point in the search space, e.g., using an\n",
"acquisition function in Bayesian optimization. Then, Lale\n",
"automatically decodes that search point into a trainable Lale\n",
"pipeline. By construction, this trainable pipeline validates against\n",
"the schemas. We can train and score the pipeline, usually with\n",
"cross-validation, to return a loss value to the CASH tool. If this\n",
"is the best pipeline found so far, the CASH tool will remember it\n",
"as the incumbent. When the search terminates, the CASH tool returns\n",
"the final incumbent, minimizing the loss. \n",
"*Types as search spaces* is another fundamental\n",
"novel concept around which we designed Lale, described in more detail\n",
"in our [paper](https://arxiv.org/pdf/1906.03957.pdf).\n",
"\n",
"| \n", " | y | \n", "X | \n", "
|---|---|---|
| 0 | \n", "rec.autos | \n", "From: lerxst@wam.umd.edu (where's my thing)\\nS... | \n", "
| 1 | \n", "comp.sys.mac.hardware | \n", "From: guykuo@carson.u.washington.edu (Guy Kuo)... | \n", "
| 2 | \n", "comp.sys.mac.hardware | \n", "From: twillis@ec.ecn.purdue.edu (Thomas E Will... | \n", "
| 3 | \n", "comp.graphics | \n", "From: jgreen@amber (Joe Green)\\nSubject: Re: W... | \n", "
| 4 | \n", "sci.space | \n", "From: jcm@head-cfa.harvard.edu (Jonathan McDow... | \n", "
![](\"img/2019-1105-search-spaces.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Customizing Schemas\n",
"\n",
"While you can use Lale operators with their schemas as-is, you can\n",
"also customize the schemas to excert more control over the automation.\n",
"For instance, you might want to reduce the number of trees in an\n",
"XGBoost forest to reduce memory consumption or to improve\n",
"explainability. Or you might want to hand-pick one of the boosters\n",
"to reduce the search space and thus hopefully speed up the search."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"import lale.schemas as schemas\n",
"Grove = XGBoost.customize_schema(\n",
" n_estimators=schemas.Int(minimum=2, maximum=6),\n",
" booster=schemas.Enum(['gbtree'], default='gbtree'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As this example demonstrates, Lale provides a simple Python API for\n",
"writing schemas, which it then converts to JSON Schema internally.\n",
"The result of customization is a new copy of the operator that can be\n",
"used in the same way as any other operator in Lale. In particular,\n",
"it can be part of a pipeline as before."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"grove_planned = ( Project(columns={'type': 'number'}) >> Norm\n",
" & Project(columns={'type': 'string'}) >> OneHot\n",
" ) >> Concat >> Grove"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given this new planned pipeline, we use hyperopt as before to search for a\n",
"good trained pipeline."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[15:02:56] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:02:57] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:02:58] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:02:59] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:02:59] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:00] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:01] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:02] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:03] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:03] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:04] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:05] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:06] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:07] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:07] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:08] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:09] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:10] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:11] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:12] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:12] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:13] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:14] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:15] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:16] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:16] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:17] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:18] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"[15:03:19] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[15:03:20] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"100%|██████████| 10/10 [00:24<00:00, 2.49s/trial, best loss: -0.7313420884048686]\n",
"[15:03:20] WARNING: /Users/travis/build/dmlc/xgboost/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
"accuracy 73.0%\n"
]
}
],
"source": [
"grove_optimizer = Hyperopt(estimator=grove_planned, cv=3, max_evals=10, verbose=True)\n",
"grove_trained = grove_optimizer.fit(train_X, train_y)\n",
"grove_y = grove_trained.predict(test_X)\n",
"print(f'accuracy {sklearn.metrics.accuracy_score(test_y, grove_y):.1%}')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result shows that the predictive performance is not quite as good\n",
"as before. As a data scientist, you can weigh that against other\n",
"needs, and possibly experiment more. Of course, you can also display\n",
"the result of automation, e.g., by pretty-printing it back as code."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"```python\n",
"project_0 = Project(columns={\"type\": \"number\"})\n",
"norm = Norm(norm=\"l1\")\n",
"project_1 = Project(columns={\"type\": \"string\"})\n",
"grove = Grove(\n",
" gamma=0.8726303533099419,\n",
" learning_rate=0.7606845221791637,\n",
" max_depth=2,\n",
" min_child_weight=11,\n",
" n_estimators=5,\n",
" reg_alpha=0.5980470775121279,\n",
" reg_lambda=0.2546844052569046,\n",
" subsample=0.8142720284737895,\n",
")\n",
"pipeline = (\n",
" ((project_0 >> norm) & (project_1 >> OneHot())) >> Concat() >> grove\n",
")\n",
"```"
],
"text/plain": [
"![](\"img/2019-1105-interop.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Please check out the Lale github repository for examples,\n",
"documentation, and code: https://github.com/ibm/lale\n",
"\n",
"This notebook started by listing three values that Lale provides:\n",
"automation, interoperability, and usability. Then, this notebook\n",
"introduced three fundamental concepts around which Lale is designed:\n",
"bindings as lifecycle, types as search spaces, and scikit-learn\n",
"compatible interoperability. As the following picture illustrates, the\n",
"three concepts sit at the intersection of the three values.\n",
"\n",
"![](\"img/2019-1105-summary.png\")
"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}