by the Oracle Cloud Infrastructure [May 2022]. Contact Alireza Dibazar for questions.
\n", "\n", "***\n", "\n", "## Overview:\n", "This notebook provides a sample Breast Tissues classification technique using OCI's data science tools.\n", "\n", "\n", "\n", "\n", "\n", "Using ultrasound guidance, a fine needle (small single arrows in the left picture) is placed in the center of the lump (large arrow) to be sampled. With the tip (double arrow) in the center of the lump, the needle is moved back and forth as aspiration is applied. The material obtained is smeared on a glass slide, stained and looked at under the microscope by a cytopathologist with specialty training in evaluating cells.\n", "\n", "\n", "This notebook demonstrates simple steps for training a model to identify malignant samples, and deploying it to production.\n", "\n", "---\n", "\n", "## Contents:\n", "\n", "* Introduction\n", " * Dataset\n", "* Exploratory Data Analysis (EDA)\n", " * Pair Plots\n", " * Feature Scaling\n", " * Distribution of the Target Classes\n", " * Split Training and Testing datasets, and Separate Target Column\n", "* Model Selection\n", " * Compare Model performances\n", " * Model Selection with ADS AutoML\n", " * Custom ADSDataset Class\n", " * AutoML Setup\n", "* Explaining the Model\n", " * Show what the model has learned\n", " * Global Explanations\n", " * What-if Scenarios\n", " * Prediction Explorer\n", " * Local Explanations\n", "* Model Deployment\n", " * Create a Serialized Model\n", " * Prepare the Model Artifacts\n", " * Verify: Test the Model Locally\n", " * Save the Model in the Model Catalog\n", " * Deploy\n", " * Predict\n", "* Invoking the Endpoint \n", "* Clean Up\n", "* References \n", "\n", "---\n", "\n", "**Important:**\n", "\n", "Some field require replacement. They are surrounded with `<>`. For example `database_name = \"| \n", " | Model | \n", "Accuracy | \n", "Train_acc | \n", "
|---|---|---|---|
| 0 | \n", "LGBMClassifier | \n", "0.938596 | \n", "0.997802 | \n", "
| 4 | \n", "XGBClassifier | \n", "0.938596 | \n", "1.000000 | \n", "
| 1 | \n", "DecisionTreeClassifier | \n", "0.921053 | \n", "0.947253 | \n", "
| 3 | \n", "GradientBoostingClassifier | \n", "0.921053 | \n", "1.000000 | \n", "
| 2 | \n", "RandomForestClassifier | \n", "0.912281 | \n", "0.936264 | \n", "
| 5 | \n", "SupportVectorClassifier | \n", "0.833333 | \n", "0.876923 | \n", "
| Training Dataset size | \n", "(455, 12) | \n", "
| Validation Dataset size | \n", "None | \n", "
| CV | \n", "5 | \n", "
| Target variable | \n", "diagnosis | \n", "
| Optimization Metric | \n", "roc_auc | \n", "
| Initial number of Features | \n", "12 | \n", "
| Selected number of Features | \n", "12 | \n", "
| Selected Features | \n", "Index(['symmetry_mean', 'texture_se', 'smoothness_se', 'compactness_se',\n", " 'concavity_se', 'concave_points_se', 'symmetry_se', 'texture_worst',\n", " 'smoothness_worst', 'compactness_worst', 'concave_points_worst',\n", " 'symmetry_worst'],\n", " dtype='object') | \n", "
| Selected Algorithm | \n", "ExtraTreesClassifier | \n", "
| End-to-end Elapsed Time (seconds) | \n", "77.019 | \n", "
| Selected Hyperparameters | \n", "{'bootstrap': False, 'ccp_alpha': 0.0, 'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'max_features': 0.777777778, 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 113, 'n_jobs': 1, 'oob_score': False, 'random_state': 0, 'verbose': 0, 'warm_start': False} | \n", "
| Mean Validation Score | \n", "0.9734 | \n", "
| AutoML n_jobs | \n", "4 | \n", "
| AutoML version | \n", "1.0 | \n", "
| Python version | \n", "3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53) \\n[GCC 9.4.0] | \n", "
| Rank based on Performance | \n", "Algorithm | \n", "#Samples | \n", "#Features | \n", "Mean Validation Score | \n", "Hyperparameters | \n", "CPU Time | \n", "Memory Usage | \n", "
|---|---|---|---|---|---|---|---|
| 2 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.9733 | \n", "{'class_weight': 'balanced', 'criterion': 'gini', 'max_features': 0.7777869446666666, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125, 'n_estimators': 128} | \n", "2.7514 | \n", "0.0000e+00 | \n", "
| 3 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.9733 | \n", "{'class_weight': 'balanced_subsample', 'criterion': 'gini', 'max_features': 0.777777778, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125, 'n_estimators': 128} | \n", "2.5746 | \n", "0.0000e+00 | \n", "
| 4 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.9733 | \n", "{'class_weight': 'balanced', 'criterion': 'gini', 'max_features': 0.777777778, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125, 'n_estimators': 128} | \n", "2.6430 | \n", "0.0000e+00 | \n", "
| 5 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.9732 | \n", "{'class_weight': 'balanced', 'criterion': 'gini', 'max_features': 0.777777778, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125, 'n_estimators': 111} | \n", "2.6791 | \n", "0.0000e+00 | \n", "
| 6 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.9730 | \n", "{'class_weight': 'balanced', 'criterion': 'gini', 'max_features': 0.777777778, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125, 'n_estimators': 112} | \n", "2.0568 | \n", "0.0000e+00 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 101 | \n", "DecisionTreeClassifier_AS | \n", "455 | \n", "12 | \n", "0.8678 | \n", "{'class_weight': None, 'max_features': 1.0, 'min_samples_leaf': 0.000625, 'min_samples_split': 0.00125} | \n", "0.1777 | \n", "7.6294e-06 | \n", "
| 102 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.8437 | \n", "{'class_weight': None, 'criterion': 'gini', 'max_features': 0.08333333333333333, 'min_samples_leaf': 0.008550093757641996, 'min_samples_split': 0.004395604395604396, 'n_estimators': 5} | \n", "0.2738 | \n", "0.0000e+00 | \n", "
| 103 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.8437 | \n", "{'class_weight': None, 'criterion': 'gini', 'max_features': 0.08333333333333333, 'min_samples_leaf': 0.008555071779620019, 'min_samples_split': 0.004395604395604396, 'n_estimators': 5} | \n", "0.2902 | \n", "0.0000e+00 | \n", "
| 104 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.7213 | \n", "{'class_weight': None, 'criterion': 'gini', 'max_features': 0.08333333333333333, 'min_samples_leaf': 0.03318719798190425, 'min_samples_split': 0.004395604395604396, 'n_estimators': 5} | \n", "0.2959 | \n", "0.0000e+00 | \n", "
| 105 | \n", "ExtraTreesClassifier_HT | \n", "455 | \n", "12 | \n", "0.7213 | \n", "{'class_weight': None, 'criterion': 'gini', 'max_features': 0.08333333333333333, 'min_samples_leaf': 0.033192176003882266, 'min_samples_split': 0.004395604395604396, 'n_estimators': 5} | \n", "0.2291 | \n", "0.0000e+00 | \n", "
| Feature | Value |
|---|---|
| texture_worst | 18.34 |
| concave_points_worst | 0.1108 |
| symmetry_se | 0.02653 |
| compactness_se | 0.05244 |
| concave_points_se | 0.0158 |
| smoothness_se | 0.009976 |
| concavity_se | 0.05278 |
| compactness_worst | 0.3635 |
| symmetry_mean | 0.2086 |
| symmetry_worst | 0.2827 |
| texture_se | 1.511 |
| smoothness_worst | 0.1312 |
| Sample Space | \n", " \n", "root_mean_squared_error | \n", " \n", "pearson Analysis | \n", "
|---|---|---|
| Nt | \n", " \n", "0.340 | \n", " \n", "Corr=0.111 p=4.1174e-15 | \n",
"
| Nv_2 | \n", " \n", "0.182 | \n", " \n", "Corr=0.116 p=0.101 | \n",
"
| Nv_4 | \n", " \n", "0.184 | \n", " \n", "Corr=0.101 p=0.155 | \n",
"
| Nv_16 | \n", " \n", "0.299 | \n", " \n", "Corr=-0.009 p=0.899 | \n",
"
Performance
| \n", " | \n", " | \n", " | Actions Needed | \n", "
|---|---|---|---|
| Step | \n", "Status | \n", "Details | \n", "\n", " |
| initiate | \n", "Done | \n", "Initiated the model | \n", "\n", " |
| prepare() | \n", "Available | \n", "Generated runtime.yaml | \n", "\n", " |
| Generated score.py | \n", "\n", " | ||
| Serialized model | \n", "\n", " | ||
| Populated metadata(Custom, Taxonomy and Provenance) | \n", "\n", " | ||
| verify() | \n", "Not Available | \n", "Local tested .predict from score.py | \n", "\n", " |
| save() | \n", "Not Available | \n", "Conducted Introspect Test | \n", "\n", " |
| Uploaded artifact to model catalog | \n", "\n", " | ||
| deploy() | \n", "Not Available | \n", "Deployed the model | \n", "\n", " |
| predict() | \n", "Not Available | \n", "Called deployment predict endpoint | \n", "\n", " |
| \n", " | \n", " | \n", " | Actions Needed | \n", "
|---|---|---|---|
| Step | \n", "Status | \n", "Details | \n", "\n", " |
| initiate | \n", "Done | \n", "Initiated the model | \n", "\n", " |
| prepare() | \n", "Done | \n", "Generated runtime.yaml | \n", "\n", " |
| Generated score.py | \n", "\n", " | ||
| Serialized model | \n", "\n", " | ||
| Populated metadata(Custom, Taxonomy and Provenance) | \n", "\n", " | ||
| verify() | \n", "Available | \n", "Local tested .predict from score.py | \n", "\n", " |
| save() | \n", "Available | \n", "Conducted Introspect Test | \n", "\n", " |
| Uploaded artifact to model catalog | \n", "\n", " | ||
| deploy() | \n", "Not Available | \n", "Deployed the model | \n", "\n", " |
| predict() | \n", "Not Available | \n", "Called deployment predict endpoint | \n", "\n", " |
| \n", " | diagnosis | \n", "radius_mean | \n", "texture_mean | \n", "perimeter_mean | \n", "area_mean | \n", "smoothness_mean | \n", "compactness_mean | \n", "concavity_mean | \n", "concave_points_mean | \n", "symmetry_mean | \n", "... | \n", "radius_worst | \n", "texture_worst | \n", "perimeter_worst | \n", "area_worst | \n", "smoothness_worst | \n", "compactness_worst | \n", "concavity_worst | \n", "concave_points_worst | \n", "symmetry_worst | \n", "fractal_dimension_worst | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 84799002 | \n", "M | \n", "14.540 | \n", "27.54 | \n", "96.73 | \n", "658.8 | \n", "0.11390 | \n", "0.15950 | \n", "0.16390 | \n", "0.07364 | \n", "0.2303 | \n", "... | \n", "17.46 | \n", "37.13 | \n", "124.10 | \n", "943.2 | \n", "0.1678 | \n", "0.6577 | \n", "0.70260 | \n", "0.17120 | \n", "0.4218 | \n", "0.13410 | \n", "
| 848406 | \n", "M | \n", "14.680 | \n", "20.13 | \n", "94.74 | \n", "684.5 | \n", "0.09867 | \n", "0.07200 | \n", "0.07395 | \n", "0.05259 | \n", "0.1586 | \n", "... | \n", "19.07 | \n", "30.88 | \n", "123.40 | \n", "1138.0 | \n", "0.1464 | \n", "0.1871 | \n", "0.29140 | \n", "0.16090 | \n", "0.3029 | \n", "0.08216 | \n", "
| 84862001 | \n", "M | \n", "16.130 | \n", "20.68 | \n", "108.10 | \n", "798.8 | \n", "0.11700 | \n", "0.20220 | \n", "0.17220 | \n", "0.10280 | \n", "0.2164 | \n", "... | \n", "20.96 | \n", "31.48 | \n", "136.80 | \n", "1315.0 | \n", "0.1789 | \n", "0.4233 | \n", "0.47840 | \n", "0.20730 | \n", "0.3706 | \n", "0.11420 | \n", "
| 849014 | \n", "M | \n", "19.810 | \n", "22.15 | \n", "130.00 | \n", "1260.0 | \n", "0.09831 | \n", "0.10270 | \n", "0.14790 | \n", "0.09498 | \n", "0.1582 | \n", "... | \n", "27.32 | \n", "30.88 | \n", "186.80 | \n", "2398.0 | \n", "0.1512 | \n", "0.3150 | \n", "0.53720 | \n", "0.23880 | \n", "0.2768 | \n", "0.07615 | \n", "
| 8510426 | \n", "B | \n", "13.540 | \n", "14.36 | \n", "87.46 | \n", "566.3 | \n", "0.09779 | \n", "0.08129 | \n", "0.06664 | \n", "0.04781 | \n", "0.1885 | \n", "... | \n", "15.11 | \n", "19.26 | \n", "99.70 | \n", "711.2 | \n", "0.1440 | \n", "0.1773 | \n", "0.23900 | \n", "0.12880 | \n", "0.2977 | \n", "0.07259 | \n", "
| 8510653 | \n", "B | \n", "13.080 | \n", "15.71 | \n", "85.63 | \n", "520.0 | \n", "0.10750 | \n", "0.12700 | \n", "0.04568 | \n", "0.03110 | \n", "0.1967 | \n", "... | \n", "14.50 | \n", "20.49 | \n", "96.09 | \n", "630.5 | \n", "0.1312 | \n", "0.2776 | \n", "0.18900 | \n", "0.07283 | \n", "0.3184 | \n", "0.08183 | \n", "
| 8510824 | \n", "B | \n", "9.504 | \n", "12.44 | \n", "60.34 | \n", "273.9 | \n", "0.10240 | \n", "0.06492 | \n", "0.02956 | \n", "0.02076 | \n", "0.1815 | \n", "... | \n", "10.23 | \n", "15.66 | \n", "65.13 | \n", "314.9 | \n", "0.1324 | \n", "0.1148 | \n", "0.08867 | \n", "0.06227 | \n", "0.2450 | \n", "0.07773 | \n", "
| 8511133 | \n", "M | \n", "15.340 | \n", "14.26 | \n", "102.50 | \n", "704.4 | \n", "0.10730 | \n", "0.21350 | \n", "0.20770 | \n", "0.09756 | \n", "0.2521 | \n", "... | \n", "18.07 | \n", "19.08 | \n", "125.10 | \n", "980.9 | \n", "0.1390 | \n", "0.5954 | \n", "0.63050 | \n", "0.23930 | \n", "0.4667 | \n", "0.09946 | \n", "
| 851509 | \n", "M | \n", "21.160 | \n", "23.04 | \n", "137.20 | \n", "1404.0 | \n", "0.09428 | \n", "0.10220 | \n", "0.10970 | \n", "0.08632 | \n", "0.1769 | \n", "... | \n", "29.17 | \n", "35.59 | \n", "188.00 | \n", "2615.0 | \n", "0.1401 | \n", "0.2600 | \n", "0.31550 | \n", "0.20090 | \n", "0.2822 | \n", "0.07526 | \n", "
| 852552 | \n", "M | \n", "16.650 | \n", "21.38 | \n", "110.00 | \n", "904.6 | \n", "0.11210 | \n", "0.14570 | \n", "0.15250 | \n", "0.09170 | \n", "0.1995 | \n", "... | \n", "26.46 | \n", "31.56 | \n", "177.00 | \n", "2215.0 | \n", "0.1805 | \n", "0.3578 | \n", "0.46950 | \n", "0.20950 | \n", "0.3613 | \n", "0.09564 | \n", "
10 rows × 31 columns
\n", "| \n", " | \n", " | \n", " | Actions Needed | \n", "
|---|---|---|---|
| Step | \n", "Status | \n", "Details | \n", "\n", " |
| initiate | \n", "Done | \n", "Initiated the model | \n", "\n", " |
| prepare() | \n", "Done | \n", "Generated runtime.yaml | \n", "\n", " |
| Generated score.py | \n", "\n", " | ||
| Serialized model | \n", "\n", " | ||
| Populated metadata(Custom, Taxonomy and Provenance) | \n", "\n", " | ||
| verify() | \n", "Done | \n", "Local tested .predict from score.py | \n", "\n", " |
| save() | \n", "Available | \n", "Conducted Introspect Test | \n", "\n", " |
| Uploaded artifact to model catalog | \n", "\n", " | ||
| deploy() | \n", "Not Available | \n", "Deployed the model | \n", "\n", " |
| predict() | \n", "Not Available | \n", "Called deployment predict endpoint | \n", "\n", " |
| \n", " | \n", " | \n", " | Actions Needed | \n", "
|---|---|---|---|
| Step | \n", "Status | \n", "Details | \n", "\n", " |
| initiate | \n", "Done | \n", "Initiated the model | \n", "\n", " |
| prepare() | \n", "Done | \n", "Generated runtime.yaml | \n", "\n", " |
| Generated score.py | \n", "\n", " | ||
| Serialized model | \n", "\n", " | ||
| Populated metadata(Custom, Taxonomy and Provenance) | \n", "\n", " | ||
| verify() | \n", "Done | \n", "Local tested .predict from score.py | \n", "\n", " |
| save() | \n", "Done | \n", "Conducted Introspect Test | \n", "\n", " |
| Uploaded artifact to model catalog | \n", "\n", " | ||
| deploy() | \n", "ACTIVE | \n", "Deployed the model | \n", "\n", " |
| predict() | \n", "Available | \n", "Called deployment predict endpoint | \n", "\n", " |
| \n", " | \n", " | \n", " | Actions Needed | \n", "
|---|---|---|---|
| Step | \n", "Status | \n", "Details | \n", "\n", " |
| initiate | \n", "Done | \n", "Initiated the model | \n", "\n", " |
| prepare() | \n", "Done | \n", "Generated runtime.yaml | \n", "\n", " |
| Generated score.py | \n", "\n", " | ||
| Serialized model | \n", "\n", " | ||
| Populated metadata(Custom, Taxonomy and Provenance) | \n", "\n", " | ||
| verify() | \n", "Done | \n", "Local tested .predict from score.py | \n", "\n", " |
| save() | \n", "Done | \n", "Conducted Introspect Test | \n", "\n", " |
| Uploaded artifact to model catalog | \n", "\n", " | ||
| deploy() | \n", "DELETED | \n", "Deployed the model | \n", "\n", " |
| predict() | \n", "Not Available | \n", "Called deployment predict endpoint | \n", "\n", " |