{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\"banner\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Working with Watson Machine Learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The notebook will train, create and deploy a Credit Risk model, configure OpenScale to monitor that deployment, and inject seven days' worth of historical records and measurements for viewing in the OpenScale Insights dashboard." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Contents\n", "\n", "- [Setup](#setup)\n", "- [Model building and deployment](#model)\n", "- [OpenScale configuration](#openscale)\n", "- [Quality monitor and feedback logging](#quality)\n", "- [Fairness, drift monitoring and explanations](#fairness)\n", "- [Custom monitors and metrics](#custom)\n", "- [Payload analytics](#analytics)\n", "- [Historical data](#historical)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1.0 Setup " ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## 1.1 Package installation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note: Some packages that are installed are dependencies for other packages. The versions are pinned to prevent warnings or errors." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm -rf /home/spark/shared/user-libs/python3.6*\n", "!pip install --upgrade ibm-ai-openscale==2.2.1 --no-cache --user | tail -n 1\n", "!pip install --upgrade opt-einsum==2.3.2 --no-cache | tail -n 1\n", "!pip install --upgrade typing-extensions==3.6.2.1 --no-cache | tail -n 1\n", "!pip install --upgrade jupyter==1 --no-cache | tail -n 1\n", "!pip install --upgrade tensorboard==1.15.0 | tail -n 1\n", "!pip install --upgrade JPype1-py3 | tail -n 1\n", "!pip install --upgrade watson-machine-learning-client-V4==1.0.93 | tail -n 1\n", "!pip install --upgrade numpy==1.18.3 --no-cache | tail -n 1\n", "!pip install --upgrade SciPy==1.4.1 --no-cache | tail -n 1\n", "!pip install --upgrade pyspark==2.3 | tail -n 1\n", "!pip install --upgrade scikit-learn==0.20.3 | tail -n 1\n", "!pip install --upgrade pandas==0.24.2 | tail -n 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Action: restart the kernel!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Configure credentials" ] }, { "cell_type": "markdown", "metadata": { "scrolled": true }, "source": [ "- WOS_CREDENTIALS (ICP)\n", "- WML_CREDENTIALS (ICP)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Replace the `username` and `password` values of `************` with your Cloud Pak for Data `username` and `password`. The value for `url` should match the `url` for your Cloud Pak for Data cluster, which you can get from the browser address bar (be sure to include the 'https://'. The credentials should look something like this (these are example values, not the ones you will use):\n", "\n", "`\n", "WOS_CREDENTIALS = {\n", " \"url\": \"https://zen.clusterid.us-south.containers.appdomain.cloud\",\n", " \"username\": \"cp4duser\",\n", " \"password\" : \"cp4dpass\"\n", " }\n", "`\n", "#### NOTE: Make sure that there is no trailing forward slash `/` in the `url`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "WOS_CREDENTIALS = {\n", " \"url\": \"************\",\n", " \"username\": \"************\",\n", " \"password\": \"************\"\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "WML_CREDENTIALS = WOS_CREDENTIALS.copy()\n", "WML_CREDENTIALS['instance_id']='openshift'\n", "WML_CREDENTIALS['version']='3.0.0'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Provide a custom name to be concatenated to model name, deployment name and open scale monitor. Sample value for CUSTOM_NAME could be ```CUSTOM_NAME = 'SAMAYA_OPENSCALE_3.0'```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CUSTOM_NAME = '************'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "MODEL_NAME = CUSTOM_NAME + \"_MODEL\"\n", "DEPLOYMENT_NAME = CUSTOM_NAME + \"_DEPLOYMENT\"\n", "MONITOR_NAME = CUSTOM_NAME + \"_MONITOR\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2.0 Model building and deployment " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section you will learn how to train Spark MLLib model and next deploy it as web-service using Watson Machine Learning service." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.1 Load the training data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "!rm german_credit_data_biased_training.csv\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/german_credit_data_biased_training.csv\n", " " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from pyspark.sql import SparkSession\n", "import pandas as pd\n", "import json\n", "\n", "spark = SparkSession.builder.getOrCreate()\n", "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n", "df_data = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n", "df_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 Explore data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "df_data.printSchema()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "print(\"Number of records: \" + str(df_data.count()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.3 Create a model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "spark_df = df_data\n", "(train_data, test_data) = spark_df.randomSplit([0.8, 0.2], 24)\n", "\n", "print(\"Number of records for training: \" + str(train_data.count()))\n", "print(\"Number of records for evaluation: \" + str(test_data.count()))\n", "\n", "spark_df.printSchema()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below creates a Random Forest Classifier with Spark, setting up string indexers for the categorical features and the label column. Finally, this notebook creates a pipeline including the indexers and the model, and does an initial Area Under ROC evaluation of the model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\n", "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n", "from pyspark.ml import Pipeline, Model\n", "\n", "si_CheckingStatus = StringIndexer(inputCol = 'CheckingStatus', outputCol = 'CheckingStatus_IX')\n", "si_CreditHistory = StringIndexer(inputCol = 'CreditHistory', outputCol = 'CreditHistory_IX')\n", "si_LoanPurpose = StringIndexer(inputCol = 'LoanPurpose', outputCol = 'LoanPurpose_IX')\n", "si_ExistingSavings = StringIndexer(inputCol = 'ExistingSavings', outputCol = 'ExistingSavings_IX')\n", "si_EmploymentDuration = StringIndexer(inputCol = 'EmploymentDuration', outputCol = 'EmploymentDuration_IX')\n", "si_Sex = StringIndexer(inputCol = 'Sex', outputCol = 'Sex_IX')\n", "si_OthersOnLoan = StringIndexer(inputCol = 'OthersOnLoan', outputCol = 'OthersOnLoan_IX')\n", "si_OwnsProperty = StringIndexer(inputCol = 'OwnsProperty', outputCol = 'OwnsProperty_IX')\n", "si_InstallmentPlans = StringIndexer(inputCol = 'InstallmentPlans', outputCol = 'InstallmentPlans_IX')\n", "si_Housing = StringIndexer(inputCol = 'Housing', outputCol = 'Housing_IX')\n", "si_Job = StringIndexer(inputCol = 'Job', outputCol = 'Job_IX')\n", "si_Telephone = StringIndexer(inputCol = 'Telephone', outputCol = 'Telephone_IX')\n", "si_ForeignWorker = StringIndexer(inputCol = 'ForeignWorker', outputCol = 'ForeignWorker_IX')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "si_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n", "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "va_features = VectorAssembler(inputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\", \"EmploymentDuration_IX\", \"Sex_IX\", \\\n", " \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\", \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \\\n", " \"LoanDuration\", \"LoanAmount\", \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\", \\\n", " \"Dependents\"], outputCol=\"features\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from pyspark.ml.classification import RandomForestClassifier\n", "classifier = RandomForestClassifier(featuresCol=\"features\")\n", "\n", "pipeline = Pipeline(stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker, si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\\\n", " si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n", "model = pipeline.fit(train_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "predictions = model.transform(test_data)\n", "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\", metricName='areaUnderROC')\n", "area_under_curve = evaluatorDT.evaluate(predictions)\n", "\n", "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\", metricName='areaUnderPR')\n", "area_under_PR = evaluatorDT.evaluate(predictions)\n", "#default evaluation is areaUnderROC\n", "print(\"areaUnderROC = %g\" % area_under_curve, \"areaUnderPR = %g\" % area_under_PR)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.4 evaluate more metrics by exporting them into pandas and numpy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import classification_report\n", "y_pred = predictions.toPandas()['prediction']\n", "y_pred = ['Risk' if pred == 1.0 else 'No Risk' for pred in y_pred]\n", "y_test = test_data.toPandas()['Risk']\n", "print(classification_report(y_test, y_pred, target_names=['Risk', 'No Risk']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.5 Publish the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, the notebook uses Watson Machine Learning to save the model (including the pipeline) to the WML instance. Previous versions of the model are removed so that the notebook can be run again, resetting all data for another demo." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from watson_machine_learning_client import WatsonMachineLearningAPIClient\n", "import json\n", "\n", "wml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5.1 Set default space" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to deploy a model, you would have to create different\n", " deployment spaces and deploy your models there. You can list all the spaces using the .list()\n", " function, or you can create new spaces by going to CP4D menu on top left corner --> analyze -->\n", " analytics deployments --> New Deployment Space. Once you know which space you want to deploy\n", " in, simply use the GUID of the space as argument for .set.default_space() function below\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "wml_client.spaces.list()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll use the `GUID` for your Deployment space as listed for the `default_space` in the method below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "wml_client.set.default_space('************')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternately, set `space_name` below and use the following cell to create a space with that name" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# space_name = \"my_space_name\"\n", "# spaces = wml_client.spaces.get_details()['resources']\n", "# space_id = None\n", "# for space in spaces:\n", "# if space['entity']['name'] == space_name:\n", "# space_id = space[\"metadata\"][\"guid\"]\n", "# if space_id is None:\n", "# space_id = wml_client.spaces.store(\n", "# meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"guid\"]\n", "#wml_client.set.default_space(space_id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5.2 Remove existing model and deployment" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "deployment_details = wml_client.deployments.get_details()\n", "for deployment in deployment_details['resources']:\n", " deployment_id = deployment['metadata']['guid']\n", " model_id = deployment['entity']['asset']['href'].split('/')[3].split('?')[0]\n", " if deployment['entity']['name'] == DEPLOYMENT_NAME:\n", " print('Deleting deployment id', deployment_id)\n", " wml_client.deployments.delete(deployment_id)\n", " print('Deleting model id', model_id)\n", " wml_client.repository.delete(model_id)\n", "wml_client.repository.list_models()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5.4 Store the model in Watson Machine Learning on CP4D" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "wml_models = wml_client.repository.get_model_details()\n", "model_uid = None\n", "\n", "for model_in in wml_models['resources']:\n", " if MODEL_NAME == model_in['entity']['name']:\n", " model_uid = model_in['metadata']['guid']\n", " break\n", "\n", "if model_uid is None:\n", " print(\"Storing model ...\")\n", " metadata = {\n", " wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n", " wml_client.repository.ModelMetaNames.TYPE: 'mllib_2.3',\n", " wml_client.repository.ModelMetaNames.RUNTIME_UID: 'spark-mllib_2.3',\n", " }\n", "\n", " published_model_details = wml_client.repository.store_model(model, metadata, training_data=df_data, pipeline=pipeline)\n", " model_uid = wml_client.repository.get_model_uid(published_model_details)\n", " print(\"Done\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "model_uid" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.6 Deploy the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next section of the notebook deploys the model as a RESTful web service in Watson Machine Learning. The deployed model will have a scoring URL you can use to send data to the model for predictions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "wml_deployments = wml_client.deployments.get_details()\n", "deployment_uid = None\n", "for deployment in wml_deployments['resources']:\n", " if DEPLOYMENT_NAME == deployment['entity']['name']:\n", " deployment_uid = deployment['metadata']['guid']\n", " break\n", "\n", "if deployment_uid is None:\n", " print(\"Deploying model...\")\n", " meta_props = {\n", " wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n", " wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n", " }\n", " deployment = wml_client.deployments.create(artifact_uid=model_uid, meta_props=meta_props)\n", " deployment_uid = wml_client.deployments.get_uid(deployment)\n", " \n", "print(\"Model id: {}\".format(model_uid))\n", "print(\"Deployment id: {}\".format(deployment_uid))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3.0 Configure OpenScale " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The notebook will now import the necessary libraries and set up a Python OpenScale client." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from ibm_ai_openscale import APIClient4ICP\n", "from ibm_ai_openscale.engines import *\n", "from ibm_ai_openscale.utils import *\n", "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n", "from ibm_ai_openscale.supporting_classes.enums import *" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ai_client = APIClient4ICP(WOS_CREDENTIALS)\n", "ai_client.version" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1 Create datamart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1.1 Set up datamart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Watson OpenScale uses a database to store payload logs and calculated metrics. If an OpenScale datamart exists in Db2, the existing datamart will be used and no data will be overwritten.\n", "\n", "Prior instances of the Credit model will be removed from OpenScale monitoring." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "try:\n", " data_mart_details = ai_client.data_mart.get_details()\n", " print('Using existing external datamart')\n", "except:\n", " print('Datamart is not set up. Please have your cluster Admin set up the DB for OpenScale')\n", " # Admin will need to setup the datamart:\n", " #ai_client.data_mart.setup(db_credentials=DATABASE_CREDENTIALS, schema=SCHEMA_NAME)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Bind machine learning engines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If this binding already exists, this code will output a warning message and use the existing binding." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "binding_uid = ai_client.data_mart.bindings.add('WML instance', WatsonMachineLearningInstance4ICP(wml_credentials=WML_CREDENTIALS))\n", "if binding_uid is None:\n", " binding_uid = ai_client.data_mart.bindings.get_details()['service_bindings'][0]['metadata']['guid']\n", "bindings_details = ai_client.data_mart.bindings.get_details()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "binding_uid" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ai_client.data_mart.bindings.list()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "ai_client.data_mart.bindings.list_assets()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.3 Subscriptions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.1 Remove existing credit risk subscriptions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This code removes previous subscriptions to the Credit model to refresh the monitors with the new model and new data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", "for subscription in subscriptions_uids:\n", " sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n", " if sub_name == MODEL_NAME:\n", " ai_client.data_mart.subscriptions.delete(subscription)\n", " print('Deleted existing subscription for', MODEL_NAME)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This code creates the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset( \n", " model_uid,\n", " problem_type=ProblemType.BINARY_CLASSIFICATION,\n", " input_data_type=InputDataType.STRUCTURED,\n", " label_column='Risk',\n", " prediction_column='predictedLabel',\n", " probability_column='probability',\n", " feature_columns = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n", " categorical_columns = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n", "))\n", "\n", "if subscription is None:\n", " print('Subscription already exists; get the existing one')\n", " subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", " for sub in subscriptions_uids:\n", " if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == MODEL_NAME:\n", " subscription = ai_client.data_mart.subscriptions.get(sub)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get subscription list" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", "ai_client.data_mart.subscriptions.list()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "subscription_details = subscription.get_details()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.2 Score the model so we can configure monitors" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "credit_risk_scoring_endpoint = None\n", "print(deployment_uid)\n", "\n", "for deployment in wml_client.deployments.get_details()['resources']:\n", " if deployment_uid in deployment['metadata']['guid']:\n", " credit_risk_scoring_endpoint = deployment['entity']['status']['online_url']['url']\n", " \n", "print(credit_risk_scoring_endpoint)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n", "values = [\n", " [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n", " [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n", " [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n", " [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n", " [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n", " [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n", " [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n", " [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n", "]\n", "\n", "payload_scoring = {\"fields\": fields,\"values\": values}\n", "payload = {\n", " wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n", "}\n", "scoring_response = wml_client.deployments.score(deployment_uid, payload)\n", "\n", "print('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4.0 Quality monitoring and feedback logging " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Enable quality monitoring" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n", "\n", "The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "time.sleep(10)\n", "subscription.quality_monitoring.enable(threshold=0.7, min_records=50)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 Feedback logging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm additional_feedback_data.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/additional_feedback_data.json" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "with open('additional_feedback_data.json') as feedback_file:\n", " additional_feedback_data = json.load(feedback_file)\n", "subscription.feedback_logging.store(additional_feedback_data['data'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscription.feedback_logging.show_table()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "run_details = subscription.quality_monitoring.run(background_mode=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscription.quality_monitoring.show_table()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "quality_pd = subscription.quality_monitoring.get_table_content(format='pandas')\n", "quality_pd.plot.barh(x='id', y='value');" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "ai_client.data_mart.get_deployment_metrics()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5.0 Fairness, drift monitoring and explanations \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n", " * Which model feature to monitor\n", " * One or more **majority** groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\n", " * One or more **minority** groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\n", " * The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 95%)\n", "\n", "Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 200 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.1 Enable fairness monitoring" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscription.fairness_monitoring.enable(\n", " features=[\n", " Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.95),\n", " Feature(\"Age\", majority=[[26,75]], minority=[[18,25]], threshold=0.95)\n", " ],\n", " favourable_classes=['No Risk'],\n", " unfavourable_classes=['Risk'],\n", " min_records=200,\n", " training_data=pd_data\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.2 Score the model again now that monitoring is configured" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This next section randomly selects 200 records from the data feed and sends those records to the model for predictions. This is enough to exceed the minimum threshold for records set in the previous section, which allows OpenScale to begin calculating fairness." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm german_credit_feed.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/german_credit_feed.json\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Score 200 randomly chosen records" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import random\n", "\n", "with open('german_credit_feed.json', 'r') as scoring_file:\n", " scoring_data = json.load(scoring_file)\n", "\n", "fields = scoring_data['fields']\n", "values = []\n", "for _ in range(200):\n", " values.append(random.choice(scoring_data['values']))\n", "payload_scoring = {\"fields\": fields, \"values\": values}\n", "payload = {\n", " wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n", "}\n", "scoring_response = wml_client.deployments.score(deployment_uid, payload)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5.3 Run fairness monitor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Kick off a fairness monitor run on current data. The monitor runs hourly, but can be manually initiated using the Python client, the REST API, or the graphical user interface. We have a 30 second sleep so that the scoring of 200 payloads above can complete." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "time.sleep(30)\n", "\n", "run_details = subscription.fairness_monitoring.run(background_mode=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "time.sleep(5)\n", "\n", "subscription.fairness_monitoring.show_table()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5.4 Configure Explainability" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we provide OpenScale with the training data to enable and configure the explainability features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import *\n", "subscription.explainability.enable(training_data=pd_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "explainability_details = subscription.explainability.get_details()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5.7 Run explanation for sample record" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "transaction_id = subscription.payload_logging.get_table_content(limit=1)['scoring_id'].values[0]\n", "\n", "print(transaction_id)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "explain_run = subscription.explainability.run(transaction_id=transaction_id, background_mode=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if explain_run == None:\n", " # explanation didn't finish within 180 seconds, if explaination is still not finished give it a minute or so then re-run this cell\n", " time.sleep(10)\n", " explain_table = subscription.explainability.get_table_content(format='pandas')\n", " explain_result = pd.DataFrame.from_dict(explain_table[explain_table['transaction_id']==transaction_id]['explanation'][0]['entity']['predictions'][0]['explanation_features'])\n", "else:\n", " explain_result = pd.DataFrame.from_dict(explain_run['entity']['predictions'][0]['explanation_features'])\n", "\n", "explain_result.plot.barh(x='feature_name', y='weight', color='g', alpha=0.8);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6.0 Custom monitors and metrics " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6.1 Register custom monitor" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_definition(monitor_name):\n", " monitors_definitions = ai_client.data_mart.monitors.get_details()['monitor_definitions']\n", " \n", " for definition in monitors_definitions:\n", " if monitor_name == definition['entity']['name']:\n", " return definition\n", " \n", " return None" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import Metric, Tag\n", "\n", "monitor_name = MONITOR_NAME\n", "metrics = [Metric(name='sensitivity', lower_limit_default=0.8), Metric(name='specificity', lower_limit_default=0.75)]\n", "tags = [Tag(name='region', description='customer geographical region')]\n", "\n", "existing_definition = get_definition(monitor_name)\n", "\n", "if existing_definition is None:\n", " my_monitor = ai_client.data_mart.monitors.add(name=monitor_name, metrics=metrics, tags=tags)\n", "else:\n", " my_monitor = existing_definition" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.1.1 Get monitors uids and details" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "monitor_uid = my_monitor['metadata']['guid']\n", "\n", "print(monitor_uid)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_monitor = ai_client.data_mart.monitors.get_details(monitor_uid=monitor_uid)\n", "print('monitor definition details', my_monitor)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.2 Enable custom monitor for subscription" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import Threshold\n", "\n", "thresholds = [Threshold(metric_uid='sensitivity', lower_limit=0.9)]\n", "subscription.monitoring.enable(monitor_uid=monitor_uid, thresholds=thresholds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 6.2.1 Get monitor configuration details" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "subscription.monitoring.get_details(monitor_uid=monitor_uid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6.3 Storing custom metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "metrics = {\"specificity\": 0.78, \"sensitivity\": 0.67, \"region\": \"us-south\"}\n", "\n", "subscription.monitoring.store_metrics(monitor_uid=monitor_uid, metrics=metrics)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.3.1 List and get custom metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "subscription.monitoring.show_table(monitor_uid=monitor_uid)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "custom_metrics = subscription.monitoring.get_metrics(monitor_uid=monitor_uid, deployment_uid='credit')\n", "custom_metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "custom_metrics_pandas = subscription.monitoring.get_table_content(monitor_uid=monitor_uid)\n", "\n", "%matplotlib inline\n", "custom_metrics_pandas.plot.barh(x='id', y='value');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 7.0 Payload analytics " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7.1 Run data distributions calculation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datetime import datetime\n", "\n", "start_date = \"2018-01-01T00:00:00.00Z\"\n", "end_date = datetime.utcnow().isoformat() + \"Z\"\n", "\n", "sex_distribution = subscription.payload_logging.data_distribution.run(\n", " start_date=start_date,\n", " end_date=end_date,\n", " group=['predictedLabel', 'Sex'],\n", " agg=['count'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7.2 Get data distributions as pandas dataframe" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sex_distribution_run_uid = sex_distribution['id']\n", "distributions_pd = subscription.payload_logging.data_distribution.get_run_result(run_id=sex_distribution_run_uid, format='pandas')\n", "distributions_pd" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "subscription.payload_logging.data_distribution.show_chart(sex_distribution_run_uid);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "credit_history_distribution = subscription.payload_logging.data_distribution.run(\n", " start_date=start_date,\n", " end_date=end_date,\n", " group=['predictedLabel', 'CreditHistory'],\n", " agg=['count'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "credit_history_distribution_run_uid = credit_history_distribution['id']\n", "\n", "subscription.payload_logging.data_distribution.show_chart(credit_history_distribution_run_uid);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 8.0 Historical data " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 8.1 Insert historical payloads" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next section of the notebook downloads and writes historical data to the payload and measurement tables to simulate a production model that has been monitored and receiving regular traffic for the last seven days. This historical data can be viewed in the Watson OpenScale user interface. The code uses the Python and REST APIs to write this data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "!rm history_payloads_with_transaction_*.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_0.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_1.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_2.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_3.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_4.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_5.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_6.json" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "historyDays = 7" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n", "import datetime\n", "import time\n", "\n", "for day in range(historyDays):\n", " print('Loading day {}'.format(day + 1))\n", " history_file = 'history_payloads_with_transaction_id_' + str(day) + '.json'\n", " with open(history_file) as f:\n", " payloads = json.load(f)\n", " hourly_records = int(len(payloads) / 24)\n", " index = 0\n", " for hour in range(24):\n", " recordsList = []\n", " for i in range(hourly_records):\n", " score_time = str(datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1))))\n", " recordsList.append(PayloadRecord(request=payloads[index]['request'], response=payloads[index]['response'], scoring_timestamp=score_time))\n", " index += 1\n", " subscription.payload_logging.store(records=recordsList)\n", "print('Finished')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "data_mart_id = subscription.get_details()['metadata']['url'].split('/service_bindings')[0].split('marts/')[1]\n", "print(data_mart_id)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "performance_metrics_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/metrics'\n", "print(performance_metrics_url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.2 Insert historical fairness metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm history_fairness.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_fairness.json" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import requests\n", "from requests.auth import HTTPBasicAuth\n", "\n", "def create_token():\n", " header = {\n", " \"Content-Type\": \"application/x-www-form-urlencoded\",\n", " \"Accept\": \"application/json\"\n", " }\n", "\n", " response = requests.Session().get(\n", " WOS_CREDENTIALS['url'] + '/v1/preauth/validateAuth',\n", " headers=header,\n", " auth=HTTPBasicAuth(\n", " WOS_CREDENTIALS['username'],\n", " WOS_CREDENTIALS['password']\n", " ),\n", " verify=False)\n", "\n", " response = handle_response(200, 'access token', response, True)\n", " token = response['accessToken']\n", "\n", " return token" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "with open('history_fairness.json', 'r') as history_file:\n", " payloads = json.load(history_file)\n", "\n", "for day in range(historyDays):\n", " print('Loading day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed\n", " \n", " qualityMetric = {\n", " 'metric_type': 'fairness',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': payloads[index]\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)\n", "\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.3 Insert historical debias metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm history_debias.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_debias.json" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "with open('history_debias.json', 'r') as history_file:\n", " payloads = json.load(history_file)\n", "\n", "for day in range(historyDays):\n", " print('Loading day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed\n", "\n", " qualityMetric = {\n", " 'metric_type': 'debiased_fairness',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': payloads[index]\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.4 Insert historical quality metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "measurements = [0.76, 0.78, 0.68, 0.72, 0.73, 0.77, 0.80]\n", "for day in range(historyDays):\n", " print('Day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " qualityMetric = {\n", " 'metric_type': 'quality',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': {\n", " 'quality': measurements[day],\n", " 'threshold': 0.7,\n", " 'metrics': [\n", " {\n", " 'name': 'auroc',\n", " 'value': measurements[day],\n", " 'threshold': 0.7\n", " }\n", " ]\n", " }\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.5 Insert historical confusion matrixes" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm history_quality_metrics.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_quality_metrics.json" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "measurements_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/measurements'\n", "print(measurements_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "with open('history_quality_metrics.json') as json_file:\n", " records = json.load(json_file)\n", "\n", "for day in range(historyDays):\n", " index = 0\n", " measurments = []\n", " print('Day', day + 1)\n", " \n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", "\n", " measurement = {\n", " \"monitor_definition_id\": 'quality',\n", " \"binding_id\": subscription.binding_uid,\n", " \"subscription_id\": subscription.uid,\n", " \"asset_id\": subscription.source_uid,\n", " 'metrics': [records[index]['metrics']],\n", " 'sources': [records[index]['sources']],\n", " 'timestamp': score_time\n", " }\n", "\n", " measurments.append(measurement)\n", " index+=1\n", "\n", " response = requests.post(measurements_url, json=measurments, headers=ai_client._get_headers(), verify=False)\n", "\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.6 Insert historical performance metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "for day in range(historyDays):\n", " print('Day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " score_count = random.randint(60, 600)\n", " score_resp = random.uniform(60, 300)\n", "\n", " performanceMetric = {\n", " 'metric_type': 'performance',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': {\n", " 'response_time': score_resp,\n", " 'records': score_count\n", " }\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[performanceMetric], headers=iam_headers, verify=False)\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.7 Insert historical manual labeling" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "manual_labeling_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/manual_labelings'\n", "print(manual_labeling_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!rm history_manual_labeling.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_manual_labeling.json" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "with open('history_manual_labeling.json', 'r') as history_file:\n", " records = json.load(history_file)\n", "\n", "for day in range(historyDays):\n", " print('Loading day', day + 1)\n", " record_json = []\n", " for hour in range(24):\n", " for record in records:\n", " if record['fastpath_history_day'] == day and record['fastpath_history_hour'] == hour:\n", " record['binding_id'] = binding_uid\n", " record['subscription_id'] = model_uid\n", " record['asset_revision'] = model_uid\n", " record['deployment_id'] = deployment_uid\n", " record['scoring_timestamp'] = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " record_json.append(record)\n", " response = requests.post(manual_labeling_url, json=record_json, headers=iam_headers, verify=False)\n", "\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.8 Additional data to help debugging" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "print('Datamart:', data_mart_id)\n", "print('Model:', model_uid)\n", "print('Deployment:', deployment_uid)\n", "print('Binding:', binding_uid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.9 Identify transactions for Explainability" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "payload_data = subscription.payload_logging.get_table_content(limit=10)\n", "payload_data.filter(items=['scoring_id', 'predictedLabel', 'probability'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Congratulations!\n", "\n", "You have finished the hands-on lab for IBM Watson OpenScale. You can now view the OpenScale dashboard by going to the ICP `Home` page, and clicking `Add-ons`. Choose the `OpenScale` tile and click the menu to `Open`. Click on the tile for the model you've created to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n", "\n", "OpenScale shows model performance over time. You have two options to keep data flowing to your OpenScale graphs:\n", " * Download, configure and schedule the [model feed notebook](https://raw.githubusercontent.com/emartensibm/german-credit/master/german_credit_scoring_feed.ipynb). This notebook can be set up with your WML credentials, and scheduled to provide a consistent flow of scoring requests to your model, which will appear in your OpenScale monitors.\n", " * Re-run this notebook. Running this notebook from the beginning will delete and re-create the model and deployment, and re-create the historical data. Please note that the payload and measurement logs for the previous deployment will continue to be stored in your datamart, and can be deleted if necessary." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Authors\n", "\n", "Eric Martens, is a technical specialist having expertise in analysis and description of business processes, and their translation into functional and non-functional IT requirements. He acts as the interpreter between the worlds of IT and business.\n", "\n", "Lukasz Cmielowski, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n", "\n", "Zilu (Peter) Tang, is a cognitive developer with experties in deep learning and enterprise AI solutions from Watson Openscale to many other cutting-edge IBM research projects." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.6", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 1 }