{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\"banner\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Working with Watson Machine Learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The notebook will train, create and deploy a Credit Risk model, configure OpenScale to monitor that deployment, and inject seven days' worth of historical records and measurements for viewing in the OpenScale Insights dashboard." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Contents\n", "\n", "- [Setup](#setup)\n", "- [Model building and deployment](#model)\n", "- [OpenScale configuration](#openscale)\n", "- [Quality monitor and feedback logging](#quality)\n", "- [Fairness, drift monitoring and explanations](#fairness)\n", "- [Custom monitors and metrics](#custom)\n", "- [Payload analytics](#analytics)\n", "- [Historical data](#historical)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1.0 Setup " ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## 1.1 Package installation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note: Some packages that are installed are dependencies for other packages. The versions are pinned to prevent warnings or errors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed ibm-ai-openscale-2.2.1\n", "Successfully installed opt-einsum-2.3.2\n", "Successfully installed typing-extensions-3.6.2.1\n", "Successfully installed jupyter-1.0.0 qtconsole-4.7.7 qtpy-1.9.0\n", "Requirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in /opt/conda/envs/Python-3.6-WMLCE/lib/python3.6/site-packages (from tensorboard==1.15.0) (0.14.1)\n", "Successfully installed JPype1-py3-0.5.5.4\n", "Successfully installed ibm-cos-sdk-2.6.0 ibm-cos-sdk-core-2.6.0 ibm-cos-sdk-s3transfer-2.6.0 watson-machine-learning-client-V4-1.0.93\n", "Successfully installed numpy-1.18.3\n", "Successfully installed SciPy-1.4.1\n", "Successfully installed py4j-0.10.6 pyspark-2.3.0\n", "Successfully installed scikit-learn-0.20.3\n", "Requirement already satisfied, skipping upgrade: six>=1.5 in /user-home/_global_/python-3 (from python-dateutil>=2.5.0->pandas==0.24.2) (1.12.0)\n" ] } ], "source": [ "!rm -rf /home/spark/shared/user-libs/python3.6*\n", "!pip install --upgrade ibm-ai-openscale==2.2.1 --no-cache --user | tail -n 1\n", "!pip install --upgrade opt-einsum==2.3.2 --no-cache | tail -n 1\n", "!pip install --upgrade typing-extensions==3.6.2.1 --no-cache | tail -n 1\n", "!pip install --upgrade jupyter==1 --no-cache | tail -n 1\n", "!pip install --upgrade tensorboard==1.15.0 | tail -n 1\n", "!pip install --upgrade JPype1-py3 | tail -n 1\n", "!pip install --upgrade watson-machine-learning-client-V4==1.0.93 | tail -n 1\n", "!pip install --upgrade numpy==1.18.3 --no-cache | tail -n 1\n", "!pip install --upgrade SciPy==1.4.1 --no-cache | tail -n 1\n", "!pip install --upgrade pyspark==2.3 | tail -n 1\n", "!pip install --upgrade scikit-learn==0.20.3 | tail -n 1\n", "!pip install --upgrade pandas==0.24.2 | tail -n 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Action: restart the kernel!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Configure credentials" ] }, { "cell_type": "markdown", "metadata": { "scrolled": true }, "source": [ "- WOS_CREDENTIALS (ICP)\n", "- WML_CREDENTIALS (ICP)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Replace the `username` and `password` values of `************` with your Cloud Pak for Data `username` and `password`. The value for `url` should match the `url` for your Cloud Pak for Data cluster, which you can get from the browser address bar (be sure to include the 'https://'. The credentials should look something like this (these are example values, not the ones you will use):\n", "\n", "`\n", "WOS_CREDENTIALS = {\n", " \"url\": \"https://zen.clusterid.us-south.containers.appdomain.cloud\",\n", " \"username\": \"cp4duser\",\n", " \"password\" : \"cp4dpass\"\n", " }\n", "`\n", "#### NOTE: Make sure that there is no trailing forward slash `/` in the `url`" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "WOS_CREDENTIALS = {\n", " \"url\": \"https://zen2-cpd-zen2.aida-cpd3-dal10-b3c-56x-f206f13e3-0001.us-south.containers.appdomain.cloud\",\n", " \"username\": \"\",\n", " \"password\": \"\"\n", "}" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "WML_CREDENTIALS = WOS_CREDENTIALS.copy()\n", "WML_CREDENTIALS['instance_id']='openshift'\n", "WML_CREDENTIALS['version']='3.0.0'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Provide a custom name to be concatenated to model name, deployment name and open scale monitor. Sample value for CUSTOM_NAME could be ```CUSTOM_NAME = 'SAMAYA_OPENSCALE_3.0'```" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "CUSTOM_NAME = 'Scottda-telco-9-22-2020'" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "MODEL_NAME = CUSTOM_NAME + \"_MODEL\"\n", "DEPLOYMENT_NAME = CUSTOM_NAME + \"_DEPLOYMENT\"\n", "MONITOR_NAME = CUSTOM_NAME + \"_MONITOR\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2.0 Model building and deployment " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section you will learn how to train Spark MLLib model and next deploy it as web-service using Watson Machine Learning service." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.1 Load the training data" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘german_credit_data_biased_training.csv’: No such file or directory\n", "--2020-09-22 12:20:36-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/german_credit_data_biased_training.csv\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 689622 (673K) [text/plain]\n", "Saving to: ‘german_credit_data_biased_training.csv’\n", "\n", "100%[======================================>] 689,622 --.-K/s in 0.02s \n", "\n", "2020-09-22 12:20:37 (36.6 MB/s) - ‘german_credit_data_biased_training.csv’ saved [689622/689622]\n", "\n" ] } ], "source": [ "!rm german_credit_data_biased_training.csv\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/german_credit_data_biased_training.csv\n", " " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "Row(CheckingStatus='0_to_200', LoanDuration=31, CreditHistory='credits_paid_to_date', LoanPurpose='other', LoanAmount=1889, ExistingSavings='100_to_500', EmploymentDuration='less_1', InstallmentPercent=3, Sex='female', OthersOnLoan='none', CurrentResidenceDuration=3, OwnsProperty='savings_insurance', Age=32, InstallmentPlans='none', Housing='own', ExistingCreditsCount=1, Job='skilled', Dependents=1, Telephone='none', ForeignWorker='yes', Risk='No Risk')" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pyspark.sql import SparkSession\n", "import pandas as pd\n", "import json\n", "\n", "spark = SparkSession.builder.getOrCreate()\n", "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n", "df_data = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n", "df_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 Explore data" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "root\n", " |-- CheckingStatus: string (nullable = true)\n", " |-- LoanDuration: integer (nullable = true)\n", " |-- CreditHistory: string (nullable = true)\n", " |-- LoanPurpose: string (nullable = true)\n", " |-- LoanAmount: integer (nullable = true)\n", " |-- ExistingSavings: string (nullable = true)\n", " |-- EmploymentDuration: string (nullable = true)\n", " |-- InstallmentPercent: integer (nullable = true)\n", " |-- Sex: string (nullable = true)\n", " |-- OthersOnLoan: string (nullable = true)\n", " |-- CurrentResidenceDuration: integer (nullable = true)\n", " |-- OwnsProperty: string (nullable = true)\n", " |-- Age: integer (nullable = true)\n", " |-- InstallmentPlans: string (nullable = true)\n", " |-- Housing: string (nullable = true)\n", " |-- ExistingCreditsCount: integer (nullable = true)\n", " |-- Job: string (nullable = true)\n", " |-- Dependents: integer (nullable = true)\n", " |-- Telephone: string (nullable = true)\n", " |-- ForeignWorker: string (nullable = true)\n", " |-- Risk: string (nullable = true)\n", "\n" ] } ], "source": [ "df_data.printSchema()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of records: 5000\n" ] } ], "source": [ "print(\"Number of records: \" + str(df_data.count()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.3 Create a model" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of records for training: 4016\n", "Number of records for evaluation: 984\n", "root\n", " |-- CheckingStatus: string (nullable = true)\n", " |-- LoanDuration: integer (nullable = true)\n", " |-- CreditHistory: string (nullable = true)\n", " |-- LoanPurpose: string (nullable = true)\n", " |-- LoanAmount: integer (nullable = true)\n", " |-- ExistingSavings: string (nullable = true)\n", " |-- EmploymentDuration: string (nullable = true)\n", " |-- InstallmentPercent: integer (nullable = true)\n", " |-- Sex: string (nullable = true)\n", " |-- OthersOnLoan: string (nullable = true)\n", " |-- CurrentResidenceDuration: integer (nullable = true)\n", " |-- OwnsProperty: string (nullable = true)\n", " |-- Age: integer (nullable = true)\n", " |-- InstallmentPlans: string (nullable = true)\n", " |-- Housing: string (nullable = true)\n", " |-- ExistingCreditsCount: integer (nullable = true)\n", " |-- Job: string (nullable = true)\n", " |-- Dependents: integer (nullable = true)\n", " |-- Telephone: string (nullable = true)\n", " |-- ForeignWorker: string (nullable = true)\n", " |-- Risk: string (nullable = true)\n", "\n" ] } ], "source": [ "spark_df = df_data\n", "(train_data, test_data) = spark_df.randomSplit([0.8, 0.2], 24)\n", "\n", "print(\"Number of records for training: \" + str(train_data.count()))\n", "print(\"Number of records for evaluation: \" + str(test_data.count()))\n", "\n", "spark_df.printSchema()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below creates a Random Forest Classifier with Spark, setting up string indexers for the categorical features and the label column. Finally, this notebook creates a pipeline including the indexers and the model, and does an initial Area Under ROC evaluation of the model." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\n", "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n", "from pyspark.ml import Pipeline, Model\n", "\n", "si_CheckingStatus = StringIndexer(inputCol = 'CheckingStatus', outputCol = 'CheckingStatus_IX')\n", "si_CreditHistory = StringIndexer(inputCol = 'CreditHistory', outputCol = 'CreditHistory_IX')\n", "si_LoanPurpose = StringIndexer(inputCol = 'LoanPurpose', outputCol = 'LoanPurpose_IX')\n", "si_ExistingSavings = StringIndexer(inputCol = 'ExistingSavings', outputCol = 'ExistingSavings_IX')\n", "si_EmploymentDuration = StringIndexer(inputCol = 'EmploymentDuration', outputCol = 'EmploymentDuration_IX')\n", "si_Sex = StringIndexer(inputCol = 'Sex', outputCol = 'Sex_IX')\n", "si_OthersOnLoan = StringIndexer(inputCol = 'OthersOnLoan', outputCol = 'OthersOnLoan_IX')\n", "si_OwnsProperty = StringIndexer(inputCol = 'OwnsProperty', outputCol = 'OwnsProperty_IX')\n", "si_InstallmentPlans = StringIndexer(inputCol = 'InstallmentPlans', outputCol = 'InstallmentPlans_IX')\n", "si_Housing = StringIndexer(inputCol = 'Housing', outputCol = 'Housing_IX')\n", "si_Job = StringIndexer(inputCol = 'Job', outputCol = 'Job_IX')\n", "si_Telephone = StringIndexer(inputCol = 'Telephone', outputCol = 'Telephone_IX')\n", "si_ForeignWorker = StringIndexer(inputCol = 'ForeignWorker', outputCol = 'ForeignWorker_IX')" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "scrolled": true }, "outputs": [], "source": [ "si_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n", "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "scrolled": true }, "outputs": [], "source": [ "va_features = VectorAssembler(inputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\", \"EmploymentDuration_IX\", \"Sex_IX\", \\\n", " \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\", \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \\\n", " \"LoanDuration\", \"LoanAmount\", \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\", \\\n", " \"Dependents\"], outputCol=\"features\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from pyspark.ml.classification import RandomForestClassifier\n", "classifier = RandomForestClassifier(featuresCol=\"features\")\n", "\n", "pipeline = Pipeline(stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker, si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\\\n", " si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n", "model = pipeline.fit(train_data)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "areaUnderROC = 0.708936 areaUnderPR = 0.631251\n" ] } ], "source": [ "predictions = model.transform(test_data)\n", "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\", metricName='areaUnderROC')\n", "area_under_curve = evaluatorDT.evaluate(predictions)\n", "\n", "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\", metricName='areaUnderPR')\n", "area_under_PR = evaluatorDT.evaluate(predictions)\n", "#default evaluation is areaUnderROC\n", "print(\"areaUnderROC = %g\" % area_under_curve, \"areaUnderPR = %g\" % area_under_PR)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.4 evaluate more metrics by exporting them into pandas and numpy" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " Risk 0.79 0.90 0.84 657\n", " No Risk 0.73 0.51 0.60 327\n", "\n", " micro avg 0.77 0.77 0.77 984\n", " macro avg 0.76 0.71 0.72 984\n", "weighted avg 0.77 0.77 0.76 984\n", "\n" ] } ], "source": [ "from sklearn.metrics import classification_report\n", "y_pred = predictions.toPandas()['prediction']\n", "y_pred = ['Risk' if pred == 1.0 else 'No Risk' for pred in y_pred]\n", "y_test = test_data.toPandas()['Risk']\n", "print(classification_report(y_test, y_pred, target_names=['Risk', 'No Risk']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.5 Publish the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, the notebook uses Watson Machine Learning to save the model (including the pipeline) to the WML instance. Previous versions of the model are removed so that the notebook can be run again, resetting all data for another demo." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "from watson_machine_learning_client import WatsonMachineLearningAPIClient\n", "import json\n", "\n", "wml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5.1 Set default space" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to deploy a model, you would have to create different\n", " deployment spaces and deploy your models there. You can list all the spaces using the .list()\n", " function, or you can create new spaces by going to CP4D menu on top left corner --> analyze -->\n", " analytics deployments --> New Deployment Space. Once you know which space you want to deploy\n", " in, simply use the GUID of the space as argument for .set.default_space() function below\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "------------------------------------ ------------------------ ------------------------\n", "GUID NAME CREATED\n", "0bd92418-0f8d-4a9d-908c-cecbd49a2adc scottda-deployment-space 2020-09-11T20:51:48.629Z\n", "------------------------------------ ------------------------ ------------------------\n" ] } ], "source": [ "wml_client.spaces.list()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll use the `GUID` for your Deployment space as listed for the `default_space` in the method below:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'SUCCESS'" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wml_client.set.default_space('0bd92418-0f8d-4a9d-908c-cecbd49a2adc')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternately, set `space_name` below and use the following cell to create a space with that name" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# space_name = \"my_space_name\"\n", "# spaces = wml_client.spaces.get_details()['resources']\n", "# space_id = None\n", "# for space in spaces:\n", "# if space['entity']['name'] == space_name:\n", "# space_id = space[\"metadata\"][\"guid\"]\n", "# if space_id is None:\n", "# space_id = wml_client.spaces.store(\n", "# meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"guid\"]\n", "#wml_client.set.default_space(space_id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5.2 Remove existing model and deployment" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "------------------------------------ --------------------- ------------------------ ---------\n", "GUID NAME CREATED TYPE\n", "af0557f3-d6ad-4f03-ace2-a5134199d1c9 scottda-9-15-20_MODEL 2020-09-15T23:38:46.002Z mllib_2.3\n", "cdf2a724-52b9-4cda-ac0c-d7d339467d51 SDAmodel-9-11-2020 2020-09-11T20:53:33.002Z mllib_2.3\n", "------------------------------------ --------------------- ------------------------ ---------\n" ] } ], "source": [ "deployment_details = wml_client.deployments.get_details()\n", "for deployment in deployment_details['resources']:\n", " deployment_id = deployment['metadata']['guid']\n", " model_id = deployment['entity']['asset']['href'].split('/')[3].split('?')[0]\n", " if deployment['entity']['name'] == DEPLOYMENT_NAME:\n", " print('Deleting deployment id', deployment_id)\n", " wml_client.deployments.delete(deployment_id)\n", " print('Deleting model id', model_id)\n", " wml_client.repository.delete(model_id)\n", "wml_client.repository.list_models()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5.4 Store the model in Watson Machine Learning on CP4D" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Storing model ...\n", "Done\n" ] } ], "source": [ "wml_models = wml_client.repository.get_model_details()\n", "model_uid = None\n", "\n", "for model_in in wml_models['resources']:\n", " if MODEL_NAME == model_in['entity']['name']:\n", " model_uid = model_in['metadata']['guid']\n", " break\n", "\n", "if model_uid is None:\n", " print(\"Storing model ...\")\n", " metadata = {\n", " wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n", " wml_client.repository.ModelMetaNames.TYPE: 'mllib_2.3',\n", " wml_client.repository.ModelMetaNames.RUNTIME_UID: 'spark-mllib_2.3',\n", " }\n", "\n", " published_model_details = wml_client.repository.store_model(model, metadata, training_data=df_data, pipeline=pipeline)\n", " model_uid = wml_client.repository.get_model_uid(published_model_details)\n", " print(\"Done\")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "'d9f69282-4c9a-4e2f-8293-8589695853da'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_uid" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.6 Deploy the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next section of the notebook deploys the model as a RESTful web service in Watson Machine Learning. The deployed model will have a scoring URL you can use to send data to the model for predictions." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deploying model...\n", "\n", "\n", "#######################################################################################\n", "\n", "Synchronous deployment creation for uid: 'd9f69282-4c9a-4e2f-8293-8589695853da' started\n", "\n", "#######################################################################################\n", "\n", "\n", "initializing\n", "ready\n", "\n", "\n", "------------------------------------------------------------------------------------------------\n", "Successfully finished deployment creation, deployment_uid='e85d6845-4b6b-44bf-a483-80a522852620'\n", "------------------------------------------------------------------------------------------------\n", "\n", "\n", "Model id: d9f69282-4c9a-4e2f-8293-8589695853da\n", "Deployment id: e85d6845-4b6b-44bf-a483-80a522852620\n" ] } ], "source": [ "wml_deployments = wml_client.deployments.get_details()\n", "deployment_uid = None\n", "for deployment in wml_deployments['resources']:\n", " if DEPLOYMENT_NAME == deployment['entity']['name']:\n", " deployment_uid = deployment['metadata']['guid']\n", " break\n", "\n", "if deployment_uid is None:\n", " print(\"Deploying model...\")\n", " meta_props = {\n", " wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n", " wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n", " }\n", " deployment = wml_client.deployments.create(artifact_uid=model_uid, meta_props=meta_props)\n", " deployment_uid = wml_client.deployments.get_uid(deployment)\n", " \n", "print(\"Model id: {}\".format(model_uid))\n", "print(\"Deployment id: {}\".format(deployment_uid))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3.0 Configure OpenScale " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The notebook will now import the necessary libraries and set up a Python OpenScale client." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from ibm_ai_openscale import APIClient4ICP\n", "from ibm_ai_openscale.engines import *\n", "from ibm_ai_openscale.utils import *\n", "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n", "from ibm_ai_openscale.supporting_classes.enums import *" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'2.1.21'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ai_client = APIClient4ICP(WOS_CREDENTIALS)\n", "ai_client.version" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1 Create datamart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1.1 Set up datamart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Watson OpenScale uses a database to store payload logs and calculated metrics. If an OpenScale datamart exists in Db2, the existing datamart will be used and no data will be overwritten.\n", "\n", "Prior instances of the Credit model will be removed from OpenScale monitoring." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using existing external datamart\n" ] } ], "source": [ "try:\n", " data_mart_details = ai_client.data_mart.get_details()\n", " print('Using existing external datamart')\n", "except:\n", " print('Datamart is not set up. Please have your cluster Admin set up the DB for OpenScale')\n", " # Admin will need to setup the datamart:\n", " #ai_client.data_mart.setup(db_credentials=DATABASE_CREDENTIALS, schema=SCHEMA_NAME)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Bind machine learning engines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If this binding already exists, this code will output a warning message and use the existing binding." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "binding_uid = ai_client.data_mart.bindings.add('WML instance', WatsonMachineLearningInstance4ICP(wml_credentials=WML_CREDENTIALS))\n", "if binding_uid is None:\n", " binding_uid = ai_client.data_mart.bindings.get_details()['service_bindings'][0]['metadata']['guid']\n", "bindings_details = ai_client.data_mart.bindings.get_details()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'c411a7d3-34ec-4c99-8a23-6c0638d0aae7'" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "binding_uid" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Service bindings

\n", " \n", " \n", " \n", "
uidnameservice_typecreated
c411a7d3-34ec-4c99-8a23-6c0638d0aae7WML instancewatson_machine_learning2020-09-22T12:23:07.383Z
3f583662-560d-40c8-9473-8826599726d5WML instancewatson_machine_learning2020-09-18T03:18:04.596Z
998WML pre_productionwatson_machine_learning2020-09-17T17:47:31.491Z
999WML productionwatson_machine_learning2020-09-17T17:47:31.275Z
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ai_client.data_mart.bindings.list()" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Available assets

\n", " \n", " \n", " \n", "
source_uidnamecreatedtypeframeworksbinding_uidis_subscribed
d9f69282-4c9a-4e2f-8293-8589695853daScottda-telco-9-22-2020_MODEL2020-09-22T12:22:26.002Zmodelmllib_2.3999False
d9f69282-4c9a-4e2f-8293-8589695853daScottda-telco-9-22-2020_MODEL2020-09-22T12:22:26.002Zmodelmllib_2.3998False
d9f69282-4c9a-4e2f-8293-8589695853daScottda-telco-9-22-2020_MODEL2020-09-22T12:22:26.002Zmodelmllib_2.3c411a7d3-34ec-4c99-8a23-6c0638d0aae7False
81a2aa32-8c5e-4e09-a5a2-944a16dda2e8JRT_WOSFullConfig_MODEL2020-09-18T03:14:13.002Zmodelmllib_2.3999True
81a2aa32-8c5e-4e09-a5a2-944a16dda2e8JRT_WOSFullConfig_MODEL2020-09-18T03:14:13.002Zmodelmllib_2.3998True
81a2aa32-8c5e-4e09-a5a2-944a16dda2e8JRT_WOSFullConfig_MODEL2020-09-18T03:14:13.002Zmodelmllib_2.33f583662-560d-40c8-9473-8826599726d5True
2121433b-bec6-4ded-ba8d-4fd34d4e24d2GermanCreditRiskModelICP2020-09-17T17:54:02.002Zmodelmllib_2.3999True
2121433b-bec6-4ded-ba8d-4fd34d4e24d2GermanCreditRiskModelICP2020-09-17T17:54:02.002Zmodelmllib_2.3998True
3732c133-bdaa-4fb1-996e-e89118ae23c0GermanCreditRiskModelPreProdICP2020-09-17T17:53:18.002Zmodelmllib_2.3999True
3732c133-bdaa-4fb1-996e-e89118ae23c0GermanCreditRiskModelPreProdICP2020-09-17T17:53:18.002Zmodelmllib_2.3998True
8bf25c2f-1ffc-499a-9eaf-48a02c9dcb53GermanCreditRiskModelChallengerICP2020-09-17T17:47:32.002Zmodelscikit-learn_0.20999True
8bf25c2f-1ffc-499a-9eaf-48a02c9dcb53GermanCreditRiskModelChallengerICP2020-09-17T17:47:32.002Zmodelscikit-learn_0.20998True
af0557f3-d6ad-4f03-ace2-a5134199d1c9scottda-9-15-20_MODEL2020-09-15T23:38:46.002Zmodelmllib_2.3999False
af0557f3-d6ad-4f03-ace2-a5134199d1c9scottda-9-15-20_MODEL2020-09-15T23:38:46.002Zmodelmllib_2.3998False
af0557f3-d6ad-4f03-ace2-a5134199d1c9scottda-9-15-20_MODEL2020-09-15T23:38:46.002Zmodelmllib_2.3c411a7d3-34ec-4c99-8a23-6c0638d0aae7False
cdf2a724-52b9-4cda-ac0c-d7d339467d51SDAmodel-9-11-20202020-09-11T20:53:33.002Zmodelmllib_2.3999False
cdf2a724-52b9-4cda-ac0c-d7d339467d51SDAmodel-9-11-20202020-09-11T20:53:33.002Zmodelmllib_2.3998False
cdf2a724-52b9-4cda-ac0c-d7d339467d51SDAmodel-9-11-20202020-09-11T20:53:33.002Zmodelmllib_2.3c411a7d3-34ec-4c99-8a23-6c0638d0aae7False
f9a9d722-e00e-4f83-9bb7-f606f0863675ChurnSparkModelV12020-09-10T03:57:40.002Zmodelmllib_2.3999True
f9a9d722-e00e-4f83-9bb7-f606f0863675ChurnSparkModelV12020-09-10T03:57:40.002Zmodelmllib_2.3998True
f9a9d722-e00e-4f83-9bb7-f606f0863675ChurnSparkModelV12020-09-10T03:57:40.002Zmodelmllib_2.33f583662-560d-40c8-9473-8826599726d5True
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ai_client.data_mart.bindings.list_assets()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.3 Subscriptions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.1 Remove existing credit risk subscriptions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This code removes previous subscriptions to the Credit model to refresh the monitors with the new model and new data." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", "for subscription in subscriptions_uids:\n", " sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n", " if sub_name == MODEL_NAME:\n", " ai_client.data_mart.subscriptions.delete(subscription)\n", " print('Deleted existing subscription for', MODEL_NAME)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This code creates the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset( \n", " model_uid,\n", " problem_type=ProblemType.BINARY_CLASSIFICATION,\n", " input_data_type=InputDataType.STRUCTURED,\n", " label_column='Risk',\n", " prediction_column='predictedLabel',\n", " probability_column='probability',\n", " feature_columns = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n", " categorical_columns = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n", "))\n", "\n", "if subscription is None:\n", " print('Subscription already exists; get the existing one')\n", " subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", " for sub in subscriptions_uids:\n", " if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == MODEL_NAME:\n", " subscription = ai_client.data_mart.subscriptions.get(sub)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get subscription list" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Subscriptions

\n", " \n", " \n", " \n", "
uidnametypebinding_uidcreated
14a4e07c-6cae-4b60-828d-40be2661ae73Scottda-telco-9-22-2020_MODELmodel9992020-09-22T12:24:50.336Z
58b43991-540e-4179-a867-445089f58c33ChurnSparkModelV1model9992020-09-18T17:12:44.963Z
e70d218c-e77a-442d-99e1-2b712caf84a4JRT_WOSFullConfig_MODELmodel9992020-09-18T03:20:35.876Z
4181e595-765c-485b-b3b7-170159194c49GermanCreditRiskModelICPmodel9992020-09-17T17:54:14.017Z
e19daf91-8069-4a70-954b-4ca1521883c5GermanCreditRiskModelPreProdICPmodel9982020-09-17T17:53:29.055Z
20cd4e73-6007-41be-8d20-4fffc029eddbGermanCreditRiskModelChallengerICPmodel9982020-09-17T17:52:46.833Z
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", "ai_client.data_mart.subscriptions.list()" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "scrolled": false }, "outputs": [], "source": [ "subscription_details = subscription.get_details()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.2 Score the model so we can configure monitors" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "e85d6845-4b6b-44bf-a483-80a522852620\n", "https://zen2-cpd-zen2.aida-cpd3-dal10-b3c-56x-f2c6cdc6801be85fd188b09d006f13e3-0001.us-south.containers.appdomain.cloud/v4/deployments/e85d6845-4b6b-44bf-a483-80a522852620/predictions\n" ] } ], "source": [ "credit_risk_scoring_endpoint = None\n", "print(deployment_uid)\n", "\n", "for deployment in wml_client.deployments.get_details()['resources']:\n", " if deployment_uid in deployment['metadata']['guid']:\n", " credit_risk_scoring_endpoint = deployment['entity']['status']['online_url']['url']\n", " \n", "print(credit_risk_scoring_endpoint)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Single record scoring result: \n", " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'EmploymentDuration_IX', 'ExistingSavings_IX', 'ForeignWorker_IX', 'Housing_IX', 'InstallmentPlans_IX', 'Job_IX', 'LoanPurpose_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'Sex_IX', 'Telephone_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n", " values: ['no_checking', 13, 'credits_paid_to_date', 'car_new', 1343, '100_to_500', '1_to_4', 2, 'female', 'none', 3, 'savings_insurance', 46, 'none', 'own', 2, 'skilled', 1, 'none', 'yes', 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, [21, [1, 3, 5, 13, 14, 15, 16, 17, 18, 19, 20], [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 13.0, 46.0, 2.0, 1.0]], [11.976118835415127, 8.023881164584871], [0.5988059417707563, 0.4011940582292436], 0.0, 'No Risk']\n" ] } ], "source": [ "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n", "values = [\n", " [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n", " [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n", " [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n", " [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n", " [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n", " [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n", " [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n", " [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n", "]\n", "\n", "payload_scoring = {\"fields\": fields,\"values\": values}\n", "payload = {\n", " wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n", "}\n", "scoring_response = wml_client.deployments.score(deployment_uid, payload)\n", "\n", "print('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4.0 Quality monitoring and feedback logging " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Enable quality monitoring" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n", "\n", "The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint." ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "scrolled": true }, "outputs": [], "source": [ "time.sleep(10)\n", "subscription.quality_monitoring.enable(threshold=0.7, min_records=50)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 Feedback logging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘additional_feedback_data.json’: No such file or directory\n", "--2020-09-22 12:25:58-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/additional_feedback_data.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 16506 (16K) [text/plain]\n", "Saving to: ‘additional_feedback_data.json’\n", "\n", "100%[======================================>] 16,506 --.-K/s in 0s \n", "\n", "2020-09-22 12:25:58 (46.7 MB/s) - ‘additional_feedback_data.json’ saved [16506/16506]\n", "\n" ] } ], "source": [ "!rm additional_feedback_data.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/additional_feedback_data.json" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "with open('additional_feedback_data.json') as feedback_file:\n", " additional_feedback_data = json.load(feedback_file)\n", "subscription.feedback_logging.store(additional_feedback_data['data'])" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Feedback_14a4e07c-6cae-4b60-828d-40be2661ae73 (binding_id=999, subscription_id=14a4e07c-6cae-4b60-828d-40be2661ae73)

\n", " \n", " \n", " \n", "
CheckingStatusLoanDurationCreditHistoryLoanPurposeLoanAmountExistingSavingsEmploymentDurationInstallmentPercentSexOthersOnLoanCurrentResidenceDurationOwnsPropertyAgeInstallmentPlansHousingExistingCreditsCountJobDependentsTelephoneForeignWorkerRiskrecord_timestamp
less_010all_credits_paid_backcar_new250500_to_10004_to_73malenone2real_estate23nonerent1skilled1noneyesNo Risk2020-09-22 12:26:01.985000+00:00
no_checking23prior_payments_delayedappliances6964100_to_5004_to_74femalenone3car_other39noneown1skilled1noneyesRisk2020-09-22 12:26:01.985000+00:00
0_to_20030outstanding_creditappliances3464100_to_500greater_73maleguarantor4savings_insurance51storesfree1skilled1yesyesRisk2020-09-22 12:26:01.986000+00:00
no_checking23outstanding_creditcar_used2681500_to_1000greater_74malenone3car_other33storesfree1unskilled1yesyesNo Risk2020-09-22 12:26:01.986000+00:00
0_to_20018prior_payments_delayedfurniture1673less_1001_to_42malenone3car_other30noneown2skilled1noneyesRisk2020-09-22 12:26:01.986000+00:00
no_checking44outstanding_creditradio_tv3476unknowngreater_74maleco-applicant4unknown60nonefree2skilled2yesyesRisk2020-09-22 12:26:01.986000+00:00
less_08no_creditseducation803less_100unemployed1malenone1savings_insurance19storesrent1skilled1noneyesNo Risk2020-09-22 12:26:01.986000+00:00
0_to_2007all_credits_paid_backcar_new250less_100unemployed1malenone1real_estate19storesrent1skilled1noneyesNo Risk2020-09-22 12:26:01.986000+00:00
0_to_20033credits_paid_to_dateradio_tv3548100_to_5001_to_43malenone4car_other28noneown2skilled1yesyesRisk2020-09-22 12:26:01.986000+00:00
no_checking24prior_payments_delayedretraining4158100_to_500greater_73femalenone2savings_insurance35storesown1unskilled2noneyesRisk2020-09-22 12:26:01.986000+00:00
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "subscription.feedback_logging.show_table()" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "================================================================================\n", "\n", " Waiting for end of quality monitoring run e9167d60-a8a0-47d7-ba0c-d6b627d5befb \n", "\n", "================================================================================\n", "\n", "\n", "\n", "initializing\n", "completed\n", "\n", "---------------------------\n", " Successfully finished run \n", "---------------------------\n", "\n", "\n" ] } ], "source": [ "run_details = subscription.quality_monitoring.run(background_mode=False)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

QualityMetrics (binding_id=999, subscription_id=14a4e07c-6cae-4b60-828d-40be2661ae73)

\n", " \n", " \n", " \n", "
tsidmeasurement_idvaluelower limitupper limittagsbinding_idsubscription_iddeployment_id
2020-09-22 12:26:08.084000+00:00true_positive_rate10533f52-37e1-4147-bc30-a0496fea40ab0.3333333333333333model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00area_under_roc10533f52-37e1-4147-bc30-a0496fea40ab0.63589743589743590.7model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00precision10533f52-37e1-4147-bc30-a0496fea40ab0.7333333333333333model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00f1_measure10533f52-37e1-4147-bc30-a0496fea40ab0.45833333333333326model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00accuracy10533f52-37e1-4147-bc30-a0496fea40ab0.7346938775510204model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00log_loss10533f52-37e1-4147-bc30-a0496fea40ab0.44464405658838824model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00false_positive_rate10533f52-37e1-4147-bc30-a0496fea40ab0.06153846153846154model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00area_under_pr10533f52-37e1-4147-bc30-a0496fea40ab0.6011337868480725model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:26:08.084000+00:00recall10533f52-37e1-4147-bc30-a0496fea40ab0.3333333333333333model_type: original99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "subscription.quality_monitoring.show_table()" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc8AAAD4CAYAAACKcG2KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5xVVb3/8dcbREhBTNTbXFEHbRRFFGT8gT8xtbxXS01MzAq0G19L5Wp5b/S1elhWV+s+yjK9hl7FypJATYNuaAqipsIMAgMo2RX8Chrij0ZRIRg/3z/OGj0ezsycPcycM8O8n4/Heczea6291mfvGfjMWnvPOYoIzMzMrHS9Kh2AmZlZd+PkaWZmlpGTp5mZWUZOnmZmZhk5eZqZmWW0XaUDsPLYddddo7q6utJhmJl1G/X19S9HxG7F6pw8e4jq6mrq6uoqHYaZWbch6bmW6rxsa2ZmlpGTp5mZWUZOnmZmZhn5nqeZmQGwadMmVq9ezYYNGyodSln169ePwYMH06dPn5KPcfLsIRrWNFI9eValw7BuatXVp1Y6BCuD1atXM2DAAKqrq5FU6XDKIiJ45ZVXWL16NUOGDCn5OC/bmpkZABs2bGDQoEE9JnECSGLQoEGZZ9tOnmZm9q6elDibteecnTy7IUnVkpam7TGSZlY6JjOznsT3PMtIuV9vFBHvVDoWM7O2dPRzEh1977x///6sX7++Q/sslWeenSzNEp+SdAOwEPispMckLZQ0XVL/1O4wSX+StFjSfEkD0rEPp7YLJR1V2bMxMzNw8iyX/YGfAycDnwdOiohDgTrgy5K2B6YB/xoRhwAnAW8DLwEnp7bnAD/JMqikiZLqJNU1vdXYcWdjZtYJvvrVr3LDDTe8u3/llVfyrW99ixNPPJFDDz2U4cOHc88992xx3Ny5cznttNPe3b/44ouZOnUqAPX19Rx//PGMGjWKj33sY7z44osdEquTZ3k8FxGPA0cCBwKPSloEjAf2JpdcX4yIBQAR8XpEbAb6ADdJagCmp2NLFhFTIqI2Imp77zCwA0/HzKzjjRs3jmnTpr27/5vf/Ibzzz+fu+++m4ULFzJnzhy+8pWvEBEl9bdp0yYuueQSZsyYQX19PRdccAFXXHFFh8Tqe57l8Wb6KuD+iDg3v1LSwUCxn4bLgLXAIeR+0elZf7lsZj3KyJEjeemll3jhhRdYt24dH/zgB6mqquKyyy5j3rx59OrVizVr1rB27Vo+9KEPtdnfihUrWLp0KSeffDIATU1NVFVVdUisTp7l9ThwvaQPR8RfJO0ADAaeBv5R0mERsUDSAHLLtgOB1RHxjqTxQO/KhW5m1vnGjh3LjBkz+Otf/8q4ceO4/fbbWbduHfX19fTp04fq6uot/iZzu+2245133nsOs7k+Ihg2bBiPPfZYh8fpZdsyioh1wATg15KWkEumQyPi7+TuaV4naTFwP9APuAEYL+lxYD/em8GamW2Txo0bxx133MGMGTMYO3YsjY2N7L777vTp04c5c+bw3HNbfkrY3nvvzfLly9m4cSONjY088MADAOy///6sW7fu3eS5adMmli1b1iFxeubZySJiFXBQ3v6DwGFF2i0gd0803zPAwXn7XyvsMyLmAnM7LmIzs5xKvC3jsGHDeOONN9hjjz2oqqrivPPO4+Mf/zi1tbWMGDGCoUOHbnHMnnvuyac+9SkOPvhgampqGDlyJADbb789M2bMYNKkSTQ2NrJ582YuvfRShg0bttVxqtQbr9a91dbWhj8M28xa89RTT3HAAQdUOoyKKHbukuojorZYey/bmpmZZeTkaWZmlpGTp5mZvasn3sprzzk7eZqZGZD7UOhXXnmlRyXQ5s/z7NevX6bj/LStmZkBMHjwYFavXs26desqHUpZ9evXj8GDB2c6xsnTzMwA6NOnD0OGDKl0GN2Cl23NzMwycvI0MzPLyMnTzMwsIydPMzOzjJw8zczMMnLyNDMzy8h/qtJDNKxppHryrEqHYWZdQCU+LWVb45mnmZlZRk6eZmZmGTl5dhJJYyTNrHQcZmbW8ba55Cmpd6VjaA9JW3X/ubuet5lZd9Ttkqek30qql7RM0sRUtl7StyU9AYyWNErSQ6ndbElVqd0XJC2QtFjSnZJ2aGWcqZLG5u2vT1/HSJoraYakpyXdLkmp7pRU9gjwybxjd5R0Sxr7SUmnp/IJkqZL+h1wXwtxjJE0T9LdkpZLulFSr2LnvXVX1szMStXtkidwQUSMAmqBSZIGATsCSyPiCOAJ4DpgbGp3C/DddOxdEXFYRBwCPAV8vp0xjAQuBQ4E9gGOltQPuAn4OHAs8KG89lcAD0bEYcAJwA8k7ZjqRgPjI+IjrYx3OPAVYDiwL+8l5nfPOyIeKTxI0kRJdZLqmt5qbOepmplZoe74pyqTJJ2ZtvcEaoAm4M5Utj9wEHB/mhD2Bl5MdQdJ+g6wM9AfmN3OGOZHxGoASYuAamA9sDIinknlvwQmpvYfBT4h6fK03w/YK23fHxGvljDes6nfXwPHADN4/3lvISKmAFMA+lbV9JwP6DMz62TdKnlKGgOcBIyOiLckzSWXiDZERFNzM2BZRBRbxpwKnBERiyVNAMa0Mtxm0sw8Lctun1e3MW+7ifeuY0sJSsBZEbGi4HyOAN5sJYZmhf027+eft5mZlUl3W7YdCLyWEudQ4MgibVYAu0kaDSCpj6RhqW4A8KKkPsB5bYy1ChiVtk8H+rTR/mlgiKR90/65eXWzgUvy7o2ObKOvQodLGpLudZ4DbLFEa2Zm5dPdkucfgO0kLQGuAh4vbBARfwfGAtdIWgwsAo5K1d8gd0/0fnLJrjU3AcdLmg+0OUOMiA3klmlnpQeGnsurvopc8l0iaWnaz+Ix4GpgKbASuDvj8WZm1oEU4VthXVlaqr48Ik7bmn76VtVE1fhrOyYoM+vW/PZ8pZFUHxG1xeq61T1Pa7/hewykzv9gzMw6RI9PnpKuAM4uKJ4eEd8t1r4T4xgO/KKgeGP685u55YzFzMxa1+OTZ0qSZU2ULcTRAIyodBxmZta27vbAkJmZWcU5eZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZOXmamZll5ORpZmaWkZOnmZlZRk6eZmZmGfX4dxjqKRrWNFI9eVanj+M3nDaznsAzTzMzs4ycPM3MzDJy8jQzM8uo05KnpEmSnpJ0ewv1YyTN7Kzx20PShZI+l7YnSPrHvLqbJR1YgZjeF4eZmVVeZz4w9CXgnyJiZSeO0aEi4sa83QnAUuCFVPcvnTWupN4R0dRC9fviMDOzyuuUmaekG4F9gHslfVXSnyQ9mb7uX6T98ZIWpdeTkgak8n+TtEDSEknfamW8aklPS7ottZ0haYdUd2Lqs0HSLZL6pvKrJS1P7f8zlV0p6XJJY4Fa4PYU0wckzZVUK+mLkr6fN/YESdel7c9Imp+O+Zmk3q3EvF7StyU9AYyW9M10rkslTVFOsThGSXpIUr2k2ZKqWhljoqQ6SXVNbzW28h0zM7MsOiV5RsSF5GZKJwD/BRwXESOBbwLfK3LI5cBFETECOBZ4W9JHgRrgcHIfEj1K0nGtDLs/MCUiDgZeB74kqR8wFTgnIoaTm2l/UdIuwJnAsNT+OwXxzwDqgPMiYkREvJ1XPQP4ZN7+OcA0SQek7aPTeTQB57US747A0og4IiIeAX4aEYdFxEHAB4DTCuMANgPXAWMjYhRwC618kHdETImI2oio7b3DwFZCMTOzLMrxwNBAYLqkpcCPgGFF2jwK/FDSJGDniNgMfDS9ngQWAkPJJdOWPB8Rj6btXwLHkEuoKyPiz6n8NuA4csl1A3CzpE8Cb5V6MhGxDnhW0pGSBqUxHgVOBEYBCyQtSvv7tNJVE3Bn3v4Jkp6Q1AB8hOLXaX/gIOD+NMbXgcGlxm5mZh2jHG+ScBUwJyLOlFQNzC1sEBFXS5oF/DPwuKSTAAH/ERE/K3GcKLKvog0jNks6nFyCGwdcTC5hlWoa8CngaeDuiAhJAm6LiK+V2MeG5vucaYZ8A1AbEc9LuhLoV+QYAcsiYnSGWM3MrIOVa+a5Jm1PKNZA0r4R0RAR15BbphwKzAYukNQ/tdlD0u6tjLOXpOakci7wCLnkVi3pw6n8s8BDqc+BEfF74FJyy8KF3gAGtDDWXcAZaZxpqewBYGxzjJJ2kbR3K/Hma06UL6fYxrYQxwpgt+bzlNRHUrEZqpmZdaJyzDy/D9wm6cvAgy20uVTSCeSWMpcD/xMRG9N9xMdykzrWA58BXmqhj6eA8ZJ+BjwD/FdEbJB0Prll4+2ABcCNwC7APWnGJ+CyIv1NBW6U9DbwvpleRLwmaTlwYETMT2XLJX0duE9SL2ATcBHwXBvXh4j4m6SbgAZgVYqzpTjGAj+RNJDc9+9aYFlbY5iZWcdRROFqZ/eTloNnpodtrIja2tqoq6urdBhmZt2GpPqIqC1W53cYMjMzy6hbfapKerr1gSJVJ3bVWWf6O86+BcWfjYiGSsRjZmZbr1slz4h4heIP93RZEXFEpWMwM7OO5WVbMzOzjJw8zczMMnLyNDMzy8jJ08zMLCMnTzMzs4ycPM3MzDJy8jQzM8vIydPMzCyjbvUmCdZ+DWsaqZ48q9JhmFXMqqtPrXQItg3xzNPMzCwjJ08zM7OMnDzNzMwycvLMSNL6Du5vrqSinxdnZmZdk5OnmZlZRk6e7aScH0haKqlB0jmpvJekGyQtkzRT0u8ljS2xz3NTX0slXZPKekuamjfOZal8kqTlkpZIuqOF/iZKqpNU1/RWY0eduplZj+c/VWm/T5L7bNFDgF2BBZLmAUcD1cBwYHfgKeCWtjqT9I/ANcAo4DXgPklnAM8DezR/2LekndMhk4EhEbExr+x9ImIKMAWgb1VNtO80zcyskGee7XcM8OuIaIqItcBDwGGpfHpEvBMRfwXmlNjfYcDciFgXEZuB24HjgGeBfSRdJ+kU4PXUfglwu6TPAJs77rTMzKwtTp7tp4zl7eovIl4jN7udC1wE3JyqTgWuJzdTrZfkVQQzszJx8my/ecA56Z7kbuRmifOBR4Cz0r3PfwDGlNjfE8DxknaV1Bs4F3hI0q5Ar4i4E/gGcKikXsCeETEH+HdgZ6B/R56cmZm1zLOV9rsbGA0sBgL494j4q6Q7gROBpcCfySXFNp/WiYgXJX2N3DKvgN9HxD2SDgFuTQkT4GtAb+CXkgamtj+KiL917OmZmVlLFOHnSDqapP4RsV7SIHKz0aPT/c+Kqa2tjbq6ukqGYGbWrUiqj4iif4fvmWfnmJmegN0euKrSidPMzDqWk2cniIgxhWWS7gaGFBR/NSJmlyUoMzPrME6eZRIRZ1Y6BjMz6xh+2tbMzCwjJ08zM7OMnDzNzMwycvI0MzPLyMnTzMwsIydPMzOzjJw8zczMMnLyNDMzy8hvktBDNKxppHryrEqHYWZWNquuPrXT+vbM08zMLCMnTzMzs4ycPM3MzDJy8qwwSb7vbGbWzbT6H7ekT7ZWHxF3dWw4XYuk3wJ7Av2AH0fEFEmnAN8DegMvR8SJkvoD1wG1QADfiog7Ja2PiP6pr7HAaRExQdJU4FVgJLBQ0jTgWuADwNvA+RGxQlJv4BrgY6nfm4DlwMXNn9Ii6WTgixHR6vfKzMw6Tluzno+nr7sDRwEPpv0TgLnANp08gQsi4lVJHwAWSLqHXAI7LiJWStoltfsG0BgRwwEkfbCEvvcDToqIJkk7pT43SzqJXHI+C5hI7jNAR6a6XYDXgOsl7RYR64DzgVuLDSBpYuqD3jvt1r4rYGZmW2g1eUbE+QCSZgIHRsSLab8KuL7zw6u4SZKaP4dzT3KJaF5ErASIiFdT3UnAuOaDIuK1EvqeHhFNaXsgcJukGnIzzD55/d4YEZvzx5P0C+Azkm4FRgOfKzZAREwBpgD0raqJEmIyM7MSlHq/rbo5cSZryc2ctlmSxpBLXqMj4i1Jc4HFwP7FmpNLeoXyy/oV1L2Zt30VMCcizpRUTW5W31q/twK/AzaQS8KbWzkVMzPrYKU+MDRX0mxJEySNB2YBczoxrq5gIPBaSpxDgSOBvsDxkoYA5C3b3gdc3Hxg3rLtWkkHSOoFnEnLBgJr0vaEvPL7gAubHypqHi8iXgBeAL4OTG3vCZqZWfuUlDwj4mLgZ8AhwAhgSkRc0pmBdQF/ALaTtITczPBxYB25pdu7JC0GpqW23wE+KGlpKj8hlU8GZpK7V5w/cy/0feA/JD1K7kGkZjcD/w9Ykvr9dF7d7cDzEbF8K87RzMzaQRG+FdYdSfop8GRE/Hcp7ftW1UTV+Gs7OSozs65ja9+eT1J9RNQWq2vrT1UeiYhjJL3B+++9CYiI2GmrIrN2kVRP7p7pV0o9ZvgeA6nrxPd5NDPrSdp62vaY9HVAecKxUkTEqErHYGbWk/kdhszMzDJy8jQzM8vIydPMzCwjJ08zM7OMnDzNzMwycvI0MzPLyMnTzMwsIydPMzOzjJw8zczMMnLyNDMzy6jUz/O0bq5hTSPVk2dVOgyzLmVr3zjcei7PPM3MzDJy8jQzM8vIydPMzCyjHpE8JU2S9JSkOyU9JmmjpMsrHZeZmXVPPeWBoS8B/0TuA6T3Bs6obDjtI2m7iNhc6TjMzHq6bX7mKelGYB/gXuC8iFgAbCrhuGpJT0u6WdJSSbdLOknSo5KekXR4arejpFskLZD0pKTT845/WNLC9DoqlVdJmidpUer32FS+Pm/ssZKmpu2pkn4oaQ5wTUvjtXAOEyXVSaprequxvZfQzMwKbPMzz4i4UNIpwAkR8XLGwz8MnA1MBBYAnwaOAT4B/F9yM9grgAcj4gJJOwPzJf0ReAk4OSI2SKoBfg3Upj5mR8R3JfUGdighjv2AkyKiSdL3io0XEW8WOfcpwBSAvlU1kfHczcysBdt88txKKyOiAUDSMuCBiAhJDUB1avNR4BN591D7AXsBLwA/lTQCaCKXACGXhG+R1Af4bUQsKiGO6RHR1MZ4T7X3JM3MLBsnz9ZtzNt+J2//Hd67dgLOiogV+QdKuhJYCxxCbnl8A0BEzJN0HHAq8AtJP4iInwP5M8N+BXHkzyqLjmdmZuWzzd/zLIPZwCWSBCBpZCofCLwYEe8AnwV6p/q9gZci4ibgv4FDU/u1kg6Q1As4sx3jmZlZmfSomaekDwF1wE7AO5IuBQ6MiNe3oturgGuBJSmhrQJOA24A7pR0NjCH92aPY4B/k7QJWA98LpVPBmYCzwNLgf4ZxzMzszJRhJ8j6Qlqa2ujrq6u0mGYmXUbkuojorZYnZdtzczMMupRy7bFSBoEPFCk6sSIeKXc8ZiZWdfX45NnSpAjKh2HmZl1H162NTMzy8jJ08zMLCMnTzMzs4ycPM3MzDJy8jQzM8vIydPMzCwjJ08zM7OMnDzNzMwy6vFvktBTNKxppHryrEqHYWZWNquuPrXT+vbM08zMLCMnTzMzs4ycPMtM0p/aqP+9pJ3LFY+ZmWXne55bQVLviGjKckxEHNVG/T9vXVRmZtbZPPNsgaRqSU9Luk3SEkkzJO0gaZWkb0p6BDhb0r6S/iCpXtLDkoam4/9B0t2SFqfXUal8ffpaJWmepEWSlko6NpWvkrRr2v5yqlsq6dK8uJ6SdJOkZZLuk/SBilwkM7MeysmzdfsDUyLiYOB14EupfENEHBMRdwBTgEsiYhRwOXBDavMT4KGIOAQ4FFhW0PengdkRMQI4BFiUXylpFHA+cARwJPAFSSNTdQ1wfUQMA/4GnFUseEkTJdVJqmt6q7F9V8DMzLbgZdvWPR8Rj6btXwKT0vY0AEn9gaOA6ZKaj+mbvn4E+BxAWtotzF4LgFsk9QF+GxGLCuqPAe6OiDfTWHcBxwL3Aivz2tcD1cWCj4gp5JI7fatqorRTNjOztnjm2brChNO8/2b62gv4W0SMyHsdUFLHEfOA44A1wC8kfa6gibY86l0b87ab8C9BZmZl5eTZur0kjU7b5wKP5FdGxOvASklnAyjnkFT9APDFVN5b0k75x0raG3gpIm4C/pvc0m6+ecAZ6T7rjsCZwMMdd2pmZtZeTp6tewoYL2kJsAvwX0XanAd8XtJicvc1T0/l/wqcIKmB3NLqsILjxgCLJD1J7p7lj/MrI2IhMBWYDzwB3BwRT3bAOZmZ2VZShG+FFSOpGpgZEQdVOJQO0beqJqrGX1vpMMzMymZr355PUn1E1Bar872yHmL4HgOp68T3eTQz60mcPFsQEauAbWLWaWZmHcv3PM3MzDJy8jQzM8vIydPMzCwjJ08zM7OMnDzNzMwycvI0MzPLyMnTzMwsIydPMzOzjJw8zczMMnLyNDMzy8jJ08zMLCO/t20P0bCmkerJsyodhpltA7b200q2BZ55mpmZZeTkaWZmlpGTZweQNEbSzErHYWZm5dGtk6ek3pWOoT0ktXmvubuem5lZT9Clk6ek30qql7RM0sRUtl7StyU9AYyWNErSQ6ndbElVqd0XJC2QtFjSnZJ2aGWcqZLG5u2vT1/HSJoraYakpyXdLkmp7pRU9gjwybxjd5R0Sxr7SUmnp/IJkqZL+h1wXwtxjJE0R9KvgIZU9mVJS9Pr0ry2n5O0JJ3fL9p7jc3MLLuu/rTtBRHxqqQPAAsk3QnsCCyNiG9K6gM8BJweEesknQN8F7gAuCsibgKQ9B3g88B17YhhJDAMeAF4FDhaUh1wE/AR4C/AtLz2VwAPRsQFknYG5kv6Y6obDRwcEa+2Mt7hwEERsVLSKOB84AhAwBOSHgL+nsY5OiJelrRLsY7SLxwTAXrvtFs7Tt3MzIrp6slzkqQz0/aeQA3QBNyZyvYHDgLuTxPC3sCLqe6glDR3BvoDs9sZw/yIWA0gaRFQDawHVkbEM6n8l6QkBXwU+ISky9N+P2CvtH1/G4mzebyVafsY4O6IeDONcxdwLBDAjIh4GaClPiNiCjAFoG9VTZR8xmZm1qoumzwljQFOAkZHxFuS5pJLRBsioqm5GbAsIkYX6WIqcEZELJY0ARjTynCbSUvYaVl2+7y6jXnbTbx3zVpKRgLOiogVBedzBPBmKzE0y2+jVsZwMjQzq5CufM9zIPBaSpxDgSOLtFkB7CZpNICkPpKGpboBwItpafe8NsZaBYxK26cDfdpo/zQwRNK+af/cvLrZwCV590ZHttFXa+YBZ0jaQdKOwJnAw8ADwKckDUpjFF22NTOzztGVk+cfgO0kLQGuAh4vbBARfwfGAtdIWgwsAo5K1d8AngDuJ5fsWnMTcLyk+eTuL7Y6Q4yIDeSWaWelB4aey6u+ilzyXSJpadpvl4hYSG4GPZ/cudwcEU9GxDJy93YfSuf9w/aOYWZm2SnCq389Qd+qmqgaf22lwzCzbUBPeXs+SfURUVusrsve87SONXyPgdT1kB94M7PO1qOSp6QrgLMLiqdHxHfLHMdwoPBvMzdGxBHljMPMzNqnRyXPlCTLmihbiKMBGFHpOMzMrH268gNDZmZmXZKTp5mZWUZOnmZmZhk5eZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZOXmamZll5ORpZmaWUY96h6GerGFNI9WTZ1U6DOumesobgZuVyjNPMzOzjJw8zczMMipL8pS0s6QvlWOsrSXpT+lrtaRP55XXSvpJBeJ5XxxmZlZ55Zp57gxskTwl9S7T+CWLiKPSZjXw6bzyuoiY1BljSmrt3vP74jAzs8orV/K8GthX0iJJCyTNkfQroCHNrJY2N5R0uaQr0/a+kv4gqV7Sw5KGtjSApKmSbkzt/izptFTeT9KtkhokPSnphFQ+TNL8FNMSSTWpfH1ezMem+sskjZE0U1IvSask7Zw39l8k/YOk3STdmc5xgaSjW4n3SklTJN0H/Dxdh4clLUyv5iReGEdvST9I/S+R9H+yfzvMzGxrlOtp28nAQRExQtIYYFbaXympupXjpgAXRsQzko4AbgA+0kr7auB4YF9gjqQPAxcBRMTwlHzvk7QfcCHw44i4XdL2QOEseDJweUQ0J+ExqZ93JN0DnAncmuJaFRFr0y8EP4qIRyTtBcwGDmgl3lHAMRHxtqQdgJMjYkNK5L8GaovEMRFojIjDJPUFHpV0X0SsLOw8tZ0I0Hun3VoJw8zMsqjUn6rML/affT5J/YGjgOmSmov7ttHvbyLiHeAZSc8CQ4FjgOsAIuJpSc8B+wGPAVdIGgzcFRHPZIh/GvBN4FZgXNoHOAk4MC/enSQNiIg3Wujn3oh4O233AX4qaQTQlGIs5qPAwZLGpv2BQA2wxfWMiCnkfgGhb1VNlHhuZmbWhkolzzfztjfz/uXjfulrL+BvETEiQ7+FCSIAFW0Y8StJTwCnArMl/UtEPFjiOI8BH5a0G3AG8J28mEfnJcS25F+Hy4C1wCGpnw0tHCPgkoiYXeIYZmbWwcp1z/MNYEALdWuB3SUNSsuQpwFExOvASklnAyjnkDbGOTvdk9wX2AdYAcwDzkt97AfsBayQtA/wbET8BLgXOLjUmCMigLuBHwJPRcQrqeo+4OLmdmkWWaqBwItp5vxZ3ltGLoxjNvBFSX2az0nSjhnGMTOzrVSW5JmSy6PpwaAfFNRtAr4NPAHMBJ7Oqz4P+LykxcAy4PQ2hloBPAT8D7l7pRvI3SftLamB3PLqhIjYCJwDLJW0iNzy7s8L+loCbJa0WNJlRcaaBnyG95ZsASYBtelBnuXk7quW6gZgvKTHyS3ZNs9KC+O4GVgOLEzX82f4naLMzMpKuUlU9ydpKjAzImZUOpauqG9VTVSNv7bSYVg35bfns55IUn1E1Bar84ylhxi+x0Dq/B+gmVmH6HbJU9IVwNkFxdMjYkIFwmmTpPOBfy0ofjQiLqpEPGZmtvW2mWVba11tbW3U1dVVOgwzs26jtWVbvzG8mZlZRk6eZmZmGTl5mpmZZeR7nj2EpDfI/R1sd7Ar8HKlgyhRd4oVule83SlW6F7xOtbS7B0RRd8YvNs9bWvttqKlG99djaQ6x9o5ulO83SlW6F7xOtat52VbMzOzjJw8zczMMnLy7DmmVDqADBxr5+lO8XanWKF7xetYt5IfGDIzM8vIM08zM7OMnDzNzMwycvLchkg6RVM+cj0AAAO6SURBVNIKSX+RNLlIfV9J01L9E5Kqyx/l++JpK97jJC2UtFnS2ErEmBdLW7F+WdLy9FmuD0jauxJx5sXTVrwXSmqQtEjSI5IOrEScKZZWY81rN1ZSSKrYny2UcF0nSFqXrusiSf9SiTjz4mnz2kr6VPrZXSbpV+WOMS+Otq7tj/Ku658l/a0Scb4rIvzaBl5Ab+B/gX2A7YHFwIEFbb4E3Ji2xwHTuni81cDB5D6ofGwXj/UEYIe0/cVucG13ytv+BPCHrhprajcAmAc8DtR21ViBCcBPK/W9b0e8NcCTwAfT/u5dNdaC9pcAt1Ty+nrmue04HPhLRDwbEX8H7gBOL2hzOnBb2p4BnChJZYwxX5vxRsSqiFgCvFOJAPOUEuuciHgr7T4ODC5zjPlKiff1vN0dgUo9OVjKzy3AVcD3gQ3lDK5AqbF2FaXE+wXg+oh4DSAiXipzjM2yXttzgV+XJbIWOHluO/YAns/bX53KiraJiM1AIzCoLNFtqZR4u4qssX4e+J9Ojah1JcUr6SJJ/0suKU0qU2yF2oxV0khgz4iYWc7Aiij15+CstHw/Q9Ke5QmtqFLi3Q/YT9Kjkh6XdErZonu/kv+NpVsiQ4AHyxBXi5w8tx3FZpCFs4lS2pRLV4qlLSXHKukzQC3wg06NqHUlxRsR10fEvsBXga93elTFtRqrpF7Aj4CvlC2ilpVyXX8HVEfEwcAfeW+lpxJKiXc7cku3Y8jN5m6WtHMnx1VMlv8PxgEzIqKpE+Npk5PntmM1kP9b7mDghZbaSNoOGAi8WpbotlRKvF1FSbFKOgm4AvhERGwsU2zFZL22dwBndGpELWsr1gHAQcBcSauAI4F7K/TQUJvXNSJeyfve3wSMKlNsxZT6f8I9EbEpIlaS+/CImjLFVxhHqT+z46jwki3gB4a2lRe53yCfJbec0XzDfVhBm4t4/wNDv+nK8ea1nUplHxgq5dqOJPfAQ003+Vmoydv+OFDXVWMtaD+Xyj0wVMp1rcrbPhN4vIv/HJwC3Ja2dyW3dDqoK8aa2u0PrCK9wU8lXxUd3K8O/mbCPwN/Tv+JX5HKvk1uJgTQD5gO/AWYD+zTxeM9jNxvpG8CrwDLunCsfwTWAovS694ufm1/DCxLsc5pLWFVOtaCthVLniVe1/9I13Vxuq5Du/jPgYAfAsuBBmBcV4017V8JXF3Ja9r88tvzmZmZZeR7nmZmZhk5eZqZmWXk5GlmZpaRk6eZmVlGTp5mZmYZOXmamZll5ORpZmaW0f8H+lGn55lMsHoAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "\n", "quality_pd = subscription.quality_monitoring.get_table_content(format='pandas')\n", "quality_pd.plot.barh(x='id', y='value');" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'deployment_metrics': [{'asset': {'asset_id': 'd9f69282-4c9a-4e2f-8293-8589695853da',\n", " 'asset_type': 'model',\n", " 'created_at': '2020-09-22T12:22:26.002Z',\n", " 'name': 'Scottda-telco-9-22-2020_MODEL',\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/models/d9f69282-4c9a-4e2f-8293-8589695853da?space_id=0bd92418-0f8d-4a9d-908c-cecbd49a2adc'},\n", " 'deployment': {'created_at': '2020-09-22T12:22:36.910Z',\n", " 'deployment_id': 'e85d6845-4b6b-44bf-a483-80a522852620',\n", " 'deployment_rn': '',\n", " 'deployment_type': 'online',\n", " 'name': 'Scottda-telco-9-22-2020_DEPLOYMENT',\n", " 'scoring_endpoint': {'request_headers': {'Content-Type': 'application/json'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/e85d6845-4b6b-44bf-a483-80a522852620/predictions'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/e85d6845-4b6b-44bf-a483-80a522852620'},\n", " 'metrics': [{'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-22T12:26:08.084Z',\n", " 'value': {'metrics': [{'name': 'true_positive_rate',\n", " 'value': 0.3333333333333333},\n", " {'name': 'area_under_roc', 'value': 0.6358974358974359},\n", " {'name': 'precision', 'value': 0.7333333333333333},\n", " {'name': 'f1_measure', 'value': 0.45833333333333326},\n", " {'name': 'accuracy', 'value': 0.7346938775510204},\n", " {'name': 'log_loss', 'value': 0.44464405658838824},\n", " {'name': 'false_positive_rate', 'value': 0.06153846153846154},\n", " {'name': 'area_under_pr', 'value': 0.6011337868480725},\n", " {'name': 'recall', 'value': 0.3333333333333333}],\n", " 'quality': 0.6358974358974359,\n", " 'threshold': 0.7}}],\n", " 'subscription': {'subscription_id': '14a4e07c-6cae-4b60-828d-40be2661ae73',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/service_bindings/999/subscriptions/14a4e07c-6cae-4b60-828d-40be2661ae73'}},\n", " {'asset': {'asset_id': 'f9a9d722-e00e-4f83-9bb7-f606f0863675',\n", " 'asset_type': 'model',\n", " 'created_at': '2020-09-10T03:57:40.002Z',\n", " 'name': 'ChurnSparkModelV1',\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/models/f9a9d722-e00e-4f83-9bb7-f606f0863675?space_id=f277dfae-8d22-477e-92a7-7e5f3bdf7898'},\n", " 'deployment': {'created_at': '2020-09-10T15:47:29.831Z',\n", " 'deployment_id': 'c0b02323-14ea-4bd7-8b7a-99bb2fce4fd6',\n", " 'deployment_rn': '',\n", " 'deployment_type': 'online',\n", " 'name': 'SparkModelOnlineDeployment',\n", " 'scoring_endpoint': {'request_headers': {'Content-Type': 'application/json'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/c0b02323-14ea-4bd7-8b7a-99bb2fce4fd6/predictions'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/c0b02323-14ea-4bd7-8b7a-99bb2fce4fd6'},\n", " 'metrics': [{'issues': 0,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-22T01:12:24Z',\n", " 'value': {'metrics': [{'name': 'auroc', 'threshold': 0.7, 'value': 0.76}],\n", " 'quality': 0.76,\n", " 'threshold': 0.7}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-22T01:12:56Z',\n", " 'value': {'records': 469, 'response_time': 167.9765650647152}},\n", " {'issues': 0,\n", " 'metric_type': 'debiased_fairness',\n", " 'timestamp': '2020-09-22T12:06:12.104437Z',\n", " 'value': {'copy_reason': 'No new data added since last fairness computation.',\n", " 'corrected_records': 0,\n", " 'debiased': True,\n", " 'evaluated_at': '2020-09-22T02:06:17.572409Z',\n", " 'is_copied': True,\n", " 'manual_labelling_store': 'Manual_Labeling_58b43991-540e-4179-a867-445089f58c33',\n", " 'metrics': [{'bias_source': {'values': []},\n", " 'fairness_threshold': 0.95,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 78.94736842105263,\n", " 'total_rows_percent': 48.717948717948715,\n", " 'values': [{'distribution': {'[26, 28]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 30]': [{'count': 17,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 32]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 34]': [{'count': 18,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[34, 36]': [{'count': 14,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 38]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[38, 40]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 42]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 44]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[44, 46]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 48]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 50]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 54]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[54, 56]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[56, 58]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[58, 60]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 62]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[70, 72]': [{'count': 17,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[72, 74]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[74, 75]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 28]': [26, 28],\n", " '[28, 30]': [28, 30],\n", " '[30, 32]': [30, 32],\n", " '[32, 34]': [32, 34],\n", " '[34, 36]': [34, 36],\n", " '[36, 38]': [36, 38],\n", " '[38, 40]': [38, 40],\n", " '[40, 42]': [40, 42],\n", " '[42, 44]': [42, 44],\n", " '[44, 46]': [44, 46],\n", " '[46, 48]': [46, 48],\n", " '[48, 50]': [48, 50],\n", " '[50, 52]': [50, 52],\n", " '[52, 54]': [52, 54],\n", " '[54, 56]': [54, 56],\n", " '[56, 58]': [56, 58],\n", " '[58, 60]': [58, 60],\n", " '[60, 62]': [60, 62],\n", " '[70, 72]': [70, 72],\n", " '[72, 74]': [72, 74],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 78.94736842105263,\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 84.21052631578947,\n", " 'total_rows_percent': 48.717948717948715,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 55,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 22]': [{'count': 39,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 10, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 24]': [{'count': 47,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 8, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 22]': [20, 22],\n", " '[22, 24]': [22, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.067,\n", " 'fav_class_percent': 84.21052631578947,\n", " 'is_biased': False,\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-22T01:13:33.475Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-22T01:10:28.895Z',\n", " 'perturbed_records': 190}},\n", " {'fairness_threshold': 0.95,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 80.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 160,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 40, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 80.0,\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 156,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 44, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.975,\n", " 'fav_class_percent': 78.0,\n", " 'is_biased': False,\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-22T01:13:33.475Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-22T01:10:28.895Z',\n", " 'perturbed_records': 200}}],\n", " 'perturbed_data_size': 390,\n", " 'response_time': '2.832253',\n", " 'rows_analyzed': 200,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 0,\n", " 'metric_type': 'fairness',\n", " 'timestamp': '2020-09-22T12:06:12.104437Z',\n", " 'value': {'copy_reason': 'No new data added since last fairness computation.',\n", " 'evaluated_at': '2020-09-22T02:06:17.572409Z',\n", " 'is_copied': True,\n", " 'manual_labelling_store': 'Manual_Labeling_58b43991-540e-4179-a867-445089f58c33',\n", " 'metrics': [{'fairness_threshold': 0.95,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 80.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 112,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 29, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 80.0,\n", " 'payload_perturb_distribution': {'male': [{'count': 160,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 40, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 51,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 8, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.975,\n", " 'fav_class_percent': 78.0,\n", " 'is_biased': False,\n", " 'payload_perturb_distribution': {'female': [{'count': 156,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 44, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-22T01:13:33.475Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-22T01:10:28.895Z',\n", " 'perturbed_records': 200}},\n", " {'bias_source': {'values': []},\n", " 'fairness_threshold': 0.95,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 78.94736842105263,\n", " 'total_rows_percent': 48.717948717948715,\n", " 'values': [{'distribution': {'[26, 28]': [{'count': 14,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 30]': [{'count': 18,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 32]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 34]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 36]': [{'count': 14,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 38]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 40]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 42]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[42, 44]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[44, 46]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 48]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 50]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 8, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 54]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 56]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[56, 58]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[58, 60]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 28]': [26, 28],\n", " '[28, 30]': [28, 30],\n", " '[30, 32]': [30, 32],\n", " '[32, 34]': [32, 34],\n", " '[34, 36]': [34, 36],\n", " '[36, 38]': [36, 38],\n", " '[38, 40]': [38, 40],\n", " '[40, 42]': [40, 42],\n", " '[42, 44]': [42, 44],\n", " '[44, 46]': [44, 46],\n", " '[46, 48]': [46, 48],\n", " '[48, 50]': [48, 50],\n", " '[50, 52]': [50, 52],\n", " '[52, 54]': [52, 54],\n", " '[54, 56]': [54, 56],\n", " '[56, 58]': [56, 58],\n", " '[58, 60]': [58, 60],\n", " '[60, 62]': [60, 62],\n", " '[70, 72]': [70, 72],\n", " '[72, 74]': [72, 74],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 78.94736842105263,\n", " 'payload_perturb_distribution': {'[26, 28]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 30]': [{'count': 17,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 32]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 34]': [{'count': 18,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[34, 36]': [{'count': 14,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 38]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[38, 40]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 42]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 44]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[44, 46]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 48]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 50]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 54]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[54, 56]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[56, 58]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[58, 60]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 62]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[70, 72]': [{'count': 17,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[72, 74]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[74, 75]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 84.21052631578947,\n", " 'total_rows_percent': 48.717948717948715,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[19, 20]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[20, 22]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[22, 24]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 22]': [20, 22],\n", " '[22, 24]': [22, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.067,\n", " 'fav_class_percent': 84.21052631578947,\n", " 'is_biased': False,\n", " 'payload_perturb_distribution': {'[18, 19]': [{'count': 55,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 22]': [{'count': 39,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 10, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 24]': [{'count': 47,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 8, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-22T01:13:33.475Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-22T01:10:28.895Z',\n", " 'perturbed_records': 190}}],\n", " 'perturbed_data_size': 390,\n", " 'response_time': '2.510657',\n", " 'rows_analyzed': 200,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-22T12:21:40.824Z',\n", " 'value': {'metrics': [{'name': 'true_positive_rate',\n", " 'value': 0.3333333333333333},\n", " {'name': 'area_under_roc', 'value': 0.6358974358974359},\n", " {'name': 'precision', 'value': 0.7333333333333333},\n", " {'name': 'f1_measure', 'value': 0.45833333333333326},\n", " {'name': 'accuracy', 'value': 0.7346938775510204},\n", " {'name': 'log_loss', 'value': 0.43616529408913673},\n", " {'name': 'false_positive_rate', 'value': 0.06153846153846154},\n", " {'name': 'area_under_pr', 'value': 0.6011337868480725},\n", " {'name': 'recall', 'value': 0.3333333333333333}],\n", " 'quality': 0.6358974358974359,\n", " 'threshold': 0.7}},\n", " {'issues': 0,\n", " 'metric_type': 'drift',\n", " 'timestamp': '2020-09-22T10:13:16.606310Z',\n", " 'value': {'metrics': [{'name': 'data_drift_magnitude', 'value': 0.0},\n", " {'threshold': 0.05, 'name': 'drift_magnitude', 'value': 0.0},\n", " {'name': 'predicted_accuracy', 'value': 0.8510000000000001}]}}],\n", " 'subscription': {'subscription_id': '58b43991-540e-4179-a867-445089f58c33',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/service_bindings/999/subscriptions/58b43991-540e-4179-a867-445089f58c33'}},\n", " {'asset': {'asset_id': '81a2aa32-8c5e-4e09-a5a2-944a16dda2e8',\n", " 'asset_type': 'model',\n", " 'created_at': '2020-09-18T03:14:13.002Z',\n", " 'name': 'JRT_WOSFullConfig_MODEL',\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/models/81a2aa32-8c5e-4e09-a5a2-944a16dda2e8?space_id=f277dfae-8d22-477e-92a7-7e5f3bdf7898'},\n", " 'deployment': {'created_at': '2020-09-18T03:14:31.455Z',\n", " 'deployment_id': 'bd7d3067-c143-4b1f-9f21-6a476405a9d7',\n", " 'deployment_rn': '',\n", " 'deployment_type': 'online',\n", " 'name': 'JRT_WOSFullConfig_DEPLOYMENT',\n", " 'scoring_endpoint': {'request_headers': {'Content-Type': 'application/json'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/bd7d3067-c143-4b1f-9f21-6a476405a9d7/predictions'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/bd7d3067-c143-4b1f-9f21-6a476405a9d7'},\n", " 'metrics': [{'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-18T02:40:43Z',\n", " 'value': {'records': 518, 'response_time': 83.89451673735343}},\n", " {'issues': 0,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-18T02:40:14Z',\n", " 'value': {'metrics': [{'name': 'auroc', 'threshold': 0.7, 'value': 0.76}],\n", " 'quality': 0.76,\n", " 'threshold': 0.7}},\n", " {'issues': 1,\n", " 'metric_type': 'fairness',\n", " 'timestamp': '2020-09-22T12:22:45.329927Z',\n", " 'value': {'copy_reason': 'No new data added since last fairness computation.',\n", " 'evaluated_at': '2020-09-18T03:23:47.153047Z',\n", " 'is_copied': True,\n", " 'manual_labelling_store': 'Manual_Labeling_e70d218c-e77a-442d-99e1-2b712caf84a4',\n", " 'metrics': [{'fairness_threshold': 0.95,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 73.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 88,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 51, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 73.0,\n", " 'payload_perturb_distribution': {'male': [{'count': 146,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 54, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 69.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 56,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.945,\n", " 'fav_class_percent': 69.0,\n", " 'is_biased': True,\n", " 'payload_perturb_distribution': {'female': [{'count': 138,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 62, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'perturbed_records': 200}},\n", " {'bias_source': {'values': []},\n", " 'fairness_threshold': 0.95,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 71.5,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 28]': [{'count': 19,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 30]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[30, 32]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 34]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 36]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[36, 38]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[38, 40]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 42]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[42, 44]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[44, 46]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 48]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 50]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 6, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 11, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 54]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[54, 56]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[56, 58]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[58, 60]': [{'count': 5,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 62]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[62, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 28]': [26, 28],\n", " '[28, 30]': [28, 30],\n", " '[30, 32]': [30, 32],\n", " '[32, 34]': [32, 34],\n", " '[34, 36]': [34, 36],\n", " '[36, 38]': [36, 38],\n", " '[38, 40]': [38, 40],\n", " '[40, 42]': [40, 42],\n", " '[42, 44]': [42, 44],\n", " '[44, 46]': [44, 46],\n", " '[46, 48]': [46, 48],\n", " '[48, 50]': [48, 50],\n", " '[50, 52]': [50, 52],\n", " '[52, 54]': [52, 54],\n", " '[54, 56]': [54, 56],\n", " '[56, 58]': [56, 58],\n", " '[58, 60]': [58, 60],\n", " '[60, 62]': [60, 62],\n", " '[62, 64]': [62, 64],\n", " '[68, 70]': [68, 70],\n", " '[70, 72]': [70, 72],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 71.5,\n", " 'payload_perturb_distribution': {'[26, 28]': [{'count': 19,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 30]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[30, 32]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 34]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 36]': [{'count': 19,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[36, 38]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[38, 40]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 42]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[42, 44]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[44, 46]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 48]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 50]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 6, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 11, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 54]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[54, 56]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[56, 58]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[58, 60]': [{'count': 5,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 62]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[62, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[68, 70]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[70, 72]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[74, 75]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[19, 20]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[20, 22]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[22, 24]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 22]': [20, 22],\n", " '[22, 24]': [22, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.091,\n", " 'fav_class_percent': 78.0,\n", " 'is_biased': False,\n", " 'payload_perturb_distribution': {'[18, 19]': [{'count': 60,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 13, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 22]': [{'count': 28,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 17, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 24]': [{'count': 41,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 9, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'perturbed_records': 200}}],\n", " 'perturbed_data_size': 400,\n", " 'response_time': '1.589982',\n", " 'rows_analyzed': 200,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 0,\n", " 'metric_type': 'debiased_fairness',\n", " 'timestamp': '2020-09-22T12:22:45.329927Z',\n", " 'value': {'copy_reason': 'No new data added since last fairness computation.',\n", " 'corrected_records': 3,\n", " 'debiased': True,\n", " 'evaluated_at': '2020-09-18T03:24:04.324214Z',\n", " 'is_copied': True,\n", " 'manual_labelling_store': 'Manual_Labeling_e70d218c-e77a-442d-99e1-2b712caf84a4',\n", " 'metrics': [{'bias_source': {'values': []},\n", " 'fairness_threshold': 0.95,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 73.5,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 28]': [{'count': 19,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 30]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[30, 32]': [{'count': 13,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 34]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 36]': [{'count': 20,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 38]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[38, 40]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 42]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 44]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[44, 46]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 48]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 50]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 6, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 11, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 54]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[54, 56]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[56, 58]': [{'count': 4,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[58, 60]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '[60, 62]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[62, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[68, 70]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[70, 72]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[74, 75]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[26, 28]': [26, 28],\n", " '[28, 30]': [28, 30],\n", " '[30, 32]': [30, 32],\n", " '[32, 34]': [32, 34],\n", " '[34, 36]': [34, 36],\n", " '[36, 38]': [36, 38],\n", " '[38, 40]': [38, 40],\n", " '[40, 42]': [40, 42],\n", " '[42, 44]': [42, 44],\n", " '[44, 46]': [44, 46],\n", " '[46, 48]': [46, 48],\n", " '[48, 50]': [48, 50],\n", " '[50, 52]': [50, 52],\n", " '[52, 54]': [52, 54],\n", " '[54, 56]': [54, 56],\n", " '[56, 58]': [56, 58],\n", " '[58, 60]': [58, 60],\n", " '[60, 62]': [60, 62],\n", " '[62, 64]': [62, 64],\n", " '[68, 70]': [68, 70],\n", " '[70, 72]': [70, 72],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 73.5,\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 73.5,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 56,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 15, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 22]': [{'count': 27,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 21, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 24]': [{'count': 39,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 10, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 22]': [20, 22],\n", " '[22, 24]': [22, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.0,\n", " 'fav_class_percent': 73.5,\n", " 'is_biased': False,\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'perturbed_records': 200}},\n", " {'fairness_threshold': 0.95,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 73.5,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 147,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 53, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 73.5,\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 74.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 148,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 52, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 1.007,\n", " 'fav_class_percent': 74.0,\n", " 'is_biased': False,\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'number_of_records': 200,\n", " 'oldest_timestamp': '2020-09-18T03:22:57.810Z',\n", " 'perturbed_records': 200}}],\n", " 'perturbed_data_size': 400,\n", " 'response_time': '1.891173',\n", " 'rows_analyzed': 200,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 2,\n", " 'metric_type': 'fairness',\n", " 'timestamp': '2020-09-18T02:39:31Z',\n", " 'value': {'manual_labelling_store': 'Manual_Labeling_3842b84b-0f31-452b-8733-d8b7d85041b9',\n", " 'metrics': [{'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 76.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 48,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 11, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 76.0,\n", " 'payload_perturb_distribution': {'male': [{'count': 76,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 24, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 72.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 29,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 12, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.947,\n", " 'fav_class_percent': 72.0,\n", " 'is_biased': True,\n", " 'payload_perturb_distribution': {'female': [{'count': 72,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 28, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2019-07-27 15:35:46.003000',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2019-07-27 15:35:46.003000',\n", " 'perturbed_records': 100}},\n", " {'bias_source': {'values': [{'fav_percent': 64.0, 'range': '[22,23]'},\n", " {'fav_percent': 93.10344827586206, 'range': '[26,34]'},\n", " {'fav_percent': 100.0, 'range': '[71,75]'}]},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 78.15126050420169,\n", " 'total_rows_percent': 39.666666666666664,\n", " 'values': [{'distribution': {'26': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '27': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '28': [{'count': 5, 'is_favourable': True, 'label': 'No Risk'}],\n", " '29': [{'count': 5, 'is_favourable': True, 'label': 'No Risk'}],\n", " '30': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '31': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '32': [{'count': 4, 'is_favourable': True, 'label': 'No Risk'}],\n", " '33': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '34': [{'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '35': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '36': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '37': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '38': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '39': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '40': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '41': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '42': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '43': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '44': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '45': [{'count': 4, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '46': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '47': [{'count': 4, 'is_favourable': True, 'label': 'No Risk'}],\n", " '48': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '49': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '53': [{'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '54': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '55': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '63': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 78.15126050420169,\n", " 'payload_perturb_distribution': {'26': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '27': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '28': [{'count': 5, 'is_favourable': True, 'label': 'No Risk'}],\n", " '29': [{'count': 4, 'is_favourable': True, 'label': 'No Risk'}],\n", " '30': [{'count': 4, 'is_favourable': True, 'label': 'No Risk'}],\n", " '31': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '32': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '33': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '34': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '35': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '36': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '37': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '38': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '39': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '40': [{'count': 6, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '41': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '42': [{'count': 1, 'is_favourable': True, 'label': 'No Risk'}],\n", " '43': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '44': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '45': [{'count': 6, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '46': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '47': [{'count': 10, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '48': [{'count': 4, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '49': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '53': [{'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '54': [{'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '55': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '56': [{'count': 5, 'is_favourable': True, 'label': 'No Risk'}],\n", " '57': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '59': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '64': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '68': [{'count': 6, 'is_favourable': True, 'label': 'No Risk'}],\n", " '71': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '73': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '74': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}]},\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 74.58563535911603,\n", " 'total_rows_percent': 60.333333333333336,\n", " 'values': [{'distribution': {'19': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '20': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '21': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '22': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'}],\n", " '24': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}],\n", " '25': [{'count': 2, 'is_favourable': True, 'label': 'No Risk'}]},\n", " 'fairness_value': 0.954,\n", " 'fav_class_percent': 74.58563535911603,\n", " 'is_biased': True,\n", " 'payload_perturb_distribution': {'18': [{'count': 30,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 8, 'is_favourable': False, 'label': 'Risk'}],\n", " '19': [{'count': 15, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '20': [{'count': 19, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 4, 'is_favourable': False, 'label': 'Risk'}],\n", " '21': [{'count': 18, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 12, 'is_favourable': False, 'label': 'Risk'}],\n", " '22': [{'count': 3, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '23': [{'count': 13, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '24': [{'count': 28, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 9, 'is_favourable': False, 'label': 'Risk'}],\n", " '25': [{'count': 9, 'is_favourable': True, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2019-07-27 15:35:46.003000',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2019-07-27 15:35:46.003000',\n", " 'perturbed_records': 200}}],\n", " 'perturbed_data_size': 300,\n", " 'response_time': '25939.003757',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'desperate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'debiased_fairness',\n", " 'timestamp': '2020-09-18T02:39:55Z',\n", " 'value': {'debiased': True,\n", " 'manual_labelling_store': 'Manual_Labeling_dd79dd1b-0afc-436e-9999-6fd6414f81c2',\n", " 'metrics': [{'bias_source': {'values': [{'fav_percent': 88.23529411764706,\n", " 'range': '[18,19]'}]},\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 96.17834394904459,\n", " 'total_rows_percent': 78.5,\n", " 'values': [{'distribution': {'26': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'No Risk'}],\n", " '28': [{'count': 5, 'is_favourable': False, 'label': 'No Risk'}],\n", " '29': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}],\n", " '30': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '31': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '32': [{'count': 4, 'is_favourable': False, 'label': 'No Risk'}],\n", " '34': [{'count': 4, 'is_favourable': False, 'label': 'No Risk'}],\n", " '35': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '36': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '37': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '39': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}],\n", " '40': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}],\n", " '41': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}],\n", " '42': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '43': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '44': [{'count': 3, 'is_favourable': False, 'label': 'No Risk'}],\n", " '45': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}],\n", " '48': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}],\n", " '49': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '52': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '59': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '60': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '66': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '67': [{'count': 1, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 96.17834394904459,\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 93.7062937062937,\n", " 'total_rows_percent': 71.5,\n", " 'values': [{'distribution': {'19': [{'count': 17,\n", " 'is_favourable': False,\n", " 'label': 'No Risk'}],\n", " '20': [{'count': 18, 'is_favourable': False, 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '21': [{'count': 13, 'is_favourable': False, 'label': 'No Risk'}],\n", " '22': [{'count': 4, 'is_favourable': False, 'label': 'No Risk'}],\n", " '23': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '24': [{'count': 1, 'is_favourable': False, 'label': 'No Risk'}],\n", " '25': [{'count': 2, 'is_favourable': False, 'label': 'No Risk'}]},\n", " 'fairness_value': 0.974,\n", " 'fav_class_percent': 93.7062937062937,\n", " 'is_biased': True,\n", " 'value': [18, 25]}]}},\n", " {'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 95.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 5,\n", " 'is_favourable': False,\n", " 'label': 'Risk'},\n", " {'count': 56, 'is_favourable': False, 'label': 'No Risk'}]},\n", " 'fav_class_percent': 95.0,\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 95.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 39,\n", " 'is_favourable': False,\n", " 'label': 'No Risk'}]},\n", " 'fairness_value': 1.0,\n", " 'fav_class_percent': 95.0,\n", " 'is_biased': False,\n", " 'value': 'female'}]}}],\n", " 'perturbed_data_size': 100,\n", " 'response_time': '237.744371',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'desperate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-22T12:21:19.436Z',\n", " 'value': {'metrics': [{'name': 'true_positive_rate',\n", " 'value': 0.36363636363636365},\n", " {'name': 'area_under_roc', 'value': 0.6587412587412588},\n", " {'name': 'precision', 'value': 0.8},\n", " {'name': 'f1_measure', 'value': 0.5000000000000001},\n", " {'name': 'accuracy', 'value': 0.7551020408163265},\n", " {'name': 'log_loss', 'value': 0.43113848847106984},\n", " {'name': 'false_positive_rate', 'value': 0.046153846153846156},\n", " {'name': 'area_under_pr', 'value': 0.6525974025974026},\n", " {'name': 'recall', 'value': 0.36363636363636365}],\n", " 'quality': 0.6587412587412588,\n", " 'threshold': 0.7}}],\n", " 'subscription': {'subscription_id': 'e70d218c-e77a-442d-99e1-2b712caf84a4',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/service_bindings/999/subscriptions/e70d218c-e77a-442d-99e1-2b712caf84a4'}},\n", " {'asset': {'asset_id': '2121433b-bec6-4ded-ba8d-4fd34d4e24d2',\n", " 'asset_type': 'model',\n", " 'created_at': '2020-09-17T17:54:02.002Z',\n", " 'name': 'GermanCreditRiskModelICP',\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/models/2121433b-bec6-4ded-ba8d-4fd34d4e24d2?space_id=c3f7e11f-c9f5-4abc-92d1-0a6641fc9b00'},\n", " 'deployment': {'created_at': '2020-09-17T17:54:05.733Z',\n", " 'deployment_id': '1d7355a8-35ba-4036-b895-91b961c07def',\n", " 'deployment_rn': '',\n", " 'deployment_type': 'online',\n", " 'name': 'GermanCreditRiskModelICP',\n", " 'scoring_endpoint': {'request_headers': {'Content-Type': 'application/json'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/1d7355a8-35ba-4036-b895-91b961c07def/predictions'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/1d7355a8-35ba-4036-b895-91b961c07def'},\n", " 'metrics': [{'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-17T16:56:51Z',\n", " 'value': {'metrics': [{'name': 'auroc', 'threshold': 0.8, 'value': 0.77}],\n", " 'quality': 0.77,\n", " 'threshold': 0.8}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 238.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 246.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 255.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 228.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 250.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 307.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 278.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 325.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 242.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 244.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 343.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 235.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 249.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 233.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 250.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 246.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 260.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 246.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 250.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 233.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 253.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 262.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 244.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 250.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 252.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 237.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 258.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 241.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 256.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 235.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 259.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 233.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 252.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 237.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 254.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 234.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 273.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 238.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 251.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 234.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 248.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 237.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 256.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 243.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 250.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 245.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 255.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 226.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 257.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 234.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 250.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 233.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 246.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 239.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 256.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 241.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 249.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 251.0}},\n", " {'issues': 0,\n", " 'metric_type': 'performance',\n", " 'timestamp': '2020-09-17T16:54:58.724Z',\n", " 'value': {'records': 1, 'response_time': 253.0}},\n", " {'issues': 0,\n", " 'metric_type': 'debiased_fairness',\n", " 'timestamp': '2020-09-22T11:54:48.957185Z',\n", " 'value': {'copy_reason': 'No new data added since last fairness computation.',\n", " 'corrected_records': 2,\n", " 'debiased': True,\n", " 'evaluated_at': '2020-09-17T18:04:56.744935Z',\n", " 'is_copied': True,\n", " 'manual_labelling_store': 'Manual_Labeling_4181e595-765c-485b-b3b7-170159194c49',\n", " 'metrics': [{'bias_source': {'values': []},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 27]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[27, 28]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[35, 36]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 37]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[37, 38]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 39]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[40, 41]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[41, 42]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 43]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[43, 44]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 49]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[49, 50]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 51]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[55, 56]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[57, 58]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[59, 60]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[63, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[69, 70]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[70, 71]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[74, 75]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[26, 27]': [26, 27],\n", " '[27, 28]': [27, 28],\n", " '[28, 29]': [28, 29],\n", " '[29, 30]': [29, 30],\n", " '[30, 31]': [30, 31],\n", " '[31, 32]': [31, 32],\n", " '[32, 33]': [32, 33],\n", " '[33, 34]': [33, 34],\n", " '[35, 36]': [35, 36],\n", " '[36, 37]': [36, 37],\n", " '[37, 38]': [37, 38],\n", " '[38, 39]': [38, 39],\n", " '[39, 40]': [39, 40],\n", " '[40, 41]': [40, 41],\n", " '[41, 42]': [41, 42],\n", " '[42, 43]': [42, 43],\n", " '[43, 44]': [43, 44],\n", " '[44, 45]': [44, 45],\n", " '[46, 47]': [46, 47],\n", " '[47, 48]': [47, 48],\n", " '[48, 49]': [48, 49],\n", " '[49, 50]': [49, 50],\n", " '[50, 51]': [50, 51],\n", " '[53, 54]': [53, 54],\n", " '[55, 56]': [55, 56],\n", " '[57, 58]': [57, 58],\n", " '[59, 60]': [59, 60],\n", " '[63, 64]': [63, 64],\n", " '[69, 70]': [69, 70],\n", " '[70, 71]': [70, 71],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 78.0,\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 82.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 28,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 21]': [{'count': 10,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[21, 22]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 23]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[23, 24]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 21]': [20, 21],\n", " '[21, 22]': [21, 22],\n", " '[22, 23]': [22, 23],\n", " '[23, 24]': [23, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.051,\n", " 'fav_class_percent': 82.0,\n", " 'is_biased': False,\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T18:00:30.509Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T18:00:05.173Z',\n", " 'perturbed_records': 100}},\n", " {'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 78,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 22, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 78.0,\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 79.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 79,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 21, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 1.013,\n", " 'fav_class_percent': 79.0,\n", " 'is_biased': False,\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T18:00:30.509Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T18:00:05.173Z',\n", " 'perturbed_records': 100}}],\n", " 'perturbed_data_size': 200,\n", " 'response_time': '2.732065',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'fairness',\n", " 'timestamp': '2020-09-22T11:54:48.957185Z',\n", " 'value': {'copy_reason': 'No new data added since last fairness computation.',\n", " 'evaluated_at': '2020-09-17T18:03:20.166312Z',\n", " 'is_copied': True,\n", " 'manual_labelling_store': 'Manual_Labeling_4181e595-765c-485b-b3b7-170159194c49',\n", " 'metrics': [{'bias_source': {'values': []},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 76.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 27]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[27, 28]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[35, 36]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 37]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[37, 38]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 39]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[40, 41]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[41, 42]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 43]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[43, 44]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 49]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[49, 50]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 51]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[55, 56]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[57, 58]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[59, 60]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[63, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 27]': [26, 27],\n", " '[27, 28]': [27, 28],\n", " '[28, 29]': [28, 29],\n", " '[29, 30]': [29, 30],\n", " '[30, 31]': [30, 31],\n", " '[31, 32]': [31, 32],\n", " '[32, 33]': [32, 33],\n", " '[33, 34]': [33, 34],\n", " '[35, 36]': [35, 36],\n", " '[36, 37]': [36, 37],\n", " '[37, 38]': [37, 38],\n", " '[38, 39]': [38, 39],\n", " '[39, 40]': [39, 40],\n", " '[40, 41]': [40, 41],\n", " '[41, 42]': [41, 42],\n", " '[42, 43]': [42, 43],\n", " '[43, 44]': [43, 44],\n", " '[44, 45]': [44, 45],\n", " '[46, 47]': [46, 47],\n", " '[47, 48]': [47, 48],\n", " '[48, 49]': [48, 49],\n", " '[49, 50]': [49, 50],\n", " '[50, 51]': [50, 51],\n", " '[53, 54]': [53, 54],\n", " '[55, 56]': [55, 56],\n", " '[57, 58]': [57, 58],\n", " '[59, 60]': [59, 60],\n", " '[63, 64]': [63, 64],\n", " '[69, 70]': [69, 70],\n", " '[70, 71]': [70, 71],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 76.0,\n", " 'payload_perturb_distribution': {'[26, 27]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[27, 28]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[35, 36]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 37]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[37, 38]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 39]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[40, 41]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[41, 42]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 43]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[43, 44]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 49]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[49, 50]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 51]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[55, 56]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[57, 58]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[59, 60]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[63, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[69, 70]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[70, 71]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[74, 75]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 82.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 7,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[20, 21]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[22, 23]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[23, 24]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 21]': [20, 21],\n", " '[21, 22]': [21, 22],\n", " '[22, 23]': [22, 23],\n", " '[23, 24]': [23, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.079,\n", " 'fav_class_percent': 82.0,\n", " 'is_biased': False,\n", " 'payload_perturb_distribution': {'[18, 19]': [{'count': 29,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[20, 21]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}],\n", " '[21, 22]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 23]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[23, 24]': [{'count': 15,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 12,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T18:00:30.509Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T18:00:05.173Z',\n", " 'perturbed_records': 100}},\n", " {'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 42,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 15, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 77.0,\n", " 'payload_perturb_distribution': {'male': [{'count': 77,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 23, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 74.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 34,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 9, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.961,\n", " 'fav_class_percent': 74.0,\n", " 'is_biased': True,\n", " 'payload_perturb_distribution': {'female': [{'count': 74,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 26, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T18:00:30.509Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T18:00:05.173Z',\n", " 'perturbed_records': 100}}],\n", " 'perturbed_data_size': 200,\n", " 'response_time': '2.383844',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-22T11:54:53.828Z',\n", " 'value': {'metrics': [{'name': 'true_positive_rate',\n", " 'value': 0.5294117647058824},\n", " {'name': 'area_under_roc', 'value': 0.7344028520499108},\n", " {'name': 'precision', 'value': 0.8181818181818182},\n", " {'name': 'f1_measure', 'value': 0.6428571428571428},\n", " {'name': 'accuracy', 'value': 0.8},\n", " {'name': 'log_loss', 'value': 0.47297854731642575},\n", " {'name': 'false_positive_rate', 'value': 0.06060606060606061},\n", " {'name': 'area_under_pr', 'value': 0.705668449197861},\n", " {'name': 'recall', 'value': 0.5294117647058824}],\n", " 'quality': 0.7344028520499108,\n", " 'threshold': 0.95}},\n", " {'issues': 1,\n", " 'metric_type': 'drift',\n", " 'timestamp': '2020-09-22T11:56:58.226963Z',\n", " 'value': {'metrics': [{'name': 'data_drift_magnitude', 'value': 0.07},\n", " {'threshold': 0.05,\n", " 'name': 'drift_magnitude',\n", " 'value': 0.052000000000000046},\n", " {'name': 'predicted_accuracy', 'value': 0.748}]}}],\n", " 'subscription': {'subscription_id': '4181e595-765c-485b-b3b7-170159194c49',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/service_bindings/999/subscriptions/4181e595-765c-485b-b3b7-170159194c49'}},\n", " {'asset': {'asset_id': '3732c133-bdaa-4fb1-996e-e89118ae23c0',\n", " 'asset_type': 'model',\n", " 'created_at': '2020-09-17T17:53:18.002Z',\n", " 'name': 'GermanCreditRiskModelPreProdICP',\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/models/3732c133-bdaa-4fb1-996e-e89118ae23c0?space_id=dfcf5e8d-51e3-4c5c-99f5-34cfc205dd08'},\n", " 'deployment': {'created_at': '2020-09-17T17:53:21.747Z',\n", " 'deployment_id': '4cb29e49-375c-43bc-952e-ae50b71fb46a',\n", " 'deployment_rn': '',\n", " 'deployment_type': 'online',\n", " 'name': 'GermanCreditRiskModelPreProdICP',\n", " 'scoring_endpoint': {'request_headers': {'Content-Type': 'application/json'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/4cb29e49-375c-43bc-952e-ae50b71fb46a/predictions'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/4cb29e49-375c-43bc-952e-ae50b71fb46a'},\n", " 'metrics': [{'issues': 1,\n", " 'metric_type': 'debiased_fairness',\n", " 'timestamp': '2020-09-17T17:54:44.330429Z',\n", " 'value': {'corrected_records': 0,\n", " 'debiased': True,\n", " 'evaluated_at': '2020-09-17T17:55:00.606623Z',\n", " 'manual_labelling_store': 'Manual_Labeling_e19daf91-8069-4a70-954b-4ca1521883c5',\n", " 'metrics': [{'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 78,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 22, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 78.0,\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 76.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 76,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 24, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.974,\n", " 'fav_class_percent': 76.0,\n", " 'is_biased': True,\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'perturbed_records': 100}},\n", " {'bias_source': {'values': []},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 27]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[27, 28]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 35]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[35, 36]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[36, 37]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[37, 38]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 39]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 41]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[41, 42]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[42, 43]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[43, 44]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[45, 46]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[49, 50]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 51]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[51, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 53]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 55]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[58, 59]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 61]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[69, 70]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[72, 73]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[74, 75]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 27]': [26, 27],\n", " '[27, 28]': [27, 28],\n", " '[28, 29]': [28, 29],\n", " '[29, 30]': [29, 30],\n", " '[30, 31]': [30, 31],\n", " '[31, 32]': [31, 32],\n", " '[32, 33]': [32, 33],\n", " '[33, 34]': [33, 34],\n", " '[34, 35]': [34, 35],\n", " '[35, 36]': [35, 36],\n", " '[36, 37]': [36, 37],\n", " '[37, 38]': [37, 38],\n", " '[38, 39]': [38, 39],\n", " '[39, 40]': [39, 40],\n", " '[40, 41]': [40, 41],\n", " '[41, 42]': [41, 42],\n", " '[42, 43]': [42, 43],\n", " '[43, 44]': [43, 44],\n", " '[44, 45]': [44, 45],\n", " '[45, 46]': [45, 46],\n", " '[46, 47]': [46, 47],\n", " '[47, 48]': [47, 48],\n", " '[49, 50]': [49, 50],\n", " '[50, 51]': [50, 51],\n", " '[51, 52]': [51, 52],\n", " '[52, 53]': [52, 53],\n", " '[53, 54]': [53, 54],\n", " '[54, 55]': [54, 55],\n", " '[58, 59]': [58, 59],\n", " '[60, 61]': [60, 61],\n", " '[69, 70]': [69, 70],\n", " '[72, 73]': [72, 73],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 77.0,\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 83.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 35,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 21]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 6, 'is_favourable': False, 'label': 'Risk'}],\n", " '[21, 22]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 23]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[23, 24]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 21]': [20, 21],\n", " '[21, 22]': [21, 22],\n", " '[22, 23]': [22, 23],\n", " '[23, 24]': [23, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.078,\n", " 'fav_class_percent': 83.0,\n", " 'is_biased': False,\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'perturbed_records': 100}}],\n", " 'perturbed_data_size': 200,\n", " 'response_time': '25.786984',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'fairness',\n", " 'timestamp': '2020-09-17T17:54:44.330429Z',\n", " 'value': {'evaluated_at': '2020-09-17T17:55:00.606623Z',\n", " 'manual_labelling_store': 'Manual_Labeling_e19daf91-8069-4a70-954b-4ca1521883c5',\n", " 'metrics': [{'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 78.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 46,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 19, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 78.0,\n", " 'payload_perturb_distribution': {'male': [{'count': 78,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 22, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 76.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 32,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.974,\n", " 'fav_class_percent': 76.0,\n", " 'is_biased': True,\n", " 'payload_perturb_distribution': {'female': [{'count': 76,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 24, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'perturbed_records': 100}},\n", " {'bias_source': {'values': []},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 27]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[27, 28]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 35]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[35, 36]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[36, 37]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[37, 38]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 39]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 41]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[41, 42]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[42, 43]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[43, 44]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[45, 46]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[49, 50]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[50, 51]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[51, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 53]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 55]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[58, 59]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 61]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 27]': [26, 27],\n", " '[27, 28]': [27, 28],\n", " '[28, 29]': [28, 29],\n", " '[29, 30]': [29, 30],\n", " '[30, 31]': [30, 31],\n", " '[31, 32]': [31, 32],\n", " '[32, 33]': [32, 33],\n", " '[33, 34]': [33, 34],\n", " '[34, 35]': [34, 35],\n", " '[35, 36]': [35, 36],\n", " '[36, 37]': [36, 37],\n", " '[37, 38]': [37, 38],\n", " '[38, 39]': [38, 39],\n", " '[39, 40]': [39, 40],\n", " '[40, 41]': [40, 41],\n", " '[41, 42]': [41, 42],\n", " '[42, 43]': [42, 43],\n", " '[43, 44]': [43, 44],\n", " '[44, 45]': [44, 45],\n", " '[45, 46]': [45, 46],\n", " '[46, 47]': [46, 47],\n", " '[47, 48]': [47, 48],\n", " '[49, 50]': [49, 50],\n", " '[50, 51]': [50, 51],\n", " '[51, 52]': [51, 52],\n", " '[52, 53]': [52, 53],\n", " '[53, 54]': [53, 54],\n", " '[54, 55]': [54, 55],\n", " '[58, 59]': [58, 59],\n", " '[60, 61]': [60, 61],\n", " '[69, 70]': [69, 70],\n", " '[72, 73]': [72, 73],\n", " '[74, 75]': [74, 75]},\n", " 'fav_class_percent': 77.0,\n", " 'payload_perturb_distribution': {'[26, 27]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[27, 28]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 35]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[35, 36]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[36, 37]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[37, 38]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[38, 39]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 41]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[41, 42]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[42, 43]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[43, 44]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[45, 46]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[49, 50]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[50, 51]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[51, 52]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[52, 53]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 3,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 55]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[58, 59]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[60, 61]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[69, 70]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[72, 73]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[74, 75]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 83.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[19, 20]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[20, 21]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[21, 22]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[22, 23]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 21]': [20, 21],\n", " '[21, 22]': [21, 22],\n", " '[22, 23]': [22, 23],\n", " '[23, 24]': [23, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.078,\n", " 'fav_class_percent': 83.0,\n", " 'is_biased': False,\n", " 'payload_perturb_distribution': {'[18, 19]': [{'count': 35,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 21]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 6, 'is_favourable': False, 'label': 'Risk'}],\n", " '[21, 22]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 23]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[23, 24]': [{'count': 8,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 11,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:54:18.007Z',\n", " 'perturbed_records': 100}}],\n", " 'perturbed_data_size': 200,\n", " 'response_time': '25.786984',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-17T17:54:35.645Z',\n", " 'value': {'metrics': [{'name': 'true_positive_rate',\n", " 'value': 0.5294117647058824},\n", " {'name': 'area_under_roc', 'value': 0.7344028520499108},\n", " {'name': 'precision', 'value': 0.8181818181818182},\n", " {'name': 'f1_measure', 'value': 0.6428571428571428},\n", " {'name': 'accuracy', 'value': 0.8},\n", " {'name': 'log_loss', 'value': 0.47297854731642586},\n", " {'name': 'false_positive_rate', 'value': 0.06060606060606061},\n", " {'name': 'area_under_pr', 'value': 0.705668449197861},\n", " {'name': 'recall', 'value': 0.5294117647058824}],\n", " 'quality': 0.7344028520499108,\n", " 'threshold': 0.95}},\n", " {'issues': 0,\n", " 'metric_type': 'drift',\n", " 'timestamp': '2020-09-17T17:54:41.073419Z',\n", " 'value': {'metrics': [{'name': 'data_drift_magnitude',\n", " 'value': 0.09900990099009901},\n", " {'threshold': 0.05,\n", " 'name': 'drift_magnitude',\n", " 'value': 0.03922772277227726},\n", " {'name': 'predicted_accuracy', 'value': 0.7607722772277228}]}}],\n", " 'subscription': {'subscription_id': 'e19daf91-8069-4a70-954b-4ca1521883c5',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/service_bindings/998/subscriptions/e19daf91-8069-4a70-954b-4ca1521883c5'}},\n", " {'asset': {'asset_id': '8bf25c2f-1ffc-499a-9eaf-48a02c9dcb53',\n", " 'asset_type': 'model',\n", " 'created_at': '2020-09-17T17:47:32.002Z',\n", " 'name': 'GermanCreditRiskModelChallengerICP',\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/models/8bf25c2f-1ffc-499a-9eaf-48a02c9dcb53?space_id=dfcf5e8d-51e3-4c5c-99f5-34cfc205dd08'},\n", " 'deployment': {'created_at': '2020-09-17T17:47:36.154Z',\n", " 'deployment_id': 'e1984ba6-d0e3-4941-9a7e-b3efa6c8d98e',\n", " 'deployment_rn': '',\n", " 'deployment_type': 'online',\n", " 'name': 'GermanCreditRiskModelChallengerICP',\n", " 'scoring_endpoint': {'request_headers': {'Content-Type': 'application/json'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/e1984ba6-d0e3-4941-9a7e-b3efa6c8d98e/predictions'},\n", " 'url': 'https://ibm-nginx-svc.zen2.svc.cluster.local/v4/deployments/e1984ba6-d0e3-4941-9a7e-b3efa6c8d98e'},\n", " 'metrics': [{'issues': 1,\n", " 'metric_type': 'debiased_fairness',\n", " 'timestamp': '2020-09-17T17:53:58.232687Z',\n", " 'value': {'corrected_records': 0,\n", " 'debiased': True,\n", " 'evaluated_at': '2020-09-17T17:54:12.407679Z',\n", " 'manual_labelling_store': 'Manual_Labeling_20cd4e73-6007-41be-8d20-4fffc029eddb',\n", " 'metrics': [{'bias_source': {'values': []},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 27]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[27, 28]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 35]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[35, 36]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 37]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[37, 38]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[38, 39]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 41]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[41, 42]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 43]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[43, 44]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[45, 46]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 49]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[49, 50]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[51, 52]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[52, 53]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 55]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[63, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[69, 70]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[70, 71]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[72, 73]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[26, 27]': [26, 27],\n", " '[27, 28]': [27, 28],\n", " '[28, 29]': [28, 29],\n", " '[29, 30]': [29, 30],\n", " '[30, 31]': [30, 31],\n", " '[31, 32]': [31, 32],\n", " '[32, 33]': [32, 33],\n", " '[33, 34]': [33, 34],\n", " '[34, 35]': [34, 35],\n", " '[35, 36]': [35, 36],\n", " '[36, 37]': [36, 37],\n", " '[37, 38]': [37, 38],\n", " '[38, 39]': [38, 39],\n", " '[39, 40]': [39, 40],\n", " '[40, 41]': [40, 41],\n", " '[41, 42]': [41, 42],\n", " '[42, 43]': [42, 43],\n", " '[43, 44]': [43, 44],\n", " '[44, 45]': [44, 45],\n", " '[45, 46]': [45, 46],\n", " '[46, 47]': [46, 47],\n", " '[47, 48]': [47, 48],\n", " '[48, 49]': [48, 49],\n", " '[49, 50]': [49, 50],\n", " '[51, 52]': [51, 52],\n", " '[52, 53]': [52, 53],\n", " '[53, 54]': [53, 54],\n", " '[54, 55]': [54, 55],\n", " '[63, 64]': [63, 64],\n", " '[69, 70]': [69, 70],\n", " '[70, 71]': [70, 71],\n", " '[72, 73]': [72, 73]},\n", " 'fav_class_percent': 77.0,\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 29,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 21]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[21, 22]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 23]': [{'count': 16,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[23, 24]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 21]': [20, 21],\n", " '[21, 22]': [21, 22],\n", " '[22, 23]': [22, 23],\n", " '[23, 24]': [23, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.0,\n", " 'fav_class_percent': 77.0,\n", " 'is_biased': False,\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'perturbed_records': 100}},\n", " {'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 79.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 79,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 21, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 79.0,\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 68.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 68,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 32, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.861,\n", " 'fav_class_percent': 68.0,\n", " 'is_biased': True,\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'perturbed_records': 100}}],\n", " 'perturbed_data_size': 200,\n", " 'response_time': '18.99171',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'fairness',\n", " 'timestamp': '2020-09-17T17:53:58.232687Z',\n", " 'value': {'evaluated_at': '2020-09-17T17:54:12.407679Z',\n", " 'manual_labelling_store': 'Manual_Labeling_20cd4e73-6007-41be-8d20-4fffc029eddb',\n", " 'metrics': [{'bias_source': {'values': []},\n", " 'fairness_threshold': 0.98,\n", " 'feature': 'Age',\n", " 'majority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[26, 27]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[27, 28]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 35]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[35, 36]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 37]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[37, 38]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[38, 39]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 41]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[41, 42]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 43]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[43, 44]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[45, 46]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 49]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[49, 50]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[51, 52]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[52, 53]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 55]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[63, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}]},\n", " 'distribution_ranges': {'[26, 27]': [26, 27],\n", " '[27, 28]': [27, 28],\n", " '[28, 29]': [28, 29],\n", " '[29, 30]': [29, 30],\n", " '[30, 31]': [30, 31],\n", " '[31, 32]': [31, 32],\n", " '[32, 33]': [32, 33],\n", " '[33, 34]': [33, 34],\n", " '[34, 35]': [34, 35],\n", " '[35, 36]': [35, 36],\n", " '[36, 37]': [36, 37],\n", " '[37, 38]': [37, 38],\n", " '[38, 39]': [38, 39],\n", " '[39, 40]': [39, 40],\n", " '[40, 41]': [40, 41],\n", " '[41, 42]': [41, 42],\n", " '[42, 43]': [42, 43],\n", " '[43, 44]': [43, 44],\n", " '[44, 45]': [44, 45],\n", " '[45, 46]': [45, 46],\n", " '[46, 47]': [46, 47],\n", " '[47, 48]': [47, 48],\n", " '[48, 49]': [48, 49],\n", " '[49, 50]': [49, 50],\n", " '[51, 52]': [51, 52],\n", " '[52, 53]': [52, 53],\n", " '[53, 54]': [53, 54],\n", " '[54, 55]': [54, 55],\n", " '[63, 64]': [63, 64],\n", " '[69, 70]': [69, 70],\n", " '[70, 71]': [70, 71],\n", " '[72, 73]': [72, 73]},\n", " 'fav_class_percent': 77.0,\n", " 'payload_perturb_distribution': {'[26, 27]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[27, 28]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[28, 29]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[29, 30]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[30, 31]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[31, 32]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[32, 33]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[33, 34]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[34, 35]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[35, 36]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[36, 37]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[37, 38]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[38, 39]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[39, 40]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[40, 41]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[41, 42]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[42, 43]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[43, 44]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[44, 45]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[45, 46]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[46, 47]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[47, 48]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[48, 49]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[49, 50]': [{'count': 2,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[51, 52]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[52, 53]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[53, 54]': [{'count': 2,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[54, 55]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[63, 64]': [{'count': 1,\n", " 'is_favourable': False,\n", " 'label': 'Risk'}],\n", " '[69, 70]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[70, 71]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[72, 73]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'value': [26, 75]}]},\n", " 'minority': {'total_fav_percent': 77.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'[18, 19]': [{'count': 6,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[19, 20]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[20, 21]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[21, 22]': [{'count': 3,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[23, 24]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}],\n", " '[24, 25]': [{'count': 1,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'}]},\n", " 'distribution_ranges': {'[18, 19]': [18, 19],\n", " '[19, 20]': [19, 20],\n", " '[20, 21]': [20, 21],\n", " '[21, 22]': [21, 22],\n", " '[22, 23]': [22, 23],\n", " '[23, 24]': [23, 24],\n", " '[24, 25]': [24, 25]},\n", " 'fairness_value': 1.0,\n", " 'fav_class_percent': 77.0,\n", " 'is_biased': False,\n", " 'payload_perturb_distribution': {'[18, 19]': [{'count': 29,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 7, 'is_favourable': False, 'label': 'Risk'}],\n", " '[19, 20]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 1, 'is_favourable': False, 'label': 'Risk'}],\n", " '[20, 21]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[21, 22]': [{'count': 5,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[22, 23]': [{'count': 16,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 2, 'is_favourable': False, 'label': 'Risk'}],\n", " '[23, 24]': [{'count': 4,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 3, 'is_favourable': False, 'label': 'Risk'}],\n", " '[24, 25]': [{'count': 9,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 5, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': [18, 25]}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'perturbed_records': 100}},\n", " {'fairness_threshold': 0.98,\n", " 'feature': 'Sex',\n", " 'majority': {'total_fav_percent': 79.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'male': [{'count': 43,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 17, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fav_class_percent': 79.0,\n", " 'payload_perturb_distribution': {'male': [{'count': 79,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 21, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'male'}]},\n", " 'minority': {'total_fav_percent': 68.0,\n", " 'total_rows_percent': 50.0,\n", " 'values': [{'distribution': {'female': [{'count': 34,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 6, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'fairness_value': 0.861,\n", " 'fav_class_percent': 68.0,\n", " 'is_biased': True,\n", " 'payload_perturb_distribution': {'female': [{'count': 68,\n", " 'is_favourable': True,\n", " 'label': 'No Risk'},\n", " {'count': 32, 'is_favourable': False, 'label': 'Risk'}]},\n", " 'value': 'female'}]},\n", " 'records_evaluated': {'additional_records': 0,\n", " 'newest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'number_of_records': 100,\n", " 'oldest_timestamp': '2020-09-17T17:53:36.275Z',\n", " 'perturbed_records': 100}}],\n", " 'perturbed_data_size': 200,\n", " 'response_time': '18.99171',\n", " 'rows_analyzed': 100,\n", " 'score_type': 'disparate impact'}},\n", " {'issues': 1,\n", " 'metric_type': 'quality',\n", " 'timestamp': '2020-09-17T17:53:53.537Z',\n", " 'value': {'metrics': [{'name': 'true_positive_rate',\n", " 'value': 0.5714285714285714},\n", " {'name': 'area_under_roc', 'value': 0.7626373626373625},\n", " {'name': 'precision', 'value': 0.8695652173913043},\n", " {'name': 'f1_measure', 'value': 0.689655172413793},\n", " {'name': 'accuracy', 'value': 0.82},\n", " {'name': 'log_loss', 'value': 0.42901577602965224},\n", " {'name': 'false_positive_rate', 'value': 0.046153846153846156},\n", " {'name': 'area_under_pr', 'value': 0.7582298136645962},\n", " {'name': 'recall', 'value': 0.5714285714285714}],\n", " 'quality': 0.7626373626373625,\n", " 'threshold': 0.95}}],\n", " 'subscription': {'subscription_id': '20cd4e73-6007-41be-8d20-4fffc029eddb',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/service_bindings/998/subscriptions/20cd4e73-6007-41be-8d20-4fffc029eddb'}}]}" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ai_client.data_mart.get_deployment_metrics()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5.0 Fairness, drift monitoring and explanations \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n", " * Which model feature to monitor\n", " * One or more **majority** groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\n", " * One or more **minority** groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\n", " * The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 95%)\n", "\n", "Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 200 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.1 Enable fairness monitoring" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "scrolled": true }, "outputs": [], "source": [ "subscription.fairness_monitoring.enable(\n", " features=[\n", " Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.95),\n", " Feature(\"Age\", majority=[[26,75]], minority=[[18,25]], threshold=0.95)\n", " ],\n", " favourable_classes=['No Risk'],\n", " unfavourable_classes=['Risk'],\n", " min_records=200,\n", " training_data=pd_data\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.2 Score the model again now that monitoring is configured" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This next section randomly selects 200 records from the data feed and sends those records to the model for predictions. This is enough to exceed the minimum threshold for records set in the previous section, which allows OpenScale to begin calculating fairness." ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘german_credit_feed.json’: No such file or directory\n", "--2020-09-22 12:26:55-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/german_credit_feed.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 3076279 (2.9M) [text/plain]\n", "Saving to: ‘german_credit_feed.json’\n", "\n", "100%[======================================>] 3,076,279 --.-K/s in 0.04s \n", "\n", "2020-09-22 12:26:55 (66.6 MB/s) - ‘german_credit_feed.json’ saved [3076279/3076279]\n", "\n" ] } ], "source": [ "!rm german_credit_feed.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/german_credit_feed.json\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Score 200 randomly chosen records" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": [ "import random\n", "\n", "with open('german_credit_feed.json', 'r') as scoring_file:\n", " scoring_data = json.load(scoring_file)\n", "\n", "fields = scoring_data['fields']\n", "values = []\n", "for _ in range(200):\n", " values.append(random.choice(scoring_data['values']))\n", "payload_scoring = {\"fields\": fields, \"values\": values}\n", "payload = {\n", " wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n", "}\n", "scoring_response = wml_client.deployments.score(deployment_uid, payload)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5.3 Run fairness monitor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Kick off a fairness monitor run on current data. The monitor runs hourly, but can be manually initiated using the Python client, the REST API, or the graphical user interface. We have a 30 second sleep so that the scoring of 200 payloads above can complete." ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "=======================================================================\n", "\n", " Counting bias for deployment_uid=e85d6845-4b6b-44bf-a483-80a522852620 \n", "\n", "=======================================================================\n", "\n", "\n", "\n", "RUNNING....\n", "FINISHED\n", "\n", "---------------------------\n", " Successfully finished run \n", "---------------------------\n", "\n", "\n" ] } ], "source": [ "time.sleep(30)\n", "\n", "run_details = subscription.fairness_monitoring.run(background_mode=False)" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

FairnessMetrics (binding_id=999, subscription_id=14a4e07c-6cae-4b60-828d-40be2661ae73)

\n", " \n", " \n", " \n", "
tsfeaturefeature_valuefairness_biasedfairness_valuefairness_fav_classbinding_idsubscription_idasset_revisiondeployment_idprocess
2020-09-22 12:27:35.998269+00:00SexfemaleTrue0.91273.099914a4e07c-6cae-4b60-828d-40be2661ae7314a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:27:35.998269+00:00Age[18, 25]False1.07784.099914a4e07c-6cae-4b60-828d-40be2661ae7314a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "time.sleep(5)\n", "\n", "subscription.fairness_monitoring.show_table()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5.4 Configure Explainability" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we provide OpenScale with the training data to enable and configure the explainability features." ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import *\n", "subscription.explainability.enable(training_data=pd_data)" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "scrolled": true }, "outputs": [], "source": [ "explainability_details = subscription.explainability.get_details()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5.7 Run explanation for sample record" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a3c82ddb-f855-4376-9519-3056f531a93a-22\n" ] } ], "source": [ "transaction_id = subscription.payload_logging.get_table_content(limit=1)['scoring_id'].values[0]\n", "\n", "print(transaction_id)" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "=====================================================================\n", "\n", " Looking for explanation for a3c82ddb-f855-4376-9519-3056f531a93a-22 \n", "\n", "=====================================================================\n", "\n", "\n", "\n", "in_progress.....\n", "finished\n", "\n", "---------------------------\n", " Successfully finished run \n", "---------------------------\n", "\n", "\n" ] } ], "source": [ "explain_run = subscription.explainability.run(transaction_id=transaction_id, background_mode=False)" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAD4CAYAAAAAX/TLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZxcVZ338c+XEEggEYREDGsnCEIgoSEVIKwBxW1AgwQhgJJRiKgMEh6ZJ+KoYYQZUEaHRcSIYXEAkZ0HkQBKWESWTgjZWINhjERoUCABEkj4PX/c03BpqrtvL9VVqXzfr1e96ta955z7O1Wd/Oqce+teRQRmZmZWX9apdgBmZmbW85zgzczM6pATvJmZWR1ygjczM6tDTvBmZmZ1aN1qB2AGMGjQoGhoaKh2GGZma5RZs2a9GBGDy21zgrea0NDQQFNTU7XDMDNbo0h6tq1tnqI3MzOrQ07wZmZmdcgJ3szMrA75GLyZmdWUt956iyVLlrBixYpqh1Iz+vXrx5Zbbknfvn0L13GCN7OqKE0rVTuEXtM0ySeQdsaSJUsYOHAgDQ0NSKp2OFUXEbz00kssWbKEoUOHFq7nKXozM6spK1asYNNNN3VyTySx6aabdnpGwwm+CiQt74V9HCopJO1Q6X11EMfJkjaoZgxmtuZxcn+vrrwfTvD1awJwH3BkleM4GXCCNzPrZT4GXyMkbQNMBwYDzcA/R8T/SjoE+DdgPeAl4OiIeF7SVGBrYFh6/u+IOC+1NQDYGzgAuBmYmtaPBU4HngcageuBecA3gf7AuIhY1E4slwK3RMS1qb3lETEgtTsVeBHYGZgFHAP8C7A5cJekFyPigB5/48ys7vX0+RqVPCfiuOOO45RTTmH48OFtlpk4cSIHH3ww48ePf8/6xYsXc//993PUUUf1SCwewdeOC4DLI2IkcAVwXlp/H7BnROwK/Br411ydHYBPArsD35fUcnrlOOC2iHgS+Luk3XJ1diFL6COALwLbR8TuwMVkCbm9WNqzK9lofTjZl4690xeO54ADyiV3SZMkNUlqam5uLrALM7PadvHFF7eb3NuzePFirrzyyh6LxQm+dowBWj7ZXwH7pOUtgRmS5gGnAjvl6vw2IlZGxIvAC8Bmaf0Esi8DpOcJuToPR8TSiFgJLAJuT+vnAQ0dxNKehyJiSUS8DczJtdWmiJgWEaWIKA0eXPZSymZmVfHDH/6Q887LxjaTJ0/mwAMPBOD3v/89xxxzDLfffjtjxoxht9124/DDD2f58uzUqrFjx75z2e1f/vKXbL/99owdO5bjjz+eE0888Z3277nnHvbaay+GDRvGtddeC8CUKVO49957aWxs5Cc/+Um3++AEX7siPZ8PXBARI4CvAv1yZVbmllcD60raFDgQuFjSYrIvBUfo3TM08nXezr1+m7YP2bTEsor0N5PaW6+9WNrrnJlZLdtvv/249957AWhqamL58uW89dZb3HfffYwYMYIzzjiDO++8k9mzZ1Mqlfjxj3/8nvrPPfccP/jBD3jggQe44447ePzxx9+zfenSpdx3333ccsstTJkyBYCzzjqLfffdlzlz5jB58uRu98EJvnbcz7snxB1NNjUPsBHw17R8bIF2xpNNr28TEQ0RsRXwZ4qNwjuKZTEwKi1/DihyxYVlwMBO7NvMrOpGjRrFrFmzWLZsGeuvvz5jxoyhqamJe++9l/79+7Nw4UL23ntvGhsbueyyy3j22ffe8+Whhx5i//33Z5NNNqFv374cfvjh79k+btw41llnHYYPH87zzz9fkT54lFUdG0haknv9Y+AkYLqkU0kntqVtU4FrJP0VeADo6CoHE4CzWq27DjgKuLpgfG3F8gvgJkkPAb8HXivQ1jTgd5KW+iQ7M1tT9O3bl4aGBi655BL22msvRo4cyV133cWiRYsYOnQoBx10EFdddVWb9SOizW0A66+/fuGyXeUEXwUR0dbMyYFlyt4E3FRm/dRWr3dOi2PLlM2fJDczt35sbnlmy7aIWNxGLM8De+ZWfbt13fT6xNzy+WSHGczM1ij77bcf55xzDtOnT2fEiBGccsopjBo1ij333JNvfOMbPP3003zkIx/h9ddfZ8mSJWy//fbv1N19992ZPHky//jHPxg4cCDXXXcdI0aMaHd/AwcOZNmyZT0WvxO8mVWFL99qRVXrb2XfffflzDPPZMyYMWy44Yb069ePfffdl8GDB3PppZcyYcIEVq7MTj8644wz3pPgt9hiC0477TT22GMPNt98c4YPH85GG23U7v5GjhzJuuuuyy677MLEiRO7fRxelZoaMOuMUqkULWeemtna7bHHHmPHHXesdhjdtnz5cgYMGMCqVas49NBD+fKXv8yhhx7a5fbKvS+SZkVE2QsF+CQ7MzOzCpg6dSqNjY3svPPODB06lHHjxvXq/j1Fb2ZmVgHnnHNOVffvEbyZmdUcHz5+r668H07wZmZWU/r168dLL73kJJ+03A++X79+HRfO8RS9mZnVlC233JIlS5bge1S8q1+/fmy55ZadquMEb2ZmNaVv374MHdrRNb2sI56iNzMzq0NO8GZmZnXICd7MzKwO+Ri8WY0oTSt7Maq65UvVmlWWR/BmZmZ1qOIJXtKHJf1a0iJJCyXdKmn7jmv22P43lvT13OsGSW9ImpPiuVxSkfual2v7Vkkbl1k/VdK3uhN3J2KYKumvqT9PSbpe0vAebL9B0lG51yVJ57VXx8zMqq+iCV6SgBuAmRGxbUQMB04DNitYv0+r1105pLAx8PVW6xZFRCMwAtgS+EIX2iUiPhMRL3elbg/7SUQ0RsR2ZPd8/4OkwUUrd/C+NpDdSx6AiGiKiJO6HKmZmfWKSo/gDwDeioiLWlZExBygj6RbWtZJukDSxLS8WNL3JN0HHC5ppqT/kHQ38E1JgyVdJ+nh9Ng71ZsqaXoq/4ykliR0FrBtGuH+KB9cRKwGHgK2SG30kfSj1O5cSV9N64dIuie1MV/SvrlYB6Xl70h6QtKdwEdzfdtW0m2SZkm6V9IOaf2lks6TdH+Kd3yuzr9KmifpUUlntddOaxFxNXA7KSm3irEkaWbu/Zom6Xbg8jRSv1fS7PTYK/f+7Zv6PlnS2JbPTtImkm5M79UDkkZ28FmYmVkvqfRJdjsDs7pQb0VE7AMg6QRg44jYP72+kmzEep+krYEZQMv983Yg+1IxEHhC0s+AKcDOacSOpIaWnUjqB+wBfDOt+grwSkSMlrQ+8MeUAD8PzIiIM9Oswgb5YCWNAo4EdiV7T2fn+j0NOCEinpK0B3AhcGDaNgTYJ8V9M3CtpE8D44A9IuJ1SZsUaKe12anNjowC9omINyRtABwUESskbQdcBZTS+/etiDg49XVsrv7pwCMRMU7SgcDlQGPa9r7PIiLeavW+TQImAWy99dYFwjUzs6Jq9Sz6q9t5/XFgeDb7D8AHJA1My7+NiJXASkkv0PahgG0lzQG2A66NiLlp/SeAkbnR9EapzMPA9HSs/sY0C5G3L3BDRLwOIOnm9DwA2Au4Jhfv+rl6N0bE28BCSS2xfhy4pKWtiPh7gXZaUzvb8m6OiDfScl/gAkmNwGqgyHkS+wCHpTj/IGlTSRulbeU+iyX5yhExjeyLC6VSyRedNjPrQZVO8AuA8WXWr+K9hwdaX0H/tXZerwOMySUmAFLiW5lbtZq2+7coIholDQFmSvpsRNxMlhj/JSJmtK4gaT/gn4BfSfpRRFzeqki5BLUO8HLL7EEZ+XiVe27dVkfttLYr0PIbpPx73d77PBl4HtgllV9RYD/lvki0xF70szAzswqo9DH4PwDrSzq+ZYWk0UAfslH4+mnE97FOtHk7cGKuvY6S3jKyaeL3iYilZFPQ306rZgBfSyN1JG0vaUNJ2wAvRMQvgF8Cu7Vq6h7gUEn902zCIan9V4E/Szo8tSdJuxTo35fTlDmSNulMO5IOI5uJuCqtWkw2FQ9ptN2GjYClaUbhi2SfEbTz/pH1++i037HAiylWMzOrsoom+Mju9XcocJCyn8ktAKYCzwG/AeYCVwCPdKLZk4BSOrFrIXBCBzG8RHYsfb5anWSX3AhskE6cuxhYCMyWNB/4OdnIcywwR9IjZEny3Fb7mE12GGEOcB1wb27z0cBXJD1KNqPxuQ7ivY3seHxTOozQ8nO79tqZnE6Cewo4BjgwIlpuw3Q6cK6ke8lG0m25EDhW0gNk0/Mto/u5wKp0wt/kVnWmkj4LspPxjm2vb2Zm1nvk++1aLSiVStHU5CubmZl1hqRZEVH2Mpi+kp2ZmVkdcoI3MzOrQ07wZmZmdcgJ3szMrA45wZuZmdUhJ3gzM7M65ARvZmZWh5zgzczM6pATvJmZWR1ygjczM6tDvsOXmVVFaVrZq2taD2qa5Ms/r808gjczM6tDTvBmZmZ1yAneCpF0qKSQtEO1YzEzs445wVtRE4D7gCOrHYiZmXXMCd46JGkAsDfwFVKCl7SOpAslLZB0i6RbJY1P20ZJulvSLEkzJA2pYvhmZmslJ3grYhxwW0Q8Cfxd0m7A54EGYARwHDAGQFJf4HxgfESMAqYDZ5ZrVNIkSU2SmpqbmyvfCzOztYh/JmdFTAD+Oy3/Or3uC1wTEW8Df5N0V9r+UWBn4A5JAH2ApeUajYhpwDSAUqkUFYvezGwt5ARv7ZK0KXAgsLOkIEvYAdzQVhVgQUSM6aUQzcysDE/RW0fGA5dHxDYR0RARWwF/Bl4EDkvH4jcDxqbyTwCDJb0zZS9pp2oEbma2NnOCt45M4P2j9euAzYElwHzg58CDwCsR8SbZl4KzJT0KzAH26r1wzcwMQBE+9GldI2lARCxP0/gPAXtHxN+60lapVIqmJl9W08ysMyTNioiy1332MXjrjlskbQysB/ygq8ndzMx6nhO8dVlEjK12DGZmVp6PwZuZmdUhJ3gzM7M65ARvZmZWh5zgzczM6pATvJmZWR1ygjczM6tDTvBmZmZ1yAnezMysDvlCN2ZWFaVpZa+uaTWkaZIvH70m8wjezMysDtVNgpe0WtKc3GNKD7W7WNKgnmirJ0jaWNLXc68bJL0h6RFJj0l6SNKxPbzPkyVtkHt9a7oGvZmZ1ah6mqJ/IyIaqx1EL9gY+DpwYW7doojYFUDSMOB6SetExCVFGpQksjsLvt1GkZOB/wFeB4iIz3Q1eDMz6x11M4JvSxqB/4ekP0lqkrSbpBmSFkk6IZUZK+keSTdIWijpIknve28knSJpfnqcnNb9QNI3c2XOlHRSavNuSb+R9KSksyQdnUbY8yRtm8oPlnSdpIfTY++0fqqk6ZJmSnpG0klpF2cB26ZZih+1jjEingFOAU7KtfOtXHzz06i/IY34LwRmA1tJ+ll6jxZIOj2VP4ns3u93Sbor954Oauc9aWn7F6mt2yX1787naGZmnVNPCb5/qyn6I3Lb/hIRY4B7gUuB8cCewL/nyuwO/B9gBLAt8Pl845JGAf8M7JHqHi9pV+CXwLGpzDrAkcAVqdouwDdTm18Eto+I3YGLgX9JZc4FfhIRo4HD0rYWOwCfTLF9X1JfYArZiL0xIk5t472Ynep25KPA5RGxa0Q8C3wn3Vd4JLC/pJERcR7wHHBARBxQ8D0B2A74aUTsBLyc+kar+pPSF4qm5ubmAuGamVlRnZqil7RhRLxWqWC6qb0p+pvT8zxgQEQsA5ZJWpE7lvxQGv0i6SpgH+DaXBv7ADe09F/S9cC+EXGepJdSYtsMeCQiXspmvXk4Ipam8ouA23NxtCTLjwPDU3mAD0gamJZ/GxErgZWSXkjtF6GOiwDwbEQ8kHv9BUmTyP4uhgDDgbnt1C/7npC933+OiDmp3CygoXXliJgGTAMolUpRMGYzMyugUIKXtBfZyHIAsLWkXYCvRsTX269ZM1am57dzyy2vW96D1gmm9ev2kubFwETgw8D0Mvttve/8ftcBxkTEG+/ZWZbw8/VXU/wL2a7AY2l5Fe+dqemXW37ny5qkocC3gNER8Q9Jl7YqW05770nr2D1Fb2bWi4pO0f+EbKr4JYCIeBTYr1JBVcnukoamafYjgPtabb8HGCdpA0kbAoeSTfkD3AB8ChgNzOjkfm8HTmx5IamjEwWXAQPb2iipATgHOD+tWgzslrbtBgxto+oHyBL+K5I2Az5dYJ/tvSdmZlZFhafoI+IvuWlkyEZltaS/pDm517dFRGd+KvcnshPYRpAlrhvyGyNidhrVPpRWXRwRj6Rtb6YT0F6OiM6+LycBP5U0l+zzuAc4oa3Cafr/j5LmA78Dfkp20t0jZCPuZcD5uTPorwO+lN6bh4En22j30dTGAuAZ4I+5zdOA30lamj8O39Z7kr5kmJlZFSmi40Ofkq4FfgxcQHYy1UlAKSKOrGx4vUPSWOBbEXFwF+uvQ3Zi2+ER8VRPxra2KJVK0dTkq2atTXwlu9rnK9nVPkmz0snR71N0BH8C2dneWwBLyKaVv9Ez4a3ZJA0HbiE72czJ3awgJw+zyiqU4CPiReDoCsdSNRExE5jZxboLgWE9GY+ZmVl3FT2LfijZ77Yb8nUi4rOVCcvMzMy6o+gU/Y1kF3T5f2Q/8TIzM7MaVjTBr0hXNDMzM7M1QNEEf66k75OdXPfOBUwiYnZFojIzM7NuKZrgW66lfiDvTtFHem1mZmY1pmiCPxQYFhFvVjIYMzMz6xlFL1X7KNl9yM3MzGwNUHQEvxnwuKSHee8xeP9MzszMrAYVTfDfr2gUZmZm1qOKXsnu7koHYtZVvqb5msmXqjWrrELH4CXtKelhScslvSlptaRXKx2cmZmZdU3Rk+wuACYATwH9gePSOqsgSZtKmpMef5P019zr9dqos0RSl06IlHRfgfvRm5nZGqAz94N/WlKfdL/zSyTdX8G4jOze70AjgKSpwPKIOKeqQZmZ2Rqh6Aj+9TRinCPph5ImAxtWMC7rgKRjJT2URvMXpnvSd1hG0rqSXpb0E0mzJd0hadNctSNTnSck7ZXa6S/pMknzUp390vrjJF0raYakpyT9Z27fn5b0p1T+akn+ezEz60VFE/wXU9kTgdeArYDDKhWUtU/SzmQXH9orIhrJZmKO7ESZjYAHImI34E/Ad/NVI2J34FTge2ndScCbEdFyRcNf5Q4R7AKMB0YCx0jaXNKHgCnAx9I+5gLfLNOPSZKaJDU1Nzd34x0xM7PWip5F/2xaXAGcXrlwrKCPA6OBJkmQnRfxl06UWQVck5b/B7gyV+/69DyL7PbAAPsAPwKIiAWSngM+krbdGRHLACQ9DmwNfBgYDtyf9r0ecF/rTkTENGAaQKlUioJ9NzOzAoreD35vYCqwDe+9H/ywyoRlHRAwPSK+29kyktYlu49AXv51y4WMVvPuZ6129rMyt9xSR8BtEfHFduqZmVkFFZ2i/yXwY7KR3Ojcw6rjTuALkgbBO2fbb92JMn2Bz6floygzum7lHuDo1M6OwBDg6XbK3w/sL2lYqrOhpO0K9czMzHpE0bPoX4mI31U0EissIuZJOh24M51c9xZwAvC/Bco8B7wC7CbpNODvwBEd7PJ84OeS5qV2vhQRb6bp93LxPS/pK8DVuWP1p5H9zNLMzHqBIjo+9CnpLKAP2fFZ3w9+DZam6F+MiJq6eVCpVIqmpq5d2cxXslsz+Up2Zt0naVZElP1PsOgIfo/0nG/E94O3muBEYWb2fkXPoj+gve2Sjo2Iy3omJKukiFiFb/1rZlb3ip5k15H3/cbZzMzMqqenEnx7P6MyMzOzXtZTCd4XKTEzM6shHsGbmZnVoZ5K8H/soXbMzMysBxRK8JI2k/RLSb9Lr4enC5kAEBEnVipAMzMz67yiI/hLgRnA5un1k8DJlQjIzMzMuq9ogh8UEb8B3oZ3fku9umJRmZmZWbcUTfCvSdqUdLa8pD3JrmduZmZmNajopWpPAW4GtpX0R2AwML5iUZlZ3fM9BOqLLxldezpM8OlOZP2A/YGPkv0k7omIeKvCsZmZmVkXdThFHxFvA/8VEasiYkFEzHdyX/tI+o6kBZLmSpojaY+Oa5mZWbUUnaK/XdJhwPVR5P6yVlckjQEOBnaLiJWSBgHrdVDNzMyqqOhJdqcA1wArJb0qaZmkVysYl9WWIWT3kF8JEBEvRsRzkkZJulvSLEkzJA2RtK6khyWNBZD0n5LOrGbwZmZro0IJPiIGRsQ6EbFeRHwgvf5ApYOzmnE7sJWkJyVdKGl/SX2B84HxETEKmA6cmX5CORH4maSDgE8Bp5drVNIkSU2Smpqbm3unJ2Zma4lCU/SS9iu3PiLu6dlwrBZFxHJJo4B9gQOAq4EzgJ2BOyQB9AGWpvILJP0K+H/AmIh4s412pwHTAEqlkg/9mJn1oKLH4E/NLfcDdgdmAQf2eERWkyJiNTATmClpHvANYEFEjGmjygjgZWCz3onQzMzyik7RH5J7HEQ2cnu+sqFZrZD0UUnb5VY1Ao8Bg9MJeEjqK2mntPx5YFNgP+A8SRv3dsxmZmu7oiP41paQJXlbOwwAzk+JehXwNDCJbHr9PEkbkf0t/bek54GzgI9FxF8kXQCcCxxbndDNzNZOKvKrN0nnky5TSzbqbwQWR8QxFYzN1iKlUimamnwlLDOzzpA0KyLKXhay6Ag+/z/vKuCqiPA94M3MzGpU0QS/cUScm18h6Zut15mZmVltKHqhm3LHTyf2YBxmZmbWg9odwUuaABwFDJV0c27TQOClSgZmZmZmXdfRFP39ZBcvGQT8V279MmBupYIyMzOz7mk3wUfEs8CzQFsXMzEzM7MaVOgYvKQ90w1Elkt6U9Jq32zGzMysdhU9ye4CYALwFNAfOI7sRiNmZmZWgwpfyS4inpbUJ12T/BJJ91cwLjMzM+uGogn+dUnrAXMk/ZDsxLsNKxeWmZmZdUfRBP9Fsun8E4HJwFbAYZUKyszqX2la2atrWh1qmuTLUFdDoQQfEc9K6g8MiYjTKxyTmZmZdVPRs+gPAeYAt6XXja0ufGNmZmY1pOhZ9FOB3YGXASJiDtBQmZDWHJI+LOnXkhZJWijpVkmTJN3SA22PLdeOpM9KmtKNdg+W9IikR1PMX03rx0kaXqB+oXJmZlZdRRP8qoh4paKRrGEkCbgBmBkR20bEcOA0YLNK7jcibo6Is7pSV1Jfsnu4HxIRuwC7AjPT5nFAkcRdtJyZmVVR0QQ/X9JRQB9J26X7w6/tP5M7AHgrIi5qWZFmNu4FBki6VtLjkq5IXwaQNErS3ZJmSZohaUha/xFJd6ZR9WxJ2+Z3JGl0GnUPkzRR0gVp/aWSzpN0v6RnJI1P69eRdKGkBZJuSTML48nuIbAu6T4CEbEyIp6QtBfwWeBHkuZI2lbS8eniRo9Kuk7SBm2UmymplPY7SNLitLyTpIdSubmStqvYJ2FmZu/TboKX9Ku0uAjYCVgJXAW8Cpxc2dBq3s7ArDa27Ur2/gwHhgF7p9Hz+cD4iBgFTAfOTOWvAH6aRtV7kf0MEYCUVC8CPhcRz5TZ1xBgH+BgoGVk/3myQygjyC5KNAYgIv4O3Aw8K+kqSUdLWici7k/rT42IxohYBFwfEaNTTI8BX2mjXFtOAM6NiEagBCxpXSAdzmiS1NTc3NxOU2Zm1lkdnUU/StI2wBFkI9b8DWc2AFZUKrA13EMRsQRAUsv5Ci+TfSm4Iw3o+wBLJQ0EtoiIGwAiYkWqB7Aj2ZT6JyLiuTb2dWNEvA0slNRyeGAf4Jq0/m+S7mopHBHHSRoBfBz4FnAQ5W/9u7OkM4CNgQHAjE6+B38CviNpS7IvC0+1LhAR01L/KJVK0cn2zcysHR0l+IvIzpwfBuR/yCgg0vq11QJgfBvbVuaWV5O9zwIWRMR7btwj6QPt7GMp0I9sRqCtBJ/fl1o9lxUR84B5aYbmz5RP8JcC4yLiUUkTgbFtNLeKd2eC+uX2caWkB4F/AmZIOi4i/tBeXGZm1nPanaKPiPMiYkdgekQMyz2GRsTanNwB/gCsL+n4lhWSRgP7t1H+CWCwpDGpbF9JO0XEq8ASSePS+vUlbZDqvEyWIP9D0thOxHYfcFg6Fr8ZKTlLGtCqnUayuwVCdgvggbltA8lmGPoCR+fWty63GBiVlt/5wiNpGPBMRJxHNq0/shPxm5lZNxU6yS4ivlbpQNY0ERHAocBB6WdyC8h+Tlh2pB0Rb5IlwLMlPUp2XYG90uYvAidJmkt28uKHc/WeBw4Bfippj4LhXUd2zHs+8HPgQeAVspH9v0p6Ih06OJ13R++/Bk5NJ/NtC3w31bsDeDzXduty5wBfU3ZvgkG5ckeQnZw5B9gBuLxg7GZm1gOU5SmrN5IGRMRySZsCDwF7R8Tfqh1XW0qlUjQ1+XKWZmadIWlWRJS97nPhu8nZGucWSRsD6wE/qOXkbmZmPc8Jvk5FxNhqx2BmZtVT9EI3ZmZmtgZxgjczM6tDTvBmZmZ1yAnezMysDjnBm5mZ1SEneDMzszrkBG9mZlaHnODNzMzqkC90Y2ZVUZpW9uqatpZqmuRLVfc0j+DNzMzqkBN8L5C0vMLtr5Y0R9ICSY9KOkVSj322kiZK2jz3+mJJw3uqfTMz63meoq8Pb0REI4CkDwFXAhsB3y/agKQ+EbG6jc0TyW49+xxARBzXrWjNzKziPIKvEknbSPq9pLnpeeu0/hBJD6b7rd8pabO0fqqk6ZJmSnpG0knl2o2IF4BJwInKTJR0QW6/t0gam5aXS/p3SQ8CYyR9T9LDkuZLmpbqjwdKwBVplqB/iqGU2pggaV6qc3ZuP8slnZlmFB5o6YeZmfUOJ/jquQC4PCJGAlcA56X19wF7RsSuwK+Bf83V2QH4JLA78H1Jfcs1HBHPkH22H+oghg2B+RGxR0TcB1wQEaMjYmegP3BwRFwLNAFHR0RjRLzRUjlN258NHAg0AqMljcu1/UBE7ALcAxzfeueSJklqktTU3NzcQahmZtYZTvDVM4ZsKh3gV8A+aXlLYIakecCpwE65Or+NiJUR8SLwAtDeqFgFYlgNXJd7fUCaPZhHlrR3Kl/tHaOBmRHRHBGryL6o7Je2vQnckpZnAQ2tK0fEtIgoRURp8ODBBcI1M7OinOBrR6Tn88lG0iOArwL9cmVW5pZX0xqtWIcAAA3tSURBVMY5FJKGpe0vAKt47+ecb29Fy3F3Sf2AC4Hxad+/aFW27K7a2fZWRLT0qc1YzcysMpzgq+d+4Mi0fDTZ1DxkJ8f9NS0f29lGJQ0GLiL7khDAYqBR0jqStiKb3i+nJZm/KGkAMD63bRkwsEydB4H9JQ2S1AeYANzd2ZjNzKzneVTVOzaQtCT3+sfAScB0SacCzcA/p21TgWsk/RV4ABhaoP3+kuYAfclG7L9K+wD4I/BnYB7ZmfCzyzUQES9L+kUqtxh4OLf5UuAiSW+QHVpoqbNU0reBu8hG87dGxE0F4jUzswrTu7OoZtVTKpWiqclXslqb+Ep2lucr2XWNpFkRUfYfk0fwZlYV/g/drLJ8DN7MzKwOOcGbmZnVISd4MzOzOuQEb2ZmVoec4M3MzOqQE7yZmVkdcoI3MzOrQ07wZmZmdcgJ3szMrA45wZuZmdUhX6rWzKrC16K3Nc2adnllj+DNzMzqkBN8BUjaUtJNkp6StEjSuZLWk9Qo6TO5clMlfatCMUyS9Hh6PCRpnwJ1Jkq6oBLxmJlZ73KC72GSBFwP3BgR2wHbAwOAM4FG4DPtVO/svvq0sf5g4KvAPhGxA3ACcKWkD/fUvs3MrLY5wfe8A4EVEXEJQESsBiYDxwE/BI6QNEfSEan8cEkzJT0j6aSWRiQdk0becyT9vCWZS1ou6d8lPQiMkXSWpIWS5ko6J1X/v8CpEfFiimE2cBnwjdTGYkmnS5otaZ6kHdrrkKQJqdx8SWfn1v9MUpOkBZJOz63vVPtmZtbznOB73k7ArPyKiHgVWAycAVwdEY0RcXXavAPwSWB34PuS+kraETgC2DsiGoHVwNGp/IbA/IjYA1gIHArsFBEjU/tlYwCa0voWL0bEbsDPgDYPE0jaHDib7ItLIzBa0ri0+TsRUQJGAvtLGtmZ9tNhhCZJTc3NzW2FYGZmXeAE3/MERCfW/zYiVqbR9gvAZsDHgFHAw5LmpNfDUvnVwHVp+VVgBXCxpM8Dr3ciruvT8yygoZ16o4GZEdEcEauAK4D90rYvSJoNPEL25WF4Z9qPiGkRUYqI0uDBg9sJwczMOssJvuctAN7z+x9JHwC2IkvOra3MLa8m++migMvSSL8xIj4aEVNTmRVp2p+UcHcnS/jjgNtSmYVkXxDydkvrW++3ZZ9tUdmV0lCykfnH0uzBb4F+XWjfzMwqwAm+5/0e2EDSl+CdE+H+C7gUeB4YWLCN8ZI+lNrYRNI2rQtJGgBsFBG3AieTTaFDdqz/bEmbpnKNwETgwi7050Gy6fdBqS8TgLuBDwCvAa9I2gz4dBfaNjOzCvHIqodFREg6FLhQ0nfJvkTdCpxGdvx8Spp2/8922lgo6d+A2yWtA7xFdoLcs62KDgRuktSPbKQ9OdW/WdIWwP2SAlgGHBMRSwt0YWLuGDvAnsC3gbvSPm6NiJsAJD1CNmPxDPDHAm2bmVkvUUS5w8JmvatUKkVT05p1lSjrHl/JztY0tXglO0mz0snO7+MRvJlVRS3+Z2lWT3wM3szMrA45wZuZmdUhJ3gzM7M65ARvZmZWh5zgzczM6pATvJmZWR1ygjczM6tDTvBmZmZ1yAnezMysDvlKdmZWFb5UrVmmUld19AjezMysDjnBm5mZ1aG6SvCSVkuak3tM6aD8rZI2bmf7yZI2KFq+QHy7S7pH0hOSHpd0cb79LrR3qaTxafliScPT8mkF6vaVdJakpyTNl/SQpB69p7ukBklH9WSbZmZWTL0dg38jIhqLFo6Iz3RQ5GTgf4DXC5Zvk6TNgGuAIyPiT5IEHEZ2T/fXc+XWjYhVnW0/Io7LvTwN+I8OqvwAGALsHBErU3z7d3a/HWgAjgKu7OF2zcysA3U1gi9H0kZpxPzR9PoqScen5cWSBknaUNJvJT2aRrNHSDoJ2By4S9Jdrco3SHpM0i8kLZB0u6T+qcxoSXMl/UnSjyTNT6F8A7gsIv4EEJlrI+J5SVMlTZN0O3C5pD6p7sOpra+mtiXpAkkLJf0W+FCunzMllSSdBfRPMxhXtNG3DYDjgX+JiJUpnucj4jeprQmS5qXyZ+f2sTy3PF7SpWn5UknnSbpf0jMtswrAWcC+KZbJPfepmplZR+otwbcktpbHERHxCnAicKmkI4EPRsQvWtX7FPBcROwSETsDt0XEecBzwAERcUCZfW0H/DQidgJeJhuNA1wCnBARY4DVufI7A7PaiX0U8LmIOAr4CvBKRIwGRgPHSxoKHAp8FBhBlqD3at1IREwhzWRExNHl+gZ8BPjfiHi1dX1JmwNnAwcCjcBoSePaibvFEGAf4GCyxA4wBbg3xfKTMvuaJKlJUlNzc3OBXZiZWVH1luBbElvL42qAiLgDmAf8FDiuTL15wMclnS1p3/SloCN/jog5aXkW0JCOzw+MiPvT+s5MTd8cEW+k5U8AX5I0B3gQ2JTsC8V+wFURsToingP+UKDdzvZtNDAzIprToYIr0n47cmNEvB0RC4HNCpQnIqZFRCkiSoMHDy5SxczMCqq3BF+WpHWAHYE3gE1ab4+IJ8lG0POA/5T0vQLNrswtryY7n0HtlF+Q9tGW1/Ihk02ft3xRGRoRt7eEWyC2d7TRt6eBrSUNLFOlvT7k992v1bb8+9FeG2Zm1gvWigQPTAYeAyYA0yX1zW9M09KvR8T/AOcAu6VNy8hOgiskIv4BLJO0Z1p1ZG7zBcCxkvbI7fcYSR8u09QM4GstcUraXtKGwD3AkekY/RCg3KEDgLdydd/Xt4h4HfglcJ6k9VK5IZKOIZsx2D+da9CH7D27O7X7vKQd0xemQwu8JZ16/8zMrOfU21n0/dO0dovbgOlk0/K7R8QySfcA/wZ8P1duBPAjSW8DbwFfS+unAb+TtLSN4/DlfAX4haTXgJnAK5CdxJbOAThH0oeAt8kS9vVl2riY7Az02els+2ZgHHAD2bHxecCTvJt4W5sGzJU0G7i8jb79G3AGsFDSCrIZhO9FxFJJ3wbuIhuJ3xoRN6U6U4BbgL8A84EBHbwXc4FVkh4FLi13HN7MzCpDEZ2a8bUOSBoQEcvT8hRgSER8s8ph1bxSqRRNTZW5XKOZWb2SNCsiyl73ud5G8LXgn9IIeF3gWWBidcMxM7O1kRN8D0tn7l9d7TjMzGzttracZGdmZrZWcYI3MzOrQ07wZmZmdchn0VtNkNRMdlJiRwYBL1Y4nFqwNvRzbegjuJ/1ptb6uU1ElL0UqBO8rVEkNbX1k5B6sjb0c23oI7if9WZN6qen6M3MzOqQE7yZmVkdcoK3Nc20agfQS9aGfq4NfQT3s96sMf30MXgzM7M65BG8mZlZHXKCNzMzq0NO8FZTJG0i6Q5JT6XnD7ZR7jZJL0u6pdX6oZIeTPWvbrnffa3pRD+PTWWeknRsbv1MSU9ImpMeH+q96Dsm6VMpvqfTXRVbb18/fT5Pp8+rIbft22n9E5I+2Ztxd1ZX+ympQdIbuc/vot6OvTMK9HM/SbMlrZI0vtW2sn/Dtaib/Vyd+zxv7r2o2xERfvhRMw/gh8CUtDwFOLuNch8DDgFuabX+N8CRafki4GvV7lNX+wlsAjyTnj+Ylj+Yts0EStXuRxt96wMsAoYB6wGPAsNblfk6cFFaPhK4Oi0PT+XXB4amdvpUu08V6GcDML/afejBfjYAI4HLgfG59W3+Ddfaozv9TNuWV7sPrR8ewVut+RxwWVq+DBhXrlBE/B5Yll8nScCBwLUd1a8BRfr5SeCOiPh7RPwDuAP4VC/F1x27A09HxDMR8Sbwa7L+5uX7fy3wsfT5fQ74dUSsjIg/A0+n9mpRd/q5JumwnxGxOCLmAm+3qrsm/Q13p581yQneas1mEbEUID13Zup5U+DliFiVXi8Btujh+HpKkX5uAfwl97p1fy5J04HfrbGk0VHc7ymTPq9XyD6/InVrRXf6CTBU0iOS7pa0b6WD7YbufCb19nm2p5+kJkkPSKqJgYXvB2+9TtKdwIfLbPpOd5sus65qvwPtgX6215+jI+KvkgYC1wFfJJs2rAVFPoe2ytTUZ9iB7vRzKbB1RLwkaRRwo6SdIuLVng6yB3TnM6m3z7M9W0fEc5KGAX+QNC8iFvVQbF3iBG+9LiI+3tY2Sc9LGhIRSyUNAV7oRNMvAhtLWjeNlrYEnutmuF3WA/1cAozNvd6S7Ng7EfHX9LxM0pVk04u1kuCXAFvlXpf7HFrKLJG0LrAR8PeCdWtFl/sZ2UHblQARMUvSImB7oKniUXdedz6TNv+Ga1C3/vYi4rn0/IykmcCuZMf0q8ZT9FZrbgZazrQ9FripaMX0n+ZdQMvZrZ2q38uK9HMG8AlJH0xn2X8CmCFpXUmDACT1BQ4G5vdCzEU9DGyXftGwHtnJZa3PKs73fzzwh/T53Qwcmc4+HwpsBzzUS3F3Vpf7KWmwpD4AacS3HdkJaLWoSD/bUvZvuEJxdleX+5n6t35aHgTsDSysWKRFVfssPz/8yD/Ijk/+HngqPW+S1peAi3Pl7gWagTfIvnl/Mq0fRpYQngauAdavdp+62c8vp748DfxzWrchMAuYCywAzqXGzjQHPgM8STaC+U5a9+/AZ9Nyv/T5PJ0+r2G5ut9J9Z4APl3tvlSin8Bh6bN7FJgNHFLtvnSzn6PTv8PXgJeABe39Ddfqo6v9BPYC5qXPcx7wlWr3JSJ8qVozM7N65Cl6MzOzOuQEb2ZmVoec4M3MzOqQE7yZmVkdcoI3MzOrQ07wZmZmdcgJ3szMrA79fytd9ONhRW1NAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "if explain_run == None:\n", " # explanation didn't finish within 180 seconds, if explaination is still not finished give it a minute or so then re-run this cell\n", " time.sleep(10)\n", " explain_table = subscription.explainability.get_table_content(format='pandas')\n", " explain_result = pd.DataFrame.from_dict(explain_table[explain_table['transaction_id']==transaction_id]['explanation'][0]['entity']['predictions'][0]['explanation_features'])\n", "else:\n", " explain_result = pd.DataFrame.from_dict(explain_run['entity']['predictions'][0]['explanation_features'])\n", "\n", "explain_result.plot.barh(x='feature_name', y='weight', color='g', alpha=0.8);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6.0 Custom monitors and metrics " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6.1 Register custom monitor" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [], "source": [ "def get_definition(monitor_name):\n", " monitors_definitions = ai_client.data_mart.monitors.get_details()['monitor_definitions']\n", " \n", " for definition in monitors_definitions:\n", " if monitor_name == definition['entity']['name']:\n", " return definition\n", " \n", " return None" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import Metric, Tag\n", "\n", "monitor_name = MONITOR_NAME\n", "metrics = [Metric(name='sensitivity', lower_limit_default=0.8), Metric(name='specificity', lower_limit_default=0.75)]\n", "tags = [Tag(name='region', description='customer geographical region')]\n", "\n", "existing_definition = get_definition(monitor_name)\n", "\n", "if existing_definition is None:\n", " my_monitor = ai_client.data_mart.monitors.add(name=monitor_name, metrics=metrics, tags=tags)\n", "else:\n", " my_monitor = existing_definition" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.1.1 Get monitors uids and details" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "scottda_telco_9_22_2020_monitor\n" ] } ], "source": [ "monitor_uid = my_monitor['metadata']['guid']\n", "\n", "print(monitor_uid)" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "monitor definition details {'entity': {'metrics': [{'expected_direction': 'increasing', 'id': 'sensitivity', 'name': 'sensitivity', 'required': True, 'thresholds': [{'default': 0.8, 'type': 'lower_limit'}]}, {'expected_direction': 'increasing', 'id': 'specificity', 'name': 'specificity', 'required': True, 'thresholds': [{'default': 0.75, 'type': 'lower_limit'}]}], 'name': 'Scottda-telco-9-22-2020_MONITOR', 'tags': [{'description': 'customer geographical region', 'id': 'region', 'name': 'region', 'required': True}]}, 'metadata': {'guid': 'scottda_telco_9_22_2020_monitor', 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/monitor_definitions/scottda_telco_9_22_2020_monitor', 'created_at': '2020-09-22T12:29:42.916Z'}}\n" ] } ], "source": [ "my_monitor = ai_client.data_mart.monitors.get_details(monitor_uid=monitor_uid)\n", "print('monitor definition details', my_monitor)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.2 Enable custom monitor for subscription" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [], "source": [ "from ibm_ai_openscale.supporting_classes import Threshold\n", "\n", "thresholds = [Threshold(metric_uid='sensitivity', lower_limit=0.9)]\n", "subscription.monitoring.enable(monitor_uid=monitor_uid, thresholds=thresholds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 6.2.1 Get monitor configuration details" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'enabled': True,\n", " 'monitor_definition': {'entity': {'metrics': [{'expected_direction': 'increasing',\n", " 'id': 'sensitivity',\n", " 'name': 'sensitivity',\n", " 'required': True,\n", " 'thresholds': [{'default': 0.9, 'type': 'lower_limit'}]},\n", " {'expected_direction': 'increasing',\n", " 'id': 'specificity',\n", " 'name': 'specificity',\n", " 'required': True}],\n", " 'name': 'Scottda-telco-9-22-2020_MONITOR',\n", " 'tags': [{'description': 'customer geographical region',\n", " 'id': 'region',\n", " 'name': 'region',\n", " 'required': True}]},\n", " 'metadata': {'guid': 'scottda_telco_9_22_2020_monitor',\n", " 'url': '/v1/data_marts/00000000-0000-0000-0000-000000000000/monitor_definitions/scottda_telco_9_22_2020_monitor',\n", " 'created_at': '2020-09-22T12:29:42.916Z'}},\n", " 'thresholds': [{'metric_id': 'sensitivity',\n", " 'type': 'lower_limit',\n", " 'value': 0.9}]}" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subscription.monitoring.get_details(monitor_uid=monitor_uid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6.3 Storing custom metrics" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'asset_id': 'd9f69282-4c9a-4e2f-8293-8589695853da',\n", " 'binding_id': '999',\n", " 'measurement_id': '31a8e59b-26d3-421a-b450-535ffb2239f2',\n", " 'metrics': [{'region': 'us-south',\n", " 'sensitivity': 0.67,\n", " 'specificity': 0.78}],\n", " 'monitor_definition_id': 'scottda_telco_9_22_2020_monitor',\n", " 'subscription_id': '14a4e07c-6cae-4b60-828d-40be2661ae73',\n", " 'timestamp': '2020-09-22T12:29:54.950570Z'}]" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metrics = {\"specificity\": 0.78, \"sensitivity\": 0.67, \"region\": \"us-south\"}\n", "\n", "subscription.monitoring.store_metrics(monitor_uid=monitor_uid, metrics=metrics)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.3.1 List and get custom metrics" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

CustomMetrics (binding_id=999, subscription_id=14a4e07c-6cae-4b60-828d-40be2661ae73)

\n", " \n", " \n", " \n", "
tsidmeasurement_idvaluelower limitupper limittagsbinding_idsubscription_iddeployment_id
2020-09-22 12:29:54.950570+00:00sensitivity31a8e59b-26d3-421a-b450-535ffb2239f20.670.9region: us-south99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
2020-09-22 12:29:54.950570+00:00specificity31a8e59b-26d3-421a-b450-535ffb2239f20.78region: us-south99914a4e07c-6cae-4b60-828d-40be2661ae73e85d6845-4b6b-44bf-a483-80a522852620
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "subscription.monitoring.show_table(monitor_uid=monitor_uid)" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'asset_id': 'd9f69282-4c9a-4e2f-8293-8589695853da',\n", " 'binding_id': '999',\n", " 'tags': [{'id': 'region', 'value': 'us-south'}],\n", " 'ts': '2020-09-22T12:29:54.950570Z',\n", " 'measurement_id': '31a8e59b-26d3-421a-b450-535ffb2239f2',\n", " 'monitor_definition_id': 'scottda_telco_9_22_2020_monitor',\n", " 'subscription_id': '14a4e07c-6cae-4b60-828d-40be2661ae73',\n", " 'metrics': [{'lower_limit': 0.9, 'id': 'sensitivity', 'value': 0.67},\n", " {'id': 'specificity', 'value': 0.78}]}]" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "custom_metrics = subscription.monitoring.get_metrics(monitor_uid=monitor_uid, deployment_uid='credit')\n", "custom_metrics" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAD4CAYAAABMtfkzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAR1ElEQVR4nO3de5CddX3H8fcXEsgoAZkAuhLKQg23kJiQVQuVWgpUSsBLzcAKjgJaBxQQSmdgip2itWOsjiAFy1Aq8QIS3JoBxTsk4oWLWQiEwMCgxukSLjHaFcTQJHz7x3k2HjabnLNkzzm/zb5fMzvnOc/ze87zOU9289nnsruRmUiSVIKdOh1AkqQhlpIkqRiWkiSpGJaSJKkYlpIkqRiTOh1gPNtrr72yu7u70zEkaVzp7+//dWbuPdIyS2k7dHd3s3z58k7HkKRxJSJ+tbVlnr6TJBXDUpIkFcNSkiQVw2tKktRiGzZsYGBggPXr13c6SltNmTKF6dOnM3ny5KbXsZQkqcUGBgaYOnUq3d3dRESn47RFZrJu3ToGBgY44IADml7P03eS1GLr169n2rRpE6aQACKCadOmjfro0FKSpDaYSIU05OW8Z0tJklQMrylJUpt1X3LbmL7e6oXzx/T1dtttN5577rkxfc1meaQkSSqGpSRJO7iLL76Yz3/+85ufX3bZZXzsYx/j2GOP5YgjjmDWrFnccsstW6y3bNkyTjrppM3Pzz33XBYtWgRAf38/b3nLW5g3bx5vfetbefLJJ8ckq6UkSTu43t5eFi9evPn5zTffzJlnnsmSJUu47777WLp0KRdddBGZ2dTrbdiwgfPOO4++vj76+/s566yzuPTSS8ckq9eUJGkHN3fuXJ555hnWrFnD2rVr2XPPPenq6uLCCy/kzjvvZKedduKJJ57g6aef5jWveU3D13v00Ud56KGHOP744wHYtGkTXV1dY5LVUpKkCWDBggX09fXx1FNP0dvbyw033MDatWvp7+9n8uTJdHd3b/EzRZMmTeLFF1/c/HxoeWYyc+ZM7rrrrjHP6ek7SZoAent7uemmm+jr62PBggUMDg6yzz77MHnyZJYuXcqvfrXlX5PYf//9efjhh3nhhRcYHBzk9ttvB+Dggw9m7dq1m0tpw4YNrFq1akxyeqQkSW021rdwN2PmzJk8++yz7LvvvnR1dXH66adz8skn09PTw5w5czjkkEO2WGe//fbjlFNOYfbs2cyYMYO5c+cCsMsuu9DX18f555/P4OAgGzdu5IILLmDmzJnbnTOavbClLfX09KR/5E9SI4888giHHnpop2N0xEjvPSL6M7NnpPGevpMkFcNSkiQVw1KSpDaYiJdKXs57tpQkqcWmTJnCunXrJlQxDf09pSlTpoxqPe++k6QWmz59OgMDA6xdu7bTUdpq6C/PjoalJEktNnny5FH99dWJzNN3kqRiWEqSpGJYSpKkYnhNaTusfGJwzP+CpCSVrpW/JskjJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxxmUpRcS3IuJV1fT5EfFIRNwQEW+LiEsarPvT6rE7Ik5rR15JUnMmdTrAy5GZJ9Y9/RDwN5n5y+r5rQ3WPaqa7AZOA24c84CSpJelZUdKEfHKiLgtIh6IiIci4tSIWB0Rn4qIe6uP11Vj946I/46In1Uff17N3y0iro+IlRHxYES8q5q/OiL2iohrgAOBWyPiwog4IyKuqsa8OiKWVNt/ICKOquY/V0VcCBwdESuqdX8UEXPq8v8kIma3av9IkrbUytN3JwBrMvP1mXk48J1q/u8y843AVcAV1bzPAZdn5huAdwHXVfP/CRjMzFmZORu4o34DmXk2sAY4JjMvH7b9K4EfZubrgSOAVcOWXwL8KDPnVOteB5wBEBEHAbtm5oPD31REfDAilkfE8k3PD45mf0iSGmhlKa0EjquOjI7OzKH/wb9a93hkNX0ccFVErKB2+m33iJhazb966AUz87ej2P5fAf9Rrbepbvtb8zXgpIiYDJwFLBppUGZem5k9mdmz8yv2GEUcSVIjLbumlJmPRcQ84ETgkxHxvaFF9cOqx52AIzPzD/WvERExbHzLZObzEfF94O3AKUBPO7YrSfqjVl5Tei3wfGZ+BfgMtVNoAKfWPd5VTX8POLdu3Tlbmb/nKCLcDpxTrbdzROw+bPmzwNRh866jdtrvZ5n5m1FsS5I0Blp5+m4WcG91Su5S4BPV/F0j4h7gI8CF1bzzgZ7qZoaHgbOr+Z8A9qxulHgAOGYU2/8IcExErAT6gZnDlj8IbKxugrgQIDP7gd8B14/mjUqSxkZktuXsWG1jEauBnsz8dds2OgrV0d0y4JDMfLHR+F27ZmTX+65oNEySdiirF87frvUjoj8zR7xEMi5/eLYVIuK9wD3Apc0UkiRp7LX1h2czs7ud2xuNzPwS8KVO55CkicwjJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSZ1OsB4NmvfPVi+cH6nY0jSDsMjJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSZta2FE/O22lmfm18c2jiRpIttmKQEnV4/7AEcBd1TPjwGWAZaSJGnMbLOUMvNMgIj4JnBYZj5ZPe8Crm59PEnSRNLsNaXuoUKqPA0c1II8kqQJrNHpuyHLIuK7wFeBBHqBpS1LJUmakJoqpcw8t7rp4ehq1rWZuaR1sSRJE1GzR0pDd9p5Y4MkqWUa3RL+48x8c0Q8S+203eZFQGbm7i1NJ0maUBrdfffm6nFqe+JIkiYyf6ODJKkYlpIkqRiWkiSpGJaSJKkYlpIkqRiWkiSpGJaSJKkYlpIkqRiWkiSpGJaSJKkYlpIkqRiWkiSpGJaSJKkYlpIkqRiWkiSpGJaSJKkYlpIkqRiWkiSpGJaSJKkYkzodYDxb+cQg3Zfc1ukYksax1QvndzpCUTxSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBXDUpIkFcNSkiQVw1KSJBVjXJZSRPy0euyOiNPq5vdExJUN1j07It5bTZ8REa9tbVpJUrMmdTrAy5GZR1WT3cBpwI3V/OXA8gbrXlP39AzgIWDNmIeUJI1ay46UIuKVEXFbRDwQEQ9FxKkRMS8ifhgR/RHx3YjoqsYui4hPRcS9EfFYRBxdzZ9ZzVsREQ9GxIxq/nPVZhYCR1fLL4yIv4yIb0bEThGxOiJeVZfn8Yh4dURcFhH/EBELgB7ghmr9+RGxpG788RHx9VbtH0nSllp5+u4EYE1mvj4zDwe+A/w7sCAz5wFfAP61bvykzHwjcAHwz9W8s4HPZeYcagUyMGwblwA/ysw5mXn50MzMfBG4BXgnQES8CVidmU/XjemjdlR1evX63wIOjYi9qyFnAtcPf1MR8cGIWB4Ryzc9Pzj6vSJJ2qpWltJK4LjqCOhoYD/gcOD7EbEC+CgwvW780FFJP7XTcgB3Af8YERcD+2fmH0ax/cXAqdV0b/V8qzIzgS8D76mOsI4Evj3CuGszsycze3Z+xR6jiCNJaqRl15Qy87GImAecCHwS+D6wKjOP3MoqL1SPm4ZyZeaNEXEPMB/4bkR8IDPvaDLCXcDrqiOfdwCfaGKd64FvAOuBr2Xmxia3JUkaA628pvRa4PnM/ArwGeBNwN4RcWS1fHJEzGzwGgcCv8jMK4FbgdnDhjwLTB1p3erIZwnwWeCRzFw3wrCXrJ+Za6jd9PBRYFGj9yhJGlutvPtuFvDpiHgR2ACcA2wEroyIPaptXwGs2sZrnErtdNoG4Cng48OWPwhsjIgHqJXI/cOWLwZ+Ru0uu5EsAq6JiD8AR1anB28A9s7Mh5t4j5KkMRS1AwoNiYirgPsz878ajd21a0Z2ve+KNqSStKNavXB+pyO0XUT0Z2bPSMvG5c8ptUpE9AO/By7qdBZJmogspTrVreqSpA4Zl79mSJK0Y7KUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFmNTpAOPZrH33YPnC+Z2OIUk7DI+UJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWwlCRJxbCUJEnFsJQkScWIzOx0hnErIp4FHu10jgb2An7d6RANmHFsmHFslJ6x9HzQOOP+mbn3SAsmtSbPhPFoZvZ0OsS2RMRyM24/M44NM26/0vPB9mX09J0kqRiWkiSpGJbS9rm20wGaYMaxYcaxYcbtV3o+2I6M3uggSSqGR0qSpGJYSpKkYlhKTYiIEyLi0Yh4PCIuGWH5rhGxuFp+T0R0F5jxLyLivojYGBEL2p2vyYx/HxEPR8SDEXF7ROxfYMazI2JlRKyIiB9HxGGlZawbtyAiMiLaevtwE/vwjIhYW+3DFRHxgXbmayZjNeaU6vNxVUTcWFrGiLi8bh8+FhH/W2DGP4mIpRFxf/V1fWLDF81MP7bxAewM/Bw4ENgFeAA4bNiYDwHXVNO9wOICM3YDs4EvAQsK3Y/HAK+ops8pdD/uXjf9NuA7pWWsxk0F7gTuBnpKygecAVzV7s/BUWacAdwP7Fk936e0jMPGnwd8obSM1G54OKeaPgxY3eh1PVJq7I3A45n5i8z8P+Am4O3Dxrwd+GI13QccGxFRUsbMXJ2ZDwIvtjFXvWYyLs3M56undwPTC8z4u7qnrwTafadQM5+PAP8C/Buwvp3haD5fJzWT8e+AqzPztwCZ+UyBGeu9G/hqW5L9UTMZE9i9mt4DWNPoRS2lxvYF/qfu+UA1b8QxmbkRGASmtSXdsO1XRsrYaaPN+H7g2y1NtKWmMkbEhyPi59T+0z+/TdmGNMwYEXOB/TLzm+0MVmn23/ld1emcvojYrz3RNmsm40HAQRHxk4i4OyJOaFu6mqa/XqrT3AcAd7QhV71mMl4GvCciBoBvUTui2yZLqbGRjniGf3fczJhW6vT2m9F0xoh4D9ADfLqliUbY9AjztsiYmVdn5p8CFwMfbXmql9pmxojYCbgcuKhtiV6qmX34DaA7M2cDP+CPZxnapZmMk6idwvtLakch10XEq1qcq95ovqZ7gb7M3NTCPCNpJuO7gUWZOR04Efhy9Tm6VZZSYwNA/Xdy09nyEHTzmIiYRO0w9TdtSTds+5WRMnZaUxkj4jjgUuBtmflCm7INGe1+vAl4R0sTbalRxqnA4cCyiFgN/Blwaxtvdmi4DzNzXd2/7X8C89qUbUizX9O3ZOaGzPwltV+8PKNN+Ya23+znYi/tP3UHzWV8P3AzQGbeBUyh9stat66dF8bG4we175h+Qe3weOhi3sxhYz7MS290uLm0jHVjF9GZGx2a2Y9zqV04nVHwv/WMuumTgeWlZRw2fhntvdGhmX3YVTf9TuDu0vYhcALwxWp6L2qnqaaVlLEadzCwmuoXIRS4H78NnFFNH0qttLaZta1vYrx+UDvsfKz6D/PSat7HqX03D7X2/xrwOHAvcGCBGd9A7Tub3wPrgFUFZvwB8DSwovq4tcCMnwNWVfmWbqsQOpVx2Ni2llKT+/CT1T58oNqHh5S2D6mdmvos8DCwEugtLWP1/DJgYbuzjWI/Hgb8pPq3XgH8daPX9NcMSZKK4TUlSVIxLCVJUjEsJUlSMSwlSVIxLCVJUjEsJUlSMSwlSVIx/h+Y1aWHhovvMgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "custom_metrics_pandas = subscription.monitoring.get_table_content(monitor_uid=monitor_uid)\n", "\n", "%matplotlib inline\n", "custom_metrics_pandas.plot.barh(x='id', y='value');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 7.0 Payload analytics " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7.1 Run data distributions calculation" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "===============================================================================\n", "\n", " Waiting for end of data distribution run a698d38e-a9ce-4b45-9cb6-9869f35b7db2 \n", "\n", "===============================================================================\n", "\n", "\n", "\n", "completed\n", "\n", "---------------------------\n", " Successfully finished run \n", "---------------------------\n", "\n", "\n" ] } ], "source": [ "from datetime import datetime\n", "\n", "start_date = \"2018-01-01T00:00:00.00Z\"\n", "end_date = datetime.utcnow().isoformat() + \"Z\"\n", "\n", "sex_distribution = subscription.payload_logging.data_distribution.run(\n", " start_date=start_date,\n", " end_date=end_date,\n", " group=['predictedLabel', 'Sex'],\n", " agg=['count'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7.2 Get data distributions as pandas dataframe" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
predictedLabelSexcount
0No Riskfemale64
1No Riskmale101
2Riskfemale7
3Riskmale36
\n", "
" ], "text/plain": [ " predictedLabel Sex count\n", "0 No Risk female 64\n", "1 No Risk male 101\n", "2 Risk female 7\n", "3 Risk male 36" ] }, "execution_count": 65, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sex_distribution_run_uid = sex_distribution['id']\n", "distributions_pd = subscription.payload_logging.data_distribution.get_run_result(run_id=sex_distribution_run_uid, format='pandas')\n", "distributions_pd" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEiCAYAAAACg5K6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAY5ElEQVR4nO3de5RU5Z3u8e/DRRgMXpA2g7RMdyIKCDiY9kpmxksSdUbF5YkZPCTBCVk9jkRJdDQadWTiMsuscMZEPZqDwQENaLxE4SQ6o3HweIwKNCIXRQ+oqB1NaFFRySAXf+eP2u0UTTXd1VXd1f3281mrV+397nfv/ate8NTut/ZFEYGZmaWlT6ULMDOz8nO4m5klyOFuZpYgh7uZWYIc7mZmCXK4m5klqF+lCwAYOnRo1NTUVLoMM7MeZfny5W9HRFWhZd0i3GtqamhoaKh0GWZmPYqk11pb5mEZM7MEOdzNzBLkcDczS1C3GHM3M2u2fft2Ghsb2bp1a6VL6TYGDhxIdXU1/fv3b/c6Dncz61YaGxsZPHgwNTU1SKp0ORUXEWzatInGxkZqa2vbvZ6HZcysW9m6dSsHHHCAgz0jiQMOOKDov2Qc7mbW7TjYd9WR34fD3cyshRtvvJHRo0czZcqUTtn+zJkzmTVrVqdsu5nH3Isxc99KV5CWmZsrXYH1ADWX/7qs29tw/d+02eeWW27h4YcfLmqMu7txuJuZ5Tn//PN55ZVXOPPMM5k8eTIvv/wyq1evZseOHcycOZNJkyYxd+5cHnzwQXbu3MmaNWu45JJL2LZtG3feeScDBgzgoYceYsiQIdx2223Mnj2bbdu2ccghh3DnnXcyaNCgXfb38ssvM336dJqamhg0aBC33XYbo0aNKvl9eFjGzCzPT3/6Uw466CAWL17Mli1bOOmkk1i2bBmLFy/m0ksvZcuWLQCsWbOGBQsWsHTpUq688koGDRrEihUrOO6447jjjjsAOPvss1m2bBkrV65k9OjRzJkzZ7f91dfXc9NNN7F8+XJmzZrFBRdcUJb30eaRu6TbgdOBjRExNq/9QuBbwA7g1xFxWdZ+BTAN2AlcFBH/XpZKzcy62COPPMKiRYs+GR/funUrr7/+OgAnnngigwcPZvDgwey7776cccYZAIwbN45Vq1YBuQ+Aq666ivfee48PP/yQU045ZZftf/jhhzz11FOcc845n7R99NFHZam9PcMyc4GbgTuaGySdCEwCxkfER5IOzNrHAJOBw4GDgN9IOjQidpalWjOzLhQR3H///Rx22GG7tC9ZsoQBAwZ8Mt+nT59P5vv06cOOHTsAOO+883jwwQc54ogjmDt3Lo8//vgu2/n444/Zb7/9eO6558pee5vDMhHxBPBOi+Z/AK6PiI+yPhuz9knA3RHxUUS8CqwHji5jvWZmXeaUU07hpptuIiIAWLFiRVHrf/DBBwwbNozt27czf/783Zbvs88+1NbWcu+99wK5D5OVK1eWXjgdH3M/FPgLSUsk/R9JR2Xtw4E38vo1Zm1mZj3O1Vdfzfbt2xk/fjxjx47l6quvLmr9a6+9lmOOOYYvfvGLrX5JOn/+fObMmcMRRxzB4YcfzsKFC8tROmr+RNpjJ6kG+FXzmLukNcB/ADOAo4BfAJ8hN3zzdET8POs3B3goIu4vsM16oB5gxIgRn3vttVZvS9x9+FTI8vKpkFbA2rVrGT16dKXL6HYK/V4kLY+IukL9O3rk3gj8MnKWAh8DQ7P2g/P6VQNvFtpARMyOiLqIqKuqKvggETMz66COhvuDwEkAkg4F9gLeBhYBkyUNkFQLjASWlqNQMzNrv/acCnkXcAIwVFIjcA1wO3B7NjyzDZgaufGd5yXdA7xA7hTJ6T5Txsys67UZ7hFxbiuLvtpK/+uA60opyszMSuMrVM3MEuRwNzNLkMPdzKxMHn/8cU4//fRKlwH4rpBm1t2V+/qSXnJ9hY/czczybNiwgVGjRvHNb36TsWPHMmXKFH7zm98wceJERo4cydKlS1m6dCnHH388EyZM4Pjjj+ell17abTtbtmzhG9/4BkcddRQTJkwo25Wn7eVwNzNrYf369cyYMYNVq1bx4osvsmDBAp588klmzZrFD37wA0aNGsUTTzzBihUr+P73v8/3vve93bZx3XXXtXq74K7gYRkzsxZqa2sZN24cAIcffjgnn3wykhg3bhwbNmxg8+bNTJ06lXXr1iGJ7du377aN1m4X3FW3VnC4m5m10NbtfK+++mpOPPFEHnjgATZs2MAJJ5yw2zZau11wV/GwjJlZkTZv3szw4bkb3s6dO7dgn1JvF1wqh7uZWZEuu+wyrrjiCiZOnMjOnYXvsFLq7YJL1a5b/na2urq6aGhoqHQZbfMtf8url5ySZsXxLX8L66pb/pqZWTfmcDczS5DD3cwsQQ53M+t2usN3gd1JR34fDncz61YGDhzIpk2bHPCZiGDTpk0MHDiwqPXa8ySm24HTgY3ND8jOW/aPwI+Aqoh4W5KAnwB/DfwROC8ini2qIjPr1aqrq2lsbKSpqanSpXQbAwcOpLq6uqh12nOF6lzgZuCO/EZJBwNfBF7Paz6N3HNTRwLHALdmr2Zm7dK/f39qa2srXUaP1+awTEQ8AbxTYNENwGVA/t9Ok4A7IucZYD9Jw8pSqZmZtVuHxtwlnQn8LiJWtlg0HHgjb74xazMzsy5U9I3DJA0CrgS+VGhxgbaC34pIqgfqAUaMGFFsGWZmtgcdOXL/LFALrJS0AagGnpX0p+SO1A/O61sNvFloIxExOyLqIqKuqqqqA2WYmVlrig73iFgdEQdGRE1E1JAL9CMj4vfAIuDryjkW2BwRb5W3ZDMza0ub4S7pLuBp4DBJjZKm7aH7Q8ArwHrgNuCCslRpZmZFaXPMPSLObWN5Td50ANNLL8vMzErhK1TNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEOdzNzBLkcDczS5DD3cwsQQ53M7MEtedJTLdL2ihpTV7bjyS9KGmVpAck7Ze37ApJ6yW9JOmUzirczMxa154j97nAqS3aHgXGRsR44P8BVwBIGgNMBg7P1rlFUt+yVWtmZu3SZrhHxBPAOy3aHomIHdnsM0B1Nj0JuDsiPoqIV8k9S/XoMtZrZmbtUI4x928AD2fTw4E38pY1Zm1mZtaFSgp3SVcCO4D5zU0FukUr69ZLapDU0NTUVEoZZmbWQofDXdJU4HRgSkQ0B3gjcHBet2rgzULrR8TsiKiLiLqqqqqOlmFmZgV0KNwlnQp8FzgzIv6Yt2gRMFnSAEm1wEhgaellmplZMfq11UHSXcAJwFBJjcA15M6OGQA8KgngmYg4PyKel3QP8AK54ZrpEbGzs4o3M7PC2gz3iDi3QPOcPfS/DriulKLMzKw0vkLVzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLUJvhLul2SRslrclrGyLpUUnrstf9s3ZJulHSekmrJB3ZmcWbmVlh7Tlynwuc2qLtcuCxiBgJPJbNA5xG7rmpI4F64NbylGlmZsVoM9wj4gngnRbNk4B52fQ84Ky89jsi5xlgP0nDylWsmZm1T0fH3D8dEW8BZK8HZu3DgTfy+jVmbWZm1oXK/YWqCrRFwY5SvaQGSQ1NTU1lLsPMrHfraLj/oXm4JXvdmLU3Agfn9asG3iy0gYiYHRF1EVFXVVXVwTLMzKyQjob7ImBqNj0VWJjX/vXsrJljgc3NwzdmZtZ1+rXVQdJdwAnAUEmNwDXA9cA9kqYBrwPnZN0fAv4aWA/8Efi7TqjZzMza0Ga4R8S5rSw6uUDfAKaXWpSZmZXGV6iamSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJcribmSXI4W5mliCHu5lZghzuZmYJKincJX1H0vOS1ki6S9JASbWSlkhaJ+kXkvYqV7FmZtY+HQ53ScOBi4C6iBgL9AUmAz8EboiIkcC7wLRyFGpmZu1X6rBMP+BPJPUDBgFvAScB92XL5wFnlbgPMzMrUofDPSJ+B8wi94Dst4DNwHLgvYjYkXVrBIaXWqSZmRWnlGGZ/YFJQC1wELA3cFqBrtHK+vWSGiQ1NDU1dbQMMzMroJRhmS8Ar0ZEU0RsB34JHA/slw3TAFQDbxZaOSJmR0RdRNRVVVWVUIaZmbVUSri/DhwraZAkAScDLwCLgS9nfaYCC0sr0czMilXKmPsScl+cPguszrY1G/gucLGk9cABwJwy1GlmZkXo13aX1kXENcA1LZpfAY4uZbtmZlYaX6FqZpYgh7uZWYIc7mZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCSrpfu69Tc3WBZUuISkbKl2AWcJKOnKXtJ+k+yS9KGmtpOMkDZH0qKR12ev+5SrWzMzap9RhmZ8A/xYRo4AjgLXA5cBjETESeCybNzOzLtThcJe0D/CXZM9IjYhtEfEeMAmYl3WbB5xVapFmZlacUo7cPwM0Af8qaYWkn0naG/h0RLwFkL0eWIY6zcysCKWEez/gSODWiJgAbKGIIRhJ9ZIaJDU0NTWVUIaZmbVUSrg3Ao0RsSSbv49c2P9B0jCA7HVjoZUjYnZE1EVEXVVVVQllmJlZSx0O94j4PfCGpMOyppOBF4BFwNSsbSqwsKQKzcysaKWe534hMF/SXsArwN+R+8C4R9I04HXgnBL3YWZmRSop3CPiOaCuwKKTS9mumZmVxrcfMDNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MEuRwNzNLkMPdzCxBDnczswQ53M3MElTqk5iQ1BdoAH4XEadLqgXuBoYAzwJfi4htpe7HzPZg5r6VriAtMzdXuoKSlePIfQawNm/+h8ANETESeBeYVoZ9mJlZEUoKd0nVwN8AP8vmBZwE3Jd1mQecVco+zMyseKUeuf8YuAz4OJs/AHgvInZk843A8BL3YWZmRepwuEs6HdgYEcvzmwt0jVbWr5fUIKmhqampo2WYmVkBpRy5TwTOlLSB3BeoJ5E7kt9PUvMXtdXAm4VWjojZEVEXEXVVVVUllGFmZi11ONwj4oqIqI6IGmAy8B8RMQVYDHw56zYVWFhylWZmVpTOOM/9u8DFktaTG4Of0wn7MDOzPSj5PHeAiHgceDybfgU4uhzbNTOzjvEVqmZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCXK4m5klyOFuZpYgh7uZWYIc7mZmCepwuEs6WNJiSWslPS9pRtY+RNKjktZlr/uXr1wzM2uPUo7cdwCXRMRo4FhguqQxwOXAYxExEngsmzczsy7U4XCPiLci4tls+gNgLTAcmATMy7rNA84qtUgzMytOWcbcJdUAE4AlwKcj4i3IfQAAB7ayTr2kBkkNTU1N5SjDzMwyJYe7pE8B9wPfjoj327teRMyOiLqIqKuqqiq1DDMzy1NSuEvqTy7Y50fEL7PmP0gali0fBmwsrUQzMytWKWfLCJgDrI2If8lbtAiYmk1PBRZ2vDwzM+uIfiWsOxH4GrBa0nNZ2/eA64F7JE0DXgfOKa1EMzMrVofDPSKeBNTK4pM7ul0zMyudr1A1M0uQw93MLEEOdzOzBDnczcwS5HA3M0uQw93MLEEOdzOzBDnczcwS5HA3M0uQw93MLEEOdzOzBJVy4zAz6yZqti6odAlJ2VDpAsrAR+5mZglyuJuZJcjhbmaWIIe7mVmCOi3cJZ0q6SVJ6yVd3ln7MTOz3XVKuEvqC/xP4DRgDHCupDGdsS8zM9tdZx25Hw2sj4hXImIbcDcwqZP2ZWZmLXTWee7DgTfy5huBY/I7SKoH6rPZDyW91Em19EZDgbcrXURb9MNKV2AV4H+b5fVnrS3orHAv9ODs2GUmYjYwu5P236tJaoiIukrXYdaS/212nc4almkEDs6brwbe7KR9mZlZC50V7suAkZJqJe0FTAYWddK+zMyshU4ZlomIHZK+Bfw70Be4PSKe74x9WUEe7rLuyv82u4giou1eZmbWo/gKVTOzBDnczcwS5HBPgKQBBdqGVKIWM+seHO5p+KWk/s0zkoYBj1awHrNPSPpcgbYzKlFLb+JwT8ODwL2S+kqqIXeW0hUVrcjsv9wmaVzzjKRzgasqWE+v4LNlEiFpOnAqUAP8fUQ8VdmKzHIkfQa4D5gCfB74OnB6RGyuaGGJc7j3YJIuzp8FvgasBlYARMS/VKIus5YkHUruL8w3gLMi4j8rXFLy/IDsnm1wi/kHWmk363KSVrPrPaWGkLuocYkkImJ8ZSrrHXzknhhJfYBPRcT7la7FejdJrd6xECAiXuuqWnojf6GaAEkLJO0jaW/gBeAlSZdWui7r3SLitSzA+wG/z6ZryT3bwePtnczhnoYx2ZH6WcBDwAhy4+9m3cH9wE5JhwBzyAX8gsqWlD6Hexr6Z+e5nwUsjIjttLh/vlkFfRwRO4CzgR9HxHeAYRWuKXkO9zT8L2ADsDfwRDbW6TF36y62Z+e2fx34VdbWfw/9rQz8hWqiJPXLjpbMKkrSGOB84OmIuEtSLfC3EXF9hUtLmsO9B5P01Yj4eYvz3T/h89zNei+f596z7Z29+rx263Yk3RMRXylwvruA8HnunctH7omStHdEbKl0HdZ7SRoWEW+1dr67z3PvXP5CtYeTNFxSXfasWiQdKOkHwLoKl2a9XES8lb2+lv8DNJK7x4x1Iod7Dybp28BzwE3AM5KmAmuBPwF2u82qWVfKLqy7QtLNkr6knAuBV4CvVLq+1HlYpgeT9ALw+Yh4R9IIYD3wlxHxTIVLM0PSQuBd4GngZGB/YC9gRkQ8V8naegOHew8m6dmIODJvfk1EjK1kTWbNJK2OiHHZdF/gbWBERHxQ2cp6B58t07NVS7oxb/7A/PmIuKgCNZk12948ERE7Jb3qYO86PnLvwbIx9lZFxLyuqsWsJUk7geYztkTuu6A/8l+nQu5Tqdp6A4e7mVmCfLaMmVmCHO5mZglyuJuZJcjhngBJ1ZIekNQk6Q+S7pdUXem6zKxyHO5p+FdgEbkHIAwH/nfWZma9lM+WSYCk5yLiz9tqM7Pew0fuaXhb0lcl9c1+vgpsqnRRZlY5PnJPQHZfmZuB48jdN/spcvfv8C1VzXoph7uZWYJ8b5keTNI/7WFxRMS1XVaMmXUrPnLvwSRdUqB5b2AacEBEfKqLSzKzbsLhnghJg4EZ5IL9HuB/RMTGylZlZpXiYZkeTtIQ4GJgCjAPODIi3q1sVWZWaQ73HkzSj4CzgdnAuIj4sMIlmVk34WGZHkzSx8BHwA5yp0B+sgjfL9usV3O4m5klyFeompklyOFuZpYgh7uZWYIc7pY0SR9mrwdJuq+Nvt+WNKjI7Z8g6VfZ9HmSbi5i3Q2ShhbRv6jtW+/mcLceR1LfYteJiDcj4sttdPs2UFS4m3VXDnfrViTVSHpR0jxJqyTdJ2lQdpT7T5KeBM6R9FlJ/yZpuaT/K2lUtn6tpKclLZN0bYvtrsmm+0qaJWl1to8LJV0EHAQslrQ46/elbFvPSrpX0qey9lOzGp8kd51BW+/pVkkNkp6X9M8tFl8qaWn2c0jWvyp7mtay7GdiGX611ss43K07OgyYHRHjgfeBC7L2rRHx+Yi4m9yFWxdGxOeAfwRuyfr8BLg1Io4Cft/K9uuBWmBCto/5EXEj8CZwYkScmA2XXAV8ISKOBBqAiyUNBG4DzgD+AvjTdryfKyOiDhgP/JWk8XnL3o+Io8ndsvnHee/hhuw9/DfgZ+3Yh9kufIWqdUdvRMRvs+mfAxdl078AyI6gjwfuldS8zoDsdSK5QAS4E/hhge1/AfhpROwAiIh3CvQ5FhgD/Dbbx17A08Ao4NWIWJfV8nNyHxZ78hVJ9eT+vw3LtrsqW3ZX3usNefWNyXtv+2T3DjJrN4e7dUctr6xrnt+SvfYB3tvDYwTbujJP7ezzaEScu0uj9OftWDe/fy25vyyOioh3Jc0FBrZSa/N0H+C4iPjPFttq727NPCxj3dIIScdl0+cCT+YvjIj3gVclnQOgnCOyxb8FJmfTU1rZ/iPA+ZL6ZesPydo/AJqPkJ8BJuaNgw+SdCjwIlAr6bN59e3JPuQ+lDZL+jRwWovlf5v3+nRefd9q7pB9oJgVxeFu3dFaYKqkVcAQ4NYCfaYA0yStBJ4HJmXtM4DpkpYB+7ay/Z8BrwOrsvX/e9Y+G3hY0uKIaALOA+7K6ngGGBURW8kNw/w6+0K15aMMz5PU2PxD7lm2K7Iabyf34ZNvgKQlWd3fydouAuqyL3tfAM5v5X2Ytcr3lrFuRVIN8KuIGFvhUsx6NB+5m5klyEfuZmYJ8pG7mVmCHO5mZglyuJuZJcjhbmaWIIe7mVmCHO5mZgn6/367GC3+aWffAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "subscription.payload_logging.data_distribution.show_chart(sex_distribution_run_uid);" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "===============================================================================\n", "\n", " Waiting for end of data distribution run 2e76fd24-cffd-4309-8a49-bcd283879fc5 \n", "\n", "===============================================================================\n", "\n", "\n", "\n", "completed\n", "\n", "---------------------------\n", " Successfully finished run \n", "---------------------------\n", "\n", "\n" ] } ], "source": [ "credit_history_distribution = subscription.payload_logging.data_distribution.run(\n", " start_date=start_date,\n", " end_date=end_date,\n", " group=['predictedLabel', 'CreditHistory'],\n", " agg=['count'])" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEiCAYAAAACg5K6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhUVbb38e8CgxCRQcB7UdSgjTJkAhKIQDPagICIDI0CmoAIKApqO+AEiFcvKu1AO11QBhUUBQFflO5WAZFBhmiYRFQ0IsJVDAaZogz7/aMqdUOojJVQ5OT3eR6e1Nlnn3NWxbhysmuftc05h4iIeEuFcAcgIiIlT8ldRMSDlNxFRDxIyV1ExIOU3EVEPEjJXUTEg84IdwAAtWvXdlFRUeEOQ0SkTElNTf3FOVcn2L7TIrlHRUWxfv36cIchIlKmmNn3ee3TsIyIiAcpuYuIeJCSu4iIB50WY+4iZcWRI0fYuXMnWVlZ4Q5FypHKlStTr149IiIiCn2MkrtIEezcuZOzzz6bqKgozCzc4Ug54JwjIyODnTt3Ur9+/UIfp2EZkSLIysqiVq1aSuxyypgZtWrVKvJfi0ruIkWkxC6nWnF+5pTcRUQ8SGPuRfD8iCXhDsFTRr7UMdwhhCxqzHsler70id2LfWz2w4C1a9ematWqHDhwoAQjK9x1W7VqxapVq0hPT2fVqlUMGDCg1K8/duxY2rZtyxVXXHFC+7Jly5g0aRKLFi0Ketz48eOpWrUqd911V0jXT0lJoUePHvTt2zek85Q03bmLyEmOHj1arONWrVoFQHp6OrNnzy7JkPI0YcKEkxK7KLmLlEm9evWiefPmNGnShClTphT5+CeeeIKYmBji4uIYM2YMAO3bt+f++++nXbt2PPvss+zZs4c+ffqQmJhIYmIiK1euBCAjI4POnTvTtGlThg8fTs6lOqtWrQrAmDFj+OSTT4iPj+fpp59my5YttGjRgvj4eGJjY/n666+DxpWenk7Dhg1JTk4mNjaWvn37cujQIcCXxBMTE4mOjmbYsGGB66akpDB37lwA/vnPf9KwYUPatGnDO++8U+D3YcOGDXTs2JEGDRowdepUAA4cOECnTp1o1qwZMTExLFy4MND/1VdfJTY2lri4OK6//vqTzvfQQw+RkpLC8ePHC7x2aStwWMbMpgE9gJ+dc9E52m8DbgWOAu855+7xt98H3AgcA0Y55/5VGoGLlGfTpk3jnHPO4fDhwyQmJtKnT59CH7t48WIWLFjAmjVriIyMZO/evYF9mZmZfPzxxwAMGDCAO+64gzZt2rBjxw66dOnC1q1befjhh2nTpg1jx47lvffeC/rLZeLEiScMidx2222MHj2agQMH8scff3Ds2LE849u2bRuvvPIKrVu3ZsiQIbzwwgvcdddd3HrrrYwdOxaA66+/nkWLFnHVVVcFjsvKyuKmm25iyZIl/OlPf6J///4Ffi82btzIp59+ysGDB2natCndu3fn3HPPZf78+VSrVo1ffvmFpKQkevbsyRdffMGjjz7KypUrqV279gnfN4B77rmHffv2MX369NPiQ/fC3LnPALrmbDCzDsDVQKxzrgkwyd/eGLgWaOI/5gUzq1iSAYsITJ48mbi4OJKSkvjhhx/yvBMO5sMPP2Tw4MFERkYCcM455wT25UyIH374Ibfeeivx8fH07NmT3377jf3797N8+XIGDRoEQPfu3alZs2aB17z88st57LHHePzxx/n++++pUqVKnn0vuOACWrduDcCgQYNYsWIFAEuXLqVly5bExMSwZMkStmzZcsJxX375JfXr16dBgwaYWSDG/Fx99dVUqVKF2rVr06FDB9auXYtzjvvvv5/Y2FiuuOIKfvzxR3766SeWLFlC3759qV279knft0ceeYTMzEz+53/+57RI7FCIO3fn3HIzi8rVfDMw0Tn3u7/Pz/72q4E3/e3fmdk3QAtgdYlFLFLOLVu2jA8//JDVq1cTGRlJ+/btizQH2jmXZwI666yzAq+PHz/O6tWrgybioiawAQMG0LJlS9577z26dOnCyy+/TMeOwT9Qz31uMyMrK4tbbrmF9evXc8EFFzB+/Pig77mocQW71qxZs9izZw+pqalEREQQFRVFVlZWvt+3xMREUlNT2bt37wlJP5yKO+Z+KfBnM1tjZh+bWaK//Xzghxz9dvrbRKSE7Nu3j5o1axIZGcmXX37Jp59+WqTjO3fuzLRp0wJj2bmHF3L2e+655wLbaWlpALRt25ZZs2YBviGeX3/99aRjzz77bPbv3x/Y/vbbb7n44osZNWoUPXv2ZOPGjXnGt2PHDlav9t0PvvHGG7Rp0yaQyGvXrs2BAwcCY+w5NWzYkO+++47t27cHji3IwoULycrKIiMjg2XLlpGYmMi+ffs499xziYiIYOnSpXz/va+qbqdOnXjrrbfIyMgATvy+de3alTFjxtC9e/cT3nc4FXcq5BlATSAJSATeMrOLgWC/1lyQNsxsGDAM4MILLyxmGCLhFcrUxeLq2rUrL730ErGxsVx22WUkJSUV+fi0tDQSEhKoVKkS3bp147HHHjup3+TJkxk5ciSxsbEcPXqUtm3b8tJLLzFu3Diuu+46mjVrRrt27YL+/xsbG8sZZ5xBXFwcKSkpZGVl8frrrxMREcF//ud/BsbOg2nUqBEzZ85k+PDhNGjQgJtvvpnIyEhuuukmYmJiiIqKIjEx8aTjKleuzJQpU+jevTu1a9emTZs2bN68Od/vRYsWLejevTs7duzgoYce4rzzzmPgwIFcddVVJCQkEB8fT8OGDQFo0qQJDzzwAO3ataNixYo0bdqUGTNmBM7Vr18/9u/fT8+ePXn//ffzHXo6FSznJ915dvINyyzK/kDVzP6Jb1hmmX97O75EPxTAOfff/vZ/AeOdc/kOyyQkJLiysFiH5rmXrLI4z33r1q00atQo3GF4Vnp6Oj169CgwKZdHwX72zCzVOZcQrH9xh2UWAB39J78UqAT8ArwLXGtmZ5pZfaABsLaY1xARkWIqzFTIN4D2QG0z2wmMA6YB08xsM/AHkOx8fwJsMbO3gC/wTZEc6ZzLe86TiJSaTZs2nTQX+8wzz2TNmjVhiuj/ZGRk0KlTp5PaP/rooxK/a58+fTrPPvvsCW2tW7fm+eefL9HrnG4KNSxT2jQsUz5pWEak8E7VsIyIiJzGlNxFRDxIyV1ExIOU3EVEPEj13EVCMb56CZ9vX8merwDt27dn0qRJJCQk0K1bt0CZ3tmzZ3PLLbeU+vVfeuklIiMjueGGG05oL2i+e1paGrt27aJbt24hx5CzHn1eHnvsMe6///6Qr3Uq6c5dxGOKW4v9/fffp0aNGmRmZvLCCy+UcFTBjRgx4qTEXhhpaWm8//77pRBRcMGe4D3dKbmLlEG564qnpKRw55130qFDB+69914OHjzIkCFDSExMpGnTpoGa5IcPH+baa68lNjaW/v37c/jw4cA5o6Ki+OWXXxgzZgzbt28nPj6eu+++m927d9O2bVvi4+OJjo7mk08+yTOuqlWr8re//Y1mzZrRqVMn9uzZA8DUqVNJTEwkLi6OPn36BOrajB8/nkmTJgGQmppKXFwcl19+eb5z0P/44w/Gjh3LnDlziI+PZ86cOezdu5devXoRGxtLUlJSvrVr8qtHH6xO/pgxYzh8+DDx8fEMHDgQgNdffz1Qn3748OH5ljAOFyV3kTJmy5YtPProoyxZsoQNGzYEHtD56quv+PDDD/n73//Oo48+SseOHVm3bh1Lly7l7rvv5uDBg7z44otERkayceNGHnjgAVJTU086/8SJE7nkkktIS0vjySefZPbs2XTp0oW0tDQ2bNhAfHx8nrEdPHiQZs2a8dlnn9GuXTsefvhhAHr37s26devYsGEDjRo14pVXXjnp2MGDBzN58uRA0bC8VKpUiQkTJtC/f3/S0tLo378/48aNo2nTpmzcuJHHHnss378GsuvRf/755/Ts2ZMdO3YE9k2bNo3U1FTWr1/P5MmTycjIYOLEiVSpUoW0tDRmzZrF1q1bmTNnDitXriQtLY2KFSsGCqmdTjTmLlLG5FVXvF+/flSs6Fs+4d///jfvvvtu4K44KyuLHTt2sHz5ckaNGgX4invFxsYWeL3ExESGDBnCkSNH6NWrV77JvUKFCoGa8IMGDaJ3794AbN68mQcffJDMzEwOHDhAly5dTjhu3759ZGZm0q5dO8C3GMfixYsL/T1ZsWIF8+bNA6Bjx45kZGSwb98+qlc/+TOR5cuXB1Zpyl2PfvLkycyfPx8gUCe/Vq1aJxz/0UcfkZqaGihedvjwYc4999xCx3qqKLmLlDF51RXPWYvdOce8efO47LLLTupX1Jrnbdu2Zfny5bz33ntcf/313H333YUeJ8++VkpKCgsWLCAuLo4ZM2awbNmyE/rlVyu9MII9aZ/f+YLtK2ydfOccycnJ/Pd//3ex4z0VNCwjUsbkV1c8W5cuXfjHP/4RSHqff/45cGIt9s2bNwcdm85di/3777/n3HPP5aabbuLGG2/ks88+yzO248ePB2qtz549mzZt2gCwf/9+6taty5EjR4IOYdSoUYPq1asHVl0qaJgjd4w539eyZcuoXbs21apVC3psXvXo86uTHxERwZEjRwDf93/u3Ln8/LNvjaK9e/cGar6fTnTnLhKKUzx1EYLXFc/toYce4vbbbyc2NhbnHFFRUSxatIibb76ZwYMHExsbS3x8PC1atDjp2Fq1atG6dWuio6O58soriY6O5sknnyQiIoKqVavy6quv5hnbWWedxZYtW2jevDnVq1dnzpw5gG8ZupYtW3LRRRcRExMTdEGL6dOnM2TIECIjI08atsmtQ4cOTJw4kfj4eO677z7Gjx8feF+RkZHMnDkzz2PzqkefX538YcOGERsbS7NmzZg1axb/9V//RefOnTl+/DgRERE8//zzXHTRRfnGfKqpcFgRqHBYyVLhMO+pWrUqBw4cCHcYnqTCYSIiomEZESm6li1b8vvvv5/Q9tprr5X4Xfu//vUv7r333hPa6tevH5jRkp/yWsc9m5K7iBTZqVrwo0uXLgWOv+dl8ODBDB48uIQjKjsKHJYxs2lm9rN/1aXc++4yM2dmtf3bZmaTzewbM9toZs1KI2gREclfYcbcZwBdczea2QXAX4AdOZqvxLduagNgGPBi6CGKiEhRFZjcnXPLgZMn0sLTwD1Azuk2VwOvOp9PgRpmVrdEIhURkUIr1mwZM+sJ/Oic25Br1/nADzm2d/rbRETkFCryB6pmFgk8AHQOtjtIW9CJ9GY2DN/QTeAhApGyJmZmTImeb1PyphI936kwY8YM1q9fz3PPPXdCffYZM2bQuXNnzjvvvHCHWC4VZ7bMJUB9YIO/PkM94DMza4HvTv2CHH3rAbuCncQ5NwWYAr6HmIoRh4iUkmPHjgWKkBXFiBEjAq9nzJhBdHS0knuYFHlYxjm3yTl3rnMuyjkXhS+hN3PO/S/wLnCDf9ZMErDPObe7ZEMWKd/S09Np1KgRN910E02aNKFz584cPnyYtLQ0kpKSiI2N5ZprrgnUTAnmm2++4YorriAuLo5mzZqxfft2li1bRocOHRgwYAAxMb6/SPKqWz59+nQuvfRS2rVrx8qVKwPnza7PPnfuXNavX8/AgQOJj4/n8OHDjBkzhsaNGxMbG8tdd91Vut8kKdRUyDeA1cBlZrbTzG7Mp/v7wLfAN8BUoPTX6RIph77++mtGjhzJli1bqFGjBvPmzeOGG27g8ccfZ+PGjcTExARqqQczcOBARo4cyYYNG1i1ahV16/rmPaxdu5ZHH32UL774Is+65bt372bcuHGsXLmSDz74gC+++OKk8/ft25eEhARmzZpFWloahw8fZv78+WzZsoWNGzfy4IMPltr3RnwKHJZxzl1XwP6oHK8dMDL0sEQkP/Xr1w/UVW/evDnbt28/oR56cnIy/fr1C3rs/v37+fHHH7nmmmsAqFy5cmBfixYtqF+/PpB33fI1a9bQvn176tSpA0D//v356quv8o23WrVqVK5cmaFDh9K9e3d69OgRwruXwlBtGZEy6Mwzzwy8rlixIpmZmYU+Nr9igblrwicnJ5OWlkZaWhrbtm1j/PjxQNFrwp9xxhmsXbuWPn36sGDBArp2PenRGSlhSu4iHlC9enVq1qwZWN/0tddeC9zF51atWjXq1avHggULAPj9998Da5rmlFfd8pYtW7Js2TIyMjI4cuQIb7/9dtDr5Ky5fuDAAfbt20e3bt145plnSEtLC/k9S/5UW0YkBKfT1MWZM2cyYsQIDh06xMUXX8z06dPz7Pvaa68xfPhwxo4dS0RERNAE3bhx46B1y5OSkhg/fjyXX345devWpVmzZkEXiE5JSWHEiBFUqVKFxYsXc/XVV5OVlYVzjqeffrpE37ucTPXci0D13EuW6rmLFJ7quYuIiIZlRLxs5MiRJ8xDBxg9enS5LoVbXii5i3hYeVmYQk6mYRkREQ9SchcR8SAldxERD1JyFxHxIH2gKhKCrQ1Lds57oy+3luj5gELVVV+wYAGXXnopjRs3LpFrpqSk0KNHD/r27cvQoUO58847S+zcJS0qKor169dTu3ZtWrVqxapVq0hPT2fVqlUMGDAg3OEVm+7cRTxuxowZ7NoVdFmFgAULFgSt7lgSXn755VOe2I8ePVqs41atWgX4yirPnj27JEM65ZTcRcqgp556iujoaKKjo3nmmWdIT08nOjo6sH/SpEmMHz++UHXVV61axbvvvsvdd99NfHw827dvZ+rUqSQmJhIXF0efPn0CtWdSUlIYNWoUrVq14uKLL2bu3LmAr8jYrbfeSuPGjenevXugHg1A+/btyX4CvWrVqjzwwAPExcWRlJTETz/9BMD27dtJSkoiMTGRsWPHUrVq1Xzf/xNPPEFMTAxxcXGMGTMmcJ3777+fdu3a8eyzz7Jnzx769OlDYmIiiYmJgfn+GRkZdO7cmaZNmzJ8+PATCqllX3fMmDF88sknxMfHl9lSCUruImVMamoq06dPZ82aNXz66adMnTo1z4U5ClNXvVWrVvTs2ZMnn3yStLQ0LrnkEnr37s26devYsGEDjRo14pVXXgmcc/fu3axYsYJFixYFEuv8+fPZtm0bmzZtYurUqYE74NwOHjxIUlISGzZsoG3btkydOhXwPVg1evRo1q1bV+DKTYsXL2bBggWsWbOGDRs2cM899wT2ZWZm8vHHH/O3v/2N0aNHc8cdd7Bu3TrmzZvH0KFDAXj44Ydp06YNn3/+OT179mTHjh0nXWPixIn8+c9/Ji0tjTvuuCPfeE5XGnMXKWNWrFjBNddcEyjP27t370A1yIIUtq765s2befDBB8nMzOTAgQN06dIlsK9Xr15UqFCBxo0bB+68ly9fznXXXUfFihU577zz6NgxeN2gSpUqBa7ZvHlzPvjgAwBWr14dqFI5YMCAfFdq+vDDDxk8eDCRkZEAnHPOOYF9/fv3P6FfzqGm3377jf3797N8+XLeeecdALp3707NmjXzvFZZVpiVmKaZ2c9mtjlH25Nm9qWZbTSz+WZWI8e++8zsGzPbZmZdgp9VRIorWLG/zMxMjh8/HtjOysoKemxh66qnpKTw3HPPsWnTJsaNG3fC+XLWks8ZS2FqvEdERAT6VaxYsVhj4865PK+Vsx798ePHWb16daAe/Y8//sjZZ59d6FjLusIMy8wAcv8EfABEO+diga+A+wDMrDFwLdDEf8wLZlb0VXZFJE9t27ZlwYIFHDp0iIMHDzJ//nyuvPJKfv75ZzIyMvj9999ZtGhRoH9h6qrn7AO+1Zrq1q3LkSNHmDVrVqFievPNNzl27Bi7d+9m6dKlRXpPSUlJzJs3D4A333wz376dO3dm2rRpgc8B9u7dm2e/5557LrCd/V7btm0beE+LFy8OOqSV+/tRFhVmmb3lZhaVq+3fOTY/Bfr6X18NvOmc+x34zsy+AVrgW4NVxHNKY+piQZo1a0ZKSgotWrQAYOjQoYEPIlu2bEn9+vVp2LBhoH9h6qpfe+213HTTTUyePJm5c+fyyCOP0LJlSy666CJiYmIKTHTXXHMNS5YsISYmJrBwdlE888wzDBo0iL///e90796d6tWr59m3a9eupKWlkZCQQKVKlejWrRuPPfbYSf0mT57MyJEjiY2N5ejRo7Rt25aXXnqJcePGcd1119GsWTPatWvHhRdeeNKxsbGxnHHGGcTFxZGSklImx90LVc/dn9wXOeeig+z7f8Ac59zrZvYc8Klz7nX/vleAxc65ufmdX/XcyyfVc5dshw4dokqVKpgZb775Jm+88QYLFy4Md1inlaLWcw/pA1UzewA4CmT/3RZsICvobw8zGwYMA4L+5hSR8iM1NZVbb70V5xw1atRg2rRp4Q6pzCt2cjezZKAH0Mn93+3/TuCCHN3qAUGfnnDOTQGmgO/OvbhxiEjZ9+c//5kNGzac0LZp0yauv/76E9rOPPNM1qxZcypDK7OKldzNrCtwL9DOOZdzZd13gdlm9hRwHtAAWBtylCJS7sTExGgh7RAUmNzN7A2gPVDbzHYC4/DNjjkT+MA/pehT59wI59wWM3sL+ALfcM1I59zJK+eKiEipKsxsmeuCNL8SpC27/6PAo6EEJSIioVH5ARERD1JyFxHxINWWEQlBST/7UJJz/8eOHUvbtm254oorSuycp6O0tDR27dpFt27dQjpPzhr0ofQJRXp6Oj169GDz5s0Fdy6AkruIBx07dowJEyYU+ZiKFctetZC0tDTWr18fcnL3Gg3LiJQx6enpNGzYkOTkZGJjY+nbty+HDh0iKiqKCRMm0KZNG95++21SUlIC9dY/+ugjmjZtSkxMDEOGDOH3338HOOmYYNq3b8/tt99Oq1atiI6OZu1a3+zmtWvX0qpVK5o2bUqrVq3Ytm0bQKBUbrbWrVuzceNGxo8fT3JyMp07dyYqKop33nmHe+65h5iYGLp27cqRI0cA3wNN7dq1o3nz5nTp0oXdu3cH4rj33ntp0aIFl156KZ988gl//PEHY8eOZc6cOcTHxzNnzhw+/vhj4uPjiY+Pp2nTpnmWTsivBn1eMeQ0YcIEEhMTiY6OZtiwYTjn2L59O82aNQv0+frrr2nevHm+50xNTSUuLo7LL7+c559/vqD//IWm5C5SBm3bto1hw4axceNGqlWrxgsvvABA5cqVWbFiBddee22gb1ZWFikpKcyZM4dNmzZx9OhRXnzxxcD+YMfkdvDgQVatWsULL7zAkCFDAGjYsCHLly/n888/Z8KECdx///2Ar9bNjBkzAPjqq6/4/fffiY2NBXyLcrz33nssXLiQQYMG0aFDBzZt2kSVKlV47733OHLkCLfddhtz584lNTWVIUOG8MADDwTiOHr0KGvXruWZZ57h4YcfplKlSkyYMIH+/fuTlpZG//79mTRpEs8//zxpaWl88sknVKlSJeh7yqsGfUExZLv11ltZt24dmzdv5vDhwyxatIhLLrmE6tWrB365TZ8+nZSUlHzPOXjwYCZPnszq1SVbgkvDMiJl0AUXXEDr1q0BGDRoEJMnTwZOrGeebdu2bdSvX59LL70UgOTkZJ5//nluv/32PI/J7brrfDOi27Zty2+//UZmZib79+8nOTmZr7/+GjML3Hn369ePRx55hCeffJJp06aRkpISOM+VV15JREQEMTExHDt2LFByOCYmhvT0dLZt28bmzZv5y1/+AviGiurWrRs4vnfv3oCvFnx6enrQWFu3bs2dd97JwIED6d27N/Xq1QvaL68a9AXFkG3p0qU88cQTHDp0iL1799KkSROuuuoqhg4dyvTp03nqqaeYM2cOa9euzfOc+/btIzMzM1Bo7frrr2fx4sUF/vcoDCV3kTIodz3y7O2c9cyzFVQcMNgxhbneQw89RIcOHZg/fz7p6em0b98egMjISP7yl7+wcOFC3nrrLXIWBcyuBV+hQoUTartXqFCBo0eP4pyjSZMmed7FZh+fXy34MWPG0L17d95//32SkpL48MMPT6iSmd/7AgqMAXx/Dd1yyy2sX7+eCy64gPHjxwdq3vfp04eHH36Yjh070rx5c2rVqsWuXbuCnjMzM7PUastrWEakDNqxY0cgUbzxxhu0adMmz74NGzYkPT2db775BoDXXnutyCV558yZA/hWgapevTrVq1dn3759nH/++QCBYZhsQ4cOZdSoUSQmJp6wUlJBLrvsMvbs2RN4b0eOHGHLli35HpO79vr27duJiYnh3nvvJSEhgS+//DLocXnVoC9MDNmJvHbt2hw4cCDw2Qb4hrm6dOnCzTffzODBg/M9Z40aNahevTorVqwAKFTt/MLSnbtICMJVtrhRo0bMnDmT4cOH06BBA26++Wb+8Y9/BO1buXJlpk+fTr9+/Th69CiJiYmMGDGiSNerWbMmrVq14rfffgtUbLznnntITk7mqaeeOmlZvebNm1OtWrVAciusSpUqMXfuXEaNGsW+ffs4evQot99+O02aNMnzmA4dOjBx4kTi4+O57777WLFiBUuXLqVixYo0btyYK6+8MuhxedWgL0wMNWrU4KabbiImJoaoqCgSExNPOPfAgQN555136Ny5c4HnnD59OkOGDCEyMvKE5QxDVah67qVN9dzLJ9VzL56SnAtdGO3bt2fSpEkkJAQtGx7Url27aN++PV9++SUVKpS/AYJJkyaxb98+HnnkkRI75ymt5y4ikturr77KAw88wFNPPVUuE/s111zD9u3bWbIkvDeDSu4iZUxUVFSp3LWPHDmSlStXntA2evRoli1bVqTz3HDDDdxwww0lGFloTnVd+Pnz55fKeYtKyV1EAEr0AZrTSXmtC1/+/mYSCdHp8DmVlC/F+ZlTchcpgsqVK5ORkaEEL6eMc46MjAwqV65cpOMKsxLTNHxrpf7snIv2t50DzAGigHTgr865X803G/9ZoBtwCEhxzn1WpIhETmP16tVj586d7NmzJ9yhSDlSuXLlPJ+0zUthxtxnAM8Br+ZoGwN85JybaGZj/Nv3AlfiWze1AdASeNH/VcQTIiIiqF+/frjDEClQgcMyzrnlwN5czVcDM/2vZzL2EZkAAA9ZSURBVAK9crS/6nw+BWqY2clFGUREpFQVd8z9P5xzuwH8X8/1t58P/JCj305/m4iInEIlPRUyWAWcoJ88mdkwYBjAhRdeWMJhlI6Oy0aGOwSP2RruAEQ8q7h37j9lD7f4v2ZXud8JXJCjXz1gV7ATOOemOOcSnHMJderUKWYYIiISTHGT+7tAsv91MrAwR/sN5pME7MsevhERkVOnMFMh3wDaA7XNbCcwDpgIvGVmNwI7gH7+7u/jmwb5Db6pkEUrCSciIiWiwOTunLsuj12dgvR1gAamRUTCTE+oioh4kJK7iIgHKbmLiHiQkruIiAcpuYuIeJCSu4iIB2klpiL46336dpWkTeEOQMTDdOcuIuJBSu4iIh6k5C4i4kFK7iIiHqTkLiLiQUruIiIepOQuIuJBSu4iIh6k5C4i4kEhJXczu8PMtpjZZjN7w8wqm1l9M1tjZl+b2Rwzq1RSwYqISOEUO7mb2fnAKCDBORcNVASuBR4HnnbONQB+BW4siUBFRKTwQh2WOQOoYmZnAJHAbqAjMNe/fybQK8RriIhIERU7uTvnfgQm4VsgezewD0gFMp1zR/3ddgLnhxqkiIgUTbHLHJpZTeBqoD6QCbwNXBmkq8vj+GHAMIALL7ywuGGcUpu+2xHuEERECiWUYZkrgO+cc3ucc0eAd4BWQA3/MA1APWBXsIOdc1OccwnOuYQ6deqEEIaIiOQWSnLfASSZWaSZGdAJ+AJYCvT190kGFoYWooiIFFUoY+5r8H1w+hm+dRcqAFOAe4E7zewboBbwSgnEKSIiRRDS0kLOuXHAuFzN3wItQjmviIiERk+oioh4kJK7iIgHKbmLiHiQkruIiAcpuYuIeJCSu4iIBym5i4h4kJK7iIgHKbmLiHiQkruIiAcpuYuIeJCSu4iIBym5i4h4kJK7iIgHKbmLiHiQkruIiAeFlNzNrIaZzTWzL81sq5ldbmbnmNkHZva1/2vNkgpWREQKJ9Q792eBfzrnGgJxwFZgDPCRc64B8JF/W0RETqFiJ3czqwa0xb9GqnPuD+dcJnA1MNPfbSbQK9QgRUSkaEK5c78Y2ANMN7PPzexlMzsL+A/n3G4A/9dzSyBOEREpglCS+xlAM+BF51xT4CBFGIIxs2Fmtt7M1u/ZsyeEMEREJLdQkvtOYKdzbo1/ey6+ZP+TmdUF8H/9OdjBzrkpzrkE51xCnTp1QghDRERyK3Zyd879L/CDmV3mb+oEfAG8CyT725KBhSFFKCIiRXZGiMffBswys0rAt8BgfL8w3jKzG4EdQL8QryEiIkUUUnJ3zqUBCUF2dQrlvCIiEho9oSoi4kFK7iIiHqTkLiLiQUruIiIepOQuIuJBSu4iIh6k5C4i4kFK7iIiHqTkLiLiQUruIiIepOQuIuJBSu4iIh6k5C4i4kFK7iIiHqTkLiLiQUruIiIeFOpKTJhZRWA98KNzroeZ1QfeBM4BPgOud879Eep1RCRvz49YEu4QPGXkSx3DHULISuLOfTSwNcf248DTzrkGwK/AjSVwDRERKYKQkruZ1QO6Ay/7tw3oCMz1d5kJ9ArlGiIiUnSh3rk/A9wDHPdv1wIynXNH/ds7gfNDvIaIiBRRsZO7mfUAfnbOpeZsDtLV5XH8MDNbb2br9+zZU9wwREQkiFDu3FsDPc0sHd8HqB3x3cnXMLPsD2rrAbuCHeycm+KcS3DOJdSpUyeEMEREJLdiJ3fn3H3OuXrOuSjgWmCJc24gsBTo6++WDCwMOUoRESmS0pjnfi9wp5l9g28M/pVSuIaIiOQj5HnuAM65ZcAy/+tvgRYlcV4RESkePaEqIuJBSu4iIh6k5C4i4kFK7iIiHqTkLiLiQUruIiIepOQuIuJBSu4iIh6k5C4i4kFK7iIiHqTkLiLiQUruIiIepOQuIuJBSu4iIh6k5C4i4kElUs+9vIjKmh3uEDwlPdwBeEjHZSPDHYLHbA13ACHTnbuIiAcVO7mb2QVmttTMtprZFjMb7W8/x8w+MLOv/V9rlly4IiJSGKHcuR8F/uacawQkASPNrDEwBvjIOdcA+Mi/LSIip1Cxk7tzbrdz7jP/6/34BqnOB64GZvq7zQR6hRqkiIgUTYmMuZtZFNAUWAP8h3NuN/h+AQDn5nHMMDNbb2br9+zZUxJhiIiIX8jJ3cyqAvOA251zvxX2OOfcFOdcgnMuoU6dOqGGISIiOYSU3M0sAl9in+Wce8ff/JOZ1fXvrwv8HFqIIiJSVKHMljHgFWCrc+6pHLveBZL9r5OBhcUPT0REiiOUh5haA9cDm8wszd92PzAReMvMbgR2AP1CC1FERIqq2MndObcCsDx2dyrueUVEJHR6QlVExIOU3EVEPEjJXUTEg5TcRUQ8SMldRMSDlNxFRDxIyV1ExIOU3EVEPEjJXUTEg5TcRUQ8SMldRMSDlNxFRDxIyV1ExIOU3EVEPEjJXUTEg5TcRUQ8qNSSu5l1NbNtZvaNmY0preuIiMjJQllmL09mVhF4HvgLsBNYZ2bvOue+KI3riZR3ja7dFe4Q5DRTKskdaAF845z7FsDM3gSuBpTcRUpBVNbscIfgKenhDqAElFZyPx/4Icf2TqBlzg5mNgwY5t88YGbbSimW8qg28Eu4gyiIPR7uCCQM9LNZsi7Ka0dpJfdgC2e7EzacmwJMKaXrl2tmtt45lxDuOERy08/mqVNaH6juBC7IsV0P0KCgiMgpUlrJfR3QwMzqm1kl4Frg3VK6loiI5FIqwzLOuaNmdivwL6AiMM05t6U0riVBabhLTlf62TxFzDlXcC8RESlT9ISqiIgHKbmLiHiQkrsHmNmZQdrOCUcsInJ6UHL3hnfMLCJ7w8zqAh+EMR6RADNrHqTtqnDEUp4ouXvDAuBtM6toZlH4ZindF9aIRP7PVDOLyd4ws+uAB8MYT7mg2TIeYWYjga5AFDDcObcqvBGJ+JjZxcBcYCDQBrgB6OGc2xfWwDxOyb0MM7M7c24C1wObgM8BnHNPhSMukdzM7FJ8f2H+APRyzh0Oc0ieV1q1ZeTUODvX9vw82kVOOTPbxIk1pc7B91DjGjPDORcbnsjKB925e4yZVQCqOud+C3csUr6ZWZ4VCwGcc9+fqljKI32g6gFmNtvMqpnZWfhq5m8zs7vDHZeUb8657/0J/Azgf/2v6+Nb20Hj7aVMyd0bGvvv1HsB7wMX4ht/FzkdzAOOmdmfgFfwJXitLlLKlNy9IcI/z70XsNA5d4Rc9fNFwui4c+4o0Bt4xjl3B1A3zDF5npK7N/wPvpXBzgKW+8c6NeYup4sj/rntNwCL/G0R+fSXEqAPVD3KzM7w3y2JhJWZNQZGAKudc2+YWX2gv3NuYphD8zQl9zLMzAY5517PNd89QPPcRcovzXMv287yf9W8djntmNlbzrm/BpnvboDTPPfSpTt3jzKzs5xzB8Mdh5RfZlbXObc7r/numudeuvSBahlnZuebWYJ/rVrM7Fwzewz4OsyhSTnnnNvt//p9zn/ATnw1ZqQUKbmXYWZ2O5AG/AP41MySga1AFeCkMqsip5L/wbr7zOw5M+tsPrcB3wJ/DXd8XqdhmTLMzL4A2jjn9prZhcA3QFvn3KdhDk0EM1sI/AqsBjoBNYFKwGjnXFo4YysPlNzLMDP7zDnXLMf2ZudcdDhjEslmZpucczH+1xWBX4ALnXP7wxtZ+aDZMmVbPTObnGP73JzbzrlRYYhJJNuR7BfOuWNm9p0S+6mjO/cyzD/Gnifn3MxTFYtIbmZ2DMiesWX4Pgs6xP9NhawWrtjKAyV3EREP0mwZEREPUnIXEfEgJXcREQ9ScvcAM6tnZvPNbI+Z/WRm88ysXrjjEpHwUXL3hunAu/gWQDgf+H/+NhEppzRbxgPMLM05F19Qm4iUH7pz94ZfzGyQmVX0/xsEZIQ7KBEJH925e4C/rsxzwOX46mavwle/QyVVRcopJXcREQ9SbZkyzMzG5rPbOeceOWXBiMhpRXfuZZiZ/S1I81nAjUAt51zVUxySiJwmlNw9wszOBkbjS+xvAX93zv0c3qhEJFw0LFPGmdk5wJ3AQGAm0Mw592t4oxKRcFNyL8PM7EmgNzAFiHHOHQhzSCJymtCwTBlmZseB34Gj+KZABnahetki5ZqSu4iIB+kJVRERD1JyFxHxICV3EREPUnIXTzOzA/6v55nZ3AL63m5mkUU8f3szW+R/nWJmzxXh2HQzq12E/kU6v5RvSu5S5phZxaIe45zb5ZzrW0C324EiJXeR05WSu5xWzCzKzL40s5lmttHM5ppZpP8ud6yZrQD6mdklZvZPM0s1s0/MrKH/+PpmttrM1pnZI7nOu9n/uqKZTTKzTf5r3GZmo4DzgKVmttTfr7P/XJ+Z2dtmVtXf3tUf4wp8zxkU9J5eNLP1ZrbFzB7OtftuM1vr//cnf/86/tW01vn/tS6Bb62UM0rucjq6DJjinIsFfgNu8bdnOefaOOfexPfg1m3OuebAXcAL/j7PAi865xKB/83j/MOA+kBT/zVmOecmA7uADs65Dv7hkgeBK5xzzYD1wJ1mVhmYClwF/Bn4z0K8nweccwlALNDOzGJz7PvNOdcCX8nmZ3K8h6f976EP8HIhriFyAj2hKqejH5xzK/2vXwdG+V/PAfDfQbcC3jaz7GPO9H9tjS8hArwGPB7k/FcALznnjgI45/YG6ZMENAZW+q9RCVgNNAS+c8597Y/ldXy/LPLzVzMbhu//t7r+827073sjx9enc8TXOMd7q+avHSRSaErucjrK/WRd9vZB/9cKQGY+ywgW9GSeFbLPB865605oNIsvxLE5+9fH95dFonPuVzObAVTOI9bs1xWAy51zh3Odq7CXFdGwjJyWLjSzy/2vrwNW5NzpnPsN+M7M+gGYT5x/90rgWv/rgXmc/9/ACDM7w3/8Of72/UD2HfKnQOsc4+CRZnYp8CVQ38wuyRFffqrh+6W0z8z+A7gy1/7+Ob6uzhHfrdkd/L9QRIpEyV1OR1uBZDPbCJwDvBikz0DgRjPbAGwBrva3jwZGmtk6oHoe538Z2AFs9B8/wN8+BVhsZkudc3uAFOANfxyfAg2dc1n4hmHe83+gmnspwxQz25n9D99atp/7Y5yG75dPTmea2Rp/3Hf420YBCf4Pe78ARuTxPkTypNoycloxsyhgkXMuOsyhiJRpunMXEfEg3bmLiHiQ7txFRDxIyV1ExIOU3EVEPEjJXUTEg5TcRUQ8SMldRMSD/j/Mp9CeI4bIvQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "credit_history_distribution_run_uid = credit_history_distribution['id']\n", "\n", "subscription.payload_logging.data_distribution.show_chart(credit_history_distribution_run_uid);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 8.0 Historical data " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 8.1 Insert historical payloads" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next section of the notebook downloads and writes historical data to the payload and measurement tables to simulate a production model that has been monitored and receiving regular traffic for the last seven days. This historical data can be viewed in the Watson OpenScale user interface. The code uses the Python and REST APIs to write this data." ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘history_payloads_with_transaction_*.json’: No such file or directory\n", "--2020-09-22 12:30:43-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_0.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5209237 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_0.json’\n", "\n", "100%[======================================>] 5,209,237 --.-K/s in 0.06s \n", "\n", "2020-09-22 12:30:44 (86.0 MB/s) - ‘history_payloads_with_transaction_id_0.json’ saved [5209237/5209237]\n", "\n", "--2020-09-22 12:30:45-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_1.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5207699 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_1.json’\n", "\n", "100%[======================================>] 5,207,699 --.-K/s in 0.06s \n", "\n", "2020-09-22 12:30:45 (79.3 MB/s) - ‘history_payloads_with_transaction_id_1.json’ saved [5207699/5207699]\n", "\n", "--2020-09-22 12:30:46-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_2.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5210448 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_2.json’\n", "\n", "100%[======================================>] 5,210,448 --.-K/s in 0.05s \n", "\n", "2020-09-22 12:30:46 (95.5 MB/s) - ‘history_payloads_with_transaction_id_2.json’ saved [5210448/5210448]\n", "\n", "--2020-09-22 12:30:47-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_3.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5211018 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_3.json’\n", "\n", "100%[======================================>] 5,211,018 --.-K/s in 0.05s \n", "\n", "2020-09-22 12:30:48 (96.6 MB/s) - ‘history_payloads_with_transaction_id_3.json’ saved [5211018/5211018]\n", "\n", "--2020-09-22 12:30:48-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_4.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5210002 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_4.json’\n", "\n", "100%[======================================>] 5,210,002 --.-K/s in 0.06s \n", "\n", "2020-09-22 12:30:49 (78.8 MB/s) - ‘history_payloads_with_transaction_id_4.json’ saved [5210002/5210002]\n", "\n", "--2020-09-22 12:30:49-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_5.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5210931 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_5.json’\n", "\n", "100%[======================================>] 5,210,931 --.-K/s in 0.05s \n", "\n", "2020-09-22 12:30:50 (106 MB/s) - ‘history_payloads_with_transaction_id_5.json’ saved [5210931/5210931]\n", "\n", "--2020-09-22 12:30:50-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_6.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 5209505 (5.0M) [text/plain]\n", "Saving to: ‘history_payloads_with_transaction_id_6.json’\n", "\n", "100%[======================================>] 5,209,505 --.-K/s in 0.06s \n", "\n", "2020-09-22 12:30:51 (85.9 MB/s) - ‘history_payloads_with_transaction_id_6.json’ saved [5209505/5209505]\n", "\n" ] } ], "source": [ "!rm history_payloads_with_transaction_*.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_0.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_1.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_2.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_3.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_4.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_5.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_payloads_with_transaction_id_6.json" ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "scrolled": true }, "outputs": [], "source": [ "historyDays = 7" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading day 1\n", "Loading day 2\n", "Loading day 3\n", "Loading day 4\n", "Loading day 5\n", "Loading day 6\n", "Loading day 7\n", "Finished\n" ] } ], "source": [ "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n", "import datetime\n", "import time\n", "\n", "for day in range(historyDays):\n", " print('Loading day {}'.format(day + 1))\n", " history_file = 'history_payloads_with_transaction_id_' + str(day) + '.json'\n", " with open(history_file) as f:\n", " payloads = json.load(f)\n", " hourly_records = int(len(payloads) / 24)\n", " index = 0\n", " for hour in range(24):\n", " recordsList = []\n", " for i in range(hourly_records):\n", " score_time = str(datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1))))\n", " recordsList.append(PayloadRecord(request=payloads[index]['request'], response=payloads[index]['response'], scoring_timestamp=score_time))\n", " index += 1\n", " subscription.payload_logging.store(records=recordsList)\n", "print('Finished')" ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "00000000-0000-0000-0000-000000000000\n" ] } ], "source": [ "data_mart_id = subscription.get_details()['metadata']['url'].split('/service_bindings')[0].split('marts/')[1]\n", "print(data_mart_id)" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://zen2-cpd-zen2.aida-cpd3-dal10-b3c-56x-f2c6cdc6801be85fd188b09d006f13e3-0001.us-south.containers.appdomain.cloud/v1/data_marts/00000000-0000-0000-0000-000000000000/metrics\n" ] } ], "source": [ "performance_metrics_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/metrics'\n", "print(performance_metrics_url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.2 Insert historical fairness metrics" ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘history_fairness.json’: No such file or directory\n", "--2020-09-22 12:32:57-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_fairness.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1424078 (1.4M) [text/plain]\n", "Saving to: ‘history_fairness.json’\n", "\n", "100%[======================================>] 1,424,078 --.-K/s in 0.03s \n", "\n", "2020-09-22 12:32:58 (44.8 MB/s) - ‘history_fairness.json’ saved [1424078/1424078]\n", "\n" ] } ], "source": [ "!rm history_fairness.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_fairness.json" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [], "source": [ "import requests\n", "from requests.auth import HTTPBasicAuth\n", "\n", "def create_token():\n", " header = {\n", " \"Content-Type\": \"application/x-www-form-urlencoded\",\n", " \"Accept\": \"application/json\"\n", " }\n", "\n", " response = requests.Session().get(\n", " WOS_CREDENTIALS['url'] + '/v1/preauth/validateAuth',\n", " headers=header,\n", " auth=HTTPBasicAuth(\n", " WOS_CREDENTIALS['username'],\n", " WOS_CREDENTIALS['password']\n", " ),\n", " verify=False)\n", "\n", " response = handle_response(200, 'access token', response, True)\n", " token = response['accessToken']\n", "\n", " return token" ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading day 1\n", "Loading day 2\n", "Loading day 3\n", "Loading day 4\n", "Loading day 5\n", "Loading day 6\n", "Loading day 7\n", "Finished\n" ] } ], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "with open('history_fairness.json', 'r') as history_file:\n", " payloads = json.load(history_file)\n", "\n", "for day in range(historyDays):\n", " print('Loading day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed\n", " \n", " qualityMetric = {\n", " 'metric_type': 'fairness',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': payloads[index]\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)\n", "\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.3 Insert historical debias metrics" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘history_debias.json’: No such file or directory\n", "--2020-09-22 12:33:26-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_debias.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 565971 (553K) [text/plain]\n", "Saving to: ‘history_debias.json’\n", "\n", "100%[======================================>] 565,971 --.-K/s in 0.01s \n", "\n", "2020-09-22 12:33:26 (37.0 MB/s) - ‘history_debias.json’ saved [565971/565971]\n", "\n" ] } ], "source": [ "!rm history_debias.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_debias.json" ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading day 1\n", "Loading day 2\n", "Loading day 3\n", "Loading day 4\n", "Loading day 5\n", "Loading day 6\n", "Loading day 7\n", "Finished\n" ] } ], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "with open('history_debias.json', 'r') as history_file:\n", " payloads = json.load(history_file)\n", "\n", "for day in range(historyDays):\n", " print('Loading day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed\n", "\n", " qualityMetric = {\n", " 'metric_type': 'debiased_fairness',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': payloads[index]\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.4 Insert historical quality metrics" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Day 1\n", "Day 2\n", "Day 3\n", "Day 4\n", "Day 5\n", "Day 6\n", "Day 7\n", "Finished\n" ] } ], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "measurements = [0.76, 0.78, 0.68, 0.72, 0.73, 0.77, 0.80]\n", "for day in range(historyDays):\n", " print('Day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " qualityMetric = {\n", " 'metric_type': 'quality',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': {\n", " 'quality': measurements[day],\n", " 'threshold': 0.7,\n", " 'metrics': [\n", " {\n", " 'name': 'auroc',\n", " 'value': measurements[day],\n", " 'threshold': 0.7\n", " }\n", " ]\n", " }\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.5 Insert historical confusion matrixes" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘history_quality_metrics.json’: No such file or directory\n", "--2020-09-22 12:34:29-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_quality_metrics.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 80099 (78K) [text/plain]\n", "Saving to: ‘history_quality_metrics.json’\n", "\n", "100%[======================================>] 80,099 --.-K/s in 0.004s \n", "\n", "2020-09-22 12:34:29 (20.1 MB/s) - ‘history_quality_metrics.json’ saved [80099/80099]\n", "\n" ] } ], "source": [ "!rm history_quality_metrics.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_quality_metrics.json" ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://zen2-cpd-zen2.aida-cpd3-dal10-b3c-56x-f2c6cdc6801be85fd188b09d006f13e3-0001.us-south.containers.appdomain.cloud/v1/data_marts/00000000-0000-0000-0000-000000000000/measurements\n" ] } ], "source": [ "measurements_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/measurements'\n", "print(measurements_url)" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Day 1\n", "Day 2\n", "Day 3\n", "Day 4\n", "Day 5\n", "Day 6\n", "Day 7\n", "Finished\n" ] } ], "source": [ "with open('history_quality_metrics.json') as json_file:\n", " records = json.load(json_file)\n", "\n", "for day in range(historyDays):\n", " index = 0\n", " measurments = []\n", " print('Day', day + 1)\n", " \n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", "\n", " measurement = {\n", " \"monitor_definition_id\": 'quality',\n", " \"binding_id\": subscription.binding_uid,\n", " \"subscription_id\": subscription.uid,\n", " \"asset_id\": subscription.source_uid,\n", " 'metrics': [records[index]['metrics']],\n", " 'sources': [records[index]['sources']],\n", " 'timestamp': score_time\n", " }\n", "\n", " measurments.append(measurement)\n", " index+=1\n", "\n", " response = requests.post(measurements_url, json=measurments, headers=ai_client._get_headers(), verify=False)\n", "\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.6 Insert historical performance metrics" ] }, { "cell_type": "code", "execution_count": 83, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Day 1\n", "Day 2\n", "Day 3\n", "Day 4\n", "Day 5\n", "Day 6\n", "Day 7\n", "Finished\n" ] } ], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "for day in range(historyDays):\n", " print('Day', day + 1)\n", " for hour in range(24):\n", " score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " score_count = random.randint(60, 600)\n", " score_resp = random.uniform(60, 300)\n", "\n", " performanceMetric = {\n", " 'metric_type': 'performance',\n", " 'binding_id': binding_uid,\n", " 'timestamp': score_time,\n", " 'subscription_id': model_uid,\n", " 'asset_revision': model_uid,\n", " 'deployment_id': deployment_uid,\n", " 'value': {\n", " 'response_time': score_resp,\n", " 'records': score_count\n", " }\n", " }\n", "\n", " response = requests.post(performance_metrics_url, json=[performanceMetric], headers=iam_headers, verify=False)\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.7 Insert historical manual labeling" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://zen2-cpd-zen2.aida-cpd3-dal10-b3c-56x-f2c6cdc6801be85fd188b09d006f13e3-0001.us-south.containers.appdomain.cloud/v1/data_marts/00000000-0000-0000-0000-000000000000/manual_labelings\n" ] } ], "source": [ "manual_labeling_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/manual_labelings'\n", "print(manual_labeling_url)" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove ‘history_manual_labeling.json’: No such file or directory\n", "--2020-09-22 12:35:06-- https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_manual_labeling.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 947956 (926K) [text/plain]\n", "Saving to: ‘history_manual_labeling.json’\n", "\n", "100%[======================================>] 947,956 --.-K/s in 0.02s \n", "\n", "2020-09-22 12:35:06 (39.3 MB/s) - ‘history_manual_labeling.json’ saved [947956/947956]\n", "\n" ] } ], "source": [ "!rm history_manual_labeling.json\n", "!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_manual_labeling.json" ] }, { "cell_type": "code", "execution_count": 86, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading day 1\n", "Loading day 2\n", "Loading day 3\n", "Loading day 4\n", "Loading day 5\n", "Loading day 6\n", "Loading day 7\n", "Finished\n" ] } ], "source": [ "iam_token = create_token()\n", "iam_headers = {\n", " 'Content-Type': 'application/json',\n", " 'Authorization': 'Bearer %s' % iam_token\n", "}\n", "\n", "with open('history_manual_labeling.json', 'r') as history_file:\n", " records = json.load(history_file)\n", "\n", "for day in range(historyDays):\n", " print('Loading day', day + 1)\n", " record_json = []\n", " for hour in range(24):\n", " for record in records:\n", " if record['fastpath_history_day'] == day and record['fastpath_history_hour'] == hour:\n", " record['binding_id'] = binding_uid\n", " record['subscription_id'] = model_uid\n", " record['asset_revision'] = model_uid\n", " record['deployment_id'] = deployment_uid\n", " record['scoring_timestamp'] = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n", " record_json.append(record)\n", " response = requests.post(manual_labeling_url, json=record_json, headers=iam_headers, verify=False)\n", "\n", "print('Finished')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.8 Additional data to help debugging" ] }, { "cell_type": "code", "execution_count": 87, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Datamart: 00000000-0000-0000-0000-000000000000\n", "Model: d9f69282-4c9a-4e2f-8293-8589695853da\n", "Deployment: e85d6845-4b6b-44bf-a483-80a522852620\n", "Binding: c411a7d3-34ec-4c99-8a23-6c0638d0aae7\n" ] } ], "source": [ "print('Datamart:', data_mart_id)\n", "print('Model:', model_uid)\n", "print('Deployment:', deployment_uid)\n", "print('Binding:', binding_uid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.9 Identify transactions for Explainability" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard." ] }, { "cell_type": "code", "execution_count": 88, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
scoring_idpredictedLabelprobability
0a3c82ddb-f855-4376-9519-3056f531a93a-1No Risk[0.647647010934288, 0.352352989065712]
1a3c82ddb-f855-4376-9519-3056f531a93a-19Risk[0.4087909352812048, 0.5912090647187953]
2a3c82ddb-f855-4376-9519-3056f531a93a-41Risk[0.19114337018658423, 0.8088566298134158]
3a3c82ddb-f855-4376-9519-3056f531a93a-42No Risk[0.7613702604004451, 0.238629739599555]
4a3c82ddb-f855-4376-9519-3056f531a93a-43No Risk[0.6848782221336038, 0.3151217778663962]
5a3c82ddb-f855-4376-9519-3056f531a93a-44No Risk[0.9527937744126087, 0.0472062255873913]
6a3c82ddb-f855-4376-9519-3056f531a93a-45No Risk[0.5576070634022912, 0.44239293659770895]
7a3c82ddb-f855-4376-9519-3056f531a93a-46No Risk[0.9228491405250228, 0.07715085947497724]
8a3c82ddb-f855-4376-9519-3056f531a93a-47Risk[0.40875977146849474, 0.5912402285315054]
9a3c82ddb-f855-4376-9519-3056f531a93a-48No Risk[0.7156516908661615, 0.2843483091338384]
\n", "
" ], "text/plain": [ " scoring_id predictedLabel \\\n", "0 a3c82ddb-f855-4376-9519-3056f531a93a-1 No Risk \n", "1 a3c82ddb-f855-4376-9519-3056f531a93a-19 Risk \n", "2 a3c82ddb-f855-4376-9519-3056f531a93a-41 Risk \n", "3 a3c82ddb-f855-4376-9519-3056f531a93a-42 No Risk \n", "4 a3c82ddb-f855-4376-9519-3056f531a93a-43 No Risk \n", "5 a3c82ddb-f855-4376-9519-3056f531a93a-44 No Risk \n", "6 a3c82ddb-f855-4376-9519-3056f531a93a-45 No Risk \n", "7 a3c82ddb-f855-4376-9519-3056f531a93a-46 No Risk \n", "8 a3c82ddb-f855-4376-9519-3056f531a93a-47 Risk \n", "9 a3c82ddb-f855-4376-9519-3056f531a93a-48 No Risk \n", "\n", " probability \n", "0 [0.647647010934288, 0.352352989065712] \n", "1 [0.4087909352812048, 0.5912090647187953] \n", "2 [0.19114337018658423, 0.8088566298134158] \n", "3 [0.7613702604004451, 0.238629739599555] \n", "4 [0.6848782221336038, 0.3151217778663962] \n", "5 [0.9527937744126087, 0.0472062255873913] \n", "6 [0.5576070634022912, 0.44239293659770895] \n", "7 [0.9228491405250228, 0.07715085947497724] \n", "8 [0.40875977146849474, 0.5912402285315054] \n", "9 [0.7156516908661615, 0.2843483091338384] " ] }, "execution_count": 88, "metadata": {}, "output_type": "execute_result" } ], "source": [ "payload_data = subscription.payload_logging.get_table_content(limit=10)\n", "payload_data.filter(items=['scoring_id', 'predictedLabel', 'probability'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Congratulations!\n", "\n", "You have finished the hands-on lab for IBM Watson OpenScale. You can now view the OpenScale dashboard by going to the ICP `Home` page, and clicking `Add-ons`. Choose the `OpenScale` tile and click the menu to `Open`. Click on the tile for the model you've created to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n", "\n", "OpenScale shows model performance over time. You have two options to keep data flowing to your OpenScale graphs:\n", " * Download, configure and schedule the [model feed notebook](https://raw.githubusercontent.com/emartensibm/german-credit/master/german_credit_scoring_feed.ipynb). This notebook can be set up with your WML credentials, and scheduled to provide a consistent flow of scoring requests to your model, which will appear in your OpenScale monitors.\n", " * Re-run this notebook. Running this notebook from the beginning will delete and re-create the model and deployment, and re-create the historical data. Please note that the payload and measurement logs for the previous deployment will continue to be stored in your datamart, and can be deleted if necessary." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Authors\n", "\n", "Eric Martens, is a technical specialist having expertise in analysis and description of business processes, and their translation into functional and non-functional IT requirements. He acts as the interpreter between the worlds of IT and business.\n", "\n", "Lukasz Cmielowski, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n", "\n", "Zilu (Peter) Tang, is a cognitive developer with experties in deep learning and enterprise AI solutions from Watson Openscale to many other cutting-edge IBM research projects." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.6", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 1 }