{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Working with Azure Machine Learning Studio engine"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook shows how to monitor the model deployed on Azure Machine Learning Studio using Watson OpenScale python sdk."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Contents\n",
"\n",
"1. [Setup](#setup)\n",
"2. [Binding machine learning engine](#binding)\n",
"3. [Subscriptions](#subsciption)\n",
"4. [Performance monitor, scoring and payload logging](#scoring)\n",
"5. [Quality monitor and feedback logging](#feedback)\n",
"6. [Fairness, Drift monitor and explainability](#monitors)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# 1. Setup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sample model creation using [Azure Machine Learning Studio](https://studio.azureml.net)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Copy [credit risk experiment](https://gallery.cortanaintelligence.com/Experiment/German-credit-risk-created-on-1-9-2019) from Azure ML Studio Gallery \n",
"- Run experiment to train a model\n",
"- Create (deploy) web service (new)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**NOTE:** Classic web services are not supported."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Installation and authentication"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"!pip install --upgrade --upgrade ibm-watson-openscale --no-cache | tail -n 1\n",
"!pip install --upgrade pyspark==2.4 --no-cache | tail -n 1\n",
"!pip install --upgrade lime --no-cache | tail -n 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Action:** Restart the kernel (Kernel->Restart)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### ACTION: Get Watson OpenScale `apikey`\n",
"\n",
"How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
"\n",
"How to get api key using bluemix console:\n",
"```\n",
"bx login --sso\n",
"bx iam api-key-create 'my_key'\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"CLOUD_API_KEY = '******'\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"DB_CREDENTIALS=None\n",
"#DB_CREDENTIALS= {\"hostname\":\"\",\"username\":\"\",\"password\":\"\",\"database\":\"\",\"port\":\"\",\"ssl\":True,\"sslmode\":\"\",\"certificate_base64\":\"\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"SCHEMA_NAME = 'azure_dm'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
"\n",
"from ibm_watson_openscale import *\n",
"from ibm_watson_openscale.supporting_classes.enums import *\n",
"from ibm_watson_openscale.supporting_classes import *\n",
"\n",
"\n",
"authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n",
"#authenticator = BearerTokenAuthenticator(bearer_token=IAM_TOKEN) ## uncomment if using IAM token\n",
"wos_client = APIClient(authenticator=authenticator)\n",
"wos_client.version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Download and preview training data set"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"!rm -rf credit_risk_training.csv\n",
"!wget \"https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/credit_risk_training.csv\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"training_data_pd = pd.read_csv(\n",
" \"credit_risk_training.csv\",\n",
" dtype={'LoanDuration': int, 'LoanAmount': int, 'InstallmentPercent': int, 'CurrentResidenceDuration': int, 'Age': int, 'ExistingCreditsCount': int, 'Dependents': int})"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"training_data_pd.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Save training data to IBM Cloud Object Storage"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit getting started with COS tutorial. You can find COS_API_KEY_ID and COS_RESOURCE_CRN variables in Service Credentials in menu of your COS instance. Used COS Service Credentials must be created with Role parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription. COS_ENDPOINT variable can be found in Endpoint field of the menu."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"COS_API_KEY_ID = \"*******\n",
"COS_RESOURCE_CRN = \"*******\" # eg \"crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003abfb5d29761c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::\"\n",
"COS_ENDPOINT = \"https://******\" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a globally unique BUCKET_NAME in COS and use it below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"BUCKET_NAME = \"*****\" #example: \"--credit-risk-training-data\"\n",
"training_data_file_name=\"credit_risk_training.csv\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import ibm_boto3\n",
"from ibm_botocore.client import Config, ClientError\n",
"\n",
"cos_client = ibm_boto3.resource(\"s3\",\n",
" ibm_api_key_id=COS_API_KEY_ID,\n",
" ibm_service_instance_id=COS_RESOURCE_CRN,\n",
" ibm_auth_endpoint=\"https://iam.bluemix.net/oidc/token\",\n",
" config=Config(signature_version=\"oauth\"),\n",
" endpoint_url=COS_ENDPOINT\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"cos_client.Object(BUCKET_NAME, training_data_file_name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"training_data_file_name"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"with open(training_data_file_name, \"rb\") as file_data:\n",
" print(file_data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"with open(training_data_file_name, \"rb\") as file_data:\n",
" cos_client.Object(BUCKET_NAME, training_data_file_name).upload_fileobj(\n",
" Fileobj=file_data\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### DataMart setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.data_marts.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false,
"tags": []
},
"outputs": [],
"source": [
"data_marts = wos_client.data_marts.list().result.data_marts\n",
"if len(data_marts) == 0:\n",
" if DB_CREDENTIALS is not None:\n",
" if SCHEMA_NAME is None: \n",
" print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n",
"\n",
" print('Setting up external datamart')\n",
" added_data_mart_result = wos_client.data_marts.add(\n",
" background_mode=False,\n",
" name=\"WOS Data Mart\",\n",
" description=\"Data Mart created by WOS tutorial notebook\",\n",
" database_configuration=DatabaseConfigurationRequest(\n",
" database_type=DatabaseType.POSTGRESQL,\n",
" credentials=PrimaryStorageCredentialsLong(\n",
" hostname=DB_CREDENTIALS['hostname'],\n",
" username=DB_CREDENTIALS['username'],\n",
" password=DB_CREDENTIALS['password'],\n",
" db=DB_CREDENTIALS['database'],\n",
" port=DB_CREDENTIALS['port'],\n",
" ssl=True,\n",
" sslmode=DB_CREDENTIALS['sslmode'],\n",
" certificate_base64=DB_CREDENTIALS['certificate_base64']\n",
" ),\n",
" location=LocationSchemaName(\n",
" schema_name= SCHEMA_NAME\n",
" )\n",
" )\n",
" ).result\n",
" else:\n",
" print('Setting up internal datamart')\n",
" added_data_mart_result = wos_client.data_marts.add(\n",
" background_mode=False,\n",
" name=\"WOS Data Mart\",\n",
" description=\"Data Mart created by WOS tutorial notebook\", \n",
" internal_database = True).result\n",
" \n",
" data_mart_id = added_data_mart_result.metadata.id\n",
" \n",
"else:\n",
" data_mart_id=data_marts[0].metadata.id\n",
" print('Using existing datamart {}'.format(data_mart_id))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# 2. Bind machine learning engines"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Bind `Azure` machine learning studio\n",
"\n",
"Provide credentials using following fields:\n",
"- `client_id`\n",
"- `client_secret`\n",
"- `subscription_id`\n",
"- `tenant`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"AZURE_ENGINE_CREDENTIALS = {\n",
" \"client_id\": \"******\",\n",
" \"client_secret\": \"******\",\n",
" \"tenant\": \"******\",\n",
" \"subscription_id\": \"******\"\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"SERVICE_PROVIDER_NAME = \"Azure Machine Learning\"\n",
"SERVICE_PROVIDER_DESCRIPTION = \"Added by Azure tutorial WOS notebook.\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false,
"tags": []
},
"outputs": [],
"source": [
"service_providers = wos_client.service_providers.list().result.service_providers\n",
"for service_provider in service_providers:\n",
" service_instance_name = service_provider.entity.name\n",
" if service_instance_name == SERVICE_PROVIDER_NAME:\n",
" service_provider_id = service_provider.metadata.id\n",
" wos_client.service_providers.delete(service_provider_id)\n",
" print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"added_service_provider_result = wos_client.service_providers.add(\n",
" name=SERVICE_PROVIDER_NAME,\n",
" description=SERVICE_PROVIDER_DESCRIPTION,\n",
" service_type=ServiceTypes.AZURE_MACHINE_LEARNING,\n",
" credentials=AzureCredentials(\n",
" subscription_id= AZURE_ENGINE_CREDENTIALS['subscription_id'], \n",
" client_id = AZURE_ENGINE_CREDENTIALS['client_id'], \n",
" client_secret= AZURE_ENGINE_CREDENTIALS['client_secret'],\n",
" tenant = AZURE_ENGINE_CREDENTIALS['tenant']\n",
" ),\n",
" background_mode=False\n",
" ).result\n",
"service_provider_id = added_service_provider_result.metadata.id"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.service_providers.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id).result\n",
"asset_deployment_details"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.1 Add deployment ID from `asset_deployment_details` above"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"## select your model deployment GUID and get model asset details\n",
"deployment_id='******'\n",
"for model_asset_details in asset_deployment_details['resources']:\n",
" if model_asset_details['metadata']['guid']==deployment_id:\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### sample scoring"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import requests\n",
"import time\n",
"import json\n",
"\n",
"scoring_url = model_asset_details['entity']['scoring_endpoint']['url']\n",
"\n",
"data = {\n",
" \"Inputs\": {\n",
" \"input1\":\n",
" [{'CheckingStatus': 'no_checking',\n",
" 'LoanDuration': 19,\n",
" 'CreditHistory': 'credits_paid_to_date',\n",
" 'LoanPurpose': 'business',\n",
" 'LoanAmount': 1867,\n",
" 'ExistingSavings': '100_to_500',\n",
" 'EmploymentDuration': '1_to_4',\n",
" 'InstallmentPercent': 4,\n",
" 'Sex': 'male',\n",
" 'OthersOnLoan': 'none',\n",
" 'CurrentResidenceDuration': 3,\n",
" 'OwnsProperty': 'car_other',\n",
" 'Age': 42,\n",
" 'InstallmentPlans': 'stores',\n",
" 'Housing': 'own',\n",
" 'ExistingCreditsCount': 2,\n",
" 'Job': 'unskilled',\n",
" 'Dependents': 1,\n",
" 'Telephone': 'none',\n",
" 'ForeignWorker': 'yes'}]\n",
" },\n",
" \"GlobalParameters\": {\n",
" }\n",
" }\n",
"\n",
"body = str.encode(json.dumps(data))\n",
"\n",
"\n",
"token = model_asset_details['entity']['scoring_endpoint']['credentials']['token']\n",
"headers = model_asset_details['entity']['scoring_endpoint']['request_headers']\n",
"headers['Authorization'] = ('Bearer ' + token)\n",
"\n",
"start_time = time.time()\n",
"response = requests.post(url=scoring_url, data=body, headers=headers)\n",
"response_time = int(time.time() - start_time)*1000\n",
"result = response.json()\n",
"\n",
"print(json.dumps(result, indent=2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# 3. Subscriptions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Add subscriptions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"List available deployments.\n",
"\n",
"**Note:** Depending on number of assets it may take some time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.subscriptions.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"feature_columns = ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker']\n",
"categorical_columns = ['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointCredentialsAzureScoringEndpointCredentials,ScoringEndpointRequest\n",
"def get_output_schema_fields_for_azure(azure_ouput_schema, target_field, target_role = \"decoded-target\", field_type = str, is_nullable = True):\n",
" fields = []\n",
" found = False\n",
" for field in azure_ouput_schema['fields']:\n",
" if field['name']==target_field:\n",
" metadata = {}\n",
" if \"metadata\" in field:\n",
" metadata = field[\"metadata\"]\n",
" metadata[\"modeling_role\"]=target_role \n",
" field[\"metadata\"] = metadata\n",
" found = True\n",
" \n",
" spark_field = SparkStructFieldObject(\n",
" name=field['name'],\n",
" type=field['type'],\n",
" nullable=field['nullable'],\n",
" )\n",
" if \"metadata\" in field:\n",
" spark_field.metadata = field[\"metadata\"]\n",
" \n",
" fields.append(spark_field) \n",
" \n",
" if found is False:\n",
" metadata = {}\n",
" metadata[\"modeling_role\"] = target_role\n",
" fields.append(SparkStructFieldObject(\n",
" name=target_field,\n",
" type=field_type,\n",
" nullable=nullable,\n",
" metadata = metadata\n",
" )) \n",
" return fields"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"azure_asset = Asset(\n",
" asset_id=model_asset_details[\"entity\"][\"asset\"][\"asset_id\"],\n",
" name=model_asset_details[\"entity\"][\"asset\"][\"name\"],\n",
" url=model_asset_details[\"entity\"][\"asset\"][\"url\"],\n",
" asset_type=AssetTypes.MODEL,\n",
" input_data_type=InputDataType.STRUCTURED,\n",
" problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"deployment_scoring_endpoint = model_asset_details['entity']['scoring_endpoint']\n",
"scoring_endpoint = ScoringEndpointRequest(url = model_asset_details['entity']['scoring_endpoint']['url']\n",
" ,request_headers = model_asset_details['entity']['scoring_endpoint']['request_headers'],\n",
" credentials= ScoringEndpointCredentialsAzureScoringEndpointCredentials(model_asset_details['entity']['scoring_endpoint']['credentials']['token']) )\n",
"\n",
"deployment = AssetDeploymentRequest(\n",
" deployment_id=model_asset_details['metadata']['guid'],\n",
" url=model_asset_details['metadata']['url'],\n",
" name=model_asset_details['entity']['name'],\n",
" description=model_asset_details['entity']['description'],\n",
" deployment_type=model_asset_details['entity']['type'],\n",
" scoring_endpoint = scoring_endpoint\n",
") "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"training_data_reference = TrainingDataReference(type='cos',\n",
" location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,\n",
" file_name = training_data_file_name),\n",
" connection=COSTrainingDataReferenceConnection(\n",
" resource_instance_id= COS_RESOURCE_CRN,\n",
" url= COS_ENDPOINT,\n",
" api_key= COS_API_KEY_ID,\n",
" iam_url=IAM_URL)\n",
" )\n",
"os = model_asset_details['entity']['asset_properties']['output_data_schema']\n",
"input_s= model_asset_details['entity']['asset_properties']['input_data_schema']\n",
"input_schema = SparkStruct(\n",
" type=model_asset_details['entity']['asset_properties']['input_data_schema']['type'],\n",
" fields=model_asset_details['entity']['asset_properties']['input_data_schema']['fields']\n",
" )\n",
"output_schema = SparkStruct(\n",
" type=model_asset_details['entity']['asset_properties']['output_data_schema']['type'],\n",
" fields=get_output_schema_fields_for_azure(model_asset_details['entity']['asset_properties']['output_data_schema'], target_field = \"Scored Labels\", target_role = \"decoded-target\")\n",
" )\n",
"output_schema.to_dict()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"asset_properties = AssetPropertiesRequest(\n",
" label_column=\"Risk\",\n",
" prediction_field='Scored Labels',\n",
" probability_fields=['Scored Probabilities'],\n",
" training_data_reference=training_data_reference,\n",
" training_data_schema=None,\n",
" input_data_schema=input_schema,\n",
" output_data_schema=output_schema,\n",
" feature_fields=feature_columns,\n",
" categorical_fields=categorical_columns\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"subscription_details = wos_client.subscriptions.add(\n",
" data_mart_id=data_mart_id,\n",
" service_provider_id=service_provider_id,\n",
" asset=azure_asset,\n",
" deployment=deployment,\n",
" asset_properties=asset_properties,\n",
" background_mode=False\n",
").result\n",
"subscription_id = subscription_details.metadata.id\n",
"subscription_id"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# 4. Scoring and payload logging"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Score the credit risk model and measure response time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import time\n",
"\n",
"time.sleep(5)\n",
"payload_data_set_id = None\n",
"payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
" target_target_id=subscription_id, \n",
" target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
"if payload_data_set_id is None:\n",
" print(\"Payload data set not found. Please check subscription status.\")\n",
"else:\n",
" print(\"Payload data set id: \", payload_data_set_id)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import requests\n",
"import time\n",
"import json\n",
"\n",
"subscription_details=wos_client.subscriptions.get(subscription_id).result.to_dict()\n",
"scoring_url = subscription_details['entity']['deployment']['scoring_endpoint']['url']\n",
"\n",
"data = {\n",
" \"Inputs\": {\n",
" \"input1\":\n",
" training_data_pd.sample(2).drop(['Risk'],axis=1).to_dict('records'),\n",
" },\n",
" \"GlobalParameters\": {\n",
" }\n",
" }\n",
"\n",
"body = str.encode(json.dumps(data))\n",
"\n",
"token = model_asset_details['entity']['scoring_endpoint']['credentials']['token']\n",
"headers = subscription_details['entity']['deployment']['scoring_endpoint']['request_headers']\n",
"headers['Authorization'] = ('Bearer ' + token)\n",
"\n",
"start_time = time.time()\n",
"response = requests.post(url=scoring_url, data=body, headers=headers)\n",
"response_time = int(time.time() - start_time)*1000\n",
"result = response.json()\n",
"\n",
"print(json.dumps(result, indent=2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Functions to adjust payload data format for OpenScale "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"def convert_user_input_2_openscale(input_data):\n",
" users_records = input_data['Inputs']['input1']\n",
" openscale_fields = list(users_records[0])\n",
" openscale_values = [[rec[k] for k in openscale_fields] for rec in users_records] \n",
"\n",
" return {'fields':openscale_fields, 'values':openscale_values}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"def convert_user_output_2_openscale(output_data):\n",
" users_records = output_data['Results']['output1']\n",
" openscale_fields = [\"Scored Labels\",\"Scored Probabilities\"]\n",
" openscale_values = [[np.double(rec[k]) if k=='Scored Probabilities' else rec[k] for k in openscale_fields ] for rec in users_records] \n",
"\n",
" return {'fields':openscale_fields, 'values':openscale_values}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"openscale_input=convert_user_input_2_openscale(data)\n",
"openscale_output=convert_user_output_2_openscale(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Store the request and response in payload logging table"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import uuid\n",
"from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
"\n",
"print(\"Performing explicit payload logging.....\")\n",
"wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n",
" scoring_id=str(uuid.uuid4()),\n",
" request=openscale_input,\n",
" response=openscale_output,\n",
" response_time=460\n",
")])\n",
"time.sleep(5)\n",
"pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
"print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"wos_client.data_sets.show_records(data_set_id=payload_data_set_id)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# 5. Feedback logging & quality (accuracy) monitoring"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Enable quality monitoring"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import time\n",
"\n",
"time.sleep(10)\n",
"target = Target(\n",
" target_type=TargetTypes.SUBSCRIPTION,\n",
" target_id=subscription_id\n",
")\n",
"parameters = {\n",
" \"min_feedback_data_size\": 10\n",
"}\n",
"thresholds = [\n",
" {\n",
" \"metric_id\": \"area_under_roc\",\n",
" \"type\": \"lower_limit\",\n",
" \"value\": .80\n",
" }\n",
" ]\n",
"quality_monitor_details = wos_client.monitor_instances.create(\n",
" data_mart_id=data_mart_id,\n",
" background_mode=False,\n",
" monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
" target=target,\n",
" parameters=parameters,\n",
" thresholds=thresholds\n",
").result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"quality_monitor_instance_id = quality_monitor_details.metadata.id\n",
"quality_monitor_instance_id"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Feedback records logging"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Feedback records are used to evaluate your model. The predicted values are compared to real values (feedback records)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"!rm additional_feedback_data_v2.json\n",
"!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Get feedback logging dataset ID"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"feedback_dataset_id = None\n",
"feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n",
" target_target_id=subscription_id, \n",
" target_target_type=TargetTypes.SUBSCRIPTION).result\n",
"#print(feedback_dataset)\n",
"feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\n",
"if feedback_dataset_id is None:\n",
" print(\"Feedback data set not found. Please check quality monitor status.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"with open('additional_feedback_data_v2.json') as feedback_file:\n",
" additional_feedback_data = json.load(feedback_file)\n",
"wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# manually records scoring and store as feedback data\n",
"\n",
"# records = [\n",
"# [\"no_checking\",\"28\",\"outstanding_credit\",\"appliances\",\"5990\",\"500_to_1000\",\"greater_7\",\"5\",\"male\",\"co-applicant\",\"3\",\"car_other\",\"55\",\"none\",\"free\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
"# [\"greater_200\",\"22\",\"all_credits_paid_back\",\"car_used\",\"3376\",\"less_100\",\"less_1\",\"3\",\"female\",\"none\",\"2\",\"car_other\",\"32\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
"# [\"no_checking\",\"39\",\"credits_paid_to_date\",\"vacation\",\"6434\",\"unknown\",\"greater_7\",\"5\",\"male\",\"none\",\"4\",\"car_other\",\"39\",\"none\",\"own\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
"# [\"0_to_200\",\"20\",\"credits_paid_to_date\",\"furniture\",\"2442\",\"less_100\",\"unemployed\",\"3\",\"female\",\"none\",\"1\",\"real_estate\",\"42\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
"# [\"greater_200\",\"4\",\"all_credits_paid_back\",\"education\",\"4206\",\"less_100\",\"unemployed\",\"1\",\"female\",\"none\",\"3\",\"savings_insurance\",\"27\",\"none\",\"own\",\"1\",\"management_self-employed\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
"# [\"greater_200\",\"23\",\"credits_paid_to_date\",\"car_used\",\"2963\",\"greater_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"46\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"none\",\"yes\",\"Risk\"],\n",
"# [\"no_checking\",\"31\",\"prior_payments_delayed\",\"vacation\",\"2673\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"2\",\"real_estate\",\"35\",\"stores\",\"rent\",\"1\",\"skilled\",\"2\",\"none\",\"yes\",\"Risk\"],\n",
"# [\"no_checking\",\"37\",\"prior_payments_delayed\",\"other\",\"6971\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"3\",\"savings_insurance\",\"54\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
"# [\"0_to_200\",\"39\",\"prior_payments_delayed\",\"appliances\",\"5685\",\"100_to_500\",\"1_to_4\",\"4\",\"female\",\"none\",\"2\",\"unknown\",\"37\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
"# [\"no_checking\",\"38\",\"prior_payments_delayed\",\"appliances\",\"4990\",\"500_to_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"50\",\"bank\",\"own\",\"2\",\"unemployed\",\"2\",\"yes\",\"yes\",\"Risk\"]]\n",
"\n",
"# fields = feature_columns.copy()\n",
"# fields.append('Risk')\n",
"\n",
"# payload_scoring = [{\"fields\": fields, \"values\": records}]\n",
"#wos_client.data_sets.store_records(feedback_dataset_id, request_body=payload_scoring, background_mode=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"time.sleep(5)\n",
"wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 6. Fairness, Drift monitoring and explanations "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Fairness monitor configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.monitor_instances.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"target = Target(\n",
" target_type=TargetTypes.SUBSCRIPTION,\n",
" target_id=subscription_id\n",
"\n",
")\n",
"parameters = {\n",
" \"features\": [\n",
" {\"feature\": \"Sex\",\n",
" \"majority\": ['male'],\n",
" \"minority\": ['female'],\n",
" \"threshold\": 0.95\n",
" },\n",
" {\"feature\": \"Age\",\n",
" \"majority\": [[26, 75]],\n",
" \"minority\": [[18, 25]],\n",
" \"threshold\": 0.95\n",
" }\n",
" ],\n",
" \"favourable_class\": [\"No Risk\"],\n",
" \"unfavourable_class\": [\"Risk\"],\n",
" \"min_records\": 40\n",
"}\n",
"\n",
"fairness_monitor_details = wos_client.monitor_instances.create(\n",
" data_mart_id=data_mart_id,\n",
" background_mode=False,\n",
" monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
" target=target,\n",
" parameters=parameters).result\n",
"fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n",
"fairness_monitor_instance_id"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Payload logging and Run fairness monitor on demand"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import requests\n",
"import time\n",
"import json\n",
"\n",
"scoring_url = subscription_details['entity']['deployment']['scoring_endpoint']['url']\n",
"\n",
"payload_data = {\n",
" \"Inputs\": {\n",
" \"input1\":\n",
" training_data_pd.sample(40).drop(['Risk'],axis=1).to_dict('records'),\n",
" },\n",
" \"GlobalParameters\": {\n",
" }\n",
" }\n",
"\n",
"pl_body = str.encode(json.dumps(payload_data))\n",
"\n",
"\n",
"token = model_asset_details['entity']['scoring_endpoint']['credentials']['token']\n",
"headers = subscription_details['entity']['deployment']['scoring_endpoint']['request_headers']\n",
"headers['Authorization'] = ('Bearer ' + token)\n",
"\n",
"start_time = time.time()\n",
"response = requests.post(url=scoring_url, data=pl_body, headers=headers)\n",
"response_time = int(time.time() - start_time)*1000\n",
"result = response.json()\n",
"\n",
"print(json.dumps(result, indent=2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Store the request and response in payload logging table"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"openscale_paylod_input=convert_user_input_2_openscale(payload_data)\n",
"openscale_payload_output=convert_user_output_2_openscale(result)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import uuid\n",
"from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
"\n",
"print(\"Performing explicit payload logging.....\")\n",
"wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n",
" scoring_id=str(uuid.uuid4()),\n",
" request=openscale_paylod_input,\n",
" response=openscale_payload_output,\n",
" response_time=460\n",
")])\n",
"time.sleep(5)\n",
"pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
"print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.data_sets.show_records(payload_data_set_id)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"time.sleep(10)\n",
"\n",
"wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explainability configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"target = Target(\n",
" target_type=TargetTypes.SUBSCRIPTION,\n",
" target_id=subscription_id\n",
")\n",
"parameters = {\n",
" \"enabled\": True\n",
"}\n",
"explainability_details = wos_client.monitor_instances.create(\n",
" data_mart_id=data_mart_id,\n",
" background_mode=False,\n",
" monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
" target=target,\n",
" parameters=parameters\n",
").result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"explainability_monitor_id = explainability_details.metadata.id\n",
"explainability_monitor_id"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run explanation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.data_sets.show_records(data_set_id=payload_data_set_id)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"pl_records_resp = wos_client.data_sets.get_list_of_records(data_set_id=payload_data_set_id, limit=1, offset=0).result\n",
"scoring_ids = [pl_records_resp[\"records\"][0][\"entity\"][\"values\"][\"scoring_id\"]]\n",
"print(\"Running explanations on scoring IDs: {}\".format(scoring_ids))\n",
"explanation_types = [\"lime\", \"contrastive\"]\n",
"result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types).result\n",
"print(result)\n",
"explanation_task_id=result.to_dict()['metadata']['explanation_task_ids'][0]\n",
"explanation_task_id"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.monitor_instances.get_explanation_tasks(explanation_task_id=explanation_task_id).result.to_dict()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Drift monitor configuration"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Drift requires a trained model to be uploaded manually for Azure. You can train, create and download a drift detection model using template given ( check for Drift detection model generation) [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/training_statistics_notebook.ipynb)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"!rm -rf creditrisk_azure_drift_detection_model.tar.gz\n",
"!wget -O creditrisk_azure_drift_detection_model.tar.gz https://github.com/IBM/watson-openscale-samples/blob/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz?raw=true "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"wos_client.monitor_instances.upload_drift_model(\n",
" model_path='creditrisk_azure_drift_detection_model.tar.gz',\n",
" data_mart_id=data_mart_id,\n",
" subscription_id=subscription_id\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
"for monitor_instance in monitor_instances:\n",
" monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
" if monitor_def_id == \"drift\" and monitor_instance.entity.target.target_id == subscription_id:\n",
" wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
" print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"target = Target(\n",
" target_type=TargetTypes.SUBSCRIPTION,\n",
" target_id=subscription_id\n",
"\n",
")\n",
"parameters = {\n",
" \"min_samples\": 40,\n",
" \"drift_threshold\": 0.1,\n",
" \"train_drift_model\": False,\n",
" \"enable_model_drift\": True,\n",
" \"enable_data_drift\": True\n",
"}\n",
"\n",
"drift_monitor_details = wos_client.monitor_instances.create(\n",
" data_mart_id=data_mart_id,\n",
" background_mode=False,\n",
" monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
" target=target,\n",
" parameters=parameters\n",
").result\n",
"\n",
"drift_monitor_instance_id = drift_monitor_details.metadata.id\n",
"drift_monitor_instance_id"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run Drift monitor on demand"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"time.sleep(5)\n",
"wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Congratulations!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You have finished the tutorial for IBM Watson OpenScale and Azure Machine Learning Studio. You can now view the [OpenScale Dashboard](https://aiopenscale.cloud.ibm.com/). Click on the tile for the German Credit model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.7",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 1
}