{ "cells": [ { "cell_type": "markdown", "id": "dbcd9f7d-ae7d-4e33-977e-8682274acd7b", "metadata": {}, "source": [ "\"Oasis\n", "


\n", "\n", "Exercise 3: Autoscaling \n", "========================\n", "\n" ] }, { "cell_type": "markdown", "id": "6a2cc386-6360-49f6-964c-0d16f07ed3c4", "metadata": {}, "source": [ "## 3.1 Autoscalling overview\n", "Each model in the OasisAPI has two new endpoints for controlling `job chunking` and `worker Autoscaling`. These match up with the meta-data files `chunking_configuration.json` and `scaling_configuration.json` which were uploaded in exercise 2. Like with the model_settings, on model registration these endpoints are updated with the json files stored in the azure file share. So fetching these values for the CHAZ model should match up with the files uploaded.\n", "\n", "\n", "\"chunking\"\n", "\"scalling\"
\n", "\n", "\n", "These two sets of options are read by seperate [kubernetes pods](https://kubernetes.io/docs/concepts/workloads/pods/) \n", "\n", "\"chunking\"
\n", "\n", "\n", "* **oasis-task-controller** - Is responsible for reading a model's `chunking_configuration` and splitting up, either `input_generation` or `loss_analyses`, requests into a fixed number of celery **sub-tasks** which can then be distributed across multiple nodes in the cluster. \n", "\n", "* **oasis-worker-controller** Manages the number of running worker pods by reading `scaling_configuration` and adjusting a model deployments [ReplicaSet](https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/) based on the current Job Queue status. (pushed as Async updates from the Oasis Websocket) \n", "\n", "\n", "\n", "### Sub-task concurrency\n", "The number of concurrently running 'slots' for sub-tasks is based on both the number of running worker pods AND how many cores are available per worker. The worker node type used in this workshop is [Standard_E2_v3](https://azureprice.net/vm/Standard_E2_v3) which has **2 vCPUs** per worker node. So if 4 workers are running, the maximum allocation per workshop account, then (4*2), **8 chunks can run in parallel** of either input or loss generation.\n", "**This limit applies across all models** \n" ] }, { "cell_type": "markdown", "id": "7977ca7b-d637-4364-8339-f45c0e9fa28c", "metadata": { "tags": [] }, "source": [ "## 3.2 Understanding the Configuration options " ] }, { "cell_type": "code", "execution_count": null, "id": "e540c016-c17e-418c-b2ab-e081521090c1", "metadata": {}, "outputs": [], "source": [ "# Edit this value to match your workshop ID, if your username is `workshop6@oasislmfenterprise.onmicrosoft.com` set 'WORKSHOP_ID=6'\n", "WORKSHOP_ID=\n", "\n", "# Install of workshop package \n", "!pip install oasis-workshop==1.0.0\n", "\n", "# Clear cell output\n", "from IPython.display import clear_output\n", "clear_output(wait=False) " ] }, { "cell_type": "code", "execution_count": null, "id": "18573956-f4ff-48c3-afa9-aa25ce49ed9d", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Python Imports \n", "from requests import get\n", "\n", "from oasis_workshop.client import APIClient\n", "from oasis_workshop.funcs import (\n", " tabulate_endpoint, \n", " tabulate_json,\n", " tabulate_analysis,\n", " tabulate_portfolio,\n", " plot_subtasks\n", ")\n", " \n", "from IPython.display import (\n", " display, \n", " Code,\n", " clear_output, \n", " HTML, \n", " JSON,\n", " Markdown \n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "58e14441-6eb7-4b33-a590-721dea68cfe5", "metadata": {}, "outputs": [], "source": [ "# Start API connection \n", "oasis_server = APIClient(\n", " api_url=f'https://oasis-workshop-{WORKSHOP_ID}.northcentralus.cloudapp.azure.com/api/',\n", " username='admin', \n", " password='password'\n", ")\n", "\n", "# Get current configs for the Chaz & PiWind models \n", "CHAZ_MODEL_ID = oasis_server.models.search({\"supplier_id__contains\":'columbia'}).json()[0]['id']\n", "PIWIND_MODEL_ID = oasis_server.models.search({\"supplier_id__contains\":'OasisLMF'}).json()[0]['id']\n", "\n", "display(JSON(oasis_server.models.chunking_configuration.get(CHAZ_MODEL_ID).json(), root='CHAZ - chunking_configuration'))\n", "display(Markdown('___'))\n", "display(JSON(oasis_server.models.scaling_configuration.get(CHAZ_MODEL_ID).json(), root='CHAZ - scaling_configuration'))\n", "#display(Markdown('___'))\n", "#display(JSON(oasis_server.models.chunking_configuration.get(PIWIND_MODEL_ID).json(), root='PiWind - chunking_configuration'))\n", "#display(Markdown('___'))\n", "#display(JSON(oasis_server.models.scaling_configuration.get(PIWIND_MODEL_ID).json(), root='PiWind - scaling_configuration'))" ] }, { "cell_type": "markdown", "id": "130ba2cf-f6c8-4519-b169-95e70398a1eb", "metadata": {}, "source": [ "## 3.2.2 Scaling options\n", "There are three modes of scaling operation which are controlled using the `scaling_strategy` key. \n", "The idle state for all Strategies is 0 worker pods running, this applies when a model's main celery queue is empty. \n", "\n", "#### **Strategy 1** - FIXED_WORKERS\n", "When one or more tasks are placed on a model queue, then a fixed number of workers will be started. Once all of the pending sub-tasks (or chunks) have completed the model deploymented will return to the idle state, where all workers are shut down.\n", "The number of workers started is controlled by the int value stored in `worker_count_fixed`. \n", "\n", "> **Note:** The control values not connected with a selected strategy have no effect when set.\n", "So when running with `FIXED_WORKERS` only the value in `worker_count_fixed` is used. \n", "Both `worker_count_max` and `chunks_per_worker` are ignored and have no effect on the number of workers started. The same applies to all other modes, only the values relevant to that Strategy are applied. " ] }, { "cell_type": "code", "execution_count": null, "id": "ea456e07-1a01-410e-9b0e-5e9b9dc31a37", "metadata": {}, "outputs": [], "source": [ "req = oasis_server.models.scaling_configuration.post(PIWIND_MODEL_ID, {\n", " \"scaling_strategy\": \"FIXED_WORKERS\",\n", " \"worker_count_fixed\": 4\n", "})" ] }, { "cell_type": "markdown", "id": "29dd2195-748d-4c84-acc9-bd242c3c8511", "metadata": {}, "source": [ "#### **Strategy 2** - QUEUE_LOAD\n", "The number of workers started depends on how many pending tasks are waiting on a model's queue. These are **main tasks** and not sub-tasks. So if `` loss analysis requests are sent to the API via `/api/v1/analyses/{id}/run/` then `` worker pods will be started, up to a maximum of `worker_count_max`. " ] }, { "cell_type": "code", "execution_count": null, "id": "0ee7cc20-c6b0-4a1a-98b2-5502053d9981", "metadata": {}, "outputs": [], "source": [ "req = oasis_server.models.scaling_configuration.post(PIWIND_MODEL_ID, {\n", " \"scaling_strategy\": \"QUEUE_LOAD\",\n", " \"worker_count_max\": 4\n", "})" ] }, { "cell_type": "markdown", "id": "3479c4d7-5292-41cb-b866-03e5aae75ff0", "metadata": {}, "source": [ "#### **Strategy 3** - DYNAMIC_TASKS\n", "The number of workers started is controlled by the number of **sub-tasks** waiting on the model queue. The number of workers started is the sum of all sub-tasks (a.k.a chunks) on the model queue divided by `chunks_per_worker`. For example, if three loss analysis requests are send and each broken up into 15 chunks and `chunks_per_worker = 5` we'll have **{3 * 15} / 5 = 9 workers** started. \n", "However, the maximum cap `worker_count_max` still applies to this strategy." ] }, { "cell_type": "code", "execution_count": null, "id": "597ad902-f2cb-43d0-83bb-35d90945b439", "metadata": {}, "outputs": [], "source": [ "CHUNKS_PER_WORKER = 2\n", "req = oasis_server.models.scaling_configuration.post(PIWIND_MODEL_ID, {\n", " \"scaling_strategy\": \"DYNAMIC_TASKS\",\n", " \"chunks_per_worker\": CHUNKS_PER_WORKER, \n", " \"worker_count_max\": 4\n", "})" ] }, { "cell_type": "markdown", "id": "36348791-7294-4cd9-858e-e1cc30b2aa7d", "metadata": {}, "source": [ "## 3.2.3 Chunking options\n", "There are two modes, **FIXED** and **DYNAMIC**, for splitting up either `lookup` or `loss` stages. These are controlled independently using `lookup_strategy` and `loss_strategy`.\n", "\n", "#### **Strategy 1** - FIXED_CHUNKS\n", "Setting a fixed value breaks every **main task** into a fixed number of chunks. So if we set the following for a model." ] }, { "cell_type": "code", "execution_count": null, "id": "80ca2f65-192c-4adb-a33c-b205f28cb0d4", "metadata": {}, "outputs": [], "source": [ "FIXED_LOOKUP_CHUNKS = 10\n", "req = oasis_server.models.chunking_configuration.post(PIWIND_MODEL_ID, {\n", " \"lookup_strategy\": \"FIXED_CHUNKS\",\n", " \"fixed_lookup_chunks\": FIXED_LOOKUP_CHUNKS\n", "})" ] }, { "cell_type": "markdown", "id": "923fcdf3-4479-41ee-b4c8-d366556c6605", "metadata": {}, "source": [ "Every `/api/v1/analyses/{id}/generate_inputs/` request will split the given **location.csv** into **FIXED_LOOKUP_CHUNKS** even partitions and run the oasis lookup on each partition in parallel (and across multiple workers). \n" ] }, { "cell_type": "code", "execution_count": null, "id": "a825bc8c-9893-4ec4-bc25-ef6b4b36d86b", "metadata": {}, "outputs": [], "source": [ "FIXED_EVENT_BATCHES = 20\n", "req = oasis_server.models.chunking_configuration.post(PIWIND_MODEL_ID, {\n", " \"loss_strategy\": \"FIXED_CHUNKS\",\n", " \"fixed_analysis_chunks\": FIXED_EVENT_BATCHES\n", "})" ] }, { "cell_type": "markdown", "id": "8892fd08-9e08-4f9d-857c-7940de5401bf", "metadata": {}, "source": [ "Now every `/api/v1/analyses/{id}/run/` request will split into **FIXED_EVENT_BATCHES** event batches. \n", "Each event batch is analogous to the ktools execution pipes, and runs in its own bash scripted.\n", "`eve 1 | modelpy | gulpy ... etc`\n", "\n", "\n", "#### Notes on Fixed chunking \n", "\n", "* **1. Minimum chunking** - If a file size is smaller than the requested chunk size, say a 4 line location is requested to be split 5 ways, then only 4 chunks will be created.\n", "\n", "* **2. Single fixed chunk** - This is treated as a special case, If a request for a fixed single chunk is sent, this task is not split and acts like tasks from the version `1.x.x` Platform. The task will run only on a single worker pod and parallelize across the resources of that single node. \n", "\n", "#### **Strategy 2** - DYNAMIC_CHUNKS\n", "\n", "Using the `DYNAMIC`chunking strategy, will automatically scale the number of sub-tasks based on the size of the input given. \n", "For lookup this is based on the number of lines in the **location.csv**. So if a location file has 100,000 lines and \n", "`dynamic_locations_per_lookup=10000` is set, then this will result in **10 chunks** each with **10,000 location lines** to process in parallel. \n", "\n", "The value of `dynamic_chunks_max` is a cap on the maximum number of chunks, so setting, `dynamic_locations_per_lookup=1` for that same file will return **200 chunks** and **not 100,000 chunks**." ] }, { "cell_type": "code", "execution_count": null, "id": "c30fb0ec-7aeb-4047-9114-310d45146681", "metadata": {}, "outputs": [], "source": [ "LOC_LINES_PER_CHUNK = 10000\n", "req = oasis_server.models.chunking_configuration.post(PIWIND_MODEL_ID, {\n", " \"lookup_strategy\": \"DYNAMIC_CHUNKS\",\n", " \"dynamic_locations_per_lookup\": LOC_LINES_PER_CHUNK, \n", " \"dynamic_chunks_max\": 200\n", "})" ] }, { "cell_type": "markdown", "id": "49ea451e-a646-441b-bec4-87811514003b", "metadata": {}, "source": [ "Using DYNAMIC chunking for loss generation scales by the size of event set selected, the selected event set from `model_settings.json` _MUST_ have a value set `number_of_events = ` Otherwise the calls to `/api/v1/analyses/{id}/run/` will return a 400 Bad Request response. \n", "\n", "\"chunking\"
\n", "\n", "\n", "The value `dynamic_events_per_analysis=n` works in a similar way as for lookup, sub-tasks are created containing batches of `n` events.\n", "Looking at PiWind, we see that there is **1447** events in set **p**. \n", "\n", "If `p` is selected with `dynamic_events_per_analysis=100`, then a total of **15 sub-tasks** will be generated." ] }, { "cell_type": "code", "execution_count": null, "id": "e81cc0ea-24da-433a-aae6-40d3d7d2395b", "metadata": {}, "outputs": [], "source": [ "display(JSON(oasis_server.models.settings.get(PIWIND_MODEL_ID).json()['model_settings']['event_set']['options'][0], root='event_set'))" ] }, { "cell_type": "code", "execution_count": null, "id": "d3ad035a-88a8-4b99-a1ae-19c7410493af", "metadata": {}, "outputs": [], "source": [ "EVENTS_PER_BATCH = 100\n", "req = oasis_server.models.chunking_configuration.post(PIWIND_MODEL_ID, {\n", " \"loss_strategy\": \"DYNAMIC_CHUNKS\",\n", " \"dynamic_events_per_analysis\": EVENTS_PER_BATCH, \n", " \"dynamic_chunks_max\": 200\n", "})" ] }, { "cell_type": "markdown", "id": "0fd21bf7-34fd-4076-93d0-74090ceee26d", "metadata": {}, "source": [ "## 3.3 Running a fixed size configuration " ] }, { "cell_type": "markdown", "id": "438c1af1-079a-4caa-b0a3-4dd86a4ca7be", "metadata": {}, "source": [ "### 3.3.1 Create PiWind Portfolio & analysis " ] }, { "cell_type": "code", "execution_count": null, "id": "a99c9793-cc7d-4ccc-b429-8bb0417e096c", "metadata": {}, "outputs": [], "source": [ "# Load exposure and create Portfolio \n", "portfolio_name = 'piwind_portfolio'\n", "analysis_name = 'fixed_run_piwind'\n", "\n", "# ---- Data Source ---------------------- #\n", "settings_url = 'https://raw.githubusercontent.com/OasisLMF/OasisPiWind/master/analysis_settings.json'\n", "base_url='https://raw.githubusercontent.com/OasisLMF/OasisPiWind/master/tests/inputs'\n", "loc_url=f'{base_url}/SourceLocOEDPiWind10.csv'\n", "#acc_url=f'{base_url}/SourceAccOEDPiWind.csv'\n", "#scp_url=f'{base_url}/SourceReinsScopeOEDPiWind.csv'\n", "#inf_url=f'{base_url}/SourceReinsInfoOEDPiWind.csv'\n", "\n", "\n", "# ---- Portfolio ---------------------- #\n", "portfolio_list = oasis_server.portfolios.search({'name': portfolio_name}).json()\n", "if len(portfolio_list) > 0:\n", " PORTFOLIO_ID = portfolio_list[-1]['id']\n", "else:\n", " PORTFOLIO_ID = oasis_server.portfolios.create(portfolio_name).json()['id']\n", "\n", "# Upload exposure data \n", "loc_data = get(loc_url).content\n", "#acc_data = get(acc_url).content \n", "#scp_data = get(scp_url).content\n", "#inf_data = get(inf_url).content\n", "\n", "oasis_server.portfolios.location_file.post(PORTFOLIO_ID, loc_data, content_type='text/csv')\n", "#oasis_server.portfolios.accounts_file.post(PORTFOLIO_ID, acc_data, content_type='text/csv')\n", "#oasis_server.portfolios.reinsurance_scope_file.post(PORTFOLIO_ID, scp_data, content_type='text/csv')\n", "#oasis_server.portfolios.reinsurance_info_file.post(PORTFOLIO_ID, inf_data, content_type='text/csv')\n", "\n", "\n", "# ---- Analysis ---------------------- #\n", "# Find or create analsysis \n", "analysis_list = oasis_server.analyses.search({'name': analysis_name}).json()\n", "if len(analysis_list) > 0:\n", " ANALYSIS = analysis_list[0]\n", " ANALYSIS_ID = ANALYSIS['id']\n", "else: \n", " ANALYSIS = oasis_server.analyses.create(analysis_name, portfolio_id=PORTFOLIO_ID, model_id=PIWIND_MODEL_ID).json()\n", " ANALYSIS_ID = ANALYSIS['id']\n", " \n", "# Upload analysis Settings \n", "analysis_settings = get(settings_url).json()\n", "oasis_server.analyses.settings.post(ANALYSIS['id'], analysis_settings).json()\n", "\n", "\n", "# ---- Display State ---------------------- #\n", "display(Markdown('#### Selected Portfolio'))\n", "display(tabulate_portfolio([oasis_server.portfolios.get(PORTFOLIO_ID).json()]))\n", "display(Markdown('#### Selected Analyses'))\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))" ] }, { "cell_type": "markdown", "id": "dfaed4f5-af8f-409a-95e8-f18f3e7a7866", "metadata": {}, "source": [ "### 3.3.2 Set PiWind to Fixed Chunks\n", "Set the number of worker pods to 4 (Max) and set the chunking options to fill all concurrent 'slots'. " ] }, { "cell_type": "code", "execution_count": null, "id": "d13141a6-681f-4ff0-a30e-a4864dc6aaab", "metadata": {}, "outputs": [], "source": [ "# Set chunking to 8 for both input & loss generation\n", "r = oasis_server.models.chunking_configuration.post(PIWIND_MODEL_ID, {\n", " \"lookup_strategy\": \"FIXED_CHUNKS\",\n", " \"fixed_lookup_chunks\": 8,\n", " \"loss_strategy\": \"FIXED_CHUNKS\",\n", " \"fixed_analysis_chunks\": 8\n", "})\n", "\n", "# Set scaling to 4 workers \n", "r = oasis_server.models.scaling_configuration.post(PIWIND_MODEL_ID, {\n", " \"scaling_strategy\": \"FIXED_WORKERS\",\n", " \"worker_count_fixed\": 4\n", "})" ] }, { "cell_type": "markdown", "id": "bd774620-d5cb-42de-a0b6-84e51e06941a", "metadata": {}, "source": [ "### 3.3.3 Generate Oasis Inputs \n", "At this point its worth opening up an [Azure cloud shell](https://shell.azure.com/) and monitoring the running pods.\n", "\n", "#### Watching running pods \n", "\"Azure `watch -n 1 kubectl get pods`\n", "\n", "#### Viewing pod logs \n", "\"Azure `kubectl logs {pod-name}`\n", "\n", "\n", "#### Restarting a pod \n", "This might be needed for the `worker-controller`, which can get stuck in an inconsistent state. If model workers are not spinning up or down try forcing a restart using this command. \n", "\n", "\"Azure `kubectl delete pod {pod-name}`\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b1c5a7cf-f646-4179-b73e-703fdcc5e2e9", "metadata": {}, "outputs": [], "source": [ "if oasis_server.analyses.status(ANALYSIS_ID) != 'READY':\n", " print('Generating Oasis Inputs')\n", " oasis_server.run_generate(ANALYSIS['id'])\n", "\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))" ] }, { "cell_type": "markdown", "id": "662c6884-8021-40f7-a631-d5de326b2c68", "metadata": {}, "source": [ "### 3.3.4 Show List of SubTasks (Optional Step)" ] }, { "cell_type": "code", "execution_count": null, "id": "e66c57f1-cc41-4d32-ad37-2bf6b216bfa8", "metadata": {}, "outputs": [], "source": [ "input_get_subtasks = oasis_server.analyses.sub_task_list(ANALYSIS['id']).json()\n", "JSON(input_get_subtasks, root='Input Generation SubTasks')" ] }, { "cell_type": "markdown", "id": "7fee660f-e399-4a46-9d66-af827255ca62", "metadata": {}, "source": [ "### 3.3.5 Display Log file from one of the SubTasks (Optional Step)" ] }, { "cell_type": "code", "execution_count": null, "id": "d28b07ed-dfbc-4f04-a3b8-9fc68de8196b", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "# Display log file from a Sub-Task\n", "CHUNK_ID = 1\n", "TASK_NAMES= [f\"prepare-keys-file-{CHUNK_ID}\", f\"generate-losses-chunk-{CHUNK_ID}\"]\n", "input_get_subtasks = oasis_server.analyses.sub_task_list(ANALYSIS_ID).json()\n", "subtask_id = [x for x in input_get_subtasks if x['slug'] in TASK_NAMES].pop()['id']\n", "subtask_output_log = oasis_server.task_status.output_log.get(subtask_id).text\n", "Code(subtask_output_log)" ] }, { "cell_type": "markdown", "id": "11e8794a-58da-4802-924a-c435244fda62", "metadata": {}, "source": [ "### 3.3.6 Graph the execution time of each lookup SubTask" ] }, { "cell_type": "code", "execution_count": null, "id": "665493ba-bd9c-4751-a57f-ccadd5ac7523", "metadata": {}, "outputs": [], "source": [ "## Graph SubTask Execution time\n", "if oasis_server.analyses.status(ANALYSIS_ID) == 'READY':\n", " input_get_subtasks = oasis_server.analyses.sub_task_list(ANALYSIS_ID).json()\n", " display(plot_subtasks(input_get_subtasks))\n", "else:\n", " print(\"Warning: Input generation has not completed\")" ] }, { "cell_type": "markdown", "id": "4634be88-b26c-45a0-ac52-5b469a1c291f", "metadata": {}, "source": [ "### 3.3.7 Update PiWind to use only one worker and Generate Losses" ] }, { "cell_type": "code", "execution_count": null, "id": "0aba6ec5-4486-4122-9e63-e2ae9cccd880", "metadata": {}, "outputs": [], "source": [ "oasis_server.models.scaling_configuration.post(PIWIND_MODEL_ID, {\n", " \"scaling_strategy\": \"FIXED_WORKERS\",\n", " \"worker_count_fixed\": 1\n", "})\n", "\n", "if oasis_server.analyses.status(ANALYSIS_ID) != 'RUN_COMPLETED':\n", " print('Starting Analysis Losses')\n", " oasis_server.run_analysis(ANALYSIS_ID)\n", "\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))" ] }, { "cell_type": "markdown", "id": "b78883b3-5eee-462f-8722-574e6178d71d", "metadata": {}, "source": [ "### 3.3.8 Graph the execution time of each loss SubTask\n", "You should see a staggered plot, its because only two subtasks can execute simultaneously. " ] }, { "cell_type": "code", "execution_count": null, "id": "d11b58e3-71f9-466f-87fc-f5e712add179", "metadata": {}, "outputs": [], "source": [ "if oasis_server.analyses.status(ANALYSIS_ID) == 'RUN_COMPLETED':\n", " loss_gen_subtasks = oasis_server.analyses.sub_task_list(ANALYSIS['id']).json()\n", " display(plot_subtasks(loss_gen_subtasks))\n", "else:\n", " print(\"Warning: Execution has not completed\")" ] }, { "cell_type": "markdown", "id": "bc28ddb7-136b-4e71-a093-0deceafbaea2", "metadata": {}, "source": [ "## 3.4 Running a dynamic configuration\n", "\n", "In this section we run the CHAZ model, only this time using Dynamic chunking.\n", "For reference the input sizes are:\n", "* `location_file` = 85803 rows\n", "* `event_set` = 38127 events " ] }, { "cell_type": "markdown", "id": "c24abffd-af55-44e7-9b3c-36ce1624e3ca", "metadata": {}, "source": [ "### 3.4.1 Update the chaz model chunk settings" ] }, { "cell_type": "code", "execution_count": null, "id": "ad4370c4-b17e-4d19-a091-895b3d28eb7b", "metadata": {}, "outputs": [], "source": [ "# Edit thses values to changes the generated sub-tasks\n", "LOC_ROWS_PER_CHUNK=2000\n", "EVENTS_PER_BATCH=1000\n", "MAX_CHUNKS=50\n", "\n", "# Chunks for each execution stage \n", "from math import ceil\n", "location_rows=85803\n", "event_set_size=38127\n", "expected_lookup_chunks = min(ceil(location_rows / LOC_ROWS_PER_CHUNK), MAX_CHUNKS)\n", "expected_loss_chunks = min(ceil(event_set_size / EVENTS_PER_BATCH), MAX_CHUNKS)\n", "\n", "# Set chunking to 8 for both input & loss generation\n", "oasis_server.models.chunking_configuration.post(CHAZ_MODEL_ID, {\n", " \"lookup_strategy\": \"DYNAMIC_CHUNKS\",\n", " \"dynamic_locations_per_lookup\": LOC_ROWS_PER_CHUNK,\n", " \"loss_strategy\": \"DYNAMIC_CHUNKS\",\n", " \"dynamic_events_per_analysis\": EVENTS_PER_BATCH,\n", " \"dynamic_chunks_max\": MAX_CHUNKS\n", "})\n", "\n", "# Set scaling to 4 workers \n", "oasis_server.models.scaling_configuration.post(CHAZ_MODEL_ID, {\n", " \"scaling_strategy\": \"FIXED_WORKERS\",\n", " \"worker_count_fixed\": 4\n", "})\n", "\n", "print(f' lookup_chunks = {expected_lookup_chunks}')\n", "print(f' loss_chunks = {expected_loss_chunks}')" ] }, { "cell_type": "markdown", "id": "ff983403-4e9a-453f-bfa3-dce51a85d45d", "metadata": {}, "source": [ "### 3.5.2 Create CHAZ Portfolio & analysis" ] }, { "cell_type": "code", "execution_count": null, "id": "43b7e67a-6841-45df-b2b9-9daf5c35ab98", "metadata": {}, "outputs": [], "source": [ "# Load exposure and create Portfolio \n", "portfolio_name = 'chaz_portfolio'\n", "analysis_name = 'chaz_dynamic_run'\n", "\n", "# Data Source \n", "base_url = 'https://raw.githubusercontent.com/OasisLMF/Workshop2022/main/examples'\n", "loc_url = f'{base_url}/oed_location_litpop.csv'\n", "run_settings = f'{base_url}/chaz_analysis_settings.json' \n", "\n", "\n", "# ---- Portfolio ---------------------- #\n", "portfolio_list = oasis_server.portfolios.search({'name': portfolio_name}).json()\n", "if len(portfolio_list) > 0:\n", " PORTFOLIO_ID = portfolio_list[-1]['id']\n", "else:\n", " PORTFOLIO_ID = oasis_server.portfolios.create(portfolio_name).json()['id']\n", "\n", "# Upload exposure data \n", "oasis_server.portfolios.location_file.post(\n", " PORTFOLIO_ID, \n", " get(loc_url).content, \n", " content_type='text/csv'\n", ")\n", "\n", "\n", "# ---- Analysis ---------------------- #\n", "# Find or create analsysis \n", "analysis_list = oasis_server.analyses.search({'name': analysis_name}).json()\n", "if len(analysis_list) > 0:\n", " ANALYSIS = analysis_list[0]\n", " ANALYSIS_ID = ANALYSIS['id']\n", "else: \n", " ANALYSIS = oasis_server.analyses.create(analysis_name, portfolio_id=PORTFOLIO_ID, model_id=CHAZ_MODEL_ID).json()\n", " ANALYSIS_ID = ANALYSIS['id']\n", " \n", "# Upload analysis Settings \n", "oasis_server.analyses.settings.post(ANALYSIS['id'], get(run_settings).json()).json()\n", "\n", "\n", "\n", "# ---- Display State ---------------------- #\n", "display(Markdown('#### Selected Portfolio'))\n", "display(tabulate_portfolio([oasis_server.portfolios.get(PORTFOLIO_ID).json()]))\n", "display(Markdown('#### Selected Analyses'))\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))" ] }, { "cell_type": "markdown", "id": "f9579450-6e35-495e-a10d-bb743e032005", "metadata": {}, "source": [ "### 3.5.3 Generate Inputs and Graph" ] }, { "cell_type": "code", "execution_count": null, "id": "f24b4e6a-a985-4153-9149-c6d7b898187e", "metadata": {}, "outputs": [], "source": [ "ANALYSIS = oasis_server.analyses.get(ANALYSIS_ID).json()\n", "if oasis_server.analyses.status(ANALYSIS_ID) != 'READY':\n", " print('Generating Oasis Inputs')\n", " oasis_server.run_generate(ANALYSIS['id'])\n", "\n", "## Graph SubTask Execution timeInputs\n", "if oasis_server.analyses.status(ANALYSIS_ID) == 'READY':\n", " input_get_subtasks = oasis_server.analyses.sub_task_list(ANALYSIS_ID).json()\n", " display(plot_subtasks(input_get_subtasks))\n", "else:\n", " print(\"Warning: Input generaInputstion has not completed\")\n", "\n", "display(Markdown('#### Completed Analyses'))\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))" ] }, { "cell_type": "markdown", "id": "804aabf8-baf9-4b5b-b516-08467f02f0c9", "metadata": {}, "source": [ "### 3.5.3 Generate Losses and Graph" ] }, { "cell_type": "code", "execution_count": null, "id": "5cbfedfe-ce65-4f84-ab03-85c692a49728", "metadata": {}, "outputs": [], "source": [ "if oasis_server.analyses.status(ANALYSIS_ID) != 'RUN_COMPLETED':\n", " print('Starting Analysis Losses')\n", " oasis_server.run_analysis(ANALYSIS_ID)\n", "\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))\n", "\n", "if oasis_server.analyses.status(ANALYSIS_ID) == 'RUN_COMPLETED':\n", " loss_gen_subtasks = oasis_server.analyses.sub_task_list(ANALYSIS['id']).json()\n", " display(plot_subtasks(loss_gen_subtasks))\n", "else:\n", " print(\"Warning: Execution has not completed\")\n", " \n", "display(Markdown('#### Selected Analyses'))\n", "display(tabulate_analysis([oasis_server.analyses.get(ANALYSIS_ID).json()]))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" } }, "nbformat": 4, "nbformat_minor": 5 }