{ "cells": [ { "cell_type": "markdown", "id": "80a38ff6-dfe6-4c7a-a378-d16a4fbb0786", "metadata": {}, "source": [ "# Cluster API\n", "\n", "IPython Parallel 7 adds a `Cluster` API for starting/stopping clusters.\n", "\n", "This is the new implementation of `ipcluster`,\n", "which can be more easily re-used in Python programs.\n", "The `ipcluster` script is\n", "\n", "Controllers and Engines are started with \"Launchers\",\n", "which are objects representing a running process.\n", "\n", "Each **Cluster** has:\n", "\n", "- a **cluster id**\n", "- a **profile directory**\n", "- one **controller**\n", "- zero or more **engine sets**\n", " - each of which has one or more **engines**\n", " \n", "The combination of `profile_dir` and `cluster_id` uniquely identifies a cluster.\n", "You can have many clusters in one profile, but each must have a distinct cluster id.\n", "\n", "To create a cluster, instantiate a Cluster object:" ] }, { "cell_type": "code", "execution_count": 1, "id": "ef0dff26-f6d2-4d79-901b-049debe0c0d1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from ipyparallel import Cluster\n", "\n", "cluster = Cluster()\n", "cluster" ] }, { "cell_type": "markdown", "id": "db2688db-d6e6-4bb2-8fec-cb38fe4d8c59", "metadata": {}, "source": [ "To start the cluster:" ] }, { "cell_type": "code", "execution_count": 2, "id": "23e385fc-6bc2-44ad-b544-0f39182c1a76", "metadata": {}, "outputs": [ { "data": { "text/plain": [ ")>" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "await cluster.start_controller()\n", "cluster" ] }, { "cell_type": "code", "execution_count": 3, "id": "8f48261b-6787-424f-a9e7-b24fb04d076d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting 4 engines with \n" ] }, { "data": { "text/plain": [ ", engine_sets=['1623263481-w75s'])>" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "engine_set_id = await cluster.start_engines(n=4)\n", "cluster" ] }, { "cell_type": "markdown", "id": "af046a0c-281c-4dad-9dc7-582a8cda6b06", "metadata": {}, "source": [ "As you can see, all methods on the Cluster object are async by default.\n", "Every async method also has a `_sync` variant, if you don't want to / can't use asyncio." ] }, { "cell_type": "code", "execution_count": 4, "id": "e239a78a-2907-43e4-80be-eaba13dfac0a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting 2 engines with \n" ] }, { "data": { "text/plain": [ "'1623263483-iafz'" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "engine_set_2 = cluster.start_engines_sync(n=2)\n", "engine_set_2" ] }, { "cell_type": "markdown", "id": "f289aaba-7417-40d3-84ea-88bf83715140", "metadata": {}, "source": [ "At this point, we have a cluster with a controller and six engines in two groups.\n", "\n", "There is also a `start_cluster` method that starts the controller and one engine set, for convenience:\n", "\n", "```python\n", "engine_set_id = await cluster.start_cluster(n=4)\n", "```\n", "\n", "We can get a client object connected to the cluster with `connect_client()`" ] }, { "cell_type": "code", "execution_count": 5, "id": "fe2dbd68-1dcb-4de9-a45f-757314261b8c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[0, 1, 2, 3, 4, 5]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rc = cluster.connect_client()\n", "rc.wait_for_engines(6)\n", "rc.ids" ] }, { "cell_type": "markdown", "id": "d9da9293-3a44-4f16-8f81-7faaf80f20cd", "metadata": {}, "source": [ "And we can use our classic `apply_async(...).get_dict()` pattern to get a dict by engine id of hostname, pid for each engine:" ] }, { "cell_type": "code", "execution_count": 6, "id": "a00c2116-269e-4016-a3d3-ec71ae1b093c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{0: {'host': 'touchy', 'pid': 81944},\n", " 1: {'host': 'touchy', 'pid': 81945},\n", " 2: {'host': 'touchy', 'pid': 81946},\n", " 3: {'host': 'touchy', 'pid': 81947},\n", " 4: {'host': 'touchy', 'pid': 81952},\n", " 5: {'host': 'touchy', 'pid': 81953}}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def identify():\n", " import os\n", " import socket\n", "\n", " return {\"host\": socket.gethostname(), \"pid\": os.getpid()}\n", "\n", "\n", "rc[:].apply_async(identify).get_dict()" ] }, { "cell_type": "markdown", "id": "1e71632f-d2d2-4c1b-b258-54ee639a8409", "metadata": {}, "source": [ "We can send signals to engine sets by id\n", "\n", "*(sending signals to just one engine is still a work in progress)*" ] }, { "cell_type": "code", "execution_count": 7, "id": "6ff6f7da-38e0-4fea-83af-ff875eebcc76", "metadata": {}, "outputs": [], "source": [ "import signal\n", "import time\n", "\n", "ar = rc[:].apply_async(time.sleep, 100)" ] }, { "cell_type": "code", "execution_count": 8, "id": "e898c30f-996b-44cd-bfad-84516d27a004", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sending signal 2 to engine(s) 1623263481-w75s\n", "Sending signal 2 to engine(s) 1623263483-iafz\n" ] }, { "ename": "CompositeError", "evalue": "one or more exceptions from call to method: sleep\n[0:apply]: KeyboardInterrupt: \n[1:apply]: KeyboardInterrupt: \n[2:apply]: KeyboardInterrupt: \n[3:apply]: KeyboardInterrupt: \n.... 2 more exceptions ...", "output_type": "error", "traceback": [ "[0:apply]: ", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m: ", "", "[1:apply]: ", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m: ", "", "[2:apply]: ", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m: ", "", "[3:apply]: ", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m: ", "", "... 2 more exceptions ..." ] } ], "source": [ "# oops! I meant 1!\n", "\n", "await cluster.signal_engines(signal.SIGINT)\n", "ar.get()" ] }, { "cell_type": "markdown", "id": "ba286e10-0e9d-41af-9016-ceb8d7943cbe", "metadata": {}, "source": [ "Now it's time to cleanup. Every `start_` method has a correspinding `stop_method`.\n", "\n", "We can stop one engine set at a time with `stop_engines`:" ] }, { "cell_type": "code", "execution_count": 9, "id": "70574035-e97b-4a3b-acfd-ca404bab7e04", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stopping engine(s): 1623263483-iafz\n" ] } ], "source": [ "await cluster.stop_engines(engine_set_2)" ] }, { "cell_type": "markdown", "id": "89026a15-60a2-40b5-beff-8b281ae624b3", "metadata": {}, "source": [ "Or stop the whole cluster" ] }, { "cell_type": "code", "execution_count": 10, "id": "81275363-a1db-4d6d-9330-514d2355a219", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stopping engine(s): 1623263481-w75s\n", "Stopping controller\n", "Controller stopped: {'exit_code': 0, 'pid': 81906}\n" ] } ], "source": [ "await cluster.stop_cluster()" ] }, { "cell_type": "markdown", "id": "7835fa97-59dc-4146-9bdf-43640bbb1801", "metadata": {}, "source": [ "## Cluster as a context manager\n", "\n", "Cluster can also be used as a Context manager,\n", "in which case:\n", "\n", "1. entering the context manager starts the cluster\n", "2. the `as` returns a connected client\n", "3. the context is only entered when all the engines are fully registered and available\n", "4. when the context exits, the cluster is torn down\n", "\n", "This makes it a lot easier to scope an IPython cluster for the duration of a computation\n", "and ensure that it is cleaned up when you are done." ] }, { "cell_type": "code", "execution_count": 11, "id": "31bf1cfb-9349-46a1-b412-50678368a017", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting 4 engines with \n", "Waiting for connection file: ~/.ipython/profile_default/security/ipcontroller-touchy-1623263508-mdel-client.json\n", "Stopping engine(s): 1623263508-5i4g\n", "Stopping controller\n" ] }, { "data": { "text/plain": [ "{0: 82284, 1: 82282, 2: 82283, 3: 82285}" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "Controller stopped: {'exit_code': 0, 'pid': 82281}\n" ] } ], "source": [ "import os\n", "\n", "with Cluster(n=4) as rc:\n", " engine_pids = rc[:].apply_async(os.getpid).get_dict()\n", "engine_pids" ] }, { "cell_type": "markdown", "id": "5d2d5f22-8116-45e8-8c82-c24e19f7e0de", "metadata": {}, "source": [ "It can also be async" ] }, { "cell_type": "code", "execution_count": 12, "id": "3c0cdb81-4288-4e3a-b25a-7b0b0ce6016a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting 2 engines with \n", "Waiting for connection file: ~/.ipython/profile_default/security/ipcontroller-touchy-1623263514-nqk6-client.json\n", "Stopping engine(s): 1623263514-b9f9\n", "Stopping controller\n" ] }, { "data": { "text/plain": [ "{0: 82407, 1: 82408}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "Controller stopped: {'exit_code': 0, 'pid': 82406}\n" ] } ], "source": [ "async with Cluster(n=2) as rc:\n", " engine_pids = rc[:].apply_async(os.getpid).get_dict()\n", "engine_pids" ] }, { "cell_type": "markdown", "id": "b7537226-dc39-4859-bcfb-16c79a16b951", "metadata": {}, "source": [ "## Launcher classes\n", "\n", "IPython's mechanism for launching controllers and engines is called `Launchers`.\n", "These are in `ipyparallel.cluster.launcher`.\n", "\n", "There are two kinds of Launcher:\n", "\n", "- ControllerLauncher, which starts a controller\n", "- EngineSetLauncher, which starts `n` engines\n", "\n", "You can use abbreviations to access the launchers that ship with IPython parallel,\n", "such as 'MPI', 'Local', or 'SGE',\n", "or you can pass classes themselves (or their import strings, such as 'mymodule.MyEngineSetLauncher').\n", "\n", "I'm going to start a cluster with engines using MPI:" ] }, { "cell_type": "code", "execution_count": 13, "id": "b85d2009-d60c-4642-a15c-84c80adcc361", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting 4 engines with \n" ] } ], "source": [ "cluster = Cluster(n=4, engine_launcher_class='MPI')\n", "await cluster.start_cluster()\n", "rc = cluster.connect_client()" ] }, { "cell_type": "code", "execution_count": 14, "id": "91062c98-3ff2-494f-ae02-6cb96648b6ef", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[0, 1, 2, 3]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rc.wait_for_engines(4)\n", "rc.ids" ] }, { "cell_type": "markdown", "id": "9274020a-a556-4d4c-a67b-170b448d1a91", "metadata": {}, "source": [ "Now I'm going to run a test with another new feature" ] }, { "cell_type": "code", "execution_count": 15, "id": "9a078e90-bdf8-4f8e-aedc-bcabf2c9ba87", "metadata": {}, "outputs": [ { "ename": "TimeoutError", "evalue": "Result not ready.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTimeoutError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/ipykernel_81840/824703262.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0mar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muhoh\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m \u001b[0mar\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/dev/ip/parallel/ipyparallel/client/asyncresult.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m 227\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 228\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 229\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTimeoutError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Result not ready.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 230\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_check_ready\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mTimeoutError\u001b[0m: Result not ready." ] } ], "source": [ "def uhoh():\n", " import time\n", " from mpi4py import MPI\n", "\n", " rank = MPI.COMM_WORLD.rank\n", " if rank == 0:\n", " print(\"rank 0: oh no.\")\n", " 1 / 0\n", " print(f\"rank {rank}: barrier\")\n", " MPI.COMM_WORLD.barrier()\n", "\n", "\n", "ar = rc[:].apply_async(uhoh)\n", "ar.get(timeout=2)" ] }, { "cell_type": "markdown", "id": "f69526d7-4c39-422e-8654-76fbac9ce14e", "metadata": {}, "source": [ "Uh oh! We are stuck in barrier because engine 0 failed.\n", "\n", "Let's try interrupting and getting the errors:" ] }, { "cell_type": "code", "execution_count": 16, "id": "ebe3e6d1-10cf-49be-83b6-da8dbaf712b1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sending signal 2 to engine(s) 1623263525-cijg\n" ] }, { "ename": "TimeoutError", "evalue": "Result not ready.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTimeoutError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/ipykernel_81840/3902026823.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0msignal\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mawait\u001b[0m \u001b[0mcluster\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msignal_engines\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msignal\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSIGINT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mar\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/dev/ip/parallel/ipyparallel/client/asyncresult.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m 227\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 228\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 229\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTimeoutError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Result not ready.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 230\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_check_ready\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mTimeoutError\u001b[0m: Result not ready." ] } ], "source": [ "import signal\n", "await cluster.signal_engines(signal.SIGINT)\n", "ar.get(timeout=2)" ] }, { "cell_type": "markdown", "id": "afd8854b-2138-458d-b2bb-69acdc009b7f", "metadata": {}, "source": [ "It didn't work! This is because MPI.barrier isn't actually interruptible 😢.\n", "\n", "We are going to have to resort to more drastic measures, and *restart* the engines:" ] }, { "cell_type": "code", "execution_count": 17, "id": "2ae1cb50-965b-400e-b39f-4567df667da2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stopping engine(s): 1623263525-cijg\n", "Starting 4 engines with \n", "engine set stopped 1623263525-cijg: {'exit_code': -9, 'pid': 82790}\n" ] } ], "source": [ "await cluster.restart_engines()" ] }, { "cell_type": "code", "execution_count": 19, "id": "c676323b-1bf7-4983-8139-5b085c2fdf91", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[4, 5, 6, 7]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rc.wait_for_engines(4)\n", "rc.ids" ] }, { "cell_type": "markdown", "id": "7c49fa83-087c-4237-acbe-7d6c8ca208c9", "metadata": {}, "source": [ "We are now back to having 4 responsive engines.\n", "Their IPP engine id may have changed, but I can get back to using them." ] }, { "cell_type": "code", "execution_count": 20, "id": "7e261d39-0793-4207-813a-24caa4ec9a92", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{4: 0, 5: 3, 6: 1, 7: 2}" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def get_rank():\n", " from mpi4py import MPI\n", "\n", " return MPI.COMM_WORLD.rank\n", "\n", "rank_map = rc[:].apply_async(get_rank).get_dict()\n", "rank_map" ] }, { "cell_type": "markdown", "id": "9a67ca12-67f5-4539-939f-80a6e61b5692", "metadata": {}, "source": [ "Finally, clean everything up" ] }, { "cell_type": "code", "execution_count": 21, "id": "060108bb-229b-4829-bf32-0dc7d0db0345", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stopping engine(s): 1623263525-cijg\n", "Stopping controller\n", "Controller stopped: {'exit_code': 0, 'pid': 82767}\n", "engine set stopped 1623263525-cijg: {'exit_code': 1, 'pid': 82998}\n" ] } ], "source": [ "await cluster.stop_cluster()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }