{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# funcX Tutorial\n", "\n", "funcX is a Function-as-a-Service (FaaS) platform for science that enables you to convert almost any computing resource into a high-performance function serving device. To do this, you deploy a funcX endpoint agent on the resource, which integrates it into the function serving fabric, allowing you to dynamically send, monitor, and receive results from function invocations. funcX is built on top of [Parsl](https://parsl-project.org), enabling a funcX endpoint to use large compute resources via traditional batch queues, where funcX will dynamically provision, use, and release resources on-demand to fulfill function requests. The function service fabric, which is run centrally as a service, is hosted in AWS.\n", "\n", "Here we provide an example of using funcX to register a function and run it on a publicly available tutorial endpoint." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## funcX Client\n", "\n", "We start by instantiating a funcX client as a programmatic means of communicating with the function service fabric. The client allows you to:\n", "- Register functions\n", "- Register containers and execution environments\n", "- Launch registered functions against endpoints\n", "- Check the status of launched functions\n", "- Retrieve outputs from functions\n", "\n", "#### Authentication\n", "\n", "Instantiating a client will force an authentication flow where you will be asked to authenticate with Globus Auth. Every interaction with funcX is authenticated to allow us to enforce access control on both functions and endpoints. As part of the authentication process we request access to your identity information (to retrieve your email address), Globus Groups management access, and Globus Search. We require Groups access in order to facilitate sharing. Globus Search allows funcX to add your functions to a searchable registry and make them discoverable to permitted users (as well as yourself!). " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from funcx.sdk.client import FuncXClient\n", "\n", "fxc = FuncXClient()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next we define a Python function, which we will later register with funcX. This function simply sums its input.\n", "\n", "When defining a function you can specify \\*args and \\*\\*kwargs as inputs. \n", "\n", "##### Note: any dependencies for a funcX function must be specified inside the function body." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def funcx_sum(items):\n", " return sum(items)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Registering a function\n", "\n", "To use a function with funcX, you must first register it with the service, using `register_function`. You can optionally include a description of the function.\n", "\n", "The registration process will serialize the function body and transmit it to the funcX function service fabric.\n", "\n", "Registering a function returns a UUID for the function, which can then be used to invoke it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "func_uuid = fxc.register_function(funcx_sum,\n", " description=\"tutorial summation\", public=True)\n", "print(func_uuid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Searching a function\n", "\n", "You can search previously registered functions to which you have access using `search_function`. The first parameter `q` is searched against all the fields, such as author, description, function name, and function source. You can navigate through pages of results with the `offset` and `limit` keyword args. \n", "\n", "The object returned is simple wrapper on a list, so you can index into it, but also can have a pretty-printed table. \n", "\n", "To make use of the results, you can either just use the `function_uuid` field returned for each result, or for functions that were registered with recent versions of the service, you can load the source code using the search results object's `load_result` method. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "search_results = fxc.search_function(\"tutorial\", offset=0, limit=5)\n", "print(search_results[0])\n", "print(search_results)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "search_results.load_result(0)\n", "result_0_uuid = search_results[0]['function_uuid']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running a function\n", "\n", "To invoke (perform) a function, you must provide the function's UUID, returned from the registration process, and an `endpoint_id`. Note: here we use the funcX public tutorial endpoint, which is running on AWS.\n", "\n", "The client's `run` function will serialize any \\*args and \\*\\*kwargs, and pass them to the function when invoking it. Therefore, as our example function simply takes an arg input (items), we can specify an input arg and it will be used by the function. Here we define a small list of integers for our function to sum.\n", "\n", "The Web service will return the UUID for the invokation of the function, which we call a task. This UUID can be used to check the status of the task and retrieve the result." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "endpoint_uuid = '4b116d3c-1703-4f8f-9f6f-39921e5864df' # Public tutorial endpoint" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "items = [1, 2, 3, 4, 5]\n", "\n", "res = fxc.run(items, endpoint_id=endpoint_uuid, function_id=func_uuid)\n", "print(res)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can now retrieve the result of the invocation using `get_result()` on the UUID of the task. \n", "\n", "##### Note: We remove the task from our database once the result has been retrieved, thus you can only retireve the result once." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fxc.get_result(res)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running batches\n", "\n", "You might want to invoke many function calls at once. This can be easily done via the batch interface:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def squared(x):\n", " return x**2\n", "squared_uuid = fxc.register_function(squared, searchable=False)\n", "\n", "inputs = list(range(10))\n", "batch = fxc.create_batch()\n", "\n", "for x in inputs:\n", " batch.add(x, endpoint_id=endpoint_uuid, function_id=squared_uuid)\n", " \n", "batch_res = fxc.batch_run(batch)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fxc.get_batch_status(batch_res)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Catching exceptions\n", "\n", "When functions fail, the exception is captured, and reraised when you try to get the result. In the following example, the 'deterministic failure' exception is raised when `fxc.get_result` is called on the failing function." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def failing():\n", " raise Exception(\"deterministic failure\")\n", "failing_uuid = fxc.register_function(failing, searchable=False)\n", "\n", "res = fxc.run(endpoint_id=endpoint_uuid, function_id=failing_uuid)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fxc.get_result(res)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 }