{ "cells": [ { "cell_type": "markdown", "source": [ "# ISB-CGC Community Notebooks\n", "\n", "```\n", "Title: How to use a workflow execution service (WES)\n", "Author: David L Gibbs\n", "Created: 2019-11-16\n", "Purpose: Introduction to using a workflow execution service, GA4GH style\n", "Repo: https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/How_to_use_a_GA4GH_tool_using_WES.ipynb\n", "Notes: Does not work on google colabs.\n", "```" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "# How to use a WES service" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "This notebook is designed to be a quick introduction to using a workflow execution service (WES)\n", "and is intended as a follow-up to a previous notebook on searching for tools using a tool registry service (TRS;\n", "How to find a tool using TRS [here](https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/How_to_find_a_tool_using_GA4GH_TRS.ipynb) ).\n", "This notebook must be run in an environment capable of running docker. Google Colab notebooks will be exteremely difficult \n", "to use. It's advised that a Jupyter-lab environment is started using the Google Cloud Console, AI platform.\n", "\n", "Software used:\n", "\n", "wes-service, a client and server implementation of the GA4GH Workflow Execution Service 1.0.0 API.\n", "\n", "https://github.com/common-workflow-language/workflow-service\n", "https://pypi.org/project/wes-service/\n", "\n", "cwl-tool, Common Workflow Language tool description reference implementation\n", "https://github.com/common-workflow-language/cwltool" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import subprocess as sp" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "!sudo pip install wes-service\n", "!sudo pip install cwltool\n", "!sudo pip install cwlref-runner" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "!wes-client --version" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "We're going to use the subprocess library to start the wes-server in the background. We submit jobs to the wes-server, \n", "which in turn runs them on a backend executor, here, cwltool." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sp.Popen( ['wes-server', '--port', '8885'] )" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# check for jobs... not yet!\n", "\n", "!wes-client --host localhost:8885 --proto http --list" ] }, { "cell_type": "markdown", "source": [ "Let's get some workflow test files to use..." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!git clone https://github.com/common-workflow-language/workflow-service" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "cd workflow-service/testdata/" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "ls -lha" ] }, { "cell_type": "markdown", "source": [ "Now, let's use the TRS to search for a tool called 'md5sum'." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "import requests\n", "response = requests.get('https://dockstore.org:8443/api/ga4gh/v1/tools/', params={\"name\": \"md5sum\"})\n", "n = (len(response.json()[0]['versions'][0])) - 2 # n was just 0 for version 1.0.0\n", "md5sum_url = response.json()[0]['versions'][n]['url'] + '/plain-CWL/descriptor/%2FDockstore.cwl'" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "# we have a url to the CWL.\n", "\n", "print(md5sum_url)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To submit a job to the wes-server, we use the wes-client, and pass the URL and a small json file that \n", "describes the input. It's surprisingly easy. \n", "\n", "Here's the file describing the input:" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "cat md5sum.json" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "Now we'll submit the job." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "!wes-client --host=localhost:8885 --proto http $md5sum_url md5sum.json" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "And we'll view the output..." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "!cat /home/jupyter/workflows/48ea8e524ae848b58bcead5eaae35052/outdir/md5sum.txt\n" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "Let's compare that result to simply running md5sum." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "!md5sum md5sum.input" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Confirmed!\n", "\n", "\n", "Now, what's in that URL?" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "!curl https://dockstore.org/api/api/ga4gh/v1/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-md5sum/versions/master/plain-CWL/descriptor/%2FDockstore.cwl\n" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "Now, for comparison's sake, we'll execute the workflow using the command given on the github readme: \n", "https://github.com/common-workflow-language/workflow-service" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!wes-client --host localhost:8885 --proto http --attachments=\"dockstore-tool-md5sum.cwl,md5sum.input\" md5sum.cwl md5sum.cwl.json\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cat /home/jupyter/workflows/0fc518dfd1fd480999315ec9499e6f69/outdir/md5sum.txt" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!md5sum md5sum.input" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "CONFIRMED !\n", "\n", "Using the wes-service was actually fairly easy, provided you have a nice CWL tool description!\n", "\n", "Please let us know if you have any questions!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.3" }, "pycharm": { "stem_cell": { "cell_type": "raw", "source": [], "metadata": { "collapsed": false } } } }, "nbformat": 4, "nbformat_minor": 4 }