{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# ISB-CGC Community Notebooks\n",
    "\n",
    "```\n",
    "Title:   How to use a workflow execution service (WES)\n",
    "Author:  David L Gibbs\n",
    "Created: 2019-11-16\n",
    "Purpose: Introduction to using a workflow execution service, GA4GH style\n",
    "Repo: https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/How_to_use_a_GA4GH_tool_using_WES.ipynb\n",
    "Notes: Does not work on google colabs.\n",
    "```"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# How to use a WES service"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "This notebook is designed to be a quick introduction to using a workflow execution service (WES)\n",
    "and is intended as a follow-up to a previous notebook on searching for tools using a tool registry service (TRS;\n",
    "How to find a tool using TRS [here](https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/How_to_find_a_tool_using_GA4GH_TRS.ipynb) ).\n",
    "This notebook must be run in an environment capable of running docker. Google Colab notebooks will be exteremely difficult \n",
    "to use. It's advised that a Jupyter-lab environment is started using the Google Cloud Console, AI platform.\n",
    "\n",
    "Software used:\n",
    "\n",
    "wes-service, a client and server implementation of the GA4GH Workflow Execution Service 1.0.0 API.\n",
    "\n",
    "https://github.com/common-workflow-language/workflow-service\n",
    "https://pypi.org/project/wes-service/\n",
    "\n",
    "cwl-tool, Common Workflow Language tool description reference implementation\n",
    "https://github.com/common-workflow-language/cwltool"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess as sp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!sudo pip install wes-service\n",
    "!sudo pip install cwltool\n",
    "!sudo pip install cwlref-runner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!wes-client --version"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We're going to use the subprocess library to start the wes-server in the background. We submit jobs to the wes-server, \n",
    "which in turn runs them on a backend executor, here, cwltool."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp.Popen( ['wes-server', '--port', '8885'] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check for jobs... not yet!\n",
    "\n",
    "!wes-client --host localhost:8885 --proto http --list"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's get some workflow test files to use..."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/common-workflow-language/workflow-service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd workflow-service/testdata/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -lha"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, let's use the TRS to search for a tool called 'md5sum'."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import requests\n",
    "response = requests.get('https://dockstore.org:8443/api/ga4gh/v1/tools/', params={\"name\": \"md5sum\"})\n",
    "n = (len(response.json()[0]['versions'][0])) - 2  # n was just 0 for version 1.0.0\n",
    "md5sum_url = response.json()[0]['versions'][n]['url'] + '/plain-CWL/descriptor/%2FDockstore.cwl'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# we have a url to the CWL.\n",
    "\n",
    "print(md5sum_url)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "To submit a job to the wes-server, we use the wes-client, and pass the URL and a small json file that \n",
    "describes the input.  It's surprisingly easy.  \n",
    "\n",
    "Here's the file describing the input:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "cat md5sum.json"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we'll submit the job."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!wes-client --host=localhost:8885 --proto http  $md5sum_url  md5sum.json"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "And we'll view the output..."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!cat /home/jupyter/workflows/48ea8e524ae848b58bcead5eaae35052/outdir/md5sum.txt\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's compare that result to simply running md5sum."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!md5sum md5sum.input"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Confirmed!\n",
    "\n",
    "\n",
    "Now, what's in that URL?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!curl https://dockstore.org/api/api/ga4gh/v1/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-md5sum/versions/master/plain-CWL/descriptor/%2FDockstore.cwl\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, for comparison's sake, we'll execute the workflow using the command given on the github readme: \n",
    "https://github.com/common-workflow-language/workflow-service"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wes-client --host localhost:8885 --proto http --attachments=\"dockstore-tool-md5sum.cwl,md5sum.input\" md5sum.cwl md5sum.cwl.json\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat /home/jupyter/workflows/0fc518dfd1fd480999315ec9499e6f69/outdir/md5sum.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!md5sum md5sum.input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "CONFIRMED !\n",
    "\n",
    "Using the wes-service was actually fairly easy, provided you have a nice CWL tool description!\n",
    "\n",
    "Please let us know if you have any questions!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}