{
"cells": [
{
"cell_type": "markdown",
"id": "0a3b715a-560d-463f-a7f4-aea8ec65a262",
"metadata": {
"tags": []
},
"source": [
"# Snakemake Overview\n",
"\n",
"A Snakemake workflow is defined in terms of rules that are written in a file named Snakefile (similar to a Makefile with GNU Make). Rules consist of a name, input file(s), output file(s), and a shell command to generate the output from the input. Dependencies between rules are handled implicitly, by matching filenames of input files against output files.\n",
"\n",
"## A first workflow\n",
"\n",
"To illustrate the use of Snakemake, we will test if a book follows the Zipf law: an empirical law which states that given a large sample of words, the frequency of any word is inversely proportional to its rank in the frequency table.\n",
"\n",
"The first rule (named `count_words`) will take a book stored in text file as input and generate a list of words sorted by the number of occurrences in the book and a second rule (named `fit_zipf`) will try to fit the data from previous step to check if it follows the Zipf law. The final step (rule named `plot_zipf`) will generate a graph from the generated data.\n",
"\n",
"These 3 rules are in the file named `Snakefile` that will be used to provide the workflow description to Snakemake.\n",
"\n",
"\n",
"Note that the first rule defined in the file is the plot_zipf rule. By default Snakemake executes the first rule in the Snakefile, thus, the rule that produce the final result should be the first rule.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "feb9ef80-8057-446f-8e01-b588a5ab8554",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile\n",
"\n",
"rule plot_zipf: \n",
" input: 'outputs/isles.dat', 'outputs/isles.fit.dat'\n",
" output: 'outputs/isles.png'\n",
" shell: 'python scripts/plotzipf.py outputs/isles.dat outputs/isles.fit.dat outputs/isles.png' \n",
"\n",
"rule fit_zipf:\n",
" input: 'outputs/isles.dat'\n",
" output: 'outputs/isles.fit.dat'\n",
" shell: 'python scripts/fitzipf.py outputs/isles.dat outputs/isles.fit.dat'\n",
"\n",
"rule count_words: \n",
" input: 'books/isles.txt' \n",
" output: 'outputs/isles.dat' \n",
" shell: 'python scripts/wordcount.py books/isles.txt outputs/isles.dat'"
]
},
{
"cell_type": "markdown",
"id": "d4db1024-1d59-4fb0-bb75-52f7b3920b59",
"metadata": {},
"source": [
"\n",
"The input and output files are defined relative to the current working directory \n",
"
\n",
"\n",
"We can check the validy of our workflow by performing a dry run. This can be done by passing the `-n` option when invoking Snakemake"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "815a2268-c608-407d-8389-9eb61a2ef68b",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --forceall -n"
]
},
{
"cell_type": "markdown",
"id": "38510eca-bdc3-43ab-b133-e760bd7059a7",
"metadata": {},
"source": [
"We use the `--forceall` option to make sure that the whole workflow is executed regardless of the eventual presence of previously generated files.\n",
"\n",
"Another thing we can do is visualize the directed acyclic graph (DAG) our workflow of our workflow with the `--dag` option and pipe the output to the Graphviz `dot` command to get a PNG from the DAG."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1b4ffff3-72e1-4580-b23f-4d0028efc96f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!snakemake --forceall --dag | dot -Tpng > dag.png\n",
"from IPython.display import Image\n",
"Image('dag.png')"
]
},
{
"cell_type": "markdown",
"id": "7982da04-d9d0-4ba2-b975-b5bbc59d92e4",
"metadata": {},
"source": [
"Now, let's run our workflow:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "51f78a4a-94fd-4f25-919e-c0d67b85e5d0",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!snakemake --forceall --cores 1\n",
"Image('outputs/isles.png')"
]
},
{
"cell_type": "markdown",
"id": "f8a0adb3-1aff-4872-949e-2315a1675fce",
"metadata": {},
"source": [
"If the output of a rule is already present on the filesystem then Snakemake will not run the job that generates it. In general, a job (rule) is executed when\n",
"\n",
"- output file is the target and does not exist\n",
"- output file needed by another executed job and does not exist\n",
"- input file newer than output file\n",
"- input file will be updated by other job\n",
"- execution is enforced (`--forceall`)\n",
"\n",
"For example, if we remove the `isles.png` file that is generated by the `plot_zipf` rule but keep the output of the other rules, then only the `plot_zipf` rule will be executed."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "67b27988-9169-48ad-a9e0-a61d888c2a13",
"metadata": {},
"outputs": [],
"source": [
"%rm outputs/isles.png\n",
"!snakemake --cores 1"
]
},
{
"cell_type": "markdown",
"id": "118636f9-4f03-4495-8a5a-162850b6c9e6",
"metadata": {},
"source": [
"We can run part of the workflow by passing the target (output) we are interested in as the last argument when invoking Snakemake. Snakemake will only run the jobs that produces the dependencies of the job that produce the output you pass as a command-line argument."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fd91f001-3a2b-4bb3-ad4b-1ef64bfe8751",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --cores 1 --forceall outputs/isles.fit.dat"
]
},
{
"cell_type": "markdown",
"id": "85403ecb-bd2c-4a2b-aedf-ad00272916db",
"metadata": {
"tags": [
"hide-input"
]
},
"source": [
"## Make our workflow more flexible and readable\n",
"\n",
"Our current workflow contains a lot of repetition. For example, we explicitly specify path for the input and output directives and repeat these parameters in the shell command. In order to avoid these repetitions we can refer to elements of the rule using `{input}` and `{output}`. In general, all local and global variables in a `Snakefile` can be accessed via their names in the [Python format minilanguage](https://docs.python.org/3/library/string.html#formatspec).\n",
"\n",
"In the case of lists or tuples, they are evaluated to a space-separated list. For example, if we have the following directives\n",
"\n",
"```python\n",
"input: 'path/to/file1', 'path/to/file2'\n",
"shell: 'mycmd {input}'\n",
"```\n",
"\n",
"then `{input}` will be evaluated as `path/to/file1 path/to/file2`. You can also refer the individual element of the list using their indexes\n",
"\n",
"```python\n",
"input: 'path/to/file1', 'path/to/file2'\n",
"shell: 'mycmd {input[0]} {input[1]}'\n",
"```\n",
"\n",
"An other option is to give a name to your inputs and access the values using the names\n",
"\n",
"```python\n",
"input: \n",
" file1='path/to/file1',\n",
" file2='path/to/file2'\n",
"shell: 'mycmd {input.file1} {input.file2}'\n",
"```\n",
"\n",
"### Wildcards\n",
"\n",
"We still have the name of the book is repeated for every rule both in the input and output directives. To avoid these repetitions and make our workflow more flexible, we can use wildcards. For example, the `count_word` rule can be rewritten as\n",
"\n",
"```python\n",
"rule count_words: \n",
" input: 'books/{book}.txt' \n",
" output: 'outputs/{book}.dat' \n",
" shell: 'python scripts/wordcount.py {input} {output}'\n",
"```\n",
"\n",
"where we have defined the wildcard `{book}`. This rule can be interpreted as: *to generate the file named `output/something.dat` (output) find a file named `books/something.txt` (input) and run `wordcount.py input output`*. With all we have discussed above, we can write our workflow as"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "08cb9a90-8745-4c64-ae70-4678d288423b",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile\n",
"\n",
"rule plot_zipf: \n",
" input: \n",
" wordcount='outputs/isles.dat',\n",
" fitting='outputs/isles.fit.dat'\n",
" output: 'outputs/isles.png'\n",
" shell: 'python scripts/plotzipf.py {input.wordcount} {input.fitting} {output}' \n",
"\n",
"rule fit_zipf:\n",
" input: 'outputs/{book}.dat'\n",
" output: 'outputs/{book}.fit.dat'\n",
" shell: 'python scripts/fitzipf.py {input} {output}'\n",
"\n",
"rule count_words: \n",
" input: 'books/{book}.txt' \n",
" output: 'outputs/{book}.dat' \n",
" shell: 'python scripts/wordcount.py {input} {output}'"
]
},
{
"cell_type": "markdown",
"id": "9e24451d-1515-4628-9aab-f9cc8ccd898a",
"metadata": {
"tags": []
},
"source": [
"By using a wildcard, we only need to specify explicitly the name of the book in the `plot_zipf` rule and let Snakemake do the wildcards substitution to determine which rules should be run. \n",
"\n",
"In our case, Snakemake will determine that in order to produce `outputs/isles.png`, `outputs/isles.dat` and `outputs/isles.fit.dat` are required and that these files may be generated by substituting the `{book}` wildcard with `isles` in the `fit_zipf` and `count_words` rules.\n",
"\n",
"### Extending the workflow\n",
"\n",
"The advantage of using wildcards is that the workflow can quickly be extended to include more books. For that we will use the fact that Snakefiles are Python code: we can include any Python code in our workflow. For example, we can define a list with the book titles:\n",
"\n",
"```python\n",
"BOOKS = ['isles', 'abyss', 'last', 'sierra']\n",
"```\n",
"\n",
"and then create a new list listing the plots we want to generate and use that list in a new rule that we will name `all`.\n",
"\n",
"```python\n",
"PLOTS = ['outputs/{book}.png'.format(book=book) for book in BOOKS]\n",
"\n",
"rule all: \n",
" input: PLOTS\n",
"```\n",
"\n",
"An alternative is to use the Snakemake `expand` function which produces the same result as Python brackets syntax:\n",
"\n",
"```python\n",
"PLOTS = expand('outputs/{book}.png', book=BOOKS)\n",
"```\n",
"\n",
"Our workflow thus becomes"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "05c8ea9f-72b0-4abe-aa00-3917140ef8cf",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile\n",
"\n",
"BOOKS = ['isles', 'abyss', 'last', 'sierra']\n",
"PLOTS = expand('outputs/{book}.png', book=BOOKS)\n",
"\n",
"rule all:\n",
" input: PLOTS\n",
"\n",
"rule plot_zipf: \n",
" input: \n",
" wordcount='outputs/{book}.dat',\n",
" fitting='outputs/{book}.fit.dat'\n",
" output: 'outputs/{book}.png'\n",
" shell: 'python scripts/plotzipf.py {input.wordcount} {input.fitting} {output}' \n",
"\n",
"rule fit_zipf:\n",
" input: 'outputs/{book}.dat'\n",
" output: 'outputs/{book}.fit.dat'\n",
" shell: 'python scripts/fitzipf.py {input} {output}'\n",
"\n",
"rule count_words: \n",
" input: 'books/{book}.txt' \n",
" output: 'outputs/{book}.dat' \n",
" shell: 'python scripts/wordcount.py {input} {output}'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8ed7ef9-dfd2-4b2b-9b11-09c426cacf38",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --forceall --dag | dot -Tpng > dag.png\n",
"Image('dag.png')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4895de84-95c2-48b6-8cad-1dd8d50c7567",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --forceall --cores 1 --quiet"
]
},
{
"cell_type": "markdown",
"id": "1bf06632-4672-4278-a474-d87952006da5",
"metadata": {
"tags": []
},
"source": [
"### Running a Python script\n",
"\n",
"Snakemake is not limited to running shell command. We can invoke a script using the `script` directive. For example, we can add script that will agreggate the data of all the books:\n",
"\n",
"```python\n",
"DATS = expand(f'{OUTPUT_DIR}/{{book}}.dat', book=BOOKS)\n",
"\n",
"rule aggregate:\n",
" input: DATS\n",
" output: 'outputs/aggregated.dat'\n",
" script: 'scripts/aggregate.py'\n",
"```\n",
"\n",
"with the `aggregate.py` script\n",
"\n",
"```python\n",
"data = {}\n",
"for input in snakemake.input:\n",
" read_input(input, data)\n",
" \n",
"sorted_data = reversed(sorted(data.items(), key = itemgetter(1)))\n",
"\n",
"with open(snakemake.output[0],'w') as outfile:\n",
" for key, value in sorted_data:\n",
" outfile.write('{:s} {:d}\\n'.format(key, value))\n",
"```\n",
"\n",
"note that inside the script we have access to a `snakemake` object and make use of it in order to iterate over all the inputs (`snakemake.input`) and determine where we need to write the output (`snakemake.input`)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdbe2f35-4d61-4b05-a63f-7e6c9a8599c4",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile\n",
"\n",
"BOOKS = ['isles', 'abyss', 'last', 'sierra']\n",
"PLOTS = expand('outputs/{book}.png', book=BOOKS)\n",
"DATS = expand('outputs/{book}.dat', book=BOOKS)\n",
"RESULTS = multiext('outputs/aggregated', '.dat', '.fit.dat', '.png')\n",
"\n",
"rule all:\n",
" input: PLOTS, RESULTS\n",
"\n",
"rule aggregate:\n",
" input: DATS\n",
" output: 'outputs/aggregated.dat'\n",
" script: 'scripts/aggregate.py'\n",
"\n",
"rule plot_zipf: \n",
" input: \n",
" wordcount='outputs/{book}.dat',\n",
" fitting='outputs/{book}.fit.dat'\n",
" output: 'outputs/{book}.png'\n",
" shell: 'python scripts/plotzipf.py {input.wordcount} {input.fitting} {output}' \n",
"\n",
"rule fit_zipf:\n",
" input: 'outputs/{book}.dat'\n",
" output: 'outputs/{book}.fit.dat'\n",
" shell: 'python scripts/fitzipf.py {input} {output}'\n",
"\n",
"rule count_words: \n",
" input: 'books/{book}.txt' \n",
" output: 'outputs/{book}.dat' \n",
" shell: 'python scripts/wordcount.py {input} {output}'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a28cc7f2-0973-44e0-8dd6-8ce8ba31e3cd",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --forceall --cores 1 --quiet"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19064ffc-d30c-4d84-9085-dd0196161872",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --forceall --dag | dot -Tpng > dag.png\n",
"Image('dag.png')"
]
},
{
"cell_type": "markdown",
"id": "5a5be7cf-197b-4747-98c0-0b72e415995c",
"metadata": {},
"source": [
"Snakemake in not limited to Python script. It can also be used with [R](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#r-and-r-markdown), [Julia](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#julia) and [Rust](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#rust). For all these languages, Snakemake provides an object that can be manipulated to gather information about the workflow."
]
},
{
"cell_type": "markdown",
"id": "9da93c2e-e0c2-4de6-9760-d34b82537d61",
"metadata": {},
"source": [
"### Using a configuration file\n",
"\n",
"We can use a configuration file In order to set the parameters of our data analysis. For that we will create a YAML file named `config.yml` with the following content:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "48bd6de2-7051-4c47-86aa-aaeafd1b921f",
"metadata": {},
"outputs": [],
"source": [
"%%writefile config.yml\n",
"books_dir: 'books'\n",
"output_dir: 'outputs'"
]
},
{
"cell_type": "markdown",
"id": "cdbe384a-cd89-459e-b083-e3b13b52121a",
"metadata": {},
"source": [
"In the Snakefile, we can make use of this configuration file using the `configfile` directive:\n",
"\n",
"```python\n",
"configfile: 'config.yml'\n",
"```\n",
"\n",
"Snakemake will read this configuration file and make the values available in a dictionary named `config`. We can then use this dictionary to extract the configuration for our workflow\n",
"\n",
"```python\n",
"OUTPUT_DIR = config['output_dir']\n",
"BOOKS_DIR = config['books_dir']\n",
"```\n",
"Then, we use this variable to define variable that will be used in the input and output directives of our workflow:\n",
"\n",
"```python\n",
"BOOK_TXT = f'{BOOKS_DIR}/{{book}}.txt'\n",
"BOOK_DAT = f'{OUTPUT_DIR}/{{book}}.dat'\n",
"BOOK_FIT = f'{OUTPUT_DIR}/{{book}}.fit.dat'\n",
"```\n",
"Note that we use double curly braces (`{{book}}`) so that we obtain the wildcard `{book}` in the output of the formatted string. If the value of the `BOOKS_DIR` is `books`, then the value of `BOOK_TXT` in example above will be `'books/{book}.txt'`. \n",
"\n",
"We can also ensure that it is no longer necessary to explicitly specify the title of books by extracting the name of the books stored as `txt` files in the `books` directory. To achieve that we can use the Snakemake `glob_wildcards` function.\n",
" \n",
"```python\n",
"BOOKS = glob_wildcards(f'{BOOKS_DIR}/{{book}}.txt').book\n",
"```\n",
"\n",
"The `glob_wildcards` function matches the given pattern against the files present in the filesystem and thereby infers the values for all wildcards in the pattern. This function returns a named tuple. In our case it will list the name of all the files in the `books` directory.\n",
"\n",
"With what as been discussed above, our workflow is now:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7c3b028d-3149-46d9-bfcb-bc0b8d2c202e",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile\n",
"\n",
"configfile: 'config.yml'\n",
"\n",
"OUTPUT_DIR = config['output_dir']\n",
"BOOKS_DIR = config['books_dir']\n",
"\n",
"BOOK_TXT = f'{BOOKS_DIR}/{{book}}.txt'\n",
"BOOK_DAT = f'{OUTPUT_DIR}/{{book}}.dat'\n",
"BOOK_FIT = f'{OUTPUT_DIR}/{{book}}.fit.dat'\n",
"\n",
"BOOKS = glob_wildcards(f'{BOOKS_DIR}/{{book}}.txt').book\n",
"\n",
"PLOTS = expand(f'{OUTPUT_DIR}/{{book}}.png', book=BOOKS)\n",
"DATS = expand(f'{OUTPUT_DIR}/{{book}}.dat', book=BOOKS)\n",
"RESULTS = multiext(f'{OUTPUT_DIR}/aggregated', '.dat', '.fit.dat', '.png')\n",
"\n",
"rule all:\n",
" input: PLOTS, RESULTS\n",
"\n",
"rule aggregate:\n",
" input: DATS\n",
" output: 'outputs/aggregated.dat'\n",
" script: 'scripts/aggregate.py'\n",
"\n",
"rule plot_zipf: \n",
" input: \n",
" wordcount=BOOK_DAT,\n",
" fitting=BOOK_FIT\n",
" output: 'outputs/{book}.png'\n",
" shell: 'python scripts/plotzipf.py {input.wordcount} {input.fitting} {output}' \n",
"\n",
"rule fit_zipf:\n",
" input: BOOK_DAT\n",
" output: BOOK_FIT\n",
" shell: 'python scripts/fitzipf.py {input} {output}'\n",
"\n",
"rule count_words: \n",
" input: BOOK_TXT \n",
" output: BOOK_DAT \n",
" shell: 'python scripts/wordcount.py {input} {output}'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0050b03-fcc6-4e12-b03e-8641fde69e08",
"metadata": {},
"outputs": [],
"source": [
"!snakemake --forceall --cores 1 --quiet"
]
},
{
"cell_type": "markdown",
"id": "655a52d8-ddf7-49d9-aae2-c2d1e8ad0065",
"metadata": {},
"source": [
"## More information about Snakemake\n",
"\n",
"- [Snakemake website](https://snakemake.github.io/)\n",
"- [Snakemake documentation](https://snakemake.readthedocs.io/en/stable/)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}