{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

Genome aliases

\n", "\n", "[TOC]\n", "\n", "TLDR; **The genome alias system in refgenie allows users to refer to assets with arbitrary strings managed with `refgenie alias` command.**\n", "\n", "## Motivation\n", "\n", "Many systems rely on human-readable identifiers of genomes, such as \"hg38\". However, two users may refer to different things with the same identifier, such as the many slight variations of the *hg38* genome assembly. Such identifier mismatches lead to compatibility issues that incur the wrath of bioinformaticians everywhere. A step toward solving this problem is to use unique identifiers that unambiguously identify a particular assembly, such as those provided by the NCBI Assembly database; however, this approach relies on a central authority, and therefore can not apply to custom genomes or custom assets. Besides, human-readable identifiers persist because there's something simple and satisfying about referring to a genome or piece of data with a simple string that makes some sense and is easy to remember, like *hg38*. \n", "\n", "## Solutions\n", "\n", "### Sequence-derived identifiers\n", "\n", "Refgenie’s approach extends the [refget](http://samtools.github.io/hts-specs/refget.html) algorithm by GA4GH, introduced in 2019 to *collections of annotated sequences*. This means that the unique sequence-derived genome identifier calculated by refgenie captures not only sequence content, but also related metadata like sequence names and length. So, instead of referring to human genome as, e.g. \"hg38\" refgenie unambiguously identifies it as `58de7f33a36ccd9d6e3b1b3afe6b9f37cd5b2867bbfb929a`. \n", "\n", "#### Genome namespace initialization\n", "\n", "The genome digest is calculated based on a FASTA file once the genome namespace is first created. This can happen when the `fasta` asset is pulled or built.\n", "\n", "To start, initialize an empty refgenie configuration file from the shell and subscribe to the desired asset server:\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initialized genome configuration file: /Users/mstolarczyk/code/refgenie/docs_jupyter/refgenie.yaml\n", "Created directories:\n", " - /Users/mstolarczyk/code/refgenie/docs_jupyter/data\n", " - /Users/mstolarczyk/code/refgenie/docs_jupyter/alias\n" ] } ], "source": [ "export REFGENIE=$(pwd)/refgenie.yaml\n", "refgenie init -c $REFGENIE -s http://rg.databio.org" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's pull a `fasta` asset, which is one way to initialize a genome:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Compatible refgenieserver instances: ['http://rg.databio.org']\n", "No local digest for genome alias: rCRSd\n", "Setting 'rCRSd' identity with server: http://rg.databio.org/v3/genomes/genome_digest/rCRSd\n", "Determined server digest for local genome alias (rCRSd): 94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4\n", "Set genome alias (94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4: rCRSd)\n", "Created alias directories: \n", " - /Users/mstolarczyk/code/refgenie/docs_jupyter/alias/rCRSd\n", "Downloading URL: http://rg.databio.org/v3/assets/archive/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta\n", "\u001b[2K94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta:default \u001b[35m100.0%\u001b[0m • • • …\u001b[0m • • ? • … …\n", "\u001b[?25hDownload complete: /Users/mstolarczyk/code/refgenie/docs_jupyter/data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta/fasta__default.tgz\n", "Extracting asset tarball: /Users/mstolarczyk/code/refgenie/docs_jupyter/data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta/fasta__default.tgz\n", "Default tag for '94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta' set to: default\n", "Initializing genome: rCRSd\n", "Loaded AnnotatedSequenceDigestList (1 sequences)\n", "Set genome alias (94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4: rCRSd)\n", "Created alias directories: \n", " - /Users/mstolarczyk/code/refgenie/docs_jupyter/alias/rCRSd/fasta/default\n" ] } ], "source": [ "refgenie pull rCRSd/fasta --force" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Following the `refgenie pull` command logs we notice that multiple steps happened:\n", "1. refgenie used the human-readable genome name from the `refgenie pull` call (`rCRSd`) to query the server for any digest associated with it\n", "2. refgenie set the digest it got back from the server as the genome identifier and set the human-readable genome name as an alias\n", "3. refgenie used the genome idenfitier (not the user-specified name) to query the server for the `fasta` asset\n", "\n", "From now on, the unique sequence-derived genome identifier will be used to query asset servers" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Compatible refgenieserver instances: ['http://rg.databio.org']\n", "Downloading URL: http://rg.databio.org/v3/assets/archive/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index\n", "\u001b[2K94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index:defau… \u001b[35m100.…\u001b[0m • •\n", "\u001b[?25hDownload complete: /Users/mstolarczyk/code/refgenie/docs_jupyter/data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index/bowtie2_index__default.tgz\n", "Extracting asset tarball: /Users/mstolarczyk/code/refgenie/docs_jupyter/data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index/bowtie2_index__default.tgz\n", "Default tag for '94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/bowtie2_index' set to: default\n", "Created alias directories: \n", " - /Users/mstolarczyk/code/refgenie/docs_jupyter/alias/rCRSd/bowtie2_index/default\n" ] } ], "source": [ "refgenie pull rCRSd/bowtie2_index --force" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Genome aliases\n", "\n", "To make the user's life easier genome aliases system in refgenie allows to set arbitrary genome aliases that can be then used to refer to a genome. Users can interact with genome aliases using `refgenie alias` command:\n", "\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: refgenie alias [-h] {remove,set,get} ...\n", "\n", "Interact with aliases.\n", "\n", "positional arguments:\n", " {remove,set,get}\n", " remove Remove aliases.\n", " set Set aliases.\n", " get Get aliases.\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n" ] } ], "source": [ "refgenie alias --help" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Set aliases\n", "\n", "To set an alias \"mito\" for genome identified by digest `94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4` one needs to issue the command below:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Set genome alias (94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4: mito)\n", "Created alias directories: \n", " - /Users/mstolarczyk/code/refgenie/docs_jupyter/alias/mito\n" ] } ], "source": [ "refgenie alias set --aliases mito --digest 94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Get aliases\n", "\n", "To get see the entire aliases collection managed by refgenie one needs to issue the command below:\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[3m Genome aliases \u001b[0m\n", "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mgenome \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1malias \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩\n", "│ 94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4 │ rCRSd, mito │\n", "└──────────────────────────────────────────────────┴─────────────┘\n" ] } ], "source": [ "refgenie alias get" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `alias` and `data` directories\n", "\n", "Refgenie stores asset data in two directories: `alias` and `data`. The `data` directory consists of the actual asset files, which are built or pulled from asset servers. The files in this directory are named using the digests, which helps refgenie to unambigously identify genomes. The `alias` holds symbolic links to asset data in `data` directory. **This way users do not need to be aware of the digest-named files at all and there is no waste of disk space due to symbolic links**. \n", "\n", "Here's a general view of the contents of both directories:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[01;34malias\u001b[00m\n", "├── \u001b[01;34mmito\u001b[00m\n", "│   ├── \u001b[01;34mbowtie2_index\u001b[00m\n", "│   │   └── \u001b[01;34mdefault\u001b[00m\n", "│   └── \u001b[01;34mfasta\u001b[00m\n", "│   └── \u001b[01;34mdefault\u001b[00m\n", "└── \u001b[01;34mrCRSd\u001b[00m\n", " ├── \u001b[01;34mbowtie2_index\u001b[00m\n", " │   └── \u001b[01;34mdefault\u001b[00m\n", " └── \u001b[01;34mfasta\u001b[00m\n", " └── \u001b[01;34mdefault\u001b[00m\n", "\n", "10 directories\n", "\u001b[01;34mdata\u001b[00m\n", "└── \u001b[01;34m94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4\u001b[00m\n", " ├── \u001b[01;34mbowtie2_index\u001b[00m\n", " │   └── \u001b[01;34mdefault\u001b[00m\n", " └── \u001b[01;34mfasta\u001b[00m\n", " └── \u001b[01;34mdefault\u001b[00m\n", "\n", "5 directories\n" ] } ], "source": [ "tree alias -d \n", "tree data -d " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the `alias` directory holds *both* of the defined aliases. Let's take a closer look at one of them" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[01;34malias/rCRSd/fasta\u001b[00m\n", "└── \u001b[01;34mdefault\u001b[00m\n", " ├── \u001b[01;36mrCRSd.chrom.sizes\u001b[00m -> ../../../../data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta/default/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4.chrom.sizes\n", " ├── \u001b[01;36mrCRSd.fa\u001b[00m -> ../../../../data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta/default/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4.fa\n", " └── \u001b[01;36mrCRSd.fa.fai\u001b[00m -> ../../../../data/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4/fasta/default/94e0d21feb576e6af61cd2a798ad30682ef2428bb7eabbb4.fa.fai\n", "\n", "1 directory, 3 files\n" ] } ], "source": [ "tree alias/rCRSd/fasta" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This explicitly shows that the files inside `alias/rCRSd/fasta/default` are in fact symbolic links that point to the actual asset files in `data` directory." ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 2 }