{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Configuration file upgrade demonstration \n",
"\n",
"In the following tutorial we will present the process of upgrading the refgenie configuration file and asset files from version **0.3** to version **0.4**.\n",
"\n",
"First, let's install the refgenie and refgenconf Python packages that support version 0.3 of refgenie configuration file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Working environment setup\n",
"\n",
"Let's install the legacy refgenconf and refgenie Python packages"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting refgenconf==0.9.3\n",
" Using cached https://files.pythonhosted.org/packages/52/c3/6aed361205272e30cd3570ca1c33feae6ad977ad32ddff8e509752046272/refgenconf-0.9.3-py3-none-any.whl\n",
"Requirement already satisfied: requests in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf==0.9.3) (2.21.0)\n",
"Requirement already satisfied: attmap>=0.12.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.9.3) (0.12.12.dev0)\n",
"Requirement already satisfied: tqdm>=4.38.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.9.3) (4.47.0)\n",
"Requirement already satisfied: pyyaml in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf==0.9.3) (5.1)\n",
"Requirement already satisfied: yacman>=0.6.9 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.9.3) (0.7.0)\n",
"Requirement already satisfied: pyfaidx in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.9.3) (0.5.9.1)\n",
"Requirement already satisfied: urllib3<1.25,>=1.21.1 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.9.3) (1.24.1)\n",
"Requirement already satisfied: idna<2.9,>=2.5 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.9.3) (2.8)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.9.3) (2019.3.9)\n",
"Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.9.3) (3.0.4)\n",
"Requirement already satisfied: ubiquerg>=0.2.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from attmap>=0.12.5->refgenconf==0.9.3) (0.6.1)\n",
"Requirement already satisfied: oyaml in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from yacman>=0.6.9->refgenconf==0.9.3) (0.9)\n",
"Requirement already satisfied: setuptools>=0.7 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from pyfaidx->refgenconf==0.9.3) (41.0.1)\n",
"Requirement already satisfied: six in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pyfaidx->refgenconf==0.9.3) (1.12.0)\n",
"Installing collected packages: refgenconf\n",
" Found existing installation: refgenconf 0.10.0.dev0\n",
" Uninstalling refgenconf-0.10.0.dev0:\n",
" Successfully uninstalled refgenconf-0.10.0.dev0\n",
"Successfully installed refgenconf-0.9.3\n",
"\u001b[33mWARNING: You are using pip version 19.2.3, however version 20.2.3 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
"Collecting refgenie==0.9.3\n",
" Using cached https://files.pythonhosted.org/packages/af/52/c1e1bc63b3543f591ebdf44caccfaab3c730708256d926b9f4b1c34d1865/refgenie-0.9.3-py3-none-any.whl\n",
"Requirement already satisfied: pyfaidx>=0.5.5.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.9.3) (0.5.9.1)\n",
"Requirement already satisfied: refgenconf>=0.9.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.9.3) (0.9.3)\n",
"Requirement already satisfied: logmuse>=0.2.6 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.9.3) (0.2.6)\n",
"Requirement already satisfied: piper>=0.12.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.9.3) (0.12.1)\n",
"Requirement already satisfied: six in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pyfaidx>=0.5.5.2->refgenie==0.9.3) (1.12.0)\n",
"Requirement already satisfied: setuptools>=0.7 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from pyfaidx>=0.5.5.2->refgenie==0.9.3) (41.0.1)\n",
"Requirement already satisfied: attmap>=0.12.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf>=0.9.1->refgenie==0.9.3) (0.12.12.dev0)\n",
"Requirement already satisfied: requests in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf>=0.9.1->refgenie==0.9.3) (2.21.0)\n",
"Requirement already satisfied: tqdm>=4.38.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf>=0.9.1->refgenie==0.9.3) (4.47.0)\n",
"Requirement already satisfied: pyyaml in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf>=0.9.1->refgenie==0.9.3) (5.1)\n",
"Requirement already satisfied: yacman>=0.6.9 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf>=0.9.1->refgenie==0.9.3) (0.7.0)\n",
"Requirement already satisfied: ubiquerg>=0.4.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from piper>=0.12.1->refgenie==0.9.3) (0.6.1)\n",
"Requirement already satisfied: psutil in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from piper>=0.12.1->refgenie==0.9.3) (5.6.1)\n",
"Requirement already satisfied: pandas in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from piper>=0.12.1->refgenie==0.9.3) (1.0.3)\n",
"Requirement already satisfied: urllib3<1.25,>=1.21.1 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf>=0.9.1->refgenie==0.9.3) (1.24.1)\n",
"Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf>=0.9.1->refgenie==0.9.3) (3.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf>=0.9.1->refgenie==0.9.3) (2019.3.9)\n",
"Requirement already satisfied: idna<2.9,>=2.5 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf>=0.9.1->refgenie==0.9.3) (2.8)\n",
"Requirement already satisfied: oyaml in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from yacman>=0.6.9->refgenconf>=0.9.1->refgenie==0.9.3) (0.9)\n",
"Requirement already satisfied: pytz>=2017.2 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pandas->piper>=0.12.1->refgenie==0.9.3) (2018.9)\n",
"Requirement already satisfied: python-dateutil>=2.6.1 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pandas->piper>=0.12.1->refgenie==0.9.3) (2.8.0)\n",
"Requirement already satisfied: numpy>=1.13.3 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pandas->piper>=0.12.1->refgenie==0.9.3) (1.17.3)\n",
"Installing collected packages: refgenie\n",
" Found existing installation: refgenie 0.10.0.dev0\n",
" Uninstalling refgenie-0.10.0.dev0:\n",
" Successfully uninstalled refgenie-0.10.0.dev0\n",
"Successfully installed refgenie-0.9.3\n",
"\u001b[33mWARNING: You are using pip version 19.2.3, however version 20.2.3 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
"source": [
"pip install refgenconf==0.9.3\n",
"pip install refgenie==0.9.3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's set up a directory that we will use for the config file and refgenie assets"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"export WORKDIR=~/Desktop/testing/refgenie/upgrade_test\n",
"rm -r $WORKDIR # remove first just to make sure the directory does not exist\n",
"mkdir -p $WORKDIR\n",
"cd $WORKDIR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's set `$REFGENIE` environment variable to point refgenie to the configuration file location and initialize it"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized genome configuration file: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/g.yml\n"
]
}
],
"source": [
"export REFGENIE=$WORKDIR/g.yml\n",
"refgenie init -c $REFGENIE -s http://rg.databio.org:82/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that we subscribe to a test instance of refgenieserver, that supports both the old and new refgenie clients. This is because it exposes different API versions, that these clients use: `v2` (refgenie v0.9.3) and `v3` (refgenie v0.10.0-dev)\n",
"\n",
"## Pull/build test assets\n",
"\n",
"Next, let's retrieve couple of assets. As mentioned above, `v2` API is used to retrieve the asset."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading URL: http://rg.databio.org:82/v2/asset/rCRSd/fasta/archive\n",
"Download complete: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/rCRSd/fasta__default.tgz\n",
"Extracting asset tarball and saving to: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/rCRSd/fasta/default\n",
"Default tag for 'rCRSd/fasta' set to: default\n",
"Downloading URL: http://rg.databio.org:82/v2/asset/human_repeats/fasta/archive\n",
"Download complete: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_repeats/fasta__default.tgz\n",
"Extracting asset tarball and saving to: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_repeats/fasta/default\n",
"Default tag for 'human_repeats/fasta' set to: default\n",
"Downloading URL: http://rg.databio.org:82/v2/asset/rCRSd/bowtie2_index/archive\n",
"Download complete: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/rCRSd/bowtie2_index__default.tgz\n",
"Extracting asset tarball and saving to: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/rCRSd/bowtie2_index/default\n",
"Default tag for 'rCRSd/bowtie2_index' set to: default\n",
"Downloading URL: http://rg.databio.org:82/v2/asset/human_repeats/bwa_index/archive\n",
"Download complete: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_repeats/bwa_index__default.tgz\n",
"Extracting asset tarball and saving to: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_repeats/bwa_index/default\n",
"Default tag for 'human_repeats/bwa_index' set to: default\n"
]
}
],
"source": [
"refgenie pull rCRSd/fasta human_repeats/fasta rCRSd/bowtie2_index human_repeats/bwa_index"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's download a small FASTA file and build a fasta asset for an arbitrary genome, which is not available at `http://rg.databio.org:82/`"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2020-10-12 17:39:25-- http://big.databio.org/refgenie_raw/files.human_alu.fasta.fasta\n",
"Resolving big.databio.org (big.databio.org)... 128.143.245.182, 128.143.245.181\n",
"Connecting to big.databio.org (big.databio.org)|128.143.245.182|:80... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 501 [application/octet-stream]\n",
"Saving to: ‘human_alu.fa.gz’\n",
"\n",
"human_alu.fa.gz 100%[===================>] 501 --.-KB/s in 0s \n",
"\n",
"2020-10-12 17:39:25 (1.19 MB/s) - ‘human_alu.fa.gz’ saved [501/501]\n",
"\n"
]
}
],
"source": [
"wget -O human_alu.fa.gz http://big.databio.org/refgenie_raw/files.human_alu.fasta.fasta\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using 'default' as the default tag for 'human_alu/fasta'\n",
"Building 'human_alu/fasta:default' using 'fasta' recipe\n",
"Saving outputs to:\n",
"- content: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu\n",
"- logs: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/_refgenie_build\n",
"### Pipeline run code and environment:\n",
"\n",
"* Command: `/Library/Frameworks/Python.framework/Versions/3.6/bin/refgenie build human_alu/fasta --files fasta=human_alu.fa.gz`\n",
"* Compute host: MichalsMBP\n",
"* Working dir: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test\n",
"* Outfolder: /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/_refgenie_build/\n",
"* Pipeline started at: (10-12 17:39:27) elapsed: 0.0 _TIME_\n",
"\n",
"### Version log:\n",
"\n",
"* Python version: 3.6.5\n",
"* Pypiper dir: `/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pypiper`\n",
"* Pypiper version: 0.12.1\n",
"* Pipeline dir: `/Library/Frameworks/Python.framework/Versions/3.6/bin`\n",
"* Pipeline version: None\n",
"\n",
"### Arguments passed to pipeline:\n",
"\n",
"* `asset_registry_paths`: `['human_alu/fasta']`\n",
"* `assets`: `None`\n",
"* `command`: `build`\n",
"* `config_file`: `/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/refgenie/refgenie.yaml`\n",
"* `docker`: `False`\n",
"* `files`: `[['fasta=human_alu.fa.gz']]`\n",
"* `genome`: `None`\n",
"* `genome_config`: `None`\n",
"* `genome_description`: `None`\n",
"* `logdev`: `False`\n",
"* `new_start`: `False`\n",
"* `outfolder`: `/Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test`\n",
"* `params`: `None`\n",
"* `recipe`: `None`\n",
"* `recover`: `False`\n",
"* `requirements`: `False`\n",
"* `silent`: `False`\n",
"* `tag_description`: `None`\n",
"* `verbosity`: `None`\n",
"* `volumes`: `None`\n",
"\n",
"----------------------------------------\n",
"\n",
"Target to produce: `/Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/_refgenie_build/human_alu_fasta__default.flag` \n",
"\n",
"> `cp human_alu.fa.gz /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/human_alu.fa.gz` (70063)\n",
"\n",
"psutil.ZombieProcess process still exists but it's a zombie (pid=70063)\n",
"Warning: couldn't add memory use for process: 70063\n",
"\n",
"Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB. \n",
" PID: 70063;\tCommand: cp;\tReturn code: 0;\tMemory used: 0.001GB\n",
"\n",
"\n",
"> `gzip -df /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/human_alu.fa.gz` (70064)\n",
"\n",
"psutil.ZombieProcess process still exists but it's a zombie (pid=70064)\n",
"Warning: couldn't add memory use for process: 70064\n",
"\n",
"Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB. \n",
" PID: 70064;\tCommand: gzip;\tReturn code: 0;\tMemory used: 0.0GB\n",
"\n",
"\n",
"> `samtools faidx /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/human_alu.fa` (70065)\n",
"\n",
"psutil.ZombieProcess process still exists but it's a zombie (pid=70065)\n",
"Warning: couldn't add memory use for process: 70065\n",
"\n",
"Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB. \n",
" PID: 70065;\tCommand: samtools;\tReturn code: 0;\tMemory used: 0.0GB\n",
"\n",
"\n",
"> `cut -f 1,2 /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/human_alu.fa.fai > /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/human_alu.chrom.sizes` (70066)\n",
"\n",
"psutil.ZombieProcess process still exists but it's a zombie (pid=70066)\n",
"Warning: couldn't add memory use for process: 70066\n",
"\n",
"Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB. \n",
" PID: 70066;\tCommand: cut;\tReturn code: 0;\tMemory used: 0GB\n",
"\n",
"\n",
"> `touch /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default/_refgenie_build/human_alu_fasta__default.flag` (70068)\n",
"\n",
"psutil.ZombieProcess process still exists but it's a zombie (pid=70068)\n",
"Warning: couldn't add memory use for process: 70068\n",
"\n",
"Command completed. Elapsed time: 0:00:00. Running peak memory: 0.001GB. \n",
" PID: 70068;\tCommand: touch;\tReturn code: 0;\tMemory used: 0GB\n",
"\n",
"\n",
"> `cd /Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test/human_alu/fasta/default; find . -type f -not -path './_refgenie_build*' -exec md5sum {} \\; | sort -k 2 | awk '{print $1}' | md5sum`\n",
"Asset digest: 9e8fa06e6125f89be4fb974879cb91a6\n",
"Default tag for 'human_alu/fasta' set to: default\n",
"\n",
"### Pipeline completed. Epilogue\n",
"* Elapsed time (this run): 0:00:00\n",
"* Total elapsed time (all runs): 0:00:00\n",
"* Peak memory (this run): 0.001 GB\n",
"* Pipeline completed time: 2020-10-12 17:39:27\n",
"Computing initial genome digest...\n",
"Initializing genome...\n",
"Finished building 'fasta' asset\n"
]
}
],
"source": [
"refgenie build human_alu/fasta --files fasta=human_alu.fa.gz"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's inspect the asset inventory"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Server subscriptions: http://rg.databio.org:82\n",
"Local genomes: human_alu, human_repeats, rCRSd\n",
"Local recipes: bismark_bt1_index, bismark_bt2_index, blacklist, bowtie2_index, bwa_index, cellranger_reference, dbnsfp, dbsnp, ensembl_gtf, ensembl_rb, epilog_index, fasta, fasta_txome, feat_annotation, gencode_gtf, hisat2_index, kallisto_index, refgene_anno, salmon_index, salmon_partial_sa_index, salmon_sa_index, star_index, suffixerator_index, tallymer_index\n",
"Local assets:\n",
" human_alu/ fasta.chrom_sizes:default, fasta.fai:default, fasta:default\n",
" human_repeats/ bwa_index:default, fasta.chrom_sizes:default, fasta.fai:default, fasta:default\n",
" rCRSd/ bowtie2_index:default, fasta.chrom_sizes:default, fasta.fai:default, fasta:default\n"
]
}
],
"source": [
"refgenie list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, assets for all three genomes are available."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Upgrade refgenie software\n",
"\n",
"Now, let's upgrade to refgenie==0.10.0-dev, which introduces the concept of sequence-derived genome identifiers to uniqly identify genomes."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting git+https://github.com/refgenie/refgenconf.git@dev_config_upgrade\n",
" Cloning https://github.com/refgenie/refgenconf.git (to revision dev_config_upgrade) to /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-req-build-kxmw8i6n\n",
" Running command git clone -q https://github.com/refgenie/refgenconf.git /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-req-build-kxmw8i6n\n",
" Running command git checkout -b dev_config_upgrade --track origin/dev_config_upgrade\n",
" Switched to a new branch 'dev_config_upgrade'\n",
" Branch 'dev_config_upgrade' set up to track remote branch 'dev_config_upgrade' from 'origin'.\n",
"Requirement already satisfied: attmap>=0.12.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.10.0.dev0) (0.12.12.dev0)\n",
"Requirement already satisfied: pyyaml in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf==0.10.0.dev0) (5.1)\n",
"Requirement already satisfied: requests in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf==0.10.0.dev0) (2.21.0)\n",
"Requirement already satisfied: yacman>=0.7.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.10.0.dev0) (0.7.0)\n",
"Requirement already satisfied: future in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.10.0.dev0) (0.17.1)\n",
"Requirement already satisfied: jsonschema in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from refgenconf==0.10.0.dev0) (3.0.1)\n",
"Requirement already satisfied: rich in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenconf==0.10.0.dev0) (3.3.0)\n",
"Requirement already satisfied: ubiquerg>=0.2.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from attmap>=0.12.5->refgenconf==0.10.0.dev0) (0.6.1)\n",
"Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.10.0.dev0) (3.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.10.0.dev0) (2019.3.9)\n",
"Requirement already satisfied: idna<2.9,>=2.5 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.10.0.dev0) (2.8)\n",
"Requirement already satisfied: urllib3<1.25,>=1.21.1 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from requests->refgenconf==0.10.0.dev0) (1.24.1)\n",
"Requirement already satisfied: oyaml in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from yacman>=0.7.0->refgenconf==0.10.0.dev0) (0.9)\n",
"Requirement already satisfied: attrs>=17.4.0 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from jsonschema->refgenconf==0.10.0.dev0) (19.1.0)\n",
"Requirement already satisfied: pyrsistent>=0.14.0 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from jsonschema->refgenconf==0.10.0.dev0) (0.14.11)\n",
"Requirement already satisfied: six>=1.11.0 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from jsonschema->refgenconf==0.10.0.dev0) (1.12.0)\n",
"Requirement already satisfied: setuptools in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from jsonschema->refgenconf==0.10.0.dev0) (41.0.1)\n",
"Requirement already satisfied: colorama<0.5.0,>=0.4.0 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from rich->refgenconf==0.10.0.dev0) (0.4.1)\n",
"Requirement already satisfied: pprintpp<0.5.0,>=0.4.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from rich->refgenconf==0.10.0.dev0) (0.4.0)\n",
"Requirement already satisfied: typing-extensions<4.0.0,>=3.7.4 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from rich->refgenconf==0.10.0.dev0) (3.7.4.2)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from rich->refgenconf==0.10.0.dev0) (2.6.1)\n",
"Requirement already satisfied: dataclasses<0.8,>=0.7; python_version >= \"3.6\" and python_version < \"3.7\" in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from rich->refgenconf==0.10.0.dev0) (0.7)\n",
"Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from rich->refgenconf==0.10.0.dev0) (0.9.1)\n",
"Building wheels for collected packages: refgenconf\n",
" Building wheel for refgenconf (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for refgenconf: filename=refgenconf-0.10.0.dev0-cp36-none-any.whl size=64959 sha256=37191046ce6136b2bd777b1aa274a2d6a5ffb508af7e4969ac0ae97c1682b1f5\n",
" Stored in directory: /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-ephem-wheel-cache-516dw93w/wheels/a8/b1/82/f79eaabaad4cf5c64fb4914e06dd04726c5c226785974aee4e\n",
"Successfully built refgenconf\n",
"Installing collected packages: refgenconf\n",
" Found existing installation: refgenconf 0.9.3\n",
" Uninstalling refgenconf-0.9.3:\n",
" Successfully uninstalled refgenconf-0.9.3\n",
"Successfully installed refgenconf-0.10.0.dev0\n",
"\u001b[33mWARNING: You are using pip version 19.2.3, however version 20.2.3 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
"Collecting git+https://github.com/refgenie/refgenie.git@dev_config_upgrade\n",
" Cloning https://github.com/refgenie/refgenie.git (to revision dev_config_upgrade) to /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-req-build-3i4zdr4w\n",
" Running command git clone -q https://github.com/refgenie/refgenie.git /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-req-build-3i4zdr4w\n",
" Running command git checkout -b dev_config_upgrade --track origin/dev_config_upgrade\n",
" Switched to a new branch 'dev_config_upgrade'\n",
" Branch 'dev_config_upgrade' set up to track remote branch 'dev_config_upgrade' from 'origin'.\n",
"Requirement already satisfied: logmuse>=0.2.6 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.10.0.dev0) (0.2.6)\n",
"Requirement already satisfied: piper>=0.12.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.10.0.dev0) (0.12.1)\n",
"Requirement already satisfied: pyfaidx>=0.5.5.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from refgenie==0.10.0.dev0) (0.5.9.1)\n",
"Requirement already satisfied: ubiquerg>=0.4.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from piper>=0.12.1->refgenie==0.10.0.dev0) (0.6.1)\n",
"Requirement already satisfied: yacman in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from piper>=0.12.1->refgenie==0.10.0.dev0) (0.7.0)\n",
"Requirement already satisfied: attmap>=0.12.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from piper>=0.12.1->refgenie==0.10.0.dev0) (0.12.12.dev0)\n",
"Requirement already satisfied: psutil in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from piper>=0.12.1->refgenie==0.10.0.dev0) (5.6.1)\n",
"Requirement already satisfied: pandas in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from piper>=0.12.1->refgenie==0.10.0.dev0) (1.0.3)\n",
"Requirement already satisfied: setuptools>=0.7 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from pyfaidx>=0.5.5.2->refgenie==0.10.0.dev0) (41.0.1)\n",
"Requirement already satisfied: six in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pyfaidx>=0.5.5.2->refgenie==0.10.0.dev0) (1.12.0)\n",
"Requirement already satisfied: oyaml in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from yacman->piper>=0.12.1->refgenie==0.10.0.dev0) (0.9)\n",
"Requirement already satisfied: pyyaml>=3.13 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from yacman->piper>=0.12.1->refgenie==0.10.0.dev0) (5.1)\n",
"Requirement already satisfied: pytz>=2017.2 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pandas->piper>=0.12.1->refgenie==0.10.0.dev0) (2018.9)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: python-dateutil>=2.6.1 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pandas->piper>=0.12.1->refgenie==0.10.0.dev0) (2.8.0)\n",
"Requirement already satisfied: numpy>=1.13.3 in /Users/mstolarczyk/Library/Python/3.6/lib/python/site-packages (from pandas->piper>=0.12.1->refgenie==0.10.0.dev0) (1.17.3)\n",
"Building wheels for collected packages: refgenie\n",
" Building wheel for refgenie (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for refgenie: filename=refgenie-0.10.0.dev0-cp36-none-any.whl size=29266 sha256=d78485a0207036ddd91c36eb66b1973bdb3588aaff925d165d5e5aed483f968c\n",
" Stored in directory: /private/var/folders/3f/0wj7rs2144l9zsgxd3jn5nxc0000gn/T/pip-ephem-wheel-cache-wmsjgl78/wheels/07/12/55/f50538357799dd2938a702a2f9e8b84a849975e61b0c59e7a0\n",
"Successfully built refgenie\n",
"Installing collected packages: refgenie\n",
" Found existing installation: refgenie 0.9.3\n",
" Uninstalling refgenie-0.9.3:\n",
" Successfully uninstalled refgenie-0.9.3\n",
"Successfully installed refgenie-0.10.0.dev0\n",
"\u001b[33mWARNING: You are using pip version 19.2.3, however version 20.2.3 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
"source": [
"pip install git+https://github.com/refgenie/refgenconf.git@dev_config_upgrade\n",
"pip install git+https://github.com/refgenie/refgenie.git@dev_config_upgrade"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"refgenie 0.10.0-dev | refgenconf 0.10.0-dev\n"
]
}
],
"source": [
"refgenie --version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Execution of refgenie commands fails since the config is incompatible:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Traceback (most recent call last):\n",
" File \"/Library/Frameworks/Python.framework/Versions/3.6/bin/refgenie\", line 10, in \n",
" sys.exit(main())\n",
" File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/refgenie/refgenie.py\", line 821, in main\n",
" skip_read_lock=skip_read_lock)\n",
" File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/refgenconf/refgenconf.py\", line 110, in __init__\n",
" raise ConfigNotCompliantError(msg)\n",
"refgenconf.exceptions.ConfigNotCompliantError: This genome config (v0.3) is not compliant with v0.4 standards. \n",
"To use current refgenconf, please use upgrade_config function to upgrade, ordowngrade refgenconf: 'pip install \"refgenconf>=0.7.0,<0.10.0\"'. \n",
"If refgenie is installed, you can use 'refgenie upgrade --target-version 0.4'; For config format documentation please see http://refgenie.databio.org/en/latest/genome_config/\n"
]
},
{
"ename": "",
"evalue": "1",
"output_type": "error",
"traceback": []
}
],
"source": [
"refgenie list "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Upgrade refgenie configuration file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's upgrade the config to v0.4, just as the error message suggests. We will use `--force` option to run the command in batch mode."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Upgrading v0.3 config file format to v0.4\n",
"Retrieved rCRSd digest from the server (511fb1178275e7d529560d53b949dba40815f195623bce8e)\n",
"Retrieved human_repeats digest from the server (ebf26d2f064462bea7029e6b4d2298967d7435bff82ed224)\n",
"Genome digest for human_alu is not available on any of the servers. Generating the digest from a local fasta file\n",
"Loaded AnnotatedSequenceDigestList (8 sequences)\n",
"Creating 'data' and 'alias' directories in '/Users/mstolarczyk/Desktop/testing/refgenie/upgrade_test'.\n",
"Copying assets to 'data' and creating alias symlinks in 'alias'. Genomes that the digest could not be determined for 'will be ignored.\n",
"Removing genome assets that have been copied to 'data' directory.\n"
]
}
],
"source": [
"refgenie upgrade --force --target-version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The upgrade succeded for all the assets that were previously managed by refgenie, regardless of the fact if the sequence-derived genome identifiers were avialable on the server. For ones that were not (`human_alu` genome) refgenie calculated the digest from the locally available FASTA file using the same algorithm that has been used to generate digests for the genomes on the server."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[3m Local refgenie assets \u001b[0m\n",
"\u001b[3m Server subscriptions: \u001b[0m\n",
"\u001b[3m http://rg.databio.org:82 \u001b[0m\n",
"┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1mgenome \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1massets \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ rCRSd │ fasta, bowtie2_index │\n",
"│ human_repeats │ fasta, bwa_index │\n",
"│ human_alu │ fasta │\n",
"└───────────────┴──────────────────────┘\n"
]
}
],
"source": [
"refgenie list"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Bash",
"language": "bash",
"name": "bash"
},
"language_info": {
"codemirror_mode": "shell",
"file_extension": ".sh",
"mimetype": "text/x-sh",
"name": "bash"
}
},
"nbformat": 4,
"nbformat_minor": 1
}