{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running in Docker container on Ostrich\n",
    "\n",
    "#### Started Docker container with the following commmand\n",
    "\n",
    "```docker run - p 8888:8888 -v /Users/sam/data/:/data -v /Users/sam/owl_home/:/owl_home -v /Users/sam/owl_web/:owl_web -v /Users/sam/gitrepos/LabDocs/jupyter_nbs/sam/:/jupyter_nbs -it 0ba43904567e```\n",
    "\n",
    "\n",
    "\n",
    "The command allows access to Jupyter Notebook over port 8888 and makes my Jupyter Notebook GitHub repo and my data files on ```Owl/home``` and ```Owl/web``` accessible to the Docker container.\n",
    "\n",
    "Once the container was started, started Jupyter Notebook with the following command inside the Docker container:\n",
    "\n",
    "```jupyter notebook```\n",
    "\n",
    "This is configured in the Docker container to launch a Jupyter Notebook without a browser on port 8888."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Nov 17 18:38:26 UTC 2016\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65e23f54bcc9\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "hostname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check computer specs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Architecture:          x86_64\n",
      "CPU op-mode(s):        32-bit, 64-bit\n",
      "Byte Order:            Little Endian\n",
      "CPU(s):                8\n",
      "On-line CPU(s) list:   0-7\n",
      "Thread(s) per core:    1\n",
      "Core(s) per socket:    8\n",
      "Socket(s):             1\n",
      "Vendor ID:             GenuineIntel\n",
      "CPU family:            6\n",
      "Model:                 26\n",
      "Model name:            Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz\n",
      "Stepping:              5\n",
      "CPU MHz:               2260.998\n",
      "BogoMIPS:              4521.99\n",
      "Hypervisor vendor:     KVM\n",
      "Virtualization type:   full\n",
      "L1d cache:             32K\n",
      "L1i cache:             32K\n",
      "L2 cache:              256K\n",
      "L3 cache:              8192K\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "lscpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List files in original download location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 62027628\n",
      "-rw-rw-rw- 1 srlab staff 7062253846 Jan 28  2016 151114_I191_FCH3Y35BCXX_L1_wHAIPI023992-37_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 7352262786 Jan 28  2016 151114_I191_FCH3Y35BCXX_L1_wHAIPI023992-37_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 7002262800 Jan 28  2016 151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 6579290112 Jan 28  2016 151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2539393024 Jan 28  2016 151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 4269542816 Jan 28  2016 151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2570830561 Jan 28  2016 160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCABDLAAPEI-62_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2900561782 Jan 28  2016 160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCABDLAAPEI-62_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1370906266 Jan 28  2016 160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCACDTAAPEI-75_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1531179372 Jan 28  2016 160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCACDTAAPEI-75_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2563497188 Jan 28  2016 160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCABDLAAPEI-62_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2932758209 Jan 28  2016 160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCABDLAAPEI-62_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1350152634 Jan 28  2016 160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCACDTAAPEI-75_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1527682296 Jan 28  2016 160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCACDTAAPEI-75_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2851433324 Jan 28  2016 160103_I137_FCH3V5YBBXX_L5_WHOSTibkDCAADWAAPEI-74_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 3249975601 Jan 28  2016 160103_I137_FCH3V5YBBXX_L5_WHOSTibkDCAADWAAPEI-74_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2734377366 Jan 28  2016 160103_I137_FCH3V5YBBXX_L6_WHOSTibkDCAADWAAPEI-74_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 3127877914 Jan 28  2016 160103_I137_FCH3V5YBBXX_L6_WHOSTibkDCAADWAAPEI-74_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff       2712 Jan 27  2016 checksums.md5\n",
      "-rw-r--r-- 1 srlab staff       1614 Jan 25  2016 md5.txt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls -l /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note that the date for the files is <em>later</em> than the dates on both of the checksum files! This is already a bad sign and I have no notes on why this would be the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List files in their permanent location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw- 1 srlab staff 7062253846 Jan 25  2016 /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L1_wHAIPI023992-37_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 7352262786 Jan 25  2016 /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L1_wHAIPI023992-37_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 7002262800 Jan 25  2016 /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 6579290112 Jan 27  2016 /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2539393024 Jan 27  2016 /root/owl_web/nightingales/O_lurida/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 4269542816 Jan 25  2016 /root/owl_web/nightingales/O_lurida/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2570830561 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCABDLAAPEI-62_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2900561782 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCABDLAAPEI-62_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1370906266 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCACDTAAPEI-75_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1531179372 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCACDTAAPEI-75_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2563497188 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCABDLAAPEI-62_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2932758209 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCABDLAAPEI-62_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1350152634 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCACDTAAPEI-75_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 1527682296 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCACDTAAPEI-75_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2851433324 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L5_WHOSTibkDCAADWAAPEI-74_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 3249975601 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L5_WHOSTibkDCAADWAAPEI-74_2.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 2734377366 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L6_WHOSTibkDCAADWAAPEI-74_1.fq.gz\n",
      "-rw-rw-rw- 1 srlab staff 3127877914 Jan 25  2016 /root/owl_web/nightingales/O_lurida/160103_I137_FCH3V5YBBXX_L6_WHOSTibkDCAADWAAPEI-74_2.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls -l /root/owl_web/nightingales/O_lurida/1[56]*.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Although it's a bit messy, there are two things to take note of:\n",
    "\n",
    "#### 1. The dates on all files are <em>earlier</em> than in the original download location!\n",
    "\n",
    "#### 2. The two problem files being investigated are the only two files with a different date!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View initial checksum of files in their permanent location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### First, view checksums stored in the checksum file that was created when the files were moved here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD5 (151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz) = 8bc0d7c7a7af3954baca31a4a7fe9f2b\n",
      "MD5 (151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz) = 08cfdc6fdc5a6190cb05cdcb81fa5b9c\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "grep -e 151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz -e 151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz /root/owl_web/nightingales/O_lurida/checksums.md5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### View current checksum values of files in their current state/location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "861c8161c1e0171532a058582db3ae6d  /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "md5sum /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f10c1f143d153f364cf6be13452eea4a  /root/owl_web/nightingales/O_lurida/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "md5sum /root/owl_web/nightingales/O_lurida/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Well, it's easy to see that these checksums don't match..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View initial checksums of files in original download location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the checksum file I made"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD5 (/Volumes/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz) = 8bc0d7c7a7af3954baca31a4a7fe9f2b\n",
      "MD5 (/Volumes/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz) = 08cfdc6fdc5a6190cb05cdcb81fa5b9c\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "grep -e 151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz -e 151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz \\\n",
    "/root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/checksums.md5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the checksum file provided by BGI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8bc0d7c7a7af3954baca31a4a7fe9f2b  151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n",
      "08cfdc6fdc5a6190cb05cdcb81fa5b9c  151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "grep -e 151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz -e 151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz \\\n",
    "/root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/md5.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Although not as legible as I'd like, you can see that the checksum values in both checksum files match."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the current checksums of the files in the original download location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "861c8161c1e0171532a058582db3ae6d  /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "md5sum /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f10c1f143d153f364cf6be13452eea4a  /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "md5sum /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### These files' checksums do not match the values stored in either checksum file. The weird part about all of this is related to the file dates. The dates of the files in the original download location are <em>later</em> than those in the permanent location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw- 1 srlab staff 6579290112 Jan 28  2016 /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls -l /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw- 1 srlab staff 6579290112 Jan 27  2016 /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls -l /root/owl_web/nightingales/O_lurida/151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw- 1 srlab staff 2539393024 Jan 28  2016 /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls -l /root/owl_home/O_lurida_BGI_F15FTSUSAT0327_OSTibkD/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw- 1 srlab staff 2539393024 Jan 27  2016 /root/owl_web/nightingales/O_lurida/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls -l /root/owl_web/nightingales/O_lurida/151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This would suggest that the files in my ```Owl/home``` directory were re-downloaded at some point. However, I can't explain why the files in both locations have the same MD5 checksum value <em>AND</em> those values differ from the checksum values recorded in the checksum files (mine and that provided by BGI)..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check MD5 checksums of remaining files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command below works as follows:\n",
    "time - Tracks how long process takes to run\n",
    "\n",
    "Uses for loop to look at all FASTQ files from BGI in the ```owl/web/nightingales/O_lurida``` directory. For each of those files:\n",
    "\n",
    "- Runs the program ```md5sum``` on each file to generate a checksum and pipes the output to awk. Awk prints the first field ($1) from the output of ```md5sum```. This field conatins the checksum value. This value is stored in the variable called ```curr_md5```.\n",
    "\n",
    "- Runs the program ```grep``` to search for the filename in the checksums.md5 file. The filename is extracted from the ```$fastq``` variable (this variable contains the full path to each file, which will screw up the search by ```grep```) using parameter expansion/substring removal. The command looks for the longest pattern matching what we provide (the ```##``` tells it to look for the longest pattern). We told bash to look for anything matching a forward slash (the ```*/``` in the command). The output from ```grep``` is piped to ```awk```. Awk prints the fourth field ($4) from the output, which corresponds to the checksum value in the ```checksums.md5``` file. This value is stored in the variable called ```orig_md5```.\n",
    "\n",
    "- Prints (```echo \"$curr_md5\"```) the value currently stored in the ```curr_md5``` variable, then prints (```echo \"$orig_md5\"```) the value currently stored in ```orig_md5```, followed by printing an empty line (```echo \"\"```) to improve readibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "810d188468dbd8bb36b2af3bf3b9fee6\n",
      "810d188468dbd8bb36b2af3bf3b9fee6\n",
      "\n",
      "cf92b18e0815dc0471d61f9107142257\n",
      "cf92b18e0815dc0471d61f9107142257\n",
      "\n",
      "3dc2137d7df0af8d6a007516908361a3\n",
      "3dc2137d7df0af8d6a007516908361a3\n",
      "\n",
      "861c8161c1e0171532a058582db3ae6d\n",
      "8bc0d7c7a7af3954baca31a4a7fe9f2b\n",
      "\n",
      "f10c1f143d153f364cf6be13452eea4a\n",
      "08cfdc6fdc5a6190cb05cdcb81fa5b9c\n",
      "\n",
      "a503043167457337a65d51151ceb5dd0\n",
      "a503043167457337a65d51151ceb5dd0\n",
      "\n",
      "a0c0177a7a4a4ca28c37bd3802361564\n",
      "a0c0177a7a4a4ca28c37bd3802361564\n",
      "\n",
      "74e51d608de7a409a29545a95ac3ec14\n",
      "74e51d608de7a409a29545a95ac3ec14\n",
      "\n",
      "e829da6f7e7a047586802a05d3b96d18\n",
      "e829da6f7e7a047586802a05d3b96d18\n",
      "\n",
      "74953d0143c392f111eec178667d3fcd\n",
      "74953d0143c392f111eec178667d3fcd\n",
      "\n",
      "9b5f7c1593f216f710814c299738493f\n",
      "9b5f7c1593f216f710814c299738493f\n",
      "\n",
      "5a6392d9c23aa85d170d7f50de4f0b54\n",
      "5a6392d9c23aa85d170d7f50de4f0b54\n",
      "\n",
      "2cf349b55558e7424fa0ed029ae0bfca\n",
      "2cf349b55558e7424fa0ed029ae0bfca\n",
      "\n",
      "5cb745671596d5d5d307986d35ea9792\n",
      "5cb745671596d5d5d307986d35ea9792\n",
      "\n",
      "b371a3b3588060bb2200f5caaf9a9d5c\n",
      "b371a3b3588060bb2200f5caaf9a9d5c\n",
      "\n",
      "dd8b0fef21fc5e330d08ae4a48c8d67b\n",
      "dd8b0fef21fc5e330d08ae4a48c8d67b\n",
      "\n",
      "50b5dbff9426738005b81efd49d66329\n",
      "50b5dbff9426738005b81efd49d66329\n",
      "\n",
      "29be6e734ad65c180aee23d1514c5c35\n",
      "29be6e734ad65c180aee23d1514c5c35\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t37m34.177s\n",
      "user\t0m4.530s\n",
      "sys\t6m46.800s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time for fastq in /root/owl_web/nightingales/O_lurida/1[56]*.gz; do \\\n",
    "curr_md5=$( md5sum \"$fastq\" | awk '{ print $1 }')\n",
    "orig_md5=$( grep ${fastq##*/} /root/owl_web/nightingales/O_lurida/checksums.md5 | awk '{ print $4}')\n",
    "echo \"$curr_md5\"\n",
    "echo \"$orig_md5\"\n",
    "echo \"\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All files current MD5 checksums match the original MD5 checksums that were generated, except those the two problem files. I'll contact BGI about re-downloading these two files..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}