{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "###Display system info" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Software:\r\n", "\r\n", " System Software Overview:\r\n", "\r\n", " System Version: OS X 10.9.5 (13F34)\r\n", " Kernel Version: Darwin 13.4.0\r\n", " Boot Volume: Hummingbird\r\n", " Boot Mode: Normal\r\n", " Computer Name: hummingbird\r\n", " User Name: Sam (Sam)\r\n", " Secure Virtual Memory: Enabled\r\n", " Time since boot: 121 days 1:26\r\n", "\r\n" ] } ], "source": [ "!system_profiler SPSoftwareDataType" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Volumes/Data/Sam/scratch\n" ] } ], "source": [ "cd /Volumes/Data/Sam/scratch/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###Quality trim all fastq.gz files using [Trimmomatic (v0.30)](http://www.usadellab.org/cms/?page=trimmomatic)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "####Code explanation of for loop below:\n", "1. ```%%bash``` specifies to use the shell for this Jupyter cell\n", "2. ```for file in /Volumes/nightingales/C_gigas/2212_lane2_[^N]*``` initiates a for loop to handle all files beginning with ```2212_lane2_``` and only those that do not have the letter \"N\" at that position in the file name.\n", "3. ```do``` tells the for loop what to do with each of the files.\n", "4. ```newname=${file##*/}``` takes the value of the ```$file``` variable (which is ```/Volumes/nightingales/C_gigas/2212_lane2_[^N]*```) and trims the longest match from the beginning of the pattern (the pattern is ```*/```; the ```##``` is a bash command to specifiy how to trim). The resulting output (which is just the file name without the full path) is then stored in the ```newname``` variable.\n", "5. This line initiates Trimmomatic and uses the following arguments to specify order of execution:\n", " 1. single end reads (```SE```)\n", " 1. number of threads (```-threads 16```), \n", " 2. type of quality score (```-phred33```),\n", " 3. input file location (```\"$file\"```),\n", " 4. output file name/location (```/Volumes/Data/Sam/scratch/20140521_trimmed_$newname```),\n", " 5. single end Illumina TruSeq adaptor trimming (```ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10```); uses fasta file with adaptor sequences; came with program,\n", " 6. trim read lengths to set length by trimming from end of read (```CROP:90```); removes last 10 bases\n", " 7. cut number of bases at beginning of read (```HEADCROP:39```)\n", " 6. cut number of bases at beginning of read if below quality threshold (```LEADING:3```)\n", " 7. cut number of bases at end of read if below quality threshold (```TRAILING:3```)\n", " 8. cut if average quality within 4 base window falls below 15 (```SLIDINGWINDOW:4:15```)\n", "6. ```done``` closes for loop." ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_CTTGTA_L002_R1_001.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_CTTGTA_L002_R1_001.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15796545 (98.73%) Dropped: 203455 (1.27%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_CTTGTA_L002_R1_002.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_CTTGTA_L002_R1_002.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15793607 (98.71%) Dropped: 206393 (1.29%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_CTTGTA_L002_R1_003.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_CTTGTA_L002_R1_003.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15784607 (98.65%) Dropped: 215393 (1.35%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_CTTGTA_L002_R1_004.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_CTTGTA_L002_R1_004.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 10634369 Surviving: 10493068 (98.67%) Dropped: 141301 (1.33%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_GCCAAT_L002_R1_001.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_GCCAAT_L002_R1_001.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15797775 (98.74%) Dropped: 202225 (1.26%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_GCCAAT_L002_R1_002.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_GCCAAT_L002_R1_002.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15794884 (98.72%) Dropped: 205116 (1.28%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_GCCAAT_L002_R1_003.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_GCCAAT_L002_R1_003.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15797804 (98.74%) Dropped: 202196 (1.26%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_GCCAAT_L002_R1_004.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_GCCAAT_L002_R1_004.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15787414 (98.67%) Dropped: 212586 (1.33%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_GCCAAT_L002_R1_005.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_GCCAAT_L002_R1_005.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 16000000 Surviving: 15789953 (98.69%) Dropped: 210047 (1.31%)\n", "TrimmomaticSE: Completed successfully\n", "TrimmomaticSE: Started with arguments: -threads 16 -phred33 /Volumes/nightingales/C_gigas/2212_lane2_GCCAAT_L002_R1_006.fastq.gz /Volumes/Data/Sam/scratch/20150521_trimmed_2212_lane2_GCCAAT_L002_R1_006.fastq.gz ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 CROP:90 HEADCROP:39 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'\n", "Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'\n", "ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences\n", "Input Reads: 255678 Surviving: 250209 (97.86%) Dropped: 5469 (2.14%)\n", "TrimmomaticSE: Completed successfully\n" ] } ], "source": [ "%%bash\n", "for file in /Volumes/nightingales/C_gigas/2212_lane2_[^N]*\n", "do\n", "newname=${file##*/} \n", "java -jar /usr/local/bioinformatics/Trimmomatic-0.30/trimmomatic-0.30.jar \\\n", "SE \\\n", "-threads 16 \\\n", "-phred33 \"$file\" \\\n", "/Volumes/Data/Sam/scratch/20150521_trimmed_$newname \\\n", "ILLUMINACLIP:/usr/local/bioinformatics/Trimmomatic-0.30/adapters/TruSeq3-SE.fa:2:30:10 \\\n", "CROP:90 \\\n", "HEADCROP:39 \\\n", "LEADING:3 \\\n", "TRAILING:3 \\\n", "SLIDINGWINDOW:4:15;\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###Concatenate two groups of sequences into single file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "####400ppm (control) sequences - Index GCCAAT" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "#gunzips all matching files in folder and appends the data to a single file:\n", "#201500521_trimmed_2212_lane2_400ppm_GCCAAT.fastq\n", "for file in 20150521_trimmed_2212_lane2_G*\n", "do\n", "gunzip -c \"$file\" >> 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq\n", "done" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "#Gzip file\n", "gzip 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "####1000ppm (acidification) sequences - Index CTTGTA" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "#gunzips all matching files in folder and appends the data to a single file:\n", "#20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq\n", "for file in 20150521_trimmed_2212_lane2_C*\n", "do\n", "gunzip -c \"$file\" >> 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq\n", "done" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "#Gzip file\n", "gzip 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###FASTQC on concatenated files using [FASTQC (v0.11.2)](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Analysis complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Analysis complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Started analysis of 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 5% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 10% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 15% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 20% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 25% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 30% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 35% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 40% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 45% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 50% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 55% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 60% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 65% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 70% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 75% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 80% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 85% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 90% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Approx 95% complete for 20150521_trimmed_2212_lane2_1000ppm_CTTGTA.fastq.gz\n", "Started analysis of 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 5% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 10% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 15% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 20% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 25% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 30% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 35% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 40% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 45% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 50% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 55% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 60% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 65% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 70% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 75% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 80% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 85% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 90% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n", "Approx 95% complete for 20150521_trimmed_2212_lane2_400ppm_GCCAAT.fastq.gz\n" ] } ], "source": [ "%%bash\n", "for file in /Volumes/Data/Sam/scratch/20150521_*[e2]_[14]*.gz; do fastqc \"$file\" --outdir=/Volumes/Eagle/Arabidopsis/; done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###Copy files to Eagle for web-based access" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "for file in 2015*e2_[14]*; do cp \"$file\" /Volumes/Eagle/Arabidopsis/; done" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }