{
 "metadata": {
  "name": "",
  "signature": "sha256:e31db52e1d58916ba69ed743a5549294cb4ba465b4c8fb53bf44b075548bd085"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## [Python Programming for Biologists, Tel-Aviv University / 0411-3122 / Spring 2015](http://py4life.github.io/TAU2015/)\n",
      "# Homework 4\n",
      "### General instructions\n",
      "- In questions 1,2 and 4 you will have to read and parse files. Download them [here](https://github.com/Py4Life/TAU2015/raw/master/files_for_hw4.rar), extract the rar file and then save the directory to your course directory (or wherever you access the notebooks from).\n",
      "- In questions 2 and 4 you will have to use the _Comma Separated Values_ (CSV) format, and the python csv module. To learn how to use it, go over the appropriate section in lecture 4. You can also learn more [here](https://docs.python.org/3/library/csv.html)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## 1) Parsing $\\lambda$ phage FASTQ\n",
      "The FASTQ file format is commonly used to store deep sequencing reads data. It is similar to the FASTA format, but includes additional information. Each record is represented by four lines:\n",
      "- Line 1 begins with a `@` character and is followed by a sequence identifier and an optional description (like a FASTA title line).\n",
      "- Line 2 is the raw sequence letters.\n",
      "- Line 3 is just a `+` character\n",
      "- Line 4 encodes the quality values for the sequence in Line 2, and must contain the same number of symbols as letters in the sequence.  \n",
      "Sequence quality is encoded with characters from:\n",
      "```\n",
      "!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\n",
      "```\n",
      "Where `!` represents the lowest quality, equivalent to a score of 1, and `~` is the highest quality with a score of 94.  \n",
      "\n",
      "The function given below translates the characters into their corresponding scores and returns a dictionary which you can use later. Make sure you understand how to work with this dictionary before proceeding. Also note the use of `\"\"\"`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def creates_scores_dict():\n",
      "    scores_string = \"\"\"!\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\"\"\"\n",
      "    scores_dict = {}\n",
      "    for i,symbol in enumerate(scores_string):\n",
      "        scores_dict[symbol] = i + 1\n",
      "    return scores_dict\n",
      "\n",
      "scores_dict = creates_scores_dict()\n",
      "print(scores_dict)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'W': 55, 'Y': 57, '7': 23, 'e': 69, 'n': 78, '>': 30, 'r': 82, ')': 9, 'l': 76, '@': 32, 's': 83, '{': 91, '1': 17, 'g': 71, 'j': 74, 'u': 85, 'X': 56, '$': 4, 'a': 65, 'F': 38, '9': 25, ';': 27, 'B': 34, '#': 3, '^': 62, '*': 10, '[': 59, '|': 92, '4': 20, 'd': 68, ']': 61, '}': 93, 'U': 53, '%': 5, 'L': 44, 'T': 52, 'y': 89, '3': 19, 'c': 67, 'o': 79, 'f': 70, 'A': 33, 'Q': 49, '-': 13, 'R': 50, 'E': 37, '!': 1, '`': 64, 'D': 36, '_': 63, ':': 26, ',': 12, 't': 84, 'I': 41, 'O': 47, 'z': 90, 'G': 39, '5': 21, '(': 8, '\\\\': 60, 'N': 46, '~': 94, 'Z': 58, 'i': 73, '6': 22, 'q': 81, '8': 24, 'v': 86, 'k': 75, 'S': 51, '/': 15, 'b': 66, 'K': 43, '=': 29, \"'\": 7, 'C': 35, '0': 16, 'M': 45, 'w': 87, '?': 31, 'J': 42, 'm': 77, 'x': 88, '&': 6, 'P': 48, '+': 11, 'h': 72, '.': 14, 'V': 54, '2': 18, '<': 28, '\"': 2, 'p': 80, 'H': 40}\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The file `files_for_hw4/lambda_reads.fq` contains 10,000 reads from the sequencing of $\\lambda$ phage. We would like to discard low quality reads. A low quality read is defined as one with a mean score lower than some predefined threshold.  \n",
      "\n",
      "**a)** Write a function `mean_score` that receives a read quality string and returns the mean score (float) of the read. \n",
      "\n",
      "For example, the quality string `!!!!!` is equivalent to the scores `1,1,1,1,1`, and thus the mean is `1.0`. \n",
      "However, the string `49@5<*>=E` is equivalent to the scores `20,25,32,21,28,10,30,29,37` and has a mean of `25.77`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def mean_score(read_quality_string):\n",
      "    # your code here. remove the pass statement.\n",
      "    pass\n",
      "    \n",
      "assert(mean_score('!!!!!') == 1.0)\n",
      "assert(round(mean_score('49@5<*>=E'),2) == 25.78)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b)** Write a function `parse_FASTQ` that parses a FASTQ file. It receives a path to a FASTQ file on the local filesystem and returns a _dictionary_ where keys are sequences and values are the mean scores of the sequences. \n",
      "\n",
      "Use the function on the provided file `files_for_hw4/lambda_reads.fq`.\n",
      "\n",
      "It is recommended to use the `readline()` method of file objects (although other solutions are also possible)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def parse_FASTQ(file):\n",
      "    # your code here. remove the pass statement.\n",
      "    pass\n",
      "    \n",
      "    \n",
      "# parse lambda reads file  \n",
      "lambda_reads_file = \"files_for_hw4/lambda_reads.fq\"\n",
      "lambda_seqs_dict = parse_FASTQ(lambda_reads_file)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c)** Write a function `filter_reads` that takes the output from section **b** and a score cutoff (integer/float) and prints out the sequences with scores higher than the cutoff. \n",
      "\n",
      "Each sequence will be printed in a separate line (no need to keep the FASTQ format). Try different cutoffs (5,10,20) on the $\\lambda$ phage reads."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def filter_reads(seqs_dict,cutoff,out_file):\n",
      "    # your code here. remove the pass statement.\n",
      "    pass\n",
      "    \n",
      "    \n",
      "# run on Lambda reads\n",
      "lambda_filtered_file = \"files_for_hw4/lambda_filtered_reads.txt\"\n",
      "filter_reads(lambda_seqs_dict, 10, lambda_filtered_file)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## 2) Endangered turtles\n",
      "In this question, we will work on data from the [Global Biodiversity Information Facility](http://www.gbif.org/) (GBIF). \n",
      "This server has lots of data about occurences of organisms around the world.  \n",
      "\n",
      "Our goal will be to get the coordinates of observations of endangered turtles species in the Southern hemisphere.  \n",
      "The CSV files under `files_for_hw4/gbif_files`, named `GBIF1.csv`, `GBIF2.csv` etc. contain observations data for various turtles genera. Each record is an observation, and contains information such as the species name and the coordinates of the observation.  \n",
      "\n",
      "**a)** The first step in the analysis will be to concatenate all the CSV files into one CSV file to make processing easier. \n",
      "\n",
      "Write a function that receives a list of file names and concatenates them (i.e., inserts all records from all files into one file). Remember to include the header line only once, at the beginning. \n",
      "\n",
      "Use the function to create a unified csv file.  \n",
      "\n",
      "__Note__: The function has no return value!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def concat_csvs(filenames, output_filename):\n",
      "    # your code here. remove the pass statement.\n",
      "    pass   \n",
      "    \n",
      "\n",
      "# create unified file\n",
      "import glob\n",
      "filenames = glob.glob(\"files_for_hw4/gbif_files/GBIF*.csv\")\n",
      "print(filenames)\n",
      "output_filename = \"files_for_hw4/gbif_files/concat_GBIF.csv\"\n",
      "append_csvs(______________, ______________)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b)** The file `files_for_hw4/gbif_files/endangered_turtles.txt` contains a list of endangered turtles species from [The IUCN Red List of Threatened Species](http://www.iucnredlist.org/). \n",
      "\n",
      "Write a function `get_species` that reads the file and returns a list of the species it includes."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def get_species(filename):\n",
      "    # your code here. remove the pass statement.\n",
      "    pass \n",
      "    \n",
      "    \n",
      "turtles_filename = \"files_for_hw4/gbif_files/endangered_turtles.txt\"\n",
      "endangered_turtles = get_species(turtles_filename)\n",
      "print(endangered_turtles)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c)** We will now use the unified CSV file and endangered species list to create a filtered CSV, containing only the records we are interested in: those of endangered species, observed in the Southern hemisphere (_i.e._ in latitude < 0).  \n",
      "\n",
      "Write a function that receives the unified filename, the species list, maximum latitude and output file, and write to the output file the records which satisfy these conditions (no return value is needed)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def filter_records(filename, species, max_lat, out_csv_file):\n",
      "    # your code here. remove the pass statement.\n",
      "    pass     \n",
      "    \n",
      "    \n",
      "filtered_file = \"files_for_hw4/gbif_files/endangered.csv\"\n",
      "filter_records(_______, _______, _______, _______)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## 3) Regex drills\n",
      "In this question, you don't have to write real code, just write the regular expression you'd use within the quotation marks.  \n",
      "\n",
      "* It is highly recommended to use [The regex Coach](http://www.weitz.de/regex-coach/) for this question.\n",
      "\n",
      "a) Write a regex that will match strings containing any kind of number: positive/negative, integer/float etc. For example, all of the following should be matched: 7 , -3 , 6.14 , -0.00054  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import re\n",
      "\n",
      "re.compile(r'')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "b) Write a regex that will match strings that __end__ with a number between 100 and 199, followed by a '.' __or__ a '\\' character.   "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "re.compile(r'')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "c) Write a regex that will match whole strings of prices in dollars, such as '100\\$', '2.99\\$', '500.90\\$', but not '7.656\\$', '80.0001\\$' or 'price is: 56.80\\$'."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "re.compile(r'')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "d) Write a regex that will match strings beginning with 3 to 8 uppercase letters, followed by at least 4 characters, which can be anything but '%' or '!', and end with 'XY' or 'QW'. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "re.compile(r'')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## 4) Plant names\n",
      "A full scientific name of a plant species consists of a genus name, a species name and an authority (usually a short for the name of the person to first describe the species). For example, in _Arabidopsis thaliana (L.) Heynh._, _Arabidopsis_ is the genus, _thaliana)_ is the species and _(L.) Heynh._ is the authority. \n",
      "\n",
      "- The genus always begins with a capital letter, followed by one or more lower case letters. \n",
      "- The species is all lowercasse. \n",
      "- The authority can include any character. \n",
      "\n",
      "In addition, a name may (or may not) include an infraspecific rank. This is added after the species name, and consists of an _epithet_ which is either _subsp._ (for subspecies) or _var._ (for variety). The epithet is followed by the name of the infraspecific rank. \n",
      "\n",
      "For example, in _Fraxinus americana var. acuminata (Lam.) K.Koch_, the genus is _Fraxinus_, the species is _americana_, the infraspecies is _var. acuminata_ and the authority is _(Lam.) K.Koch_.  \n",
      "\n",
      "The file `files_for_hw4/plant_names.txt` contains a list of plant names. The goal is to break these names into their components. \n",
      "\n",
      "Write a program that reads the names from the file and writes a new CSV file with the following column names: _Genus_, _Species_, _Infraspecific_ and _Authority_. \n",
      "Each plant name in `plant_names.txt` should then be processed (use regular expressions) and its components inserted into the CSV file."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "### your code here\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}