{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Making a data frame tall\n",
"\n",
"[Data set download](https://s3.amazonaws.com/bebi103.caltech.edu/data/Reeves2012_data.xlsx)\n",
"\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"nbsphinx": "hide",
"tags": []
},
"outputs": [],
"source": [
"# Colab setup ------------------\n",
"import os, sys, subprocess\n",
"if \"google.colab\" in sys.modules:\n",
" cmd = \"pip install --upgrade polars iqplot watermark\"\n",
" process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
" stdout, stderr = process.communicate()\n",
" data_path = \"https://s3.amazonaws.com/bebi103.caltech.edu/data/\"\n",
"else:\n",
" data_path = \"../data/\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" \n",
"
\n",
"
Loading BokehJS ...\n",
"
\\n\"+\n",
" \"
\\n\"+\n",
" \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
" \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
" \"
\\n\"+\n",
" \"
\\n\"+\n",
" \"- re-rerun `output_notebook()` to attempt to load from CDN again, or
\\n\"+\n",
" \"- use INLINE resources instead, as so:
\\n\"+\n",
" \"
\\n\"+\n",
" \"
\\n\"+\n",
" \"from bokeh.resources import INLINE\\n\"+\n",
" \"output_notebook(resources=INLINE)\\n\"+\n",
" \"\\n\"+\n",
" \"
\\n\"+\n \"
\\n\"+\n \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n \"
\\n\"+\n \"
\\n\"+\n \"- re-rerun `output_notebook()` to attempt to load from CDN again, or
\\n\"+\n \"- use INLINE resources instead, as so:
\\n\"+\n \"
\\n\"+\n \"
\\n\"+\n \"from bokeh.resources import INLINE\\n\"+\n \"output_notebook(resources=INLINE)\\n\"+\n \"\\n\"+\n \"
\n",
"\n",
"In this part of the lesson, we will perform some wrangling on a data set that involves:\n",
"\n",
"- Using Polars's `read_excel()` function to read in data from an MS Excel spread sheet\n",
"- Parsing column names to extract useful metadata en route to tidy data\n",
"- Performing an unpivot operation to efficiently tidy a data set."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The data set\n",
"\n",
"We will use a data set from [Angela Stathopoulos's lab](http://www.stathlab.caltech.edu), acquired to study morphogen profiles in developing fruit fly embryos. The original paper is [Reeves, Trisnadi, et al., Dorsal-ventral gene expression\n",
"in the *Drosophila* embryo reflects the dynamics and precision of the Dorsal nuclear gradient, *Dev. Cell.*, **22**, 544-557, 2012](https://doi.org/10.1016/j.devcel.2011.12.007), and the data set may be downloaded here: [https://s3.amazonaws.com/bebi103.caltech.edu/data/Reeves2012_data.xlsx](https://s3.amazonaws.com/bebi103.caltech.edu/data/Reeves2012_data.xlsx).\n",
"\n",
"In this experiment, Reeves, Trisnadi, and coworkers measured expression levels of a fusion of Dorsal, a morphogen transcription factor important in determining the dorsal-ventral axis of the developing organism, and Venus, a yellow fluorescent protein along the dorsal/ventral- (DV) coordinate. They put this construct on the third chromosome, while the wild type *dorsal* is on the second. Instead of the wild type, they had a homozygous *dorsal*-null mutant on the second chromosome. The Dorsal-Venus construct rescues wild type behavior, so they could use this construct to study Dorsal gradients.\n",
"\n",
"Dorsal shows higher expression on the ventral side of the organism, thus giving a *gradient* in expression from dorsal to ventral which can be ascertained by the spatial distribution of Venus fluorescence intensity.\n",
"\n",
"This can be seen in the image below, which is a cross section of a fixed embryo with anti-Dorsal staining. The bottom of the image is the ventral side and the top is the dorsal side of the embryo. The DV coordinate system is defined by the yellow line. The image is adapted from the Reeves, Trisnadi, et al. paper.\n",
"\n",
"\n",
"\n",
"\n",
" \n",
"
\n",
"\n",
"*A quick note on nomenclature:* Dorsal (capital D) is the name of the protein product of the gene *dorsal* (italicized). The dorsal (adjective) side of the embryo is its back. The ventral side is its belly. Dorsal is expressed more strongly on the _ventral side_ of the developing embryo. This can be confusing.\n",
"\n",
"To quantify the gradient, Reeves, Trisnadi, and coworkers had to first choose a metric for describing it. They chose to fit the measured profile of fluorescence intensity with a Gaussian peak (plus background) and use the standard deviation of that Gaussian as a metric for the width of the Dorsal gradient.\n",
"\n",
"In this lesson, we will use the gradient widths as outputted from this procedure. The units of the widths are dimensionless, consistent with the coordinate system shown in the image above. I asked one of the authors for the data sets used in making the figures. She sent me a MS Excel file that had a separate sheet for each of several figures in the paper that I asked about. We will focus on the data used for Fig. 1F of the paper. In this figure, the authors seek to demonstrate that live imaging with their Venus-Dorsal construct gives a Dorsal gradient of similar width as would be obtained by fixing wild type cells and doing Dorsal antibody staining (the gold standard). These wild type embryos were analyzed as whole mounts and also as cross sections. They also tried anti-Dorsal staining and anti-Venus staining in the Venus-Dorsal construct. Finally, they also measured gradient widths of a GFP-Dorsal construct that fails to complete development."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loading in an Excel sheet\n",
"\n",
"Generally, you should store your data sets in portable formats, like CSV, JSON, XML, HDF5, OME-TIFF, etc., and not proprietary formats. Nonetheless, software like Microsoft Excel is widely used, and you will often receive data sets in this format. Fortunately, Polars can read Excel files, provided they are from fairly recent versions of Excel.\n",
"\n",
"To read in this data set, we will use `pl.read_excel()`. Importantly, because an Excel document may have many sheets, we need to specify the sheet name we want, in this case `'Fig 1F'`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
shape: (5, 8)| wt wholemounts | wt cross-sections | anti-Dorsal dl1/+dl-venus/+ | anti-gfp dl1/+dl-venus/+ | Venus (live) dl1/+dl-venus/+ | anti-Dorsal dl1/+dl-GFP/+ | anti-gfp dl1/+dl-GFP/+ | GFP (live) dl1/+dl-GFP/+ |
|---|
| f64 | f64 | f64 | f64 | f64 | f64 | f64 | f64 |
| 0.1288 | 0.1327 | 0.1482 | 0.1632 | 0.1666 | 0.2248 | 0.2389 | 0.2412 |
| 0.1554 | 0.1457 | 0.1503 | 0.1671 | 0.1753 | 0.1891 | 0.2035 | 0.1942 |
| 0.1306 | 0.1447 | 0.1577 | 0.1704 | 0.1705 | 0.1705 | 0.1943 | 0.2186 |
| 0.1413 | 0.1282 | 0.1711 | 0.1779 | null | 0.1735 | 0.2 | 0.2104 |
| 0.1557 | 0.1487 | 0.1342 | 0.1483 | null | 0.2135 | 0.256 | 0.2463 |
\n",
"
shape: (5, 8)| WT_whole mount | WT_cross-section | dl1/+ ; dl-venus/+_anti-Dorsal | dl1/+ ; dl-venus/+_anti-Venus | dl1/+ ; dl-venus/+_Venus (live) | dl1/+ ; dl-gfp/+_anti-Dorsal | dl1/+ ; dl-gfp/+_anti-GFP | dl1/+ ; dl-gfp/+_GFP (live) |
|---|
| f64 | f64 | f64 | f64 | f64 | f64 | f64 | f64 |
| 0.1288 | 0.1327 | 0.1482 | 0.1632 | 0.1666 | 0.2248 | 0.2389 | 0.2412 |
| 0.1554 | 0.1457 | 0.1503 | 0.1671 | 0.1753 | 0.1891 | 0.2035 | 0.1942 |
| 0.1306 | 0.1447 | 0.1577 | 0.1704 | 0.1705 | 0.1705 | 0.1943 | 0.2186 |
| 0.1413 | 0.1282 | 0.1711 | 0.1779 | null | 0.1735 | 0.2 | 0.2104 |
| 0.1557 | 0.1487 | 0.1342 | 0.1483 | null | 0.2135 | 0.256 | 0.2463 |
\n",
"
shape: (5, 2)| genotype_method | gradient width |
|---|
| str | f64 |
| "WT_whole mount" | 0.1288 |
| "WT_whole mount" | 0.1554 |
| "WT_whole mount" | 0.1306 |
| "WT_whole mount" | 0.1413 |
| "WT_whole mount" | 0.1557 |
\n",
"
shape: (5, 3)| genotype | method | gradient width |
|---|
| str | str | f64 |
| "WT" | "whole mount" | 0.1288 |
| "WT" | "whole mount" | 0.1554 |
| "WT" | "whole mount" | 0.1306 |
| "WT" | "whole mount" | 0.1413 |
| "WT" | "whole mount" | 0.1557 |