![](\"https://raw.githubusercontent.com/jupyterlab/jupyterlab-toc/master/toc.gif\")
\n",
"\n",
"When you need to navigate the directory structure or access files in the file browser, switch back to the file browser by clicking on the 'folder' icon \n",
"on the top of the left sidebar. \n",
"\n",
"## Saving files or bundles of files from the session to local\n",
"\n",
"This notebook includes ways to make various files. You'll need to download those from this **temporary, remote session** to your local computer. \n",
"\n",
"To make this easier once you start to generate a lot of files, the ability to right-click on any directory in the file navigator and choose to save it as an archive has been added. It will resemble the following animation:\n",
"\n",
"![](\"https://raw.githubusercontent.com/hadim/jupyter-archive/master/archive.gif\")
\n",
"\n",
"For example, if you unpack the archive as instructed below you may end up making things in `unpack` and want that entire directory. Therefore in the file navigator panel on the left, you'd go to a point in the directory hierarchy above that `unpack` directory and right-click on it and choose `Download as an archive`. You can make a new directory any time using the 'New folder' icon above the left panel and then use drag and drop in the file navigator to move any files you want into it.\n",
"\n",
"The archive defaults to being `.gzip` format. You can alter that format if you'd prefer by going to the JupyterLab menu bar along the top of your browser window and selecting `Settings` > `Advanced Settings Editor`, and then selecting `Archive` in the `Settings` window that comes up in the main window. There you can edit the '\n",
"`User Preferences` on the right to override the system default. For example maybe you'd prefer `tar.gz`, and so you'd edit the contents of `User Preferences` to be `{\"format\": \"tar.gz\"}`. Be sure to click on the disk icon in the upper right to save your new settings."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Accessing the bendIt results: Starting up an active session\n",
"\n",
"If you already have an active Jupyter session going, you can skip this step.\n",
"\n",
"However, if you have a previously saved result from the demo pipeline and you are now returning to it and looking to access the contents of the archive, the best way to get started is to start a session by going [here](https://github.com/fomightez/bendit-binder) and clicking on the `launch bend.it` badge. When the session starts, select the `Accessing_and_reviewing_the_bendit_results.ipynb` notebook from the navigation panel at the right and work through the steps below this.\n",
"\n",
"The uncompressing step below should work on an unix-style command line. However, some of the advanced steps under the section entitled 'Using Python to review the data or customize the plots' involve modules/packages that aren't necessarily in a standard Python installation. By running in the recommended environment as directed above, the code is more likely to work without hiccups."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Accessing the bendIt results: Uncompress the Archive\n",
"\n",
"This notebook assumes, you just ran the pipeline and are trying to access the results. There is then a number of files in the current directory beside the archive. To make things easier, I am going to suggest making a new directory with a simple name and then you could drag the archive into that folder and work there. To make things match witih the demo, I am going to write out those steps as commands, too. Feel free to run the cells or to do it by hand. If you make a different 'unpack' directory, you'll need to adjust things below accordingly.\n",
"\n",
"The exclamation marks in front of the shell commands, tells Jupyter to run those commands as shell commands and not Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"!mkdir unpack\n",
"!mv bendit_analysis*.tar.gz unpack/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To make things in this notebook work, you'll need to switch the current working directory over to where we are going to unpack the archive. We'll use the Jupyter magic command `%cd` to change the working directory for all subsequent cells in the notebook. If we just used `!cd`, it would only change the directory for that cell."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/jovyan/unpack\n"
]
}
],
"source": [
"%cd unpack"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note you can check anytime what is the current working directory with `pwd`. (Note most shell commands need an exclamation point but a few were added in to Jupyter so they work without it and `pwd` is one.) Also of note, is that this location is completely independent of where the file navigation pane on the left side of this window may be showing."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'/home/jovyan/unpack'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pwd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To **unpack the archive**, we'll run the command in the following cell. \n",
"You need to edit the command so it will extract your file. In other words, the part after the `xzf` has to be changed to match the actual file name of the archive you are working with in particular. To make it easier you should be able to click after the default code in the cell below and hit `Tab` button on your keyboard and Jupyter will auto-complete the file name for you after a slight pause.\n",
"\n",
"(Note that I am keeping this untarring step as generic as possible so that it would work as written on any unix-style command line without the exclamation point.)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"!tar xzf bendit_analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you ran the above cell and saw anything like `tar (child): bendit_analysisZZZZZZZZZ.tar.gz: Cannot open: No such file or directory` (where `ZZZZZZZZZ` is used to represent the date time stamp), it simply means you had the file name wrong. You'll need to edit it and run the cell again.\n",
"\n",
"If things worked, you should just see the asterisk to the left of the cell turn to a number. If you changed the file navigation pane on the left-side of this browser window over to the 'unpack' directory, after a moment you should see more files show up in the file pane. Don't worry if you didn't switch. We are going to explore the contents using commands next anyways."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overview of the Unpacked Items\n",
"\n",
"If all is correct and you used the `%cd` command to previously switch to the 'unpack directory, running the next cell will show the contents of the current working directory so we can begin to explore the contents of the-now-unpacked archive."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 1.3M\n",
"-rw-r--r-- 1 jovyan root 11K Feb 21 20:27 A_output.png\n",
"-rw-r--r-- 1 jovyan root 365K Feb 21 20:28 bendit_analysisFeb2120202027.tar.gz\n",
"-rw-r--r-- 1 jovyan root 10K Feb 21 20:27 B_output.png\n",
"-rw-r--r-- 1 jovyan root 3.1K Feb 21 20:27 demo_A.pkl\n",
"-rw-r--r-- 1 jovyan root 38K Feb 21 20:27 demo_A.png\n",
"-rw-r--r-- 1 jovyan root 43K Feb 21 20:27 demo_A.svg\n",
"-rw-r--r-- 1 jovyan root 1.6K Feb 21 20:27 demo_A.tsv\n",
"-rw-r--r-- 1 jovyan root 3.1K Feb 21 20:27 demo_B.pkl\n",
"-rw-r--r-- 1 jovyan root 38K Feb 21 20:27 demo_B.png\n",
"-rw-r--r-- 1 jovyan root 44K Feb 21 20:27 demo_B.svg\n",
"-rw-r--r-- 1 jovyan root 1.6K Feb 21 20:27 demo_B.tsv\n",
"-rw-r--r-- 1 jovyan root 1.3K Feb 21 20:27 demo_cassettesGC.pkl\n",
"-rw-r--r-- 1 jovyan root 171 Feb 21 20:27 demo_cassettesGC.tsv\n",
"-rw-r--r-- 1 jovyan root 1.1K Feb 21 20:27 demo_mergedGC.pkl\n",
"-rw-r--r-- 1 jovyan root 115 Feb 21 20:27 demo_mergedGC.tsv\n",
"-rw-r--r-- 1 jovyan root 65 Feb 21 20:14 demo_sample_set.fa\n",
"-rw-r--r-- 1 jovyan root 5.3K Feb 21 20:27 LOG_baFeb2120202027.txt\n",
"-rw-r--r-- 1 jovyan root 100K Feb 21 20:27 plots4review_from-baFeb2120202027.ipynb\n",
"-rw-r--r-- 1 jovyan root 544K Feb 21 20:27 seqs_dfs_and_plots_for_each_set.pkl\n"
]
}
],
"source": [
"ls -lh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That lists out the contents of the directory. The options added along with the `ls` command make the output more redable by showing. In particular, the `h` in `-lh` means the file sizes are human readable and the `l` in `-lh` says to list the details in long form and not just the names.\n",
"\n",
"You'll see there is much more than the compressed archive that was originally added when we made the directory just a few cells back. These are the results of the bendIt run. The following section will go through what these are and how to use them.\n",
"\n",
"----"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## About the Contents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you used your own data, your results will be different but the types will be the same. I am going describe the contents as if you ran the demo sequences; however, mostly you just need to pay attention to the file extensions, or sometimes the start of the name, to tell which file types correspond.\n",
"\n",
"After a brief overview, I am going to add some details about most of the types."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Typical contents overview:\n",
"\n",
"- Log file\n",
"- Image files for the plots\n",
"- a Jupyter notebook for reviewing all the plots en masse\n",
"- serialized (pickeled) bendIt data \n",
"- bendIt data as tabular text\n",
"- Nucleotide composition detail\n",
"- serialized processing information and results on a per sample basis\n",
"- input files\n",
"- raw gnuplots\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Log file\n",
"\n",
"The Log file will look something like `LOG_ba| \n", " | Position | \n", "Sequence | \n", "Predicted_curvature | \n", "Bendability | \n", "
|---|---|---|---|---|
| 0 | \n", "3 | \n", "a | \n", "0.0000 | \n", "0.0633 | \n", "
| 1 | \n", "4 | \n", "a | \n", "0.0000 | \n", "1.3522 | \n", "
| 2 | \n", "5 | \n", "a | \n", "0.0000 | \n", "3.9733 | \n", "
| 3 | \n", "6 | \n", "c | \n", "8.9765 | \n", "6.1887 | \n", "
| 4 | \n", "7 | \n", "g | \n", "9.2796 | \n", "6.2942 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 76 | \n", "79 | \n", "a | \n", "6.8233 | \n", "5.9429 | \n", "
| 77 | \n", "80 | \n", "g | \n", "0.0000 | \n", "6.8829 | \n", "
| 78 | \n", "81 | \n", "c | \n", "0.0000 | \n", "6.8913 | \n", "
| 79 | \n", "82 | \n", "t | \n", "0.0000 | \n", "6.1950 | \n", "
| 80 | \n", "83 | \n", "g | \n", "0.0000 | \n", "0.0000 | \n", "
81 rows × 4 columns
\n", "| \n", " | Position | \n", "Sequence | \n", "Predicted_curvature | \n", "Bendability | \n", "
|---|---|---|---|---|
| 0 | \n", "3 | \n", "a | \n", "0.0000 | \n", "0.0633 | \n", "
| 1 | \n", "4 | \n", "a | \n", "0.0000 | \n", "1.3522 | \n", "
| 2 | \n", "5 | \n", "a | \n", "0.0000 | \n", "3.9733 | \n", "
| 3 | \n", "6 | \n", "c | \n", "8.9765 | \n", "6.1887 | \n", "
| 4 | \n", "7 | \n", "g | \n", "9.2796 | \n", "6.2942 | \n", "
| \n", " | Position | \n", "Sequence | \n", "Predicted_curvature | \n", "Bendability | \n", "
|---|---|---|---|---|
| 76 | \n", "79 | \n", "a | \n", "6.8233 | \n", "5.9429 | \n", "
| 77 | \n", "80 | \n", "g | \n", "0.0000 | \n", "6.8829 | \n", "
| 78 | \n", "81 | \n", "c | \n", "0.0000 | \n", "6.8913 | \n", "
| 79 | \n", "82 | \n", "t | \n", "0.0000 | \n", "6.1950 | \n", "
| 80 | \n", "83 | \n", "g | \n", "0.0000 | \n", "0.0000 | \n", "
| \n", " | Position | \n", "Predicted_curvature | \n", "Bendability | \n", "
|---|---|---|---|
| count | \n", "81.000000 | \n", "81.000000 | \n", "81.000000 | \n", "
| mean | \n", "43.000000 | \n", "3.651504 | \n", "5.332964 | \n", "
| std | \n", "23.526581 | \n", "2.100518 | \n", "1.584404 | \n", "
| min | \n", "3.000000 | \n", "0.000000 | \n", "0.000000 | \n", "
| 25% | \n", "23.000000 | \n", "2.476700 | \n", "4.875700 | \n", "
| 50% | \n", "43.000000 | \n", "3.351800 | \n", "5.416200 | \n", "
| 75% | \n", "63.000000 | \n", "4.963900 | \n", "6.195000 | \n", "
| max | \n", "83.000000 | \n", "9.279600 | \n", "9.082300 | \n", "
| \n", " | seq | \n", "A | \n", "C | \n", "T | \n", "G | \n", "Total_nts | \n", "%G+C | \n", "rank_on_GC | \n", "
|---|---|---|---|---|---|---|---|---|
| A | \n", "ACCTGCGATGACTGACTCCCCTTTAGAGG | \n", "6 | \n", "9 | \n", "7 | \n", "7 | \n", "29 | \n", "0.551724 | \n", "2.0 | \n", "
| B | \n", "AGGAGCTAGACTTTAGGACTATAGCTCTC | \n", "8 | \n", "6 | \n", "8 | \n", "7 | \n", "29 | \n", "0.448276 | \n", "1.0 | \n", "
| \n", " | G | \n", "T | \n", "A | \n", "C | \n", "Total_nts | \n", "%G+C | \n", "rank_on_GC | \n", "
|---|---|---|---|---|---|---|---|
| A | \n", "25 | \n", "19 | \n", "22 | \n", "20 | \n", "86 | \n", "0.523256 | \n", "2.0 | \n", "
| B | \n", "25 | \n", "20 | \n", "24 | \n", "17 | \n", "86 | \n", "0.488372 | \n", "1.0 | \n", "