{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# R2. Git/Github tips and traps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purposes of this material, it is assumed that the you have read through [Lesson 2](../../lessons/02/index.rst). We'll start by reviewing some of these concepts and then covering some simple tricks to prevent a whole host of headaches in using GitHub within your groups. The bulk of this lesson, however, will extend on these materials. Specifically, we cover **merging**, including using [jupytext](https://github.com/mwouts/jupytext) to deal with the oddities of performing these operations on notebooks. If you are looking for a more in-depth tutorial on various Git concepts, reference this tutorial from [software carpentry](http://swcarpentry.github.io/git-novice/). The goal for today is to get you comfortable with git. \n",
    "\n",
    "We will also do a quick tutorial of GitHub with Google Colab and talk about the beauty and dangers of this feature.\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple practices can save from huge pains\n",
    "\n",
    "Git is extremely useful and powerful, but it can cause serious annoyances if not used carefully. Follow Justin's suggestion from [homework 1](../../homework/01/hw1.4.ipynb): \"it is wise to commit and push often.\" And from [Lesson 2](../../lessons/02/version_control_with_git.ipynb), pull frequently and \"always `git pull` when you start working.\" These practices will go a long way to prevent clashes where two team members have edited the same file at the same time without knowing. **Even if you feel that a particular piece of work is not yet finished, commit and push it any time you step away from it.** It is much better for a teammate to look at work you might feel insecure about due to its incompleteness than to have a horrible merge issue later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the sandbox exceedingly liberally\n",
    "\n",
    "The simplest and easiest way to avoid clashing with group members work is to use the sandbox provided in your GitHub repository extensively. When doing this, each member can work on a differently named file. For example, I might work on `hw3.2_ar.ipynb`, where the `ar` signifies my initials, while my teammate Liana works on `hw3.2_lm.ipynb`. Each team member can have their own version of various pieces of the homework for scratch work. The key reason to do this is that **you might not know when a team member is working on the same file**. If two people are working on the same file at the same time, a **merge conflict** is certain in the near future. One will push to the repository first, leaving changes that the other does not have in the file they are working with. This means that if the second forces a push, their changes will be kept at the expense of the other's. Without additional tools, merging notebooks with disparate changes is very non-trivial. Instead, to avoid all this, both members can work on their own files in the sandbox, discuss various parts, and combine and edit a final submission when the time comes.\n",
    "\n",
    "Note that if you open two Jupyter notebooks next to each other in JupyterLab it is quite easy to create a combined version of a document. This is much better than overwriting each other repeatedly. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use `checkout` to avoid destroying others work\n",
    "\n",
    "When you use separate files for each team member in the sandbox, you will often want to look at and build in the work of your teammates. This means that you will often pull material from the repo and open the notebooks of others. If you run this notebook and poke around in it, you may make a \"change\" in the document that you do not intend to. If you commit and push this change, you might cause a merge conflict with changes the owner of this file has made since you last pulled. If you insist on pushing, you might destroy another's work. If you check `git status` and see a change in such a file that you don't want to push, use `git checkout FILE_NAME_HERE` to discard your non-change-changes and prevent later merge conflicts. Your teammates will thank you!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & basics\n",
    "\n",
    "If you haven't done so already, make sure to configure your Git parameters. Namely, you must set your contact information and name. Ignore if you already did this when setting up for the class.\n",
    "\n",
    "```bash\n",
    "git config --global user.name \"YOUR NAME\"\n",
    "git config --global user.email \"YOUR EMAIL ADDRESS\"\n",
    "```\n",
    "\n",
    "Naturally, you fill in the name and email with your details.\n",
    "\n",
    "For the purposes of this exercise, we will be playing around with a repository set up specifically for practice. Justin recommends to store cloned and forked GitHub repos in a folder called `git` in your home directory. I will assume this convention for this tutorial. For today, please fork the repository located at https://github.com/AnkitaRoychoudhury/gitbrachmerge. This can be done in GitHub using the browser, using the 'Fork' button on the upper right side of the page when you navigate to the repository. Select that you want to fork it to your personal GitHub organization, *not* the BE/Bi 103 one, to avoid clutter there. Once you do this fork, clone the repository by copying the URL and performing\n",
    "\n",
    "```bash\n",
    "cd ~/git\n",
    "git clone URL_YOU_COPIED\n",
    "```\n",
    "\n",
    "You should additionally already be familiar with the following commands from [Lesson 2](../../lessons/02/version_control_with_git.ipynb):\n",
    "\n",
    "| Command                      | Description                                                    |\n",
    "|------------------------------|----------------------------------------------------------------|\n",
    "| git status                 | Check the difference between your local files and GitHub files |\n",
    "| git add                      | Add all files you want to push to GitHub repo                  |\n",
    "| git commit -m \"your_message\" | Use this to commit with a descriptive message                              |\n",
    "| git push origin main         | Push the commit onto remote repo (on GitHub)                                 |\n",
    "| git pull                     | Retrieve files from remote repo (on GitHub)   \n",
    "\n",
    "For additional help with these commands and their uses (as well as a bunch of other useful commands) reference [this cheat sheet](https://education.github.com/git-cheat-sheet-education.pdf) from GitHub. This very handy reference shows all the commands you are likely to need and then some. Common descriptions were copied over above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Syncing your forked repository to the upstream repository\n",
    "\n",
    "For this course, you will do all your work with your teammates in a single repo. You will not need to deal with forking repos, but for today, this is the easiest way to demonstrate merge clashes and how to deal with them. You want to be able to sync your repository with my `gitbrachmerge` repository so you can retrieve any updates in it. The original repository is typically called the **upstream repository**, since presumably you are changing it, so you are downstream. You want the upstream repository to be a **remote** repository, which is just what we call a repository we track and fetch and merge from. To see which repositories are remote, do\n",
    "\n",
    "    git remote -v\n",
    "\n",
    "The `-v` just means \"verbose,\" so it will also tell you the URLs. Entering that now will show a single repository, `origin`, which you can fetch from and push to. In your case, `origin` is your fork of the gitbranchmerge repository.\n",
    "\n",
    "We now want to add the upstream repository. To do this, copy the URL of my gitbrachmerge repository and do:\n",
    "\n",
    "    git remote add upstream https://github.com/AnkitaRoychoudhury/gitbrachmerge.git\n",
    "\n",
    "Now try doing `git remote -v`, and you will see that you are now also tracking the upstream repository.\n",
    "\n",
    "Now, when you want to pull from the upstream repository, you do\n",
    "\n",
    "    git pull upstream master\n",
    "    \n",
    "This will pull in all the changes from the upstream repository. If you want to pull in changes to your own forked repository, it's still just \n",
    "\n",
    "    git pull\n",
    "    \n",
    "which is shorthand for\n",
    "\n",
    "    git pull origin master"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merging\n",
    "\n",
    "### If possible, avoid merging\n",
    "\n",
    "The best way to merge is to avoid merging in the first place. This is doubly true when working with Jupyter notebooks. The simple practices above are meant to help avoid the need to merge. Here are some more excellent tips on avoiding the need to merge from a tutorial from [software carpentry](http://swcarpentry.github.io/git-novice/09-conflict/index.html). These include some more advanced features of git than we've covered today, like branches, but they are nevertheless good advice, especially the project management strategies:\n",
    "\n",
    ">Git’s ability to resolve conflicts is very useful, but conflict resolution costs time and effort, and can introduce errors if conflicts are not resolved correctly. If you find yourself resolving a lot of conflicts in a project, consider these technical approaches to reducing them:\n",
    ">\n",
    ">- Pull from upstream more frequently, especially before starting new work\n",
    ">- Make smaller more atomic commits\n",
    ">- Where logically appropriate, break large files into smaller ones so that it is less likely that two authors will alter the same file simultaneously\n",
    ">\n",
    ">Conflicts can also be minimized with project management strategies:\n",
    ">\n",
    ">- Clarify who is responsible for what areas with your collaborators\n",
    ">- Discuss what order tasks should be carried out in with your collaborators so that tasks expected to change the same lines won’t be worked on simultaneously\n",
    ">- If the conflicts are stylistic churn (e.g. tabs vs. spaces), establish a project convention that is governing and use code style tools (e.g. htmltidy, perltidy, rubocop, etc.) to enforce, if necessary\n",
    "\n",
    "Daisie Huang and Ivan Gonzalez (eds): \"Software Carpentry: Version\n",
    "Control with Git.\"  Version 2016.06, June 2016,\n",
    "https://github.com/swcarpentry/git-novice, 10.5281/zenodo.57467."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Merging with simple text files\n",
    "\n",
    "#### Exercise 1\n",
    "\n",
    "We will first play around with the simplest case of merges by dealing with text files. In the repository you forked, there is a text file called `text_merge/edit_me.txt`. We will play around with editing this file concurrently with two people and deal with the resulting merge issues. Open this file in your favorite text editor and add a few lines. This could be accomplished, for example, with\n",
    "\n",
    "```bash\n",
    "cd ~/git/gitbrachmerge/text_merging/\n",
    "vim edit_me.txt \n",
    "\n",
    "OR double click 'edit_me.txt' to open as a new Jupyter window\n",
    "```\n",
    "\n",
    "When you are done editing the file save the file and exit. (In `vim`, tap `i` for insert, then navigate with arrow keys and make your edits. When done, tap `ESC`, then type `:wq`, for write & quit, then `Enter`. If you make a mistake and want to bail without saving, use `ESC`, then `:q`, `Enter`).\n",
    "\n",
    "In the mean time, I have also edited this document, committed, and pushed it to the upstream master repo. If you pull from the upstream master, you will get a prompt to merge. Try this now:\n",
    "\n",
    "```\n",
    "git pull upstream master\n",
    "```\n",
    "\n",
    "If we didn't edit the same lines, Git should be able to automerge the `edit_me.txt` file. But if we did make edits to the same lines, Git doesn't know which version to keep, yours or mine! **To handle this merge, you need to manually edit the document and deal with the discrepancies.** Otherwise, you'll end up publishing material with ugly merge issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`<<<<<<<HEAD` indicates the change (in the above case an insertion) from the master, and the material below the `=======` shows changes from your local end. \n",
    "\n",
    "For a simple file like this, doing this manual merge is not impossible. You can use your favorite text editor to manually edit the document and decide which lines to keep or remove. When you have done this, you can add, commit, and push the changes to your forked version of the repo.\n",
    "\n",
    "```\n",
    "git add edit_me.txt\n",
    "git commit -m 'merge edit_me.txt with upstream edits'\n",
    "git push origin master\n",
    "```\n",
    "\n",
    "You have now dealt with the discrepancies like a champ!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Merging Jupyter Notebooks\n",
    "\n",
    "#### Exercise 2\n",
    "We will now run through the same experiment as before, but this time each will edit the simple Jupyter notebook stored in `notebook_merging/simple_notebook.ipynb`. The resulting merge will not be so simple to deal with since the internal formatting of Jupyter notebooks is not very human readable. Open the document, change a plot, maybe add a cell or two the the file, and save. Try to reconcile this with upstream master after I push a change myself.\n",
    "\n",
    "Note also that if you have a merge conflict with notebooks, the inserts that git adds to flag the conflict will break the formatting of the notebook, so the notebook will become unreadable to Jupyterlab. If this happens, you can use `git merge --abort` to at least return your notebook to a readable format. Since there is no easy way to merge notebooks, your least bad option may actually be to naively memorize a standard git workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Jupytext\n",
    "\n",
    "Jupytext is a relatively new package that promises to combine the best of both worlds of notebooks (interactivity, inline plots, and beyond) and plain python scripts (easy version control and merging). Its integration with Jupyterlab is a work in progess, so **_caveat emptor_**. On my system, the newer versions of `jupytext` prevent me from saving notebooks at all (though I haven't ruled out that the problem is on my end and not intrinsic to the package). We will instead explore the command line options available.\n",
    "\n",
    "The valuable addition that such a package provides is a conversion of notebooks into scripts of various types (`.py`, `.md`) that can then be easily version controlled with meaningful and human-readable merges. To begin, you will need to install jupytext. This is a simple pip install. **Note the explicit version number** since we do not want the newest version available.\n",
    "\n",
    "```\n",
    "pip install jupytext==0.8.6\n",
    "```\n",
    "\n",
    "This is all you need to have to start using the tools. You can explore the ways of automatically going back and forth between a text file and notebook by editing the metadata preferences for your notebooks, but I have found this to be more trouble than it's worth. To make a python script file that represents your Jupyter notebook, you can simply use\n",
    "\n",
    "```\n",
    "jupytext --to python notebook_name_here.ipynb\n",
    "```\n",
    "\n",
    "The result of this script is a `.py` file with your code and markdown cells formatted together. Better yet, since `.py` files are really just text, merge tools of git will work for them! You can store your notebooks on GitHub as this `.py` file, and then easily convert it to a notebook after pulling it. This conversion is achieved with\n",
    "\n",
    "```\n",
    "jupytext --to notebook file_name_here.py\n",
    "```\n",
    "\n",
    "(The conversion from `.py` to `.ipynb` may give you a deprecation warning; it works fine, but has security vulnerabilities, so maybe don't run this on random scripts from the internet.)\n",
    "\n",
    "Jupytext has been heavily tested, ensuring that the result you get from a conversion and back gives the same notebook as you started with. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 3\n",
    "\n",
    "Try out the same experiment we did with the text file and the Jupyter notebook on the jupytext document. Convert the file to a notebook with the above command, add a cell or two, save, convert back to a `.py` file, and try to merge with changes I made to the same file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A new workflow in your repos (with jupytext)\n",
    "\n",
    "Based on the power of merging with a text based file, a better workflow for your repository would be to store your sandbox work in `.py` files generated with jupytext instead of pushing as notebooks. With this, you need not bother version controlling your `.ipynb` files at all. In addition to aiding with merges, this also helps limit the size of your repository: the text file is much smaller than a notebook, especially one that has glyph-heavy plots in it. The suggested work flow is as follows:\n",
    "\n",
    "- Pull all work \n",
    "- Edit your work in JupyterLab\n",
    "- Save your notebook and keep it locally\n",
    "- Convert your notebook to a `.py` file\n",
    "- Add, commit, and push this `.py` file to GitHub\n",
    "- Pull, convert to a notebook, and repeat\n",
    "\n",
    "### A workflow without jupytext\n",
    "\n",
    "- Pull all work \n",
    "- Edit your work in JupyterLab \n",
    "- Save your notebook and keep it locally\n",
    "- Add, commit, and push the notebook to GitHub\n",
    "- Pull and repeat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## GitHub with Google Colab\n",
    "\n",
    "Refer to this part of [Lesson 2](../../lessons/02/git_with_colab.ipynb) to refresh on how to use GitHub with Google Colab! \n",
    "\n",
    "This can end up being very helpful to accomplish pairs coding remotely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example of GitHub + Colab\n",
    "\n",
    "First, everybody should have the Google Chrome Extension already setup, as mentioned in [Lesson 2](../../lessons/02/git_with_colab.ipynb). \n",
    "\n",
    "Navigate to your GitHub gitbrachmerge repo that you forked earlier in this recitation. Go to `notebook_merging`, click on `simple_notebook.ipynb`, and click on the Google Colab Chrome Extension.\n",
    "\n",
    "Now your notebook has automatically opened on Google Colab! This is effectively performing the 'git pull' onto the Google platform.\n",
    "\n",
    "Now **change a few lines of code**. Click `File`, then `Save a copy in GitHub`. You will be redirected to the updated Git repository and should see the changes you've made! As we can see, there's no need to perform git add, commit, push commands when using Colab.\n",
    "\n",
    "However, be **extra** careful if two people are working on the same Colab notebook on their own local machines. Google Colab does **not** work like Google Drive—changes are not updated live. If you push a file with your edits after your teammate pushes the same file with different edits, you will overwrite what they have done. This is why the sandbox is your friend!\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A final note on commit messages\n",
    "\n",
    "Git really comes into its own for managing larger projects than your homework assignments in this course. One key ingredient of good version control in this context is writing informative commit messages. Justin emphasized in [Lesson 0](../../lessons/00/intro_to_jupyterlab.ipynb) that the emphasis of comments in code should be _why_, not _how_. The important information is _why_ was a design choice made, what was it's context, and not the implementation details (the _how_), which should be fairly self-evident from good self-documenting code. Future You is the most important reader.\n",
    "\n",
    "All of these same thoughts apply to commit messages. Their goal is to give context and explain why, not how, changes were made. I highly recommend [this](https://chris.beams.io/posts/git-commit/) post on how to write good commit messages. This may sound like a silly thing to harp on, so read that blog post for more insight on why your `git log --oneline` should never look like [this](https://xkcd.com/1296/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*This recitation was originally written by Muir Morrison and was added to by Julian Wagner and Ankita Roychoudhury.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}