{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Useful_Google_Colab_snippets.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Dene33/mlcourse.ai/blob/master/jupyter_english/tutorials/Useful_Google_Colab_snippets.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "qgq9XRjSYknq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For now I mainly use Google Colab for my ML researches. I love the way of having machine with Linux and GPU in browser for free. I can start it without messing with virtual environments, restart it in few clicks if I did something wrong, upload and download files, etc. I find it really flexible.\n",
        "\n",
        "But sometimes that's not so easy, for example, to replicate someone's implementation of Neural-Networks. That's because of dependencies that are not installed and lack of terminal in Google Colab.\n",
        "\n",
        "I've gathered few useful (at least for myself) snippets of code for Google Colab that could save a lot of time if you're trying to run some Neural-Network implementations from github (2d and 3d pose estimations in my case)."
      ]
    },
    {
      "metadata": {
        "id": "3uieS9GNxhjt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Install Python 3.5 (or any other version of your choice)"
      ]
    },
    {
      "metadata": {
        "id": "Dgp8iT06yxGC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The code snippet below will install Python 3.5 without any Colab pre-installed libraries (such as Tensorflow). You can install them later with pip, like `!pip install tensorflow`\n",
        "\n",
        "To run python scripts with 3.5 version, use `!python3.5`"
      ]
    },
    {
      "metadata": {
        "id": "TEUYMtKsw-u2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!add-apt-repository ppa:deadsnakes/ppa\n",
        "!apt-get update\n",
        "!apt-get install python3.5\n",
        "!apt-get install python3.5-dev\n",
        "\n",
        "!wget https://bootstrap.pypa.io/get-pip.py && python3.5 get-pip.py\n",
        "  \n",
        "import sys\n",
        "\n",
        "sys.path[2] = '/usr/lib/python35.zip'\n",
        "sys.path[3] = '/usr/lib/python3.5'\n",
        "sys.path[4] = '/usr/lib/python3.5/lib-dynload'\n",
        "sys.path[5] = '/usr/local/lib/python3.5/dist-packages'\n",
        "sys.path[7] ='/usr/local/lib/python3.5/dist-packages/IPython/extensions'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "75bVaKUxzXg4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Install Caffe2"
      ]
    },
    {
      "metadata": {
        "id": "ZTVQnY5v4MZm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Trying a lot of different (not working) approaches of installing Caffe2, I ended up with this one. The code snippet below will install Caffe2.\n",
        "\n",
        "Because Caffe2 runs with Python 2.7, it's better to use Google Colab **Python 2 runtime type**. "
      ]
    },
    {
      "metadata": {
        "id": "Lo29NqX5zZA_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget https://anaconda.org/pytorch/pytorch-nightly/1.0.0.dev20181206/download/linux-64/pytorch-nightly-1.0.0.dev20181206-py2.7_cuda9.2.148_cudnn7.4.1_0.tar.bz2\n",
        "!tar xvjf pytorch-nightly-1.0.0.dev20181206-py2.7_cuda9.2.148_cudnn7.4.1_0.tar.bz2\n",
        "!cp -r lib/python2.7/site-packages/* /usr/local/lib/python2.7/dist-packages/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sVLAD7L238bq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# To check if Caffe2 build was successful\n",
        "!python2 -c 'from caffe2.python import core' 2>/dev/null && echo \"Success\" || echo \"Failure\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XjRu4yoD8IG_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Execute .py file as cell \n",
        "This will be useful if you have a lot of code what just clutters up your notebook. Notice that this snippet will run code exactly like cell (i.e. loading all the variables into botebook, etc), not just like a python script.\n"
      ]
    },
    {
      "metadata": {
        "id": "RjL1kqjj9KCN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "exec(open('PATH_TO_PYTHON_FILE').read())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "k5xaS3-c-LLa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Overwrite or create new files within cell"
      ]
    },
    {
      "metadata": {
        "id": "eHKKEUYk-kUB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "That's a really useful magic command to easily create new files or overwrite existing ones."
      ]
    },
    {
      "metadata": {
        "id": "vOTj__Gr-Td0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%writefile YOUR_FILE.py\n",
        "\n",
        "##Anything you'd like"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gekgVDVQMVH3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Better way of mounting Google Drive"
      ]
    },
    {
      "metadata": {
        "id": "pH5n5OomMfct",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This snippet uses `/content/gdrive` path instead of Colab's default snippet `/gdrive` so you can manage all your files visualy in `Files` menu on the left.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "o5niQbseLJLo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5qW7krBs5A08",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Download multiple files from Google Colab"
      ]
    },
    {
      "metadata": {
        "id": "lpfqHWWm5FEs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The code snippet below will download multiple files from defined path from Google Colab."
      ]
    },
    {
      "metadata": {
        "id": "Ziuhm-cg5ATv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "import os\n",
        "\n",
        "data_path = 'sample_data'\n",
        "\n",
        "for file in os.listdir(data_path):\n",
        "  files.download(data_path + '/' + file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NMjsfAAwT76H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Install Kaggle-API"
      ]
    },
    {
      "metadata": {
        "id": "T-SHisQ5USsW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, create an API key in Kaggle.\n",
        "\n",
        "To do this, go to [kaggle.com](https://www.kaggle.com) and open your user settings page.\n",
        "\n",
        "![alt text](https://i.stack.imgur.com/jxGQv.png)"
      ]
    },
    {
      "metadata": {
        "id": "DaIk5lnWUd4o",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, scroll down to the API access section and click generate to download an API key. \n",
        "![alt text](https://i.stack.imgur.com/Hzlhp.png)\n",
        "\n",
        "This will download a file called kaggle.json to your computer. You'll use this file in Colab to access Kaggle datasets and competitions.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "o-MUwA6hT-qP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Run this cell and select the kaggle.json file downloaded\n",
        "# from the Kaggle account settings page.\n",
        "from google.colab import files\n",
        "files.upload()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sipZJjdhUCr-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Let's make sure the kaggle.json file is present.\n",
        "!ls -lha kaggle.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6WQ3rqVUULNN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Next, install the Kaggle API client.\n",
        "!pip install -q kaggle"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pOJpkvKQUOIa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# The Kaggle API client expects this file to be in ~/.kaggle,\n",
        "# so move it there.\n",
        "!mkdir -p ~/.kaggle\n",
        "!cp kaggle.json ~/.kaggle/\n",
        "\n",
        "# This permissions change avoids a warning on Kaggle tool startup.\n",
        "!chmod 600 ~/.kaggle/kaggle.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ag3E5_hhURhH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# List available datasets.\n",
        "!kaggle datasets list"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JHYqnsBNVMKc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Copy the stackoverflow data set locally.\n",
        "!kaggle datasets download -d stackoverflow/stack-overflow-2018-developer-survey"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SytTUVCvVMl9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!head ~/.kaggle/datasets/stackoverflow/stack-overflow-2018-developer-survey/survey_results_public.csv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "R42VKK6BAav2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Processing videos with ffmpeg"
      ]
    },
    {
      "metadata": {
        "id": "njl5f6fTD222",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!apt install ffmpeg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9bU0Gu5GDsmI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "####Crop"
      ]
    },
    {
      "metadata": {
        "id": "sU-2oclsAhUp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#out_w is the width of the output rectangle\n",
        "#out_h is the height of the output rectangle\n",
        "#x and y specify the top left corner of the output rectangle\n",
        "!ffmpeg -i INPUT_FILE_NAME.mp4 -filter:v \"crop=out_w:out_h:x:y\" OUTPUT_FILE_NAME.mp4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sRYTTXRnEAWv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "####Trim"
      ]
    },
    {
      "metadata": {
        "id": "P3UdA9raEAeO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# -ss: Used with -i, this seeks in the input file (input.mp4) to position. \n",
        "# 00:01:00: This is the time your trimmed video will start with. \n",
        "# -to: This specifies duration from start (00:01:40) to end (00:02:12). \n",
        "!ffmpeg -ss 00:01:00 -i INPUT_FILE_NAME.mp4 -to 00:02:00 -c copy OUTPUT_FILE_NAME.mp4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "S9JBO-4TDwMW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "####Video to images"
      ]
    },
    {
      "metadata": {
        "id": "dd-wSJxXDydc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!ffmpeg -i INPUT_FILE_NAME.mpg -r DESIRED_FPS_RATE 'OUTPUT_FILE_NAME%03d.png'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2mmgB_aSHZXg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "####Resize"
      ]
    },
    {
      "metadata": {
        "id": "WvWo9TpoHbpc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Fixed width and height \n",
        "!ffmpeg -i INPUT_FILE_NAME.avi -vf scale=\"OUTPUT_WIDTH:OUTPUT_HEIGHT\" OUTPUT_FILE_NAME.avi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "duz3aL7wH7O0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Retain aspect ratio\n",
        "#resize based on the width\n",
        "!ffmpeg -i INPUT_FILE_NAME.avi -vf scale=\"NEW_WIDTH:-1\" OUTPUT_FILE_NAME.avi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yTWa_77fIQdG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Scale based on input\n",
        "!ffmpeg -i INPUT_FILE_NAME.avi -vf scale=\"INPUT_WIDTH/1:INPUT_HEIGHT/2\" OUTPUT_FILE_NAME.avi"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}