{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GCP에서 Apache Spark Cluster 설치 \n",
    "\n",
    "## Reference\n",
    "\n",
    "https://zzsza.github.io/data/2018/06/09/apache-spark-cluster/\n",
    "\n",
    "## GCP 가입\n",
    "\n",
    "* [GCP 가입](https://zzsza.github.io/gcp/2018/01/01/gcp-intro/)\n",
    "\n",
    "## VM 인스턴스 생성\n",
    "\n",
    "* Compute Engine 클릭\n",
    "* 만들기 버튼 클릭\n",
    "\n",
    "### 인스턴스 만들기\n",
    "\n",
    "* CPU 1개, 메모리 3.75G, Ubuntu 18.04 LTS 선택\n",
    "\n",
    "## 스파크 환경 설치\n",
    "\n",
    "* SSH 클릭\n",
    "\n",
    "* keygen 등록\n",
    "\n",
    "```\n",
    "~$ ssh-keygen\n",
    "~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys \n",
    "~$ chmod og-wx ~/.ssh/authorized_keys\n",
    "```\n",
    "\n",
    "* Spark 다운로드\n",
    "\n",
    "```\n",
    "~$ wget http://mirror.navercorp.com/apache/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz\n",
    "```\n",
    "\n",
    "* 자바설치\n",
    "참고주소 : https://tecadmin.net/install-oracle-java-8-ubuntu-via-ppa/\n",
    "\n",
    "```\n",
    "~$ sudo add-apt-repository ppa:webupd8team/java\n",
    "~$ sudo apt-get update\n",
    "~$ sudo apt-get install oracle-java8-installer\n",
    "```\n",
    "\n",
    "* Spark 설치 및 실행\n",
    "    + Local : 클러스터 구성없이 Spark 머신 하나로 에뮬레이션 되도록 실행.\n",
    "\n",
    "```\n",
    "~$ tar xvfz spark-2.3.0-bin-hadoop2.7.tgz\n",
    "~$ cd spark-2.3.0-bin-hadoop2.7/\n",
    "~/spark-2.3.0-bin-hadoop2.7$ ./bin/spark-shell\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 마스터 주소 확인\n",
    "\n",
    "```\n",
    "~$ cd spark-2.3.0-bin-hadoop2.7/sbin\n",
    "~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-master.sh\n",
    "starting org.apache.spark.deploy.master.Master, logging to /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out\n",
    "\n",
    "~/spark-2.3.0-bin-hadoop2.7/sbin$ grep \"spark://\" /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out\n",
    "\n",
    "2018-06-18 14:33:38 INFO  Master:54 - Starting Spark master at spark://instance-1.c.arctic-compass-206702.internal:7077\n",
    "```\n",
    "    - 예제의 마스터주소 : spark://instance-1.c.arctic-compass-206702.internal:7077\n",
    "    \n",
    "* 스파크 실행\n",
    "\n",
    "```\n",
    "~/spark-2.3.0-bin-hadoop2.7/sbin$ cd ../bin\n",
    "~/spark-2.3.0-bin-hadoop2.7/bin$ ./spark-shell --master spark://instance-1.c.arctic-compass-206702.internal:7077\n",
    "```\n",
    "\n",
    "* RDD 생성 및 count\n",
    "\n",
    "```\n",
    "scala> sc.makeRDD(List(1,2,3)).count\n",
    "```\n",
    "    - 워커가 없어서 실행 안됨."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 방화벽 규칙 만들기\n",
    "\n",
    "* 방화벽 규칙 클릭\n",
    "\n",
    "* 방화벽 규칙 만들기\n",
    "    - 이름: spark-1\n",
    "    - 대상 : 네트워크의 모든 인스턴스\n",
    "    - 소스 IP 범위 : 0.0.0.0/0\n",
    "    - 프로토콜 및 포트 : 모두 허용\n",
    "* 만들기 클릭"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 스냅샷 만들기\n",
    "\n",
    "* 기존 instance-1을 종료하고 다음 작업을 진행합니다.\n",
    "\n",
    "* Compute Engine -> 스냅샷\n",
    "* 스냅샷 만들기 클릭\n",
    "    - 소스 디스크 : instance-1 선택"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Slave VM 생성\n",
    "\n",
    "* 만들고 싶은 slave 개수만큼 다음을 반복합니다.\n",
    "\n",
    "## VM 인스턴스 생성\n",
    "\n",
    "* Compute Engine 클릭\n",
    "* 만들기 버튼 클릭\n",
    "\n",
    "### 인스턴스 만들기\n",
    "\n",
    "* 부팅 디스크 -> 변경 클릭 -> snapshot-1 선택\n",
    "* CPU 1개, 메모리 3.75G"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 클러스터에서 Worker 띄우기\n",
    "\n",
    "## Slave 컴퓨터에 접속해서 각각 띄우기\n",
    "\n",
    "```\n",
    "~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-slave.sh\n",
    "Usage: ./sbin/start-slave.sh [options] <master>\n",
    "```\n",
    "\n",
    "## Master 컴퓨터에서 Worker들을 띄우기\n",
    "\n",
    "* 기존 instance-1을 시작 버튼을 눌러 켭니다.\n",
    "\n",
    "* 마스터 띄우기\n",
    "\n",
    "```\n",
    "~$ cd spark-2.3.0-bin-hadoop2.7/\n",
    "~/spark-2.3.0-bin-hadoop2.7$ cd sbin\n",
    "~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-master.sh\n",
    "starting org.apache.spark.deploy.master.Master, logging to /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out\n",
    "```\n",
    "\n",
    "* 마스터 주소 다시 확인\n",
    "\n",
    "```\n",
    "~/spark-2.3.0-bin-hadoop2.7/bin$ grep \"spark://\" /home/sdrlurker2/spark-2.3.0-bin-hadoop2.7/logs/spark-sdrlurker2-org.apache.spark.deploy.master.Master-1-instance-1.out\n",
    "2018-06-19 05:15:04 INFO  Master:54 - Starting Spark master at spark://instance-1.c.arctic-compass-206702.internal:7077\n",
    "~/spark-2.3.0-bin-hadoop2.7/bin$ ./spark-shell --master=spark://instance-1.c.arctic-compass-206702.internal:7077\n",
    "```\n",
    "\n",
    "### slaves 등록\n",
    "\n",
    "```\n",
    "~$ cd spark-2.3.0-bin-hadoop2.7/conf\n",
    "~/spark-2.3.0-bin-hadoop2.7/conf$ cp -p slaves.template slaves\n",
    "~/spark-2.3.0-bin-hadoop2.7/conf$ vi slaves\n",
    "# slaves 내부 IP 또는 외부 IP 추가.\n",
    "```\n",
    "\n",
    "### slaves 실행\n",
    "\n",
    "```\n",
    "~/spark-2.3.0-bin-hadoop2.7/sbin$ ./start-slaves.sh\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spark Web UI 확인."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html><html>\n",
       "      <head>\n",
       "        <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"/><link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/vis.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/webui.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/timeline-view.css\" type=\"text/css\"/><script src=\"/static/sorttable.js\"></script><script src=\"/static/jquery-1.11.1.min.js\"></script><script src=\"/static/vis.min.js\"></script><script src=\"/static/bootstrap-tooltip.js\"></script><script src=\"/static/initialize-tooltips.js\"></script><script src=\"/static/table.js\"></script><script src=\"/static/additional-metrics.js\"></script><script src=\"/static/timeline-view.js\"></script><script src=\"/static/log-view.js\"></script><script src=\"/static/webui.js\"></script><script>setUIRoot('')</script>\n",
       "        \n",
       "        <title>Spark Master at spark://instance-1.c.arctic-compass-206702.internal:7077</title>\n",
       "      </head>\n",
       "      <body>\n",
       "        <div class=\"container-fluid\">\n",
       "          <div class=\"row-fluid\">\n",
       "            <div class=\"span12\">\n",
       "              <h3 style=\"vertical-align: middle; display: inline-block;\">\n",
       "                <a style=\"text-decoration: none\" href=\"/\">\n",
       "                  <img src=\"/static/spark-logo-77x50px-hd.png\"/>\n",
       "                  <span class=\"version\" style=\"margin-right: 15px;\">2.3.0</span>\n",
       "                </a>\n",
       "                Spark Master at spark://instance-1.c.arctic-compass-206702.internal:7077\n",
       "              </h3>\n",
       "            </div>\n",
       "          </div>\n",
       "          <div class=\"row-fluid\">\n",
       "          <div class=\"span12\">\n",
       "            <ul class=\"unstyled\">\n",
       "              <li><strong>URL:</strong> spark://instance-1.c.arctic-compass-206702.internal:7077</li>\n",
       "              <li>\n",
       "                    <strong>REST URL:</strong> spark://instance-1.c.arctic-compass-206702.internal:6066\n",
       "                    <span class=\"rest-uri\"> (cluster mode)</span>\n",
       "                  </li>\n",
       "              <li><strong>Alive Workers:</strong> 5</li>\n",
       "              <li><strong>Cores in use:</strong> 5 Total,\n",
       "                5 Used</li>\n",
       "              <li><strong>Memory in use:</strong>\n",
       "                13.0 GB Total,\n",
       "                5.0 GB Used</li>\n",
       "              <li><strong>Applications:</strong>\n",
       "                1 <a href=\"#running-app\">Running</a>,\n",
       "                0 <a href=\"#completed-app\">Completed</a> </li>\n",
       "              <li><strong>Drivers:</strong>\n",
       "                0 Running,\n",
       "                0 Completed </li>\n",
       "              <li><strong>Status:</strong> ALIVE</li>\n",
       "            </ul>\n",
       "          </div>\n",
       "        </div><div class=\"row-fluid\">\n",
       "          <div class=\"span12\">\n",
       "            <h4> Workers (5) </h4>\n",
       "            <table class=\"table table-bordered table-condensed table-striped sortable\">\n",
       "      <thead><th width=\"\" class=\"\">Worker Id</th><th width=\"\" class=\"\">Address</th><th width=\"\" class=\"\">State</th><th width=\"\" class=\"\">Cores</th><th width=\"\" class=\"\">Memory</th></thead>\n",
       "      <tbody>\n",
       "        <tr>\n",
       "      <td>\n",
       "        <a href=\"http://10.146.0.2:8082\">\n",
       "              worker-20180619051632-10.146.0.2-37773\n",
       "            </a>\n",
       "      </td>\n",
       "      <td>10.146.0.2:37773</td>\n",
       "      <td>ALIVE</td>\n",
       "      <td>1 (1 Used)</td>\n",
       "      <td sorttable_customkey=\"2669.1024\">\n",
       "        2.6 GB\n",
       "        (1024.0 MB Used)\n",
       "      </td>\n",
       "    </tr><tr>\n",
       "      <td>\n",
       "        <a href=\"http://10.146.0.2:8081\">\n",
       "              worker-20180619051632-10.146.0.2-42601\n",
       "            </a>\n",
       "      </td>\n",
       "      <td>10.146.0.2:42601</td>\n",
       "      <td>ALIVE</td>\n",
       "      <td>1 (1 Used)</td>\n",
       "      <td sorttable_customkey=\"2669.1024\">\n",
       "        2.6 GB\n",
       "        (1024.0 MB Used)\n",
       "      </td>\n",
       "    </tr><tr>\n",
       "      <td>\n",
       "        <a href=\"http://10.146.0.3:8081\">\n",
       "              worker-20180619051637-10.146.0.3-44837\n",
       "            </a>\n",
       "      </td>\n",
       "      <td>10.146.0.3:44837</td>\n",
       "      <td>ALIVE</td>\n",
       "      <td>1 (1 Used)</td>\n",
       "      <td sorttable_customkey=\"2669.1024\">\n",
       "        2.6 GB\n",
       "        (1024.0 MB Used)\n",
       "      </td>\n",
       "    </tr><tr>\n",
       "      <td>\n",
       "        <a href=\"http://10.146.0.5:8081\">\n",
       "              worker-20180619051637-10.146.0.5-38961\n",
       "            </a>\n",
       "      </td>\n",
       "      <td>10.146.0.5:38961</td>\n",
       "      <td>ALIVE</td>\n",
       "      <td>1 (1 Used)</td>\n",
       "      <td sorttable_customkey=\"2669.1024\">\n",
       "        2.6 GB\n",
       "        (1024.0 MB Used)\n",
       "      </td>\n",
       "    </tr><tr>\n",
       "      <td>\n",
       "        <a href=\"http://10.146.0.4:8081\">\n",
       "              worker-20180619051639-10.146.0.4-45191\n",
       "            </a>\n",
       "      </td>\n",
       "      <td>10.146.0.4:45191</td>\n",
       "      <td>ALIVE</td>\n",
       "      <td>1 (1 Used)</td>\n",
       "      <td sorttable_customkey=\"2669.1024\">\n",
       "        2.6 GB\n",
       "        (1024.0 MB Used)\n",
       "      </td>\n",
       "    </tr>\n",
       "      </tbody>\n",
       "    </table>\n",
       "          </div>\n",
       "        </div><div class=\"row-fluid\">\n",
       "          <div class=\"span12\">\n",
       "            <h4 id=\"running-app\"> Running Applications (1) </h4>\n",
       "            <table class=\"table table-bordered table-condensed table-striped sortable\">\n",
       "      <thead><th width=\"\" class=\"\">Application ID</th><th width=\"\" class=\"\">Name</th><th width=\"\" class=\"\">Cores</th><th width=\"\" class=\"\">Memory per Executor</th><th width=\"\" class=\"\">Submitted Time</th><th width=\"\" class=\"\">User</th><th width=\"\" class=\"\">State</th><th width=\"\" class=\"\">Duration</th></thead>\n",
       "      <tbody>\n",
       "        <tr>\n",
       "      <td>\n",
       "        <a href=\"app?appId=app-20180619052812-0000\">app-20180619052812-0000</a>\n",
       "        <form action=\"app/kill/\" method=\"POST\" style=\"display:inline\">\n",
       "        <input type=\"hidden\" name=\"id\" value=\"app-20180619052812-0000\"/>\n",
       "        <input type=\"hidden\" name=\"terminate\" value=\"true\"/>\n",
       "        <a href=\"#\" onclick=\"if (window.confirm('Are you sure you want to kill application app-20180619052812-0000 ?')) { this.parentNode.submit(); return true; } else { return false; }\" class=\"kill-link\">(kill)</a>\n",
       "      </form>\n",
       "      </td>\n",
       "      <td>\n",
       "        <a href=\"http://instance-1.c.arctic-compass-206702.internal:4040\">Spark shell</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        5\n",
       "      </td>\n",
       "      <td sorttable_customkey=\"1024\">\n",
       "        1024.0 MB\n",
       "      </td>\n",
       "      <td>2018/06/19 05:28:12</td>\n",
       "      <td>sdrlurker2</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td>4.4 min</td>\n",
       "    </tr>\n",
       "      </tbody>\n",
       "    </table>\n",
       "          </div>\n",
       "        </div><div>\n",
       "          \n",
       "        </div><div class=\"row-fluid\">\n",
       "          <div class=\"span12\">\n",
       "            <h4 id=\"completed-app\"> Completed Applications (0) </h4>\n",
       "            <table class=\"table table-bordered table-condensed table-striped sortable\">\n",
       "      <thead><th width=\"\" class=\"\">Application ID</th><th width=\"\" class=\"\">Name</th><th width=\"\" class=\"\">Cores</th><th width=\"\" class=\"\">Memory per Executor</th><th width=\"\" class=\"\">Submitted Time</th><th width=\"\" class=\"\">User</th><th width=\"\" class=\"\">State</th><th width=\"\" class=\"\">Duration</th></thead>\n",
       "      <tbody>\n",
       "        \n",
       "      </tbody>\n",
       "    </table>\n",
       "          </div>\n",
       "        </div><div>\n",
       "          \n",
       "        </div>\n",
       "        </div>\n",
       "      </body>\n",
       "    </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "r = requests.get(\"http://localhost:8080\")\n",
    "\n",
    "from IPython.display import HTML\n",
    "HTML(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spark Web Worker UI 확인.\n",
    "\n",
    "* 8081, 8082, ... 식으로 증가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html><html>\n",
       "      <head>\n",
       "        <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"/><link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/vis.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/webui.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/timeline-view.css\" type=\"text/css\"/><script src=\"/static/sorttable.js\"></script><script src=\"/static/jquery-1.11.1.min.js\"></script><script src=\"/static/vis.min.js\"></script><script src=\"/static/bootstrap-tooltip.js\"></script><script src=\"/static/initialize-tooltips.js\"></script><script src=\"/static/table.js\"></script><script src=\"/static/additional-metrics.js\"></script><script src=\"/static/timeline-view.js\"></script><script src=\"/static/log-view.js\"></script><script src=\"/static/webui.js\"></script><script>setUIRoot('')</script>\n",
       "        \n",
       "        <title>Spark Worker at 10.146.0.2:42601</title>\n",
       "      </head>\n",
       "      <body>\n",
       "        <div class=\"container-fluid\">\n",
       "          <div class=\"row-fluid\">\n",
       "            <div class=\"span12\">\n",
       "              <h3 style=\"vertical-align: middle; display: inline-block;\">\n",
       "                <a style=\"text-decoration: none\" href=\"/\">\n",
       "                  <img src=\"/static/spark-logo-77x50px-hd.png\"/>\n",
       "                  <span class=\"version\" style=\"margin-right: 15px;\">2.3.0</span>\n",
       "                </a>\n",
       "                Spark Worker at 10.146.0.2:42601\n",
       "              </h3>\n",
       "            </div>\n",
       "          </div>\n",
       "          <div class=\"row-fluid\"> <!-- Worker Details -->\n",
       "        <div class=\"span12\">\n",
       "          <ul class=\"unstyled\">\n",
       "            <li><strong>ID:</strong> worker-20180619051632-10.146.0.2-42601</li>\n",
       "            <li><strong>\n",
       "              Master URL:</strong> spark://instance-1.c.arctic-compass-206702.internal:7077\n",
       "            </li>\n",
       "            <li><strong>Cores:</strong> 1 (1 Used)</li>\n",
       "            <li><strong>Memory:</strong> 2.6 GB\n",
       "              (1024.0 MB Used)</li>\n",
       "          </ul>\n",
       "          <p><a href=\"http://instance-1.c.arctic-compass-206702.internal:8080\">Back to Master</a></p>\n",
       "        </div>\n",
       "      </div><div class=\"row-fluid\"> <!-- Executors and Drivers -->\n",
       "        <div class=\"span12\">\n",
       "          <h4> Running Executors (1) </h4>\n",
       "          <table class=\"table table-bordered table-condensed table-striped sortable\">\n",
       "      <thead><th width=\"\" class=\"\">ExecutorID</th><th width=\"\" class=\"\">Cores</th><th width=\"\" class=\"\">State</th><th width=\"\" class=\"\">Memory</th><th width=\"\" class=\"\">Job Details</th><th width=\"\" class=\"\">Logs</th></thead>\n",
       "      <tbody>\n",
       "        <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td sorttable_customkey=\"1024\">\n",
       "        1024.0 MB\n",
       "      </td>\n",
       "      <td>\n",
       "        <ul class=\"unstyled\">\n",
       "          <li><strong>ID:</strong> app-20180619052812-0000</li>\n",
       "          <li><strong>Name:</strong>\n",
       "          <a href=\"http://instance-1.c.arctic-compass-206702.internal:4040\"> Spark shell</a>\n",
       "          </li>\n",
       "          <li><strong>User:</strong> sdrlurker2</li>\n",
       "        </ul>\n",
       "      </td>\n",
       "      <td>\n",
       "        <a href=\"http://instance-1.c.arctic-compass-206702.internal:8081/logPage?appId=app-20180619052812-0000&amp;executorId=1&amp;logType=stdout\">stdout</a>\n",
       "        <a href=\"http://instance-1.c.arctic-compass-206702.internal:8081/logPage?appId=app-20180619052812-0000&amp;executorId=1&amp;logType=stderr\">stderr</a>\n",
       "      </td>\n",
       "    </tr>\n",
       "      </tbody>\n",
       "    </table>\n",
       "          \n",
       "          \n",
       "          \n",
       "        </div>\n",
       "      </div>\n",
       "        </div>\n",
       "      </body>\n",
       "    </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(\"http://localhost:8081\")\n",
    "HTML(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spark Web Console\n",
    "\n",
    "* 클러스터 모드로 Spark shell 실행\n",
    "```\n",
    "~/spark-2.3.0-bin-hadoop2.7/bin$ ./spark-shell --master=spark://instance-1.c.arctic-compass-206702.internal:7077\n",
    "```\n",
    "\n",
    "* 다음처럼 Spark Web Console 4040 포트로 확인 가능."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html><html>\n",
       "      <head>\n",
       "        <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"/><link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/vis.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/webui.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/timeline-view.css\" type=\"text/css\"/><script src=\"/static/sorttable.js\"></script><script src=\"/static/jquery-1.11.1.min.js\"></script><script src=\"/static/vis.min.js\"></script><script src=\"/static/bootstrap-tooltip.js\"></script><script src=\"/static/initialize-tooltips.js\"></script><script src=\"/static/table.js\"></script><script src=\"/static/additional-metrics.js\"></script><script src=\"/static/timeline-view.js\"></script><script src=\"/static/log-view.js\"></script><script src=\"/static/webui.js\"></script><script>setUIRoot('')</script>\n",
       "        \n",
       "        \n",
       "        <title>Spark shell - Spark Jobs</title>\n",
       "      </head>\n",
       "      <body>\n",
       "        <div class=\"navbar navbar-static-top\">\n",
       "          <div class=\"navbar-inner\">\n",
       "            <div class=\"brand\">\n",
       "              <a href=\"/\" class=\"brand\">\n",
       "                <img src=\"/static/spark-logo-77x50px-hd.png\"/>\n",
       "                <span class=\"version\">2.3.0</span>\n",
       "              </a>\n",
       "            </div>\n",
       "            <p class=\"navbar-text pull-right\">\n",
       "              <strong title=\"Spark shell\">Spark shell</strong> application UI\n",
       "            </p>\n",
       "            <ul class=\"nav\"><li class=\"active\">\n",
       "        <a href=\"/jobs/\">Jobs</a>\n",
       "      </li><li class=\"\">\n",
       "        <a href=\"/stages/\">Stages</a>\n",
       "      </li><li class=\"\">\n",
       "        <a href=\"/storage/\">Storage</a>\n",
       "      </li><li class=\"\">\n",
       "        <a href=\"/environment/\">Environment</a>\n",
       "      </li><li class=\"\">\n",
       "        <a href=\"/executors/\">Executors</a>\n",
       "      </li></ul>\n",
       "          </div>\n",
       "        </div>\n",
       "        <div class=\"container-fluid\">\n",
       "          <div class=\"row-fluid\">\n",
       "            <div class=\"span12\">\n",
       "              <h3 style=\"vertical-align: bottom; display: inline-block;\">\n",
       "                Spark Jobs\n",
       "                <sup>\n",
       "      (<a data-toggle=\"tooltip\" data-placement=\"bottom\" title=\"A job is triggered by an action, like count() or saveAsTextFile(). Click on a job to see information about the stages of tasks inside it.\">?</a>)\n",
       "    </sup>\n",
       "              </h3>\n",
       "            </div>\n",
       "          </div>\n",
       "          <div>\n",
       "        <ul class=\"unstyled\">\n",
       "          <li>\n",
       "            <strong>User:</strong>\n",
       "            sdrlurker2\n",
       "          </li>\n",
       "          <li>\n",
       "            <strong>Total Uptime:</strong>\n",
       "            4.4 min\n",
       "          </li>\n",
       "          <li>\n",
       "            <strong>Scheduling Mode: </strong>\n",
       "            FIFO\n",
       "          </li>\n",
       "          \n",
       "          \n",
       "          \n",
       "        </ul>\n",
       "      </div><span class=\"expand-application-timeline\">\n",
       "      <span class=\"expand-application-timeline-arrow arrow-closed\"></span>\n",
       "      <a data-toggle=\"tooltip\" title=\"Shows when jobs started and ended and when executors joined or left. Drag to scroll.\n",
       "       Click Enable Zooming and use mouse wheel to zoom in/out.\" data-placement=\"right\">\n",
       "        Event Timeline\n",
       "      </a>\n",
       "    </span><div id=\"application-timeline\" class=\"collapsed\">\n",
       "      <div class=\"control-panel\">\n",
       "        <div id=\"application-timeline-zoom-lock\">\n",
       "          <input type=\"checkbox\"></input>\n",
       "          <span>Enable zooming</span>\n",
       "        </div>\n",
       "      </div>\n",
       "    </div><script type=\"text/javascript\">\n",
       "      drawApplicationTimeline(\n",
       "[\n",
       "  {\n",
       "    'id': 'executors',\n",
       "    'content': '<div>Executors</div><div class=\"legend-area\"><svg width=\"150px\" height=\"55px\">      <rect class=\"executor-added-legend\" x=\"5px\" y=\"5px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"17px\">Added</text>      <rect class=\"executor-removed-legend\" x=\"5px\" y=\"30px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"42px\">Removed</text>    </svg></div>',\n",
       "  },\n",
       "  {\n",
       "    'id': 'jobs',\n",
       "    'content': '<div>Jobs</div><div class=\"legend-area\"><svg width=\"150px\" height=\"85px\">      <rect class=\"succeeded-job-legend\" x=\"5px\" y=\"5px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"17px\">Succeeded</text>      <rect class=\"failed-job-legend\" x=\"5px\" y=\"30px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"42px\">Failed</text>      <rect class=\"running-job-legend\" x=\"5px\" y=\"55px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"67px\">Running</text>    </svg></div>',\n",
       "  }\n",
       "]\n",
       "        ,[\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1529386096781),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"bottom\"' +\n",
       "    'data-title=\"Executor 0<br>' +\n",
       "    'Added at 2018/06/19 05:28:16\"' +\n",
       "    'data-html=\"true\">Executor 0 added</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1529386106423),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"bottom\"' +\n",
       "    'data-title=\"Executor 1<br>' +\n",
       "    'Added at 2018/06/19 05:28:26\"' +\n",
       "    'data-html=\"true\">Executor 1 added</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1529386097113),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"bottom\"' +\n",
       "    'data-title=\"Executor 2<br>' +\n",
       "    'Added at 2018/06/19 05:28:17\"' +\n",
       "    'data-html=\"true\">Executor 2 added</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1529386096716),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"bottom\"' +\n",
       "    'data-title=\"Executor 3<br>' +\n",
       "    'Added at 2018/06/19 05:28:16\"' +\n",
       "    'data-html=\"true\">Executor 3 added</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1529386106255),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"bottom\"' +\n",
       "    'data-title=\"Executor 4<br>' +\n",
       "    'Added at 2018/06/19 05:28:26\"' +\n",
       "    'data-html=\"true\">Executor 4 added</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1529386092824),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"bottom\"' +\n",
       "    'data-title=\"Executor driver<br>' +\n",
       "    'Added at 2018/06/19 05:28:12\"' +\n",
       "    'data-html=\"true\">Executor driver added</div>'\n",
       "}\n",
       "         ], 1529386090861, 0);\n",
       "    </script>\n",
       "        </div>\n",
       "      </body>\n",
       "    </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(\"http://localhost:4040/Jobs\")\n",
    "HTML(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"600\"\n",
       "            src=\"https://www.zepl.com/viewer/notebooks/bm90ZTovL1NEUkx1cmtlci84MjlmMTM4ZDEzZmY0Yjk0YTQ2MDQyNGFjMmZjMTcwYy9ub3RlLmpzb24\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f90957509b0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame('https://www.zepl.com/viewer/notebooks/bm90ZTovL1NEUkx1cmtlci84MjlmMTM4ZDEzZmY0Yjk0YTQ2MDQyNGFjMmZjMTcwYy9ub3RlLmpzb24', width='100%', height=600)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}