{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8a2fe751",
   "metadata": {},
   "source": [
    "# ⭐ Scaling Machine Learning in Three Week course ⭐\n",
    "\n",
    "## Intro to PySpark\n",
    "\n",
    "In this excercise, you will use:\n",
    " * RDD\n",
    " * DataFrame\n",
    " * Spark SQL\n",
    " * Loading data from file\n",
    " * Getting a feel for the data\n",
    "\n",
    "\n",
    "\n",
    "This excercise is part of the [Scaling Machine Learning with Spark book](https://learning.oreilly.com/library/view/scaling-machine-learning/9781098106812/)\n",
    "available on the O'Reilly platform or on [Amazon](https://amzn.to/3WgHQvd).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "28acad89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create SparkSession from builder\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.master(\"local[1]\") \\\n",
    "                    .appName('Scalling_ml_with_spark') \\\n",
    "                    .getOrCreate()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5a0e61ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *\n",
    "from pyspark.sql.functions import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afdb2330",
   "metadata": {},
   "source": [
    "## Get started with PySpark RDD\n",
    "\n",
    "**RDD transformations** – Transformations are lazy operations. When you run a transformation(for example update), instead of updating a current RDD, these operations return another RDD.\n",
    "\n",
    "**RDD actions** – operations that trigger computation and return RDD values to the driver.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee32f489",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Create RDD from parallelize    \n",
    "dataList = [(\"scaling\", 20000), (\"machine-learning\", 100000), (\"pyspark\", 3000)]\n",
    "rdd = spark.sparkContext.parallelize(dataList)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "28b945f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('scalling', 20000)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffa761f2",
   "metadata": {},
   "source": [
    " * Find intresting operations that you can do on top of RDD and share in the chat!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f448c834",
   "metadata": {},
   "source": [
    "## Get started with DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "586c38ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "data = [('Adi','','Polak','1991-04-01','M',3000),\n",
    "  ('Michael','Smith','','2000-05-19','M',4000),\n",
    "  ('Robert','','Jhonie','1978-09-05','M',4000),\n",
    "  ('Maria','Anne','Swiss','1967-12-01','F',4000),\n",
    "  ('Jen','Condo','Brown','1980-02-17','F',-1)\n",
    "]\n",
    "\n",
    "columns = [\"firstname\",\"middlename\",\"lastname\",\"dob\",\"gender\",\"salary\"]\n",
    "df = spark.createDataFrame(data=data, schema = columns)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9a6bc85b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- firstname: string (nullable = true)\n",
      " |-- middlename: string (nullable = true)\n",
      " |-- lastname: string (nullable = true)\n",
      " |-- dob: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- salary: long (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "08f392df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+----------+--------+----------+------+------+\n",
      "|firstname|middlename|lastname|       dob|gender|salary|\n",
      "+---------+----------+--------+----------+------+------+\n",
      "|      Adi|          |   Polak|1991-04-01|     M|  3000|\n",
      "|  Michael|     Smith|        |2000-05-19|     M|  4000|\n",
      "|   Robert|          |  Jhonie|1978-09-05|     M|  4000|\n",
      "|    Maria|      Anne|   Swiss|1967-12-01|     F|  4000|\n",
      "|      Jen|     Condo|   Brown|1980-02-17|     F|    -1|\n",
      "+---------+----------+--------+----------+------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9038ce1a",
   "metadata": {},
   "source": [
    "✅ **Task :** \n",
    "\n",
    "Explore the funationalitis of the RDD and the DataFrame. what is the difference between them?\n",
    "\n",
    "How can you convert RDD to a DataFrame? name it df2\n",
    "\n",
    "\n",
    "\n",
    "<details><summary>Hint</summary>\n",
    "<p>\n",
    "    \n",
    "Try out the following python code snippet:\n",
    "    \n",
    "```python\n",
    "df2 = rdd.toDF()\n",
    "df2.printSchema()\n",
    "df2.show(truncate=False)\n",
    "    \n",
    "```\n",
    "</p>\n",
    "</details>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5fcaa005",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer is going here..\n"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO8AAACCCAYAAACuGp89AAABX2lDQ1BJQ0MgUHJvZmlsZQAAKJFtkD9LQmEUxh/NEM2oIYqiwaEl0LBrEC2BfyICBzEla7teTQP/vFxvRC3ZF2iSvkBQfYJcHJqag6CgqSBoaylsKLk9r1ZqdV4O58fDOed9OIDVpQqRtwEoFA09thx0J9c33PYnWOHEIBwYU7WyCESjEbbgu/ZG4wYWWa+9ctfea3U/ePjoPXiL158fLhf/9veEM50pa6wfTEUTugFYfOTojiEkV8gjOk2Rq5KzbT6TnGpzvdUTj4XIV+RhLaemyfdkT6pLz3ZxIb+tfXmQ7l2ZYmKVdZQ5iTCWEOFzIwEF85hlhnmj/2fmWjMhlCCwCx1byCIHg9MBKgJ5ZMgrKELDDDxkBT6mX9769w07mqCXBf7ad9zRUhNAjR7GHR1t6hQYSgLnFaHq6s9lLQ1bedOvtHmgBvQfmebLGmCfBpq3pvleM83mCfffAReNTyryY9bIyxdhAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAADvoAMABAAAAAEAAACCAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdD+iCLEAAAHWaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjEzMDwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4yMzk8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpVc2VyQ29tbWVudD5TY3JlZW5zaG90PC9leGlmOlVzZXJDb21tZW50PgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4Khb0TvgAAK5JJREFUeAHtnQeYVbUSgMMC8tSnzw5WFkQRG4gUUQRUFLBXVOyK2FDAXlCxYsGO7YmigBTB3kWRYkNFpYgKqFhQUZ9drHhf/oE5ZM/efs9ezu7N7Hf3nJNM2iSTTJKZpNaiRYsSJgv44osvzMYbb5wFpkfxFPAUKAYFyoqRiE/DU8BTIHoKeOaNnqY+Rk+BolDAM29RyOwT8RSIngKeeaOnqY/RU6AoFPDMWxQy+0Q8BaKngGfe6GnqY/QUKAoFPPMWhcw+EU+B6CngmTd6mvoYPQWKQgHPvEUhs0/EUyB6CnjmjZ6mPkZPgaJQIHLmve2228y4ceOKknmfSM2iQCm1nZ49e5q5c+cWVIGRM+9bb71lZs+eXVCmfODSpEAptZ1nn33WfP755wVVdOTMW1BufGBPAU+BrClQJ2vMNIjffPNNIAJgfbR48WLzyiuvSIjNN9/crLbaamlCe69SpkAptZ158+aZr7/+Wqr7xx9/NDNmzDD16tUzZWVlplWrVqZOndzYsVYUJoHnn3++GThwYNI2OHjwYHPKKaeYzz77zPzwww+VcNZee23ToEED88cff5g5c+ZU8q9Vq5Zp1qyZqV27tnnvvffM33//XQmnvLzcrLLKKua7774zCxYsqOS/4oormiZNmmSVhs+nMcWkZzZtp6bUe5s2bcwbb7xRqX3iMG3aNNOyZcukfikdYd5sfrbXSGQDxx13XOKSSy6phLrjjjtiN1zp16NHD8F98cUXK/kpvq28hGXulP533XWXxEG6GsZ9WsbOKg2QfD5Nopj0lIpZ+i9Z26lJ9e6WdYMNNkhMmDDBdcr5Pbdx2nJEvnD55ZcnHRU32WQTibJ58+Zm5MiRlaJn5G3YsKFZYYUVzAMPPJB05G3fvr2E6969u9H43IhWXXXVrNIAyefTmGLS062nZO81qd6Tla8Qt0jEZjcDJ510kll//fVN//79XWf/7imQkQKl1HY4lWbo0KGmQ4cOGemSCiFy5mXBit5yrbXWSpWmd/cUSEqBUmo7zONhYHglX4icefPNiA/nKeApkBsF/D5vbvTy2J4CsaGAZ97YVIXPiKdAbhTwzJsbvTy2p0BsKOCZNzZV4TPiKZAbBSJnXjRIvGFCbpXgsZdQoJTazqOPPmq+//77gqo+cuZ97LHHzJQpUwrKlA9cmhQopbZz1VVXGbbGCoGiaVhlk0kMGlDWxqihbt26plevXtkE8zieAgYDhxEjRpiPPvpIFB923313s/LKK9doykQ+8hZCrbFjx5qOHTsaeiVv0F8IJUsrLBY6KDyMGjXKNGrUSLT7DjrooBpPhFgx7yGHHCKWQf369avxhPcFjI4CmNTBtOPHjzenn366GTJkiHn66afNwoULo0skhjHFinmhT642jTGkqc9SkSmAOej06dPNf/7zH0l5nXXWkWehx8wUuRg5Jxc75s25BD6Ap0CIAnfeeaeBgbfbbruQT836jNWCVc0irS/N8qDA6NGjzQ033GDYiqnpUpwfeZdHC/NpVgkFJk+ebA499FBz5ZVXmr333rtK0ohTpJ5541QbPi95U2Dq1KmyU9G7d29z7rnn5h1PdQoYK+blHCv26Tik6+effzYc2JXs3KvqRGCf16qnAOeO7brrrmbTTTc1++67r5k4caI5+OCD5Vn1qS+/FGI15+XQbZb6FTjShv06jr/x4CmQigIo9dDZ8+vcuXOAxskcNRliNfKyv2tP4arw84xbk5tfNGVjlA23G747deoUTQIxjSVWzBtTGvlseQrEkgJFEZtPOOEE89tvv6UkwHrrrScqkSkRvEdJUuDTTz/NeJAhc90jjjiiJOkTOfPuueeelRTCUXv866+/UhL43//+d0o/71E6FAi3nTXWWMMcfvjhaQmw0UYbpfWPq+dZZ51lGLQKAX8AXSHU82E9BZYjBfycdzkS3yftKVAIBTzzFkI9H9ZTYDlSwDPvciS+T9pToBAKRM68pXS7eSGE92ErU6CU2k7Pnj2Da3ErUyI7l8iZt5RuN8+OxB4rWwqUUtt59tlnzeeff54taZLiRc68SVPxjp4CngKRUyCSfd5Sut088hoo8QhLqe1gaIPRDcC5Wxy2WK9ePcMxPq1atcrZ/jiSfd5sbjf3N84X98Z5LLTmzJlTqWvgvuNmzZqZ2rVrmzjcOJ9N24lDPqOgZ5s2bQxnUyeDadOmmZYtWybzSu22aNGiRDY/22tkdXN3stvNCehvnC/ujfMvvvhiwtZ60p9lhkRcbpx3G1WythOXfEZBT7esG2ywQWLChAmuU87vkYjNqbuGZT7+xvni3jjfvHlzM3LkyGUVsPSNkbdhw4ZyLywWW3///XclnPbt24tb9+7dDWaZYVh11VXFKVMaIGWq93Dc4W/ur41DPjOVNZt8hstW6HckYrObiVK63dwtt38vnAKl1HY4Z3ro0KFyQHy+lIuceUvpdvN8ie7DJadAKbUd5vEwMCN2vhA58+abER/OU8BTIDcK+H3e3OjlsT0FYkMBz7yxqQqfEU+B3CjgmTc3enlsT4HYUMAzb2yqwmfEUyA3CkTOvKV0u3lupPbYmShQSm2H61i+//77TCRJ6x8589ak283Zuvjnn3/SEtB7RkeBmtR2MlGFO6hpX4VA0TSsCslkLmG5q0Z7tEsuucS0bdu2QvBrrrnGWLU0ceNAd6uSV8FfP1AaR6vm5ZdfNttvv706++dyosBzzz0n9bbiiiuaAw44wGy55ZaVcvLMM8/Ivbzl5eWGE0tXWmmlCjizZs0y999/vxgDHHXUUXKnr4uAthmKE9Q9dU5bCkOmOML4Vfkd+chblZnNJm5uXOD37rvvmgULFlQKsv/++4v/v/71L6mkSghLHbgiEo0fGoKH5UsBLg7r0qWLKDR8++23ZquttjKvvfZahUw99NBDplu3bnJH77333muoZxe4RgeG/PLLL6Xeuf7zu+++c1HMGWecIUfNrr322sL8dPQuZBOHi1/V78tl5KWHS3f9Iv5YvaCHmwpSxdG6dWsJovq34fBNmjQx/OilFy9eHPYOvhs0aGA42SETEAd59VB1FODY4P/+97/m+OOPl0Rmz54t+s7u/bt33HGHOfnkk82ll15q9ttvP7HQwQSPugbGjBlj1l13XXP33XcbLISoX8T0o48+WvytcY65+eabJV4kstVXX130ss8880wx2csmDomoiP+KNvL+/vvv0rOhEla3bl0RSWEwiKbwwQcfmK5du4o/t5xzmPYWW2wR4HzyySeGs33r168vOBwlkmx01fjyfXI9JOkiNofnJTA0fozuKPDTCZ1yyikVLkSjYxk0aJD477bbbub222+XuHbZZZd8s1TS4S6++OKAcSEE5x1Pnz49oAn0Hj9+vLQNHLfZZhu5XHvKlCkBzvPPPy+jMR0t4rReSKYInOIBcIg7QDvE9pbRViFTHIpXrGfRmJdGP2LECDN27FixM916663lYii9SQFxiFFz7ty5cnsCIgsrcvSyGC4DMCo94uDBg80TTzxhqBxEpKiBA7H79+8v4tWvv/5aIXoqlTxwgTPMy4gwfPhw88gjjwR4XDFJHJtttpnhUHBGBOZR9OIeCqMAbYB25B7GTtsB3ClO06ZNzVdffRUkxpEzjRs3Dr4Zkd2OeeHChWaVVVYxq622muDoYe64K2SKQ/GK9Sya2AxhAK7sZNHh+uuvlwuQORUfgAm55Y0FASXctttuazBgZn4KMGfZcMMN5epGmBxzNRj8ggsuEP+o/lnbY4OkkAxoAJjUkW9WDAHy/PTTT4sIRrjrrrvO3HnnnaZXr17ij3iHyNehQwf59v/yo8Avv/wibQZ6uotJuAO0KwVu4fjpp5/0M2h36sDo614fS9tD2lPghAsglzg0bLGeRRt56Sk333xzg+jIkwUh5h+6FfP+++/LyKuMCwEYiRGJGOkAxGT8YXQWHjidQ0dlQSjiP3cVm15cR2jt7XfYYYcgN2ofGzj4l5wpwLy3R48ecmjbww8/HHToRLTmmmtKfC6jsePAwpMCc1zXn3ZDG1RYa621KrQlrU/cFTLFoXjFehaNedmAZ1EBMWTmzJkiPnOCHgtHAEwJjktg3LlsCmBlEGZHZHrhhRfMLbfcYljujxvo/TNIBAquSK1u/pk9BVgUZGFp4sSJsl0EE7mAqItkx9QEYEGK1WhXTEaM1nktOBw7w2XcCkhTjL4ff/yxONFGAdwVMsWheMV6Fo15mevuvvvu0nOy4KOEg2AAd6wCxx57rBCexStGWioA8QYxhwpihGYezCIQS/uMzIzCAKM4CwysMiK+4s67O7dhzoQbnQG9M+8ffvhhIAHQeeCGWA5QmXzr3IcFNvL8v//9TxoJ6XOIGuHozbHPZH8ZUZ4Rl995550ncfl/+VGA9QdOBRkwYIAsIo0aNUrEZ42NXYljjjlG1iGoA90lYLFQgcvu6PxpP1zGzTTnwAMPVG/ZfqJNMp1j1OXJ9pQ7OmeKI4isWC/ZnF8Fjm3AWZ2xYwmdsCNsJVy7gFPhPCXLiAk7D0zYxh/gWoImLLECvJ133jlhR+fA3zJsgnCWNgnOALJMIu+EAaZOnRqEBUd/hAGs6BWEVz992n1hwendu3cQTv142nmW+GuauNlGlHjqqacC/NNOO01w+GcbW8KOFgniAw982ygCf/9SmQKp2o5lqoDGWiecieaCXbRK2EXQAM9KPq63tDMrqQX+55xzTgV/PqzkF7QP2pTtwCvg0FYzxVEhQJoP2pNdK0mDkdkrcmP8Cy+80FjGkk1uS+gAWFW22ZG9W0YtRJ9Ue72MlIhCYQ0ZImPuw0in4mmQQIxf0NpBokAySLd3HeMiFCVrqdpOtonTvpCi2M9deeWVkwajbdHu3BHVRfzzzz9F2mJLM137TBeHG1+q93bt2pkhQ4bItmMqnEzuRVttdlcCYe50kI4x2SNO558u3mL6Mc9lXsX2woMPPmjY+PeMW7U1AH1VKSNVSpnaDtMe5rbpIFMc6cJG6Ve0OW+Uma4OcSEhcE4RjYE94WQnOVaHcvg8xpcCkYvNdt4pIksyxfH4ksHnLA4UKKW2gy72TjvtFGyD5kP/yJk3n0z4MJ4CngK5UyBysbmUDKpzJ7cPkY4CpdR20ANQ09V0NEnnFznzVheDavbyCiVeOsLWND92CFQPvarKVl3aThTlj8IYP3LmjaJg+cZh91lFEQSlikzApr6rH5sJP2p/0sbIgR9zvbhDo0aNxGRueeUTZRm06zD6SAVo6/Xp00cWCF1rNcVHBx2FGZQ9VJNK/XiicHPXXXeZU0891aAIkgwyxUEnB2NidcZBDlUJNYp5ORUDzRkImAnQnnE1bDLhR+2f6dCAqNMrND6MLFx97ULjyyU8KrJo2g0cONBcffXVgTacG0ccjPFR47SKRWLxxrWdaNe99NJLbjajfS+WhpXqi9hNcH0Vjafgw3mxygwJSwjHpeIr/q5mlutrqZOweq3ilArHxU/3ni4PhENji18hYI00EnYfOGUU2aSRSx5S0SSVe8qMJfEoNI5UGlYkRRlff/110ZBKVi/WDjdhTS8lV1aHWfBcDSl7GkcCTS3yiKYbWndWeUbw+YcbbcdeaiZu1jBftP3ctDLFYRlV4rAGMxLHPvvsI1p28hH6F4WGVVFGXnSU0WemJ2Lfk280TDCrQ99UAYMDcOi10JJB48YSW71Fx5mziTDdQsMFkRP8sNjJniqmg2ym33jjjUF4XjgLibzw49QFFzAtwwQRI3z80PLi/cknn3TRRO8ZPW0URvhhvM9BAVECutXp0rCNWeiohxuA6x4Ngw4veaecE61CP2aK4GKhRXnAxZ+bADn0AHpSDvTGXbjiiisCeiHVuJDNwQTgszhDXVHn1157rbyTdjLR1Y3ffU+l7QQObSQOxviTJk2SEzxUCckyr7Hqs24xIn0vCvOeeOKJYr6HeRXnEcGwqD5yRhTfCjQ4mIdGQiVzPeSrr74q3rbjkoPHmIvA/BjAY1zAvALTQBdGjx4tcyPS7devXwV/LjCGMblgOjw3xgYU0RvrFLSiMPrHfBHVRtIHSJMOA0ZngQVNKowdUIJPNs9y85XtezZpkD54Z599tljaYPOMMYQCVlrQD4MJysShAdDisMMOk7JQfuyjMRhh8Q56Ug7o7gKHvUEvDD3U3FH9YchMBxM8/vjjcmoFHXLHjh3lhBGsyZiy0EFHAXExxoc+HMCgwDoBp3G4A5D6RfEsinpkq1atZLS0SvqmU6dOhtvQrWK4zGM4MQMrHSvGyIkTLAhMnjxZVjaZ57AIgXG8FZmkkcK8WHcAjDbY/Ib1WC+2x6ZwegWgJoSYlAF0IDRI0kEPNgw0MIAOhnzDlDQyRiRGMToGKoRGuf766wsuTEOc9gJms8cee2S8cV4CpfmXTRowzT333COniWAlQ2+PnSvMyvFCdI577bWXLM4g7TACqF20Js1hB9AdGoHPDzqhi61AY+SHllgYqJ90BxOAj3UO9aULQMRPx4E9tB6yEI431++4GONDe1cNWN/p1FOdqZZrWV38ojBvhQSteAYg0mqBsL9EL5UFEUSpzp07yzlVVIqaDKqJHsyhgNg9Z86cSgfA0SgV0FNlZMkVWrRoIUHUdlTjQOURcI3xxcH+Y3RCsZ3ROhmwkonUkAkypUF4Rn3EMiQJpBUV25M1FI7fCTOu5oHOD6YFGCmU3uqfzdOlBbrFbqeIdKOdLXHREdBhRAlxMcbH2GH+/PlB0fSgiKjLqwkUnXk14fCTeRniKhVPjw644rDOI9jIZ0VPAQsj5sDay6l7VT0ZbakMRCRt9OG0Mt3kHsYPf2eTBlseHK2DOAxAl/CcNBzv8vhmlEUiIK+IzkgJ+XQQ6fLuGuNzNpoa4yPdKSQzpHdFXNcYn04sW2N8Nw7CqYRBuhySR0deVQYpRZnzogzBiIRxPGZxAKKnAu86/6HAnHjAXI6FFcJijM+5RRhLM29jAQTG5oRGwsH4AHNAQOdmjNyIMhjSs8AD0DMyGjBPwo93fjov0XzpU+dTzAcBRFEaH6MZc3QkBcRMjvfhdEEACyL2ccM/FtEAaJDu0IBs0mDxCfM2xHnmrXogGyMw83NGYMoFDcDj3V0goryUjbLoWU5KNz3cAJopfVDQ0MMNdHQhDcKzNQfDEKd7MAFlZV+WOS7njXGWcj7bTdCL7SKVLpDCNI+kAXPEwRgfSQgrMhiY9sNRskceeSRZrBqwFZDI5mcrMbTYnfwz2XK/3VOVJXRbgoQVCWWZ3o5eCduwxB0jasAuDAV4tpITdsFJvu1qqPhbETmBO/HwIxxL+mwdAVYEDPwssyUsowffGGfbBhB8axz6xOjfNsTAn20FwIqlgZttPOI2bty4CocGUBbSsh2N+Gf6Z1fHgzg1fZ7Eo5ApDbYlrIQSxGMXpwJDcmhs90QDPzcN26gkCXtrROBvFw4TtuMLvrXstlMI3Nw4eAc/24MJ7KJjgjSoX7vAJ/m0q8Na1OCZrO3gqVtEbh6gle0sgrC2I4qFMb49eDCgmV3fSXkAQxRbRZEbJhRqUM1oaPfWUs7RbAXKaExvrHMd3IoNttVI7885wBx0hkgYNWSTBj090kd1OfidURMJil0EpCkXCm070CsOxvhIMkiMiNGpoFoZ46cqRNhdF7HC7u53qsUXF6eq3xHVqtooO5s0dC2gqstbSPzMH7kjiKkTIj4LO8xNowboFQdjfObg/Koaoh8uqjrHPv5qRwGYinUH5sPcLsE8PdViX7Ur3HLMcORicykZVC/HequRSZdS2/HG+DWyCftCeQpkRwERm5no60+DsSCEG6B+6sY3i0rqxxM/AE0o1Y8FT8PoExx9D8ev3y4O77i7cfGeKn03DjeMpunGrbg89Yc/4OKrn7rx7dNPXv/QRoF3pZk+XdoqLk9+bttx8dVf460Jz0iM8ZUwSiwaJW4KuDNn0Z/i840fgBvfAJo/qDfiR1xuOA3DU8NofFGl78bn069e9KftoNxPHSq47Q93vt2f4lW3ZxTG+HWUULrVoY2fb4jEk8133GEGtiSUCSGYhtdw+q3xgavhlOjEAR7fCoqv8YTTJ4zGky594iOsxhd1+mzAs3LKNZJY3Gh5Nd9VnX4m+lf39Mk/tAzXPwYoKOOgDIL1Eyqh1K3Sn3f04NEyKy8vl3PDw4ti6LNTd1wiZg9Pr7SVQztH4QZNP4w2ludhDdAhE5RR+DAzaIOHuWAYJaQSS5mZyJV4vGvD4h0cwvMEiEfT4gku/uqmjKzfhCMMT9LnPQ7pc+IfKn9qZE1+FUqh/JSXOtFy6xMaRFF+4qGe3fq3CiWi2cb+KRpj7JFiTAGe4mMF1q1bN1GV5SI6tLlcyOZWe67PsYoism+P6SlGM3EGGXnZfwN0tIJwyihUDu48lQHBpaL0m3f8lRlx54cNprrxJB5+qN0RvxI/VfqkQ7z8AE2PuEgTwM9NX91wT5e+BLb/CJspffILHumjkIF10Ztvvhk04nzSL6T85F3zk2/545g+ZeGn7Y9y8o1VGQYoKOXwzciIlZSaN1LPXKmKJRnmi/vtt58YbKDaqfu+Y8aMEWUWrMUYvTE4QUw/eqm1GaqeqDOil456K7oEmFSiBqt8QX7iBGVhQsEYuPHUH42Wd0AZGSICPPGngStz4a5MSjiU0REz2ePDOJ7Ri2NoIKQyMiZ1GGjzgzHQxME6iG/OpiJudHOPs7ap9evXF9EHDR30gPFDrxlRCgsXdKKpQN7R2sEiSBkMY3zSxxwNXWP8CQ9QFi7FRoEASxjygL4quNiugqfllwD2HzbD4GPdgzUQcfBzy8+3pg9tFYdnMvpDE5f24GgeNX3CAjzxc+mPW3VNn7yHy09Z9B5n6Ig4DPO5nT/G+JiIArQ1FEG4fF0h0632eoOgPZFDglDf6CczYscVRGzWiqeRaOPSBoMfPRVPRijFxfyNdyU2BdSGRGPip3GgRM8cApO4AQMGGMztOIFCjfGJh16Txo8f+FQORu+4wUik3bdvX1GCR2SlMkgPBXXCY0iP0fzbb78tIhMVjHjLmUfMkVCsB4/4sNNltY+elYUD5lPKAMxzaASIYZ06dRKbXeJgxNXyK2OhdEBHQ9qIW/Ty0Ao8t/y4Kb2UZuQdd6Wfpg/NAPA0DuLTn7prfBqeMBpndU1f6RouP2WmTKhU0ilDc0Zd6ANjqxEJ1mi48cOKSA0tiC/TrfYokVh96UAzSu+J1tshiSNuUIfMUemA26urWKVPdDWVSOApQHAdVXBD/GBeghWOhgVfxRercC+6raSpp2XAMBCbRQRWqrGDRBSikhitMdfCAgajcvzQX0Ws4b5ftqWwaiE+mBXgdA6OdyH9na35IEyNmMVIDCMjTqEQQIOgkjgNg4qnHPTYao/LoQE0FuJBlNKD7bCkoUdmBEdPF2mC/EIf8gE+Dcgtv9JYGxzfLr35VjpqHBpe49M40tFfw1bH9JGQ0G2H6SijW37aGZZFSF90xFgYUU8MArQ5AAlJw3FAA3EpMF92zUbp3NWaChzqFNNSBRa1ADcO9YvLs9bn8xdglWNqW2JZ2xzJFwTQEUAIaJZuFZXZ7SFplODaLST7B9Bg+OOrFkTnjwb8j12dLqtt/l68ZA4tjc4mofiExc2n7+mfT/tbYaW6MlLChEhqXOyG5EabYlGLDpdTOgGmXyxC6QkrTJeQ0p544gnx59mjR4+AWelIkKjYe+bAgqghEsOEv/9aIqb9s3jZto0VPIK8wpSVYRmu6yeYlmFd0HiJR9+T+bt+Pv1lNPT0T97+atepLSMwoyVSGCKvvWNZmJcpBQfsYROuEIUxvsYVl+cy+TcuOfL58BTIggJIdyq1Ickx5eKcLaZ39hJ2iYHzxxRYN2GBlAMUWEBlP9g9t3urrbaSKRBnbjHq8mRRlWlUXCE2x+DElUA+X/GlAPN6XQDkWCBWiDlgEODoHVdJg10DGFzXRTgih0VUF1DgYI2EU0Nh2qq+8cBNO5/3gkfe+Z/Ml8UkEnff88lMnMLMnDVTFrTyzdOsd2fJCnM4PItkM2bOCDvn9e3Se87cOUE9pIrMxU+Fk697qvJ+/c3XZu68uUmjZRGSPOUDMg2zazW6qAXTMudlZGXBlC1E/BRYDER5g4VUFmnZZQgDuxDkiTjA00XWMF5cvgtm3hcmPG9mzFrSGN33QgvIuUsTJ00sKBp65QcfGpdXHPfce7f54ccf8gpLoPuG3xecqeVGwmo5cUcBLr3HP/+cmT5zetpoXfxUiPnSPVV5p09/x7zw4gtJk4Phxz8/PqlfNo66oo7YzDsLn5xeoUfK4h4GDlBIJwqzo8H8mN2DuEPBzFtVBfzyqy/N1NdfKyh6Rrmnn306UDApKLKYBz7lpN5ml512KTiXUdDdzcSunXczJx5/ousUybuOqrpXrpHCdLKrsdQhGQMrbnV/1nn40YfNy6++LFtA27fb3hyw34Gi+DBk6BBZxfvgg/fN4JtuNW+8+YZ5/MnHzPd2WX7zzZqZY446NujhMhGB1b+h991jZr//nlndHg+y1x57m9atWpuPPv7IPPrYI6ZfnyXL+ZOnTDILv15oWm/bWkau3//43Zx57pnmqiuuMsPvH27qrVDPzLSjPPt87Xdobw7c/yBJGpzLB1wu+UHcGX7/MHPu2eeZiy69SPzPOu8s0+u4XtLjMprvs9c+FSoYpI9tXoaPHGH3GL807bbbXrazJLD9x2hEnJ98+olpuFFDc3iPI0R54/kXxptf7R4j8QGUsfnWLUzLbVrK9+SXJpvXpr5q96fr2TLvaXZs30Hc3X/Q/6WXp8gedtfdupqOHTq53vLupr/eeuubo444ymy4wYYV8IaNGGY2syNGm9ZtK+Q3Ff6jjz9qFRc+Myf0OtHUqb1klJk//+NKdGc3IlndVUh86cfzVgp76+1pUt7Dehxmmm/V3Lz8ysvm088+NYcefKhgjRw90tLkNVkdbmxHSQWY8B5Lv3dnv2vWWXsd0+OQHqbJxk3UO+mTkVbFZmVmtjgRkRVgXvVTtzg80eQr9BilsjenvWHOtw39/HMusIR/y7xjxRz2ZT+1DbVtm7bmumuuF4a6b8R90mivHXitKbPEGTn6/qxpAC5hCEvDJy6YFEL/9PNPQTww+aJFv5ny8kbSQDdu1NgMumqQNK5ff/nVVuwsc9YZZ5v+510oncnrbyy5GvMHu8LIvjNA3jXOSy+6VNxIt+mmTWUlcoYVLRmRw3DrnbeZdm23Mzded5N0MMybFG4afKPZtuW25qbrb5bnzbfeJF50IvwUfrGrlG7cXy38ylx5+UDT89ieZszYMcJUisuTRvzO9LfNhRdcZE49+TQDQ9HQw0B6LZpvY24YtCQfycTuX3/9xfyxtFyZ8EnnfduRHndMz4BxSTMZ3VPVXTiPfJdZPYCrr7zGdNixgxk7bqyg/PHnH7ZOlxx6T+cMcw64aIDpe2pf84HVV1YYNWaUWcEqXFDf3bp0M4Nvu0VEYfUPP92tRZhYRWjFU4bVp7rH5YnhRKFnsZV1aN/R6o2uKat0V1420GzTYhsp32p276xNqzaiQ8yCw6abbCo/5hNddu1i3v/gg6zpAC5hCKvxpFrESBdpxx1tXldfQ/K6w/Y7VKj8dOHUr7HtDC7uP6CSxMCiyu+WCRHxmPN067q7qb10zsMCBlsHnXfZVbR5ePKNeyboulsXs9KKK0mZmzbdzMz7sOLCDSMzCgYvvTRFRiy0eebMqUhXVDDpSLrYuND6oWFThlSQCZ81gMefeEykHZ0bpooL91zqrv327SWP7dq2s9polenDolpHy9jUIeqm7e0NGQpIH3R8SDMLbadHmb/48gv1rvRENNatIphXv3WkVabFr6aCLfOyFTm3kHWdCTtaVWjAKCCWLLYjXLYArivKiDaNjROgx1RYnIHQVJBCmdXccitGe2LymisQD5phClS8UkW0wxw/cMDFHXDp4JYFP/KoIGX+p/ICytpWRFzdNmZ+3Q/sbvcam2oQeYpEYfOTLWTCRzxFdRBxPhtIV3fh8EwPAFQWk801qd9wHbpxsGLs0mLVVVZ1vSu8Ez8/2hX1RR2qyMy7pu+2uwoR1ICPMua7iKv0eoNuGGRYAQzDxo2byCiHGAhRJk2eaJfRNwmjpfwGlzCEJY737ehCnByP+ZVdtkdnGGZApFJglGG112WIV157VZT5EVWnvj41uBYFKUHz/c6MZSuusnhhK1d1kmm4ybYm6q9T35TZBqBiOKOAqoc2qN/Aipa1Az9w+MaddOd9OE9oxyoyc0YXJk2eJI2KMr9n1w70GhfFQRSvXbtM5v/oXSN2a7qKQ97qWWmAuSNA+jfcdL16V3pmwu9x6GGmjxVZHxj3gOQ9HEGY7qnqLhwum2+mQVqHi35bJHWo4aDFYqvlx7rLlltsab60+s11V6ir3pWftj9zaeV2CpWRa6ZLnUbljc3Z551tG9liWWxpZhejFnyxoEJpuTtn3332NVdfe5VlssVm/XXXM6xuKiwbp+yoFIxZ6mvM4Ycebm6741Zz2umnScPfd+99gxv22rZuYy646HyzstUjhSEUmKPWtgspp/btbW6+8RZxXtNeLHbO+WdLpbVo3sIgpgF7dN3DLrTca0Y/MMpuFTQWN/7REIn/wgH9Td/T+sp8+rEnHjUDLrwkwOGFnrtXzxPM0GFDzTC7MEbaK620coBDWZln4reKzaeWvWWLlvZmwAmm7xl9ZP4SPquXAbOP9aPTYjFqow03kuNPNeKdOu4kawtnnnOmzNm3aLZ5pYUocEnv7qFDzChbPsp0/LHHaxTL6E1iSyETPtOHg7sfYm6383xEcPes7DDd09WdppfsqWKr2x526rSz7PmeflY/KUfjRhsHQQ+zbeTWOwabPqf3EVp03XXJlCNACL1QWk0DL2gMA4fdGIXjOPpy4RyKIlwDky/Umj/3k4SKgLrqmC4yRmjmhflAqrC4Q+BkRCZv5GvwbYNlsYhFNHrccB7AQ3QOu5NPjeOpZ56y89VfzEEHdE+ZfaSQVHNBRm6YJwxIAq7FiuvPXjONKlnZFI/8JaxIjbiZDtLlLVm4XPHdOJRm6paq7tQ/lyc0gR7JRktoDB2S+blpIFX98sfPYqBPnYNPOKUzTxgZpnYZ2o1jeb5jSz5s2DCxTMs3H7JHkA3TagLJmEP9Mj1ThU3lTnyaN5hG9/CS4QvesilmhaxoHCwysUWVDlIxLmGSMS7uqRgXv0wMCU66vOOvkC5viuM+c8V3wyrN1C0ZzdUv12c6mqSicTgNmJKfy6zgMM2inegzrswbLk8+38K8+QQsdpjjj1smKuab9pGHH5lvUB8uZhRglhBmYGVmRmHEZSDTCF7MYs2bNy+wiWeNhON86KzII6qZdDq5gIjNuQTwuJ4CcaBAnbp1zLc/fCOHKWBIjwE/jZ+RWOe5SEScyoKYzhlYYUCcxlCBMFwZ6xrnKy5bWtgKI85nioNTWtxFNI2jvLxclFI4pol7lJMBetkcpZQLeObNhVoeNzYUKLOr9MNG3ifG9ZgCXnbZZZXyho0ve+cTJ05MObeE4bggu0OHDhXOvNLIMNDH2ihTHOwkpBL5Of6JBSoXIpnzMmRDCBZ7sNRgy4T9SU7BKKtl9z6teEJPRi/FHzgsrtBbgaNu4OIHLvMlPUUDUYb5hw0u+6N8AyrOkK5P39M/5/Zn2+jOO+8s7ZCjirAAok3RvhiBeeqxNpyiMXLkSJd35J02zRFLAOeZoVobBl0NzhQHawKcPJls5OUstqqAWtZmMaGrcTyV2VT8gBBkiMUeeguX+cCHWSEaDMqT86Ow2uCmcmVQwihhKYTi404cbvr4a1z4Sydh4wY0f7wTR7L0NS3y7NNfoniiNOGptIPGfMeJ/vZCcGk7nDmmdaftkG/eyS/viMRscbH4xY+y0Fbx1zYDHt9xBMR5DnhnxM8XanHbPAXUitQKVjd153A1eincYQwlquKRAd6Ze7DKyZ4nYSGgMjjfOooTHtBvTYcnoPGqO9/qni598DVvvPv0qw/9aWMwIse8av1T59Sh/rRuGenAUYbFX9uUjrxa/9JwYvYPcR0GLmQVv9b8+fMTFFx7YiUS3xBHmQbLlkaNGsk3DAdALPAACIa7+inTgcM7gJ+G0XhxzyZ9wmp+NA2Nizh8+jWP/rQLrXPaEPWNG8BTt5zcd213+lR8CVTD/tVRkYNCQihlNN5xU6ahh4B4yjBKFJ6Kx9MlKCMk8RAH8RKH4vOE8YkPP77BVdBvzQ/xKj5P/AGe/EiDp09/iaIHtKgJ9KeO3baonTRPbS/aJsF13fmuyVDLHlouYrMSSCucb37KKIjD5eXl4oY7TMVTGdMloBIMfwVlMI2Pb0A7B96Jg/QBwoIDPu/8NE333ae/pGOsqfTX9qHthae+U2bt3JWpaRsKiqffNe1Zy562l1AGoXBaeGUy3CAQNxM0Wio24wZRlXiE4RtigatP3vm5zIafmx7fhFUAl5/iaFj8NS7effqlQ3/qW0HbB20BhgW0vfEEwCkFkNK7hVamoPDKWDAphOIH0dQv/ARfmQ4/JTDujKiEx1/jwR0cfSqj8iRN/eZd4/Pplx79aR/aFqQh2H+4Abjrk3far7qJRw3+VycVEZg7QgieKkpDBxhPieTSBaZSwhEnDAej8s6T+a7646fvSmh9qjvheOdJ+hqfT7/06O+2I9qBtg1tMzxpf+DhXyogzKvE4Am4zMU3DAyBXMYBV4mmT42Hb2VCiKoExV/DEZd2ChqOJwA+FQEQXjsSn/4yxlU6Qp+aTn/qnfZEu9C2Qpl51/LTVrT98F4KIMzrMooSRJ8QgXclDERUN94hnvopIXkCSnTewdV4YFoNRyUkS5+K0ngIpx2CT9/Tn/ZEm9D2R1uhfZQa/B9pbJHHsiOcegAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "id": "36e185a5",
   "metadata": {},
   "source": [
    "Now that you have a df, let's look at it and add names to the columns _1, _2.\n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eff44d67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- name: string (nullable = true)\n",
      " |-- id: long (nullable = true)\n",
      "\n",
      "+----------------+------+\n",
      "|name            |id    |\n",
      "+----------------+------+\n",
      "|scalling        |20000 |\n",
      "|machine-learning|100000|\n",
      "|pyspark         |3000  |\n",
      "+----------------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "deptColumns = [\"name\",\"id\"]\n",
    "df2 = rdd.toDF(deptColumns)\n",
    "df2.printSchema()\n",
    "df2.show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c70f6e2b",
   "metadata": {},
   "source": [
    "The book mentioned critical best practice around struct. since data in spark is bounded to a specific struct. many times it is better to not have it hardcoded and rather add a configuration file. let's see how that would look and feel with a code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c18d9399",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- name: string (nullable = true)\n",
      " |-- id: string (nullable = true)\n",
      "\n",
      "+----------------+------+\n",
      "|name            |id    |\n",
      "+----------------+------+\n",
      "|scalling        |20000 |\n",
      "|machine-learning|100000|\n",
      "|pyspark         |3000  |\n",
      "+----------------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.types import StructType,StructField, StringType\n",
    "\n",
    "deptSchema = StructType([       \n",
    "    StructField('name', StringType(), True),\n",
    "    StructField('id', StringType(), True)\n",
    "])\n",
    "\n",
    "deptDF1 = spark.createDataFrame(rdd, schema = deptSchema)\n",
    "deptDF1.printSchema()\n",
    "deptDF1.show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20fd8dea",
   "metadata": {},
   "source": [
    "## Temp views and running SQL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8bf93b74",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- firstname: string (nullable = true)\n",
      " |-- middlename: string (nullable = true)\n",
      " |-- lastname: string (nullable = true)\n",
      " |-- dob: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- salary: long (nullable = true)\n",
      "\n",
      "+---------+----------+--------+----------+------+------+\n",
      "|firstname|middlename|lastname|       dob|gender|salary|\n",
      "+---------+----------+--------+----------+------+------+\n",
      "|      Adi|          |   Polak|1991-04-01|     M|  3000|\n",
      "|  Michael|     Smith|        |2000-05-19|     M|  4000|\n",
      "|   Robert|          |  Jhonie|1978-09-05|     M|  4000|\n",
      "|    Maria|      Anne|   Swiss|1967-12-01|     F|  4000|\n",
      "|      Jen|     Condo|   Brown|1980-02-17|     F|    -1|\n",
      "+---------+----------+--------+----------+------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.createOrReplaceTempView(\"PERSON_DATA\")\n",
    "df2 = spark.sql(\"SELECT * from PERSON_DATA\")\n",
    "df2.printSchema()\n",
    "df2.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95e5a94",
   "metadata": {},
   "source": [
    "✅ **Task :** \n",
    "\n",
    "Now that you know that you can run SQL like operations using ```spark.sql``` Group the person data by gender using a sql query.\n",
    "\n",
    "\n",
    "Share how did you go about that! Use hint only if necessary. the goal is for you to experiment with pyspark api.\n",
    "\n",
    "\n",
    "\n",
    "<details><summary>Hint</summary>\n",
    "<p>\n",
    "    \n",
    "Try out the following python code snippet:\n",
    "    \n",
    "```python\n",
    "groupDF = spark.sql(\"SELECT gender, count(*) from PERSON_DATA group by gender\")\n",
    "groupDF.show()\n",
    "    \n",
    "```\n",
    "</p>\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b621da1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer is going here..\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "764205ba",
   "metadata": {},
   "source": [
    "## ⭐ - Time to start with our Twitter data journey ⭐ \n",
    "Now that you feel a bit more comfortable. it is a good time to level up and start working with out Twitter data.\n",
    "This exercise will also continue into week 2 class.\n",
    "\n",
    "In this part, you will:\n",
    "\n",
    "• Task 1: Load data into a Spark DataFrame (DF)\n",
    "\n",
    "• Task 2: Immutability\n",
    "\n",
    "• Task 3: Get a feel for the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8bbc7a",
   "metadata": {},
   "source": [
    "## ✅ **Task :**  load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "95884953",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.csv ('../datasets/bot_data.csv', header= True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9d7e98a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2840"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# understand what is the data size:\n",
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb8d8118",
   "metadata": {},
   "source": [
    "## ✅ **Task :**  Immutability\n",
    "DataFrame in Spark is **immutable**.\n",
    "\n",
    "What does that mean? It means that every action we do on DataFrame doesn't change the actual DataFrame!\n",
    "\n",
    "Instead, it creates a new DataFrame. Run the next commands and get a feel for working with DataFrame.\n",
    "\n",
    "Don't worry if you don't understand everything completely, the next exercises go deeper into it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a88b81b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new = df.select('bot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "88bc24b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>id_str</th>\n",
       "      <th>screen_name</th>\n",
       "      <th>location</th>\n",
       "      <th>description</th>\n",
       "      <th>url</th>\n",
       "      <th>followers_count</th>\n",
       "      <th>friends_count</th>\n",
       "      <th>listed_count</th>\n",
       "      <th>created_at</th>\n",
       "      <th>favourites_count</th>\n",
       "      <th>verified</th>\n",
       "      <th>statuses_count</th>\n",
       "      <th>lang</th>\n",
       "      <th>status</th>\n",
       "      <th>default_profile</th>\n",
       "      <th>default_profile_image</th>\n",
       "      <th>has_extended_profile</th>\n",
       "      <th>name</th>\n",
       "      <th>bot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.16E+17</td>\n",
       "      <td>\"\"\"815745789754417152\"\"\"</td>\n",
       "      <td>\"\"\"HoustonPokeMap\"\"\"</td>\n",
       "      <td>\"\"\"Houston</td>\n",
       "      <td>TX\"\"\"</td>\n",
       "      <td>\"\"\"Rare and strong PokŽmon in Houston</td>\n",
       "      <td>TX. See more PokŽmon at https://t.co/dnWuDbFR...</td>\n",
       "      <td>\"\"\"https://t.co/dnWuDbFRkt\"\"\"</td>\n",
       "      <td>1291</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>\"\"\"Mon Jan 02 02:25:26 +0000 2017\"\"\"</td>\n",
       "      <td>0</td>\n",
       "      <td>FALSE</td>\n",
       "      <td>78554</td>\n",
       "      <td>\"\"\"en\"\"\"</td>\n",
       "      <td>\"{      \"\"created_at\"\": \"\"Sun Mar 12 15:44:04 ...</td>\n",
       "      <td>\"\"id\"\": 840951532543737900</td>\n",
       "      <td>\"\"id_str\"\": \"\"840951532543737856\"\"</td>\n",
       "      <td>\"\"text\"\": \"\"[Southeast Houston] Chansey ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4843621225</td>\n",
       "      <td>4843621225</td>\n",
       "      <td>kernyeahx</td>\n",
       "      <td>Templeville town, MD, USA</td>\n",
       "      <td>From late 2014 Socium Marketplace will make sh...</td>\n",
       "      <td>None</td>\n",
       "      <td>1</td>\n",
       "      <td>349</td>\n",
       "      <td>0</td>\n",
       "      <td>2/1/2016 7:37</td>\n",
       "      <td>38</td>\n",
       "      <td>FALSE</td>\n",
       "      <td>31</td>\n",
       "      <td>en</td>\n",
       "      <td>null</td>\n",
       "      <td>TRUE</td>\n",
       "      <td>FALSE</td>\n",
       "      <td>FALSE</td>\n",
       "      <td>Keri Nelson</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id                    id_str           screen_name  \\\n",
       "0    8.16E+17  \"\"\"815745789754417152\"\"\"  \"\"\"HoustonPokeMap\"\"\"   \n",
       "1  4843621225                4843621225             kernyeahx   \n",
       "\n",
       "                    location  \\\n",
       "0                 \"\"\"Houston   \n",
       "1  Templeville town, MD, USA   \n",
       "\n",
       "                                         description  \\\n",
       "0                                              TX\"\"\"   \n",
       "1  From late 2014 Socium Marketplace will make sh...   \n",
       "\n",
       "                                     url  \\\n",
       "0  \"\"\"Rare and strong PokŽmon in Houston   \n",
       "1                                   None   \n",
       "\n",
       "                                     followers_count  \\\n",
       "0   TX. See more PokŽmon at https://t.co/dnWuDbFR...   \n",
       "1                                                  1   \n",
       "\n",
       "                   friends_count listed_count     created_at favourites_count  \\\n",
       "0  \"\"\"https://t.co/dnWuDbFRkt\"\"\"         1291              0               10   \n",
       "1                            349            0  2/1/2016 7:37               38   \n",
       "\n",
       "                               verified statuses_count   lang status  \\\n",
       "0  \"\"\"Mon Jan 02 02:25:26 +0000 2017\"\"\"              0  FALSE  78554   \n",
       "1                                 FALSE             31     en   null   \n",
       "\n",
       "  default_profile                              default_profile_image  \\\n",
       "0        \"\"\"en\"\"\"  \"{      \"\"created_at\"\": \"\"Sun Mar 12 15:44:04 ...   \n",
       "1            TRUE                                              FALSE   \n",
       "\n",
       "               has_extended_profile                                      name  \\\n",
       "0        \"\"id\"\": 840951532543737900        \"\"id_str\"\": \"\"840951532543737856\"\"   \n",
       "1                             FALSE                               Keri Nelson   \n",
       "\n",
       "                                                 bot  \n",
       "0        \"\"text\"\": \"\"[Southeast Houston] Chansey ...  \n",
       "1                                                  1  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.limit(2) .toPandas ()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "004a9751",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new = df.select('bot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cd36188",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new.limit(2) .toPandas ()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40206a90",
   "metadata": {},
   "source": [
    "You probably notice that ```df_new```, and ```df``` are different! They are pointers to two different DataFrames.\n",
    "\n",
    "Try the next command:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6b45da78",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"\"text\"\": \"\"[Southeast Houston] Chansey ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 bot\n",
       "0        \"\"text\"\": \"\"[Southeast Houston] Chansey ...\n",
       "1                                                  1"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.select('bot').limit(2) .toPandas ()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "327eac58",
   "metadata": {},
   "source": [
    "The last ```toPandas ()``` commands printed different results,\n",
    "\n",
    "## why?\n",
    "```df.select('bot')``` functionality returns pointer to a new immutable DataFrame! AHA!\n",
    "\n",
    "Let's have ```df_new``` and ```df``` point to the same DataFrame. By doing this, we release the pointer from df_new and it can be erased from memory.\n",
    "\n",
    "If we wish to have access to it again, we will need to rerun the logic. Bear that in mind for working with ```Apache Spark```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7b29049",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new = df\n",
    "df_new.limit(2) .toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b72683de",
   "metadata": {},
   "source": [
    "By the Way! ```limit(2)``` returns a pointer to a DataFrame with 2 rows.\n",
    "\n",
    "Interesting! This is what **Immutability** means!!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec434e22",
   "metadata": {},
   "source": [
    "## ✅ **Task :**  Get a feel for the data\n",
    "\n",
    "Query the DF using SQL to get a feel for the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ef114f7",
   "metadata": {},
   "source": [
    "Look at 2 records from the DataFrame to understand the values better before filter: use take() function\n",
    "\n",
    "df.take(insert an integer here)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6a8398db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(id='8.16E+17', id_str='\"\"\"815745789754417152\"\"\"', screen_name='\"\"\"HoustonPokeMap\"\"\"', location='\"\"\"Houston', description=' TX\"\"\"', url='\"\"\"Rare and strong PokŽmon in Houston', followers_count=' TX. See more PokŽmon at https://t.co/dnWuDbFRkt\"\"\"', friends_count='\"\"\"https://t.co/dnWuDbFRkt\"\"\"', listed_count='1291', created_at='0', favourites_count='10', verified='\"\"\"Mon Jan 02 02:25:26 +0000 2017\"\"\"', statuses_count='0', lang='FALSE', status='78554', default_profile='\"\"\"en\"\"\"', default_profile_image='\"{      \"\"created_at\"\": \"\"Sun Mar 12 15:44:04 +0000 2017\"\"', has_extended_profile='      \"\"id\"\": 840951532543737900', name='      \"\"id_str\"\": \"\"840951532543737856\"\"', bot='      \"\"text\"\": \"\"[Southeast Houston] Chansey (F) (IV: 73%) until 11:11:37AM at 2511 Winbern St https://t.co/HYRIyq4mF7 https://t.co/bydOOKsEEI\"\"'),\n",
       " Row(id='4843621225', id_str='4843621225', screen_name='kernyeahx', location='Templeville town, MD, USA', description='From late 2014 Socium Marketplace will make shopping for fundamental business services more simple, more cost effective and more about you.', url=None, followers_count='1', friends_count='349', listed_count='0', created_at='2/1/2016 7:37', favourites_count='38', verified='FALSE', statuses_count='31', lang='en', status='null', default_profile='TRUE', default_profile_image='FALSE', has_extended_profile='FALSE', name='Keri Nelson', bot='1')]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "df.take(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f7a8a11",
   "metadata": {},
   "source": [
    "Check out the schema stracture of the DataFrame.\n",
    "\n",
    "What are the values types? Use:\n",
    "\n",
    "```df.printSchema()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "133234bb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "410bdfc4",
   "metadata": {},
   "source": [
    "Run the next function:\n",
    "\n",
    "> ```df.limit(25) .toPandas ()```\n",
    "What happened here? toPandas function took the Spark DataFrame and translated it into Pandas DataFrame.\n",
    "\n",
    "### Run this function only on small data sets and when exploring the data.\n",
    "Otherwise, you might throw an out of memory exception!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ce2c37b",
   "metadata": {},
   "source": [
    "## Well Done! 👏👏👏\n",
    "## You just finished: Intro to PySpark\n",
    "## Next Week: Feature Engineering and Marshaling the data"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}