{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Petite Pandas Data Analysis using Pandas and NumPy\n",
    "> A series of lessons based on the utilization of Pandas and NumPy to read and create interesting things with files and data. \n",
    "- toc: true"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Analysis\n",
    "Predictive analysis is the use of _________, data mining, and machine learning techniques to analyze current and historical data in order to make predictions about future events or behaviors. It involves identifying ________ and trends in data, and then using that information to forecast what is likely to happen in the future.\n",
    "\n",
    "Predictive analysis is used in a wide range of applications, from forecasting sales and demand, to predicting customer behavior, to detecting fraudulent transactions. It involves collecting and analyzing data from a variety of sources, including historical data, customer data, financial data, and social media data, among others.\n",
    "\n",
    "The process of predictive analysis typically involves the following steps:\n",
    "1. Defining the problem and identifying the relevant data sources\n",
    "2. ______________________\n",
    "3. Exploring and analyzing the data to identify patterns and trends\n",
    "4. Selecting an appropriate model or algorithm to use for predictions\n",
    "5. Training and validating the model using historical data\n",
    "6. ______________________\n",
    "7. Monitoring and evaluating the performance of the model over time\n",
    "\n",
    "Predictive analysis can help organizations make more informed decisions, improve efficiency, and gain a competitive advantage by leveraging insights from data.\n",
    "\n",
    "It is most commonly used in __________, where workers try to predict which products would be most popular and try to advertise those products as much as possible, and also ___________, where algorithms analyze patterns and reveal prerequisites for diseases and suggest preventive treatment, predict the results of various treatments and choose the best option for each patient individually, and predict disease outbreaks and epidemics."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Intro to NumPy and the features it consists\n",
    "\n",
    "Numpy, by definition, is the fundamental package for ____________ in Python which can be used to perform __________________, provide ______________, and makes ____________ much ________. Numpy is very important and useful when it comes to data analysis, as it can easily use its features to complete and perform any mathematical operation, as well as analyze data files. \n",
    "\n",
    "If you don't already have numpy installed, you can do so using ```conda install numpy``` or ```pip install numpy```\n",
    "\n",
    "Once that is complete, to import numpy in your code, all you must do is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Using NumPy to create arrays\n",
    "An array is the central ______________ of the NumPy library. They are used as ______________ which are able to store more than one item at the same time. Using the function ```np.array``` is used to create an array, in which you can create multidimensional arrays. \n",
    "\n",
    "Shown below is how to create a 1D array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "print(a) \n",
    "# this creates a 1D array"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How could you create a 3D array based on knowing how to make a 1D array?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create 3D array here"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arrays can be printed in different ways, especially a more readable format. As we have seen, arrays are printed in rows and columns, but we can change that by using the ```reshape``` function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
    "print(c.reshape(1, 9)) # organizes it all in a single line of output"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code segment below, we can also specially select certain rows and columns from the array to further analyze selective data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(c[1:, :2])\n",
    "# the 1: means \"start at row 1 and select all the remaining rows\"\n",
    "# the :2 means \"select the first two columns\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Basic array operations\n",
    "\n",
    "One of the most basic operations that can be performed on arrays is _______________. With numpy, it is very easy to perform arithmetic operations on arrays. You can ________, _________, __________ and __________ arrays, just like you would with regular numbers. When performing these operations, numpy applies the operation element-wise, meaning that it performs the operation on each element in the array separately. This makes it easy to perform operations on large amounts of data quickly and efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "print(a + b) # adds each value based on the column the integer is in\n",
    "print(a - b) # subtracts each value based on the column the integer is in\n",
    "print(a * b) # multiplies each value based on the column the integer is in\n",
    "print(a / b) # divides each value based on the column the integer is in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = np.exp(b)\n",
    "e = np.sqrt(b)\n",
    "print(d)\n",
    "print(e)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the knowledge of how to use more advanced mathematical expressions than the basic 4 mathematical operations such as exponent and square root, now can you code how to calculate the 3 main trig expressions (sin, cos, tan), natural log, and log10 of a 1D array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate sin\n",
    "# calculate cos\n",
    "# calculate tan\n",
    "# calculate natural log\n",
    "# calculate log10"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Data analysis using numpy\n",
    "Numpy provides a convenient and powerful way to perform data analysis tasks on ____________. One of the most common tasks in data analysis is finding the ________, ________, and __________ of a dataset. Numpy provides functions to perform these operations quickly and easily. The mean function calculates the average value of the data, while the median function calculates the middle value in the data. The standard deviation function calculates how spread out the data is from the mean. Additionally, numpy provides functions to find the minimum and maximum values in the data. These functions are very useful for gaining insight into the properties of large datasets and can be used for a wide range of data analysis tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.array([2, 5, 12, 13, 19])\n",
    "print(np.mean(data)) # finds the mean of the dataset\n",
    "print(np.median(data)) # finds the median of the dataset\n",
    "print(np.std(data)) # finds the standard deviation of the dataset\n",
    "print(np.min(data)) # finds the min of the dataset\n",
    "print(np.max(data)) # finds the max of the dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now from learning this, can you find a different way from how we can solve the sum or products of a dataset other than how we learned before?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a different way of solving the sum or products of a dataset from what we learned above"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy also has the ability to handle CSV files, which are commonly used to store and exchange large datasets. By importing CSV files into numpy arrays, we can easily perform complex operations and analysis on the data, making numpy an essential tool for data scientists and researchers.\n",
    "\n",
    "```genfromtxt``` and ```loadtxt``` are two functions in the numpy library that can be used to read data from text files, including CSV files."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```genfromtxt``` is a more advanced function that can be used to read text files that have more complex structures, including CSV files. ```genfromtxt``` can handle files that have missing or invalid data, or files that have columns of different data types. It can also be used to skip header lines or to read only specific columns from the file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "padres = np.genfromtxt('files/padres.csv', delimiter=',', dtype=str, encoding='utf-8')\n",
    "# delimiter indicates that the data is separated into columns which is distinguished by commas\n",
    "# genfromtxt is used to read the csv file itself\n",
    "# dtype is used to have numpy automatically detect the data type in the csv file\n",
    "\n",
    "print(padres)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```loadtxt``` is a simpler function that can be used to read simple text files that have a regular structure, such as files that have only one type of data (such as all integers or all floats). ```loadtxt``` can be faster than ```genfromtxt``` because it assumes that the data in the file is well-structured and can be easily parsed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "padres = np.loadtxt('files/padres.csv', delimiter=',', dtype=str, encoding='utf-8')\n",
    "print(padres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in padres:\n",
    "    print(\",\".join(i))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas\n",
    "### What is Pandas\n",
    "Pandas is a Python library used for working with data sets. A python library is something  It has functions for analyzing, cleaning, exploring, and manipulating data.\n",
    "\n",
    "### Why Use Pandas?\n",
    "Pandas allows us to analyze big data and make conclusions based on statistical theories. Pandas can clean messy data sets, and make them readable and relevant. Also it is a part of data analysis, and data manipulation.\n",
    "\n",
    "### What Can Pandas Do?\n",
    "Pandas gives you answers about the data. Like:\n",
    "- Is there a correlation between two or more columns?\n",
    "- What is average value\n",
    "- Max value\n",
    "- Min value\n",
    "- How to load data \n",
    "- Delete data \n",
    "- Sort Data.\n",
    "\n",
    "Pandas are also able to delete rows that are not relevant, or contains wrong values, like empty or NULL values. This is called cleaning the data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basics of Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# What this does is it calls the python pandas library and this code segment is needed whenever incorporating pandas."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DICTIONARIES AND DATASETS\n",
    "- One way you are able to manipulate a pandas data set is by creating a dictionary and calling it as seen with the dict data 1 and pd.dataframe which is a way to print the set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data1 = {\n",
    "  'teams': [\"BARCA\", \"REAL\", \"ATLETICO\"],\n",
    "  'standings': [1, 2, 3]\n",
    "}\n",
    "\n",
    "myvar = pd.DataFrame(data1)\n",
    "\n",
    "print(myvar)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing and manipulaton of data through lists.\n",
    "- With pandas you can also organize the data which is one of its biggest perks, we call this indexing, this is when we define the first column in a data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is an example using lists and an index.\n",
    "import pandas as pd \n",
    "\n",
    "score = [5/5, 5/5, 1/5]\n",
    "\n",
    "myvar = pd.Series(score, index = [\"math\", \"science\", \"pe\"])\n",
    "\n",
    "print(myvar)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Classes \n",
    "Within pandas the library consits of a lot of functions which allow you to manipulate datasets in lists dictionsaries and csv files here are some of the ones we are going to cover (hint: take notes on these)\n",
    "- Series\n",
    "- Index\n",
    "- PeriodIndex\n",
    "- DataframeGroupedBy\n",
    "- Categorical\n",
    "- Time Stamp\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  PeriodIndex \n",
    "- This allows for a way to repeat data over time that it occurs as seen from january 2022 to december 2023. You can use Y for years, M for months, and D for days.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "time = pd.period_range('2022-01', '2022-12', freq='M')\n",
    "\n",
    "\n",
    "print(time)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now implement a way to show a period index from June 2022 to July 2023 in days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use period index to show - in days - June 2022 to 2023"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataframe Grouped By \n",
    "- This allows for you to organize your data and calculate the different functions such as\n",
    "- count(): returns the number of non-null values in each group.\n",
    "- sum(): returns the sum of values in each group.\n",
    "- mean(): returns the mean of values in each group.\n",
    "- min(): returns the minimum value in each group.\n",
    "- max(): returns the maximum value in each group.\n",
    "- median(): returns the median of values in each group.\n",
    "- var(): returns the variance of values in each group.\n",
    "- agg(): applies one or more functions to each group and returns a new DataFrame with the results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = {\n",
    "    'Category': ['E', 'F', 'E', 'F', 'E', 'F', 'E', 'F'],\n",
    "    'Value': [100, 250, 156, 255, 240, 303, 253, 3014]\n",
    "}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "\n",
    "grouped = df.groupby('Category').#GUESS WHAT THIS WOULD BE IF WE WERE LOOKING FOR COMBINED TOTALS!()\n",
    "\n",
    "print(grouped)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical \n",
    "- This sets up a category for something and puts it within the categories and allows for better orginzation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "colors = pd.Categorical(['yellow', 'orange', 'blue', 'yellow', 'orange'], categories=['yellow', 'orange', 'blue'])\n",
    "\n",
    "print(colors)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timestamp Class\n",
    "- This allows to display a single time which can be useful when working with datasets that deal with time allowing you to manipulate the time you do something and how you do it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "timing = pd.Timestamp('2023-02-05 02:00:00')\n",
    "\n",
    "print(#WHAT WOULD THIS BE)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSV FILES!\n",
    "- A csv file contains data and within pandas you are able to call the function and you are able to manipulate the data with the certain data classes talked about above. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Name, Position, Average, HR, RBI, OPS, JerseyNumber\n",
    "- Manny Machado, 3B, .298, 32, 102, .897, 13\n",
    "- Tatis Jr, RF, .281, 42, 97, .975, 23\n",
    "- Juan Soto, LF, .242, 27, 62, .853, 22\n",
    "- Xanger Bogaerts, SS, .307, 15, 73, .833, 2\n",
    "- Nelson Cruz, DH, .234, 10, 64, .651, 32\n",
    "- Matt Carpenter, DH, .305, 15, 37, 1.138, 14\n",
    "- Cronezone, 1B, .239, 17, 88, .722, 9\n",
    "- Ha-Seong Kim, 2B, .251, 11, 59, .708, 7\n",
    "- Trent Grisham, CF, .184, 17, 53, .626, 1\n",
    "- Luis Campusano, C, .250, 1, 5, .593, 12\n",
    "- Austin Nola, C, .251, 4, 40, .649, 26\n",
    "- Jose Azocar, OF, .257, 0, 10, .630, 28\n",
    "\n",
    "QUESTION: WHAT DO YOU GUYS THINK THE INDEX FOR THIS WOULD BE?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you explain what is going on in this code segment below. (hint: define what ascending= false means, and df. head means)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#read csv and sort 'Duration' largest to smallest\n",
    "df = pd.read_csv('files/padres.csv').sort_values(by=['Name'], ascending=False)\n",
    "\n",
    "print(\"--Duration Top 10---------\")\n",
    "print(df.head(10))\n",
    "\n",
    "print(\"--Duration Bottom 10------\")\n",
    "print(df.tail(10))\n",
    "print(', '.join(df.tail(10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "df = pd.read_csv(\"./files/housing.csv\")\n",
    "\n",
    "\n",
    "mode_total_rooms = df['total_rooms'].mode()\n",
    "\n",
    "\n",
    "print(f\"The mode of the 'total_rooms' column is: {mode_total_rooms}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"./files/housing.csv\")\n",
    "\n",
    "\n",
    "grouped_df = df.groupby('total_rooms')\n",
    "\n",
    "\n",
    "agg_df = grouped_df.agg({'total_rooms': 'sum', 'population': 'mean', 'longitude': 'count'})\n",
    "\n",
    "# WHAT DO YOU GUYS THINK df.agg means in context of pandas and what does it stand for.\n",
    "\n",
    "\n",
    "print(agg_df)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Our Frontend Data Analysis Project\n",
    "[Link](https://paravsalaniwal.github.io/T3Project/DataAnalysisProject/)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Popcorn Hacks\n",
    "- Complete fill in the blanks for Predictive Analysis Numpy\n",
    "- Takes notes on Panda where it asks you to\n",
    "- Complete code segment tasks in Panda and Numpy\n",
    "\n",
    "# Main Hack\n",
    "- Make a data file - content is up to you, just make sure there are integer values - and print\n",
    "- Run Panda and Numpy commands\n",
    "    - Panda:\n",
    "        - Find Min and Max values\n",
    "        - Sort in order - can be order of least to greatest or vice versa\n",
    "        - Create a smaller dataframe and merge it with your data file\n",
    "    - Numpy:\n",
    "        -  Random number generation\n",
    "        - create a multi-dimensional array (multiple elements)\n",
    "        - create an array with linearly spaced intervals between values\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grading\n",
    "The grading will be binary - all or nothing; no partial credit\n",
    "- 0.3 for all the popcorn hacks\n",
    "- 0.6 for the main hack - CSV file\n",
    "- 0.1 for going above and beyond in the main hack"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}