{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Scientific Python\n",
"\n",
"Python has a large number of tools available for doing data science, sometimes referred to as 'scientific Python'. \n",
"\n",
"The scientific Python ecosystem revolves around some a set of core modules, including:\n",
"\n",
"- `scipy`\n",
"- `numpy`\n",
"- `pandas`\n",
"- `matplotlib`\n",
"- `scikit-learn`\n",
"\n",
"Here we will explore the basics of these modules and what they do, starting with scipy."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" \n",
" \n",
"\n",
" \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"Scipy is an 'ecosystem', including a collection of open-source packages for scientific computing in Python.\n",
"
\n",
"\n",
"
\n",
"The scipy organization website is \n",
"here,\n",
"including a \n",
"description\n",
"of the 'ecosystem', materials for \n",
"getting started, \n",
"and extensive \n",
"tutorials.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# You can import the full scipy package, typically shortened to 'sp'\n",
"import scipy as sp\n",
"\n",
"# However, it is perhaps more common to import particular submodules\n",
"# For example, let's import the stats submodule\n",
"import scipy.stats as sts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Scipy has a broad range of functionality.\n",
"\n",
"For a simple / random example, let's use it's stats module to model flipping a coin with [Bernouilli Distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution), which is a distribution that can model a random variable that can be either 0 (call it Tails) or 1 (call it Heads). "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Let's model a fair coin - with 0.5 probability of being Heads\n",
"sts.bernoulli.rvs(0.5)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The first ten coin flips are: [1, 0, 0, 1, 0, 1, 1, 0, 1, 0]\n",
"The percent of heads from this sample is: 59.0 %\n"
]
}
],
"source": [
"# Let's flip a bunch of coins!\n",
"coin_flips = [sts.bernoulli.rvs(0.5) for ind in range(100)]\n",
"print('The first ten coin flips are: ', coin_flips[:10])\n",
"print('The percent of heads from this sample is: ', sum(coin_flips) / len(coin_flips) * 100, '%')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" \n",
" \n",
"\n",
" \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"Numpy contains an array object (for multi-dimensional data, typically of uniform type), and operations for linear algrebra and analysis on arrays. \n",
"
\n",
"\n",
"
\n",
"The numpy website is\n",
"here,\n",
"including their official \n",
"quickstart tutorial.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Note: \n",
"An array is a 'a systematic arrangement of similar objects, usually in rows and columns' (definition from [Wikipedia](https://en.wikipedia.org/wiki/Array))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Numpy is standardly imported as 'np'\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# Numpy's specialty is linear algebra and arrays of (uniform) data \n",
"\n",
"# Define some arrays\n",
"# Arrays can have different types, but all the data within an array needs to be the same type\n",
"arr_1 = np.array([1, 2, 3])\n",
"arr_2 = np.array([4, 5, 6])\n",
"bool_arr = np.array([True, False, True])\n",
"str_arr = np.array(['a', 'b', 'c'])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n"
]
}
],
"source": [
"# Note that if you try to make a mixed-data-type array, numpy won't fail, \n",
"# but it will (silently)\n",
"arr = np.array([1, 'b', True])\n",
"\n",
"# Check the type of array items\n",
"print(type(arr[0]))\n",
"print(type(arr[2]))"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True\n",
"False\n"
]
}
],
"source": [
"# These array will therefore not act like you might expect\n",
"# The last item looks like a Boolen\n",
"print(arr[2])\n",
"\n",
"# However, since it's actually a string, it won't evaluate like a Boolean\n",
"print(arr[2] == True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"For more practice with numpy, check out the collection \n",
"numpy exercises.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Pandas main data object is the DataFrame, which is a powerful data object to store mixed data types together with labels. \n",
"\n",
"Pandas dataframes also offer a large range of available methods for processing and analyzing data.\n",
"\n",
"If you are familiar with R, pandas dataframes object and approaches are quite similar to R."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Pandas is standardly imported as pd\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# Let's start with an array of data, but we also have a label for each data item\n",
"dat_1 = np.array(['London', 'Washington', 'London', 'Budapest'])\n",
"labels = ['Ada', 'Alonzo', 'Alan', 'John']"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# Pandas offers the 'Series' data object to store 1d data with axis labels\n",
"pd.Series?"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Ada London\n",
"Alonzo Washington\n",
"Alan London\n",
"John Budapest\n",
"dtype: object"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Let's make a Series with out data, and check it out\n",
"ser_1 = pd.Series(dat_1, labels)\n",
"ser_1.head()"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Ada 36\n",
"Alonzo 92\n",
"Alan 41\n",
"John 53\n",
"dtype: int64"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# If we have some different data (with the same labels) we can make another Series\n",
"dat_2 = [36, 92, 41, 53]\n",
"ser_2 = pd.Series(dat_2, labels)\n",
"\n",
"ser_2.head()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"# However, having a collection of series can quickly get quite messy\n",
"# Pandas therefore offer the dataframe - a powerful data object to store mixed type data with labels\n",
"pd.DataFrame?"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"# There are several ways to initialize a dataframe\n",
"# Here, we provide a dictionary made up of our series\n",
"df = pd.DataFrame(data={'Col-A': ser_1, 'Col-B':ser_2}, index=labels)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"London 2\n",
"Washington 1\n",
"Budapest 1\n",
"Name: Col-A, dtype: int64"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# For categorical data, we can check how many of each value there are\n",
"df['Col-A'].value_counts()"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"pandas.core.series.Series"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Note that dataframes are actually collections of Series\n",
"# When we index the df, as above, we actually pull out a Series\n",
"# So, the '.value_counts()' is actually a Series method\n",
"type(df['Col-A'])"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Col-B 55.5\n",
"dtype: float64"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Pandas also gives us tons an ways to directly explore and analyze data in dataframes\n",
"# For example, the mean for all numberic data columns\n",
"df.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"For more practice with pandas, you can try some collections of exercises, including\n",
"this one\n",
"and\n",
" this one.\n",
"
\n",
"Matplotlib is a library for plotting, in particular for 2D plots.\n",
"
\n",
"\n",
"
\n",
"The official numpy \n",
"website\n",
"includes the official\n",
"tutorial\n",
"as well as a \n",
"gallery\n",
"of examples that you can start from and modify.\n",
"
\n",
"Scikit-Learn is a packages for data mining, data analysis, and machine learning. \n",
"
\n",
"\n",
"
\n",
"Here is the official scikit-learn\n",
"website\n",
"including their official\n",
"tutorial.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"# Import sklearn\n",
"import sklearn as skl"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"# Check out module description\n",
"skl?"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"We will get into machine learning and working with sklearn later on in the tutorials."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## External Resources\n",
"\n",
"There are many, many resources to learn how to use those packages. \n",
"\n",
"The links above include the official documentation and tutorials, which are the best place to start.\n",
"\n",
"You can also search google for other resources and exercises."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
![](\"https://raw.githubusercontent.com/COGS108/Tutorials/master/img/scipy.png\")
\n",
"![](\"https://raw.githubusercontent.com/COGS108/Tutorials/master/img/numpy.png\")
\n",
"![](\"https://raw.githubusercontent.com/COGS108/Tutorials/master/img/pandas.png\")
\n",
"![](\"https://raw.githubusercontent.com/COGS108/Tutorials/master/img/matplotlib.png\")
\n",
"![](\"https://raw.githubusercontent.com/COGS108/Tutorials/master/img/sklearn.png\")
\n",
"