{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This walkthrough is distributed under a [Creative Commons Attribution 4.0 International (CC BY 4.0) License](https://creativecommons.org/licenses/by/4.0/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting data into R\n",
    "\n",
    "## Ways to get data into R\n",
    "In order to use your data in R, you must import it and turn it into an R *object*. There are many ways to get data into R.\n",
    "\n",
    "* **Manually**: You can manually create it as we did at the end of last session. To create a data.frame, use the `data.frame()` and specify your variables. \n",
    "* **Import it from a file** Below is a very incomplete list\n",
    "+ Text: TXT (`readLines()` function)\n",
    "+ Tabular data: CSV, TSV (`read.table()` function or `readr` package)\n",
    "+ Excel: XLSX (`xlsx` package)\n",
    "+ Google sheets: (`googlesheets` package)\n",
    "+ Statistics program: SPSS, SAS (`haven` package)\n",
    "+ Databases: MySQL (`RMySQL` package)\n",
    "* **Gather it from the web**: You can connect to webpages, servers, or APIs directly from within R, or you can create a data scraped from HTML webpages using the `rvest` package. \n",
    "- For example, connect to the Twitter API with the [`twitteR`](https://sites.google.com/site/miningtwitter/questions/talking-about/wordclouds/wordcloud1) package, or Altmetrics data with [`rAltmetric`](https://cran.r-project.org/web/packages/rAltmetric/vignettes/intro-to-altmetric.html), or World Bank's World Development Indicators with [`WDI`](https://cran.r-project.org/web/packages/WDI/WDI.pdf).\n",
    "\n",
    "## `readr`\n",
    "R has some base functions for reading a local data file into your R session--namely `read.table()` and `read.csv()`, but these have some idiosyncrasies that were improved upon in the `readr` package, which is installed and loaded with `tidyverse`. You can either load `tidyverse`, which will automatically load `readr`, or you can load `readr` individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "eval": "FALSE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "library(readr)\n",
    "library(purrr)\n",
    "library(dplyr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this session, we will be reading a CSV from a web connection rather than saving the data to our computer and loading it into R. However, to do that, see the below section on [Loading data from a local file](#loadlocal). \n",
    "\n",
    "To get our sample data into our R session, we will use the `read_csv()` function and connect to a CSV saved on my GitHub using the `url()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "eval": "TRUE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "books_url <- url(\"https://raw.githubusercontent.com/ciakovx/ciakovx.github.io/master/data/books.csv\")\n",
    "books <- readr::read_csv(books_url)\n",
    "books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notice a warning message telling you that because you did not specify the data type for each column, `read_csv()` parsed it automatically. For example, `LOCATION` was parsed as a `col_character()` field. You should now have an R object called `books` in the Environment pane: 5991 observations of 12 variables. We will be using this data file in the next module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data exploration\n",
    "\n",
    "After you read in the data, you want to examine it not only to make sure it was read in correctly, but also to gather some basic information about it. Here I am working with the data file that was provided to you during the webinar session. Read this file by saving it to your computer, setting your working directory, and typing the expression found in the **TRY IT YOURSELF** exercise at the end of Section 2.\n",
    "\n",
    "## Exploring dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "eval": "FALSE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# Use dim() to obtain the dimensions\n",
    "dim(books)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the column names\n",
    "names(books)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nrow() is number of rows. \n",
    "# ncol() is the number of columns\n",
    "nrow(books)\n",
    "ncol(books)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use head() and tail() to get the first and last 6 observations\n",
    "# View more by adding the n argument\n",
    "head(books)\n",
    "head(books, n = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `map()` series of function from `purrr` is a useful way of running a function on all variables in a data frame or list. Here we call `class()` on `books` using `map_chr()`, which will return a character vector of the classes for each variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "map_chr(books, class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`TOT.CHKOUT` (the total number of checkouts) was read into R as `numeric`. Everything else was read in as `character`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring variables\n",
    "\n",
    "### Dollar sign\n",
    "The dollar sign `$` is used to distinguish a specific variable (column, in Excel-speak) in a data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# print the first six book titles\n",
    "head(books$X245.ab)\n",
    "\n",
    "# print the mean number of checkouts\n",
    "mean(books$TOT.CHKOUT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `unique()`, `table()`, and `duplicated()`\n",
    "Use `unique()` to see all the distinct values in a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "unique(books$LOCATION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take that one step further with `table()` to get quick frequency counts on a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "table(books$LOCATION)\n",
    "\n",
    "# you can use it with relational operators \n",
    "# Here we find that 9 books have over 50 checkouts\n",
    "table(books$TOT.CHKOUT > 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`duplicated()` will give you the a logical vector of duplicated values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "eval": "TRUE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# The books dataset doesn't have much duplication, we'll create a new vector to test this.\n",
    "mydupes <- tibble(\"identifier\" = c(\"111\", \"222\", \"111\", \"333\", \"444\"),\n",
    "                  \"birthYear\" = c(1980, 1940, 1980, 2000, 1960))\n",
    "\n",
    "mydupes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The second 111 is duplicated\n",
    "duplicated(mydupes$identifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can put an exclamation mark before it to get non-duplicated values\n",
    "!duplicated(mydupes$identifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or run a table of duplicated values\n",
    "table(duplicated(mydupes$identifier))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# which() is also a useful function for identifying the specific element\n",
    "# in the vector that is duplicated\n",
    "which(duplicated(mydupes$identifier))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring missing values\n",
    "You may also need to know the number of missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# How many total missing values?\n",
    "sum(is.na(books))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Total missing values per column\n",
    "colSums(is.na(books))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use table() and is.na() in combination\n",
    "table(is.na(books$ISN))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return only observations that have no missing values\n",
    "booksNoNA <- na.omit(books)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**TRY IT YOURSELF**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the class() of the TOT.CHKOUT variable?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The publication date variable is represented with X008.Date.One. \n",
    "# Use the table() function to get frequency counts of values in this variable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use table() and is.na() to find out how many NA values are in the ISN variable. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use which() and is.na() to find out which rows are NA in the ISN variable. What happened? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use which() and !is.na() to find out which rows are **not** NA in ISN. \n",
    "# tip: !is.na() is the same thing as complete.cases()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use brackets [] to subset the books$ISN vector along with !is.na() to \n",
    "# include only those values that are not NA.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call summary(books$TOT.CHKOUT). \n",
    "# What can we infer when we compare the mean, median, and max?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist() will print a rudimentary histogram, which displays frequency counts. \n",
    "# Call hist(books$TOT.CHKOUT). What is this telling us?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Logical tests\n",
    "R contains a number of operators you can use to compare values. Use `help(Comparison)` to read the R help file. Note that **two equal signs** (`==`) are used for evaluating equality (because one equals sign (`=`) is used for assigning variables)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "echo": "F,",
     "eval": "TRUE",
     "id": "",
     "results": "'asis',"
    }
   },
   "outputs": [],
   "source": [
    "help(Comparison)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you need to do multiple logical tests (think Boolean logic). Use `help(Logic)` to read the help file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "echo": "F,",
     "eval": "TRUE",
     "id": "",
     "results": "'asis',"
    }
   },
   "outputs": [],
   "source": [
    "help(Logic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**TRY IT YOURSELF**\n",
    "\n",
    "1. Evaluate the following expressions yourself and consider the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 == 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 < 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 == 16 - 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 == 8 & 8 < 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 == 8 | 8 > 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 == 9 | 8 > 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "any(8 == 8, 8 == 9, 8 == 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all(8 == 8, 8 == 9, 8 == 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 %in% c(6, 7, 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "8 %in% c(5, 6, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!(8 %in% c(5, 6, 7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if(8 == 8) {print(\"eight equals eight\")}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if(8 > 16){\n",
    "  print(\"eight is greater than sixteen\")\n",
    "} else {\n",
    "  print(\"eight is less than sixteen\")\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data cleaning & transformation with dplyr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now entering the data cleaning and transforming phase. While it is possible to do much of the following using Base R functions (in other words, without loading an external package) `dplyr` makes it much easier. Like many of the most useful R packages, `dplyr` was developed by [http://hadley.nz/](Hadley Wickham), a data scientist and professor at Rice University. \n",
    "\n",
    "## Renaming variables\n",
    "It is often necessary to rename variables to make them more meaningful. If you print the names of the sample `books` dataset you can see that some of the vector names are not particularly helpful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "eval": "TRUE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# print names of the books data frame to the console\n",
    "names(books)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways to rename variables in R, but I find the `rename()` function in the `dplyr` package to be the easiest and most straightforward. The new variable name comes first. See `help(rename)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# rename the X245.ab variable. Make sure you return (<-) the output to your \n",
    "# variable, otherwise it will just print it to the console\n",
    "books <- dplyr::rename(books,\n",
    "                       title = X245.ab)\n",
    "\n",
    "# rename multiple variables at once\n",
    "books <- dplyr::rename(books,\n",
    "                       author = X245.c,\n",
    "                       callnumber = CALL...BIBLIO.,\n",
    "                       isbn = ISN,\n",
    "                       pubyear = X008.Date.One,\n",
    "                       subCollection = BCODE1,\n",
    "                       format = BCODE2)\n",
    "books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Side note: where does `X245.ab` come from? That is the MARC field 245|ab. However, because R variables cannot start with a number, R automatically inserted and X, and because pipes | are not allowed in variable names, R replaced it with a period. \n",
    "\n",
    "R does this automatically when you use `read_csv`, but sometimes you need to force it. The `clean_names()` function from `janitor` can be used to clean names of data frames. As per the help file, \"Resulting names are unique and consist only of the _ character, numbers, and letters.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# print column names\n",
    "names(books)\n",
    "\n",
    "# use clean_names() to make them easier to work with\n",
    "books <- janitor::clean_names(books)\n",
    "names(books)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recoding values\n",
    "It is often necessary to recode or reclassify values in your data. For example, in the sample dataset provided to you, the `sub_collection` (formerly `BCODE1`) and `format` (formerly `BCODE2`) variables contain single characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do this easily using the `recode()` function, also in the `dplyr` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "FALSE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# first print to the console all of the unique values you will need to recode\n",
    "unique(books$sub_collection)\n",
    "unique(books$format)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the recode function to assign them. \n",
    "# Unlike rename, the old value comes first here. \n",
    "books$sub_collection <- dplyr::recode(books$sub_collection,\n",
    "                                      \"-\" = \"general collection\",\n",
    "                                      u = \"government documents\",\n",
    "                                      r = \"reference\",\n",
    "                                      b = \"k-12 materials\",\n",
    "                                      j = \"juvenile\",\n",
    "                                      s = \"special collections\",\n",
    "                                      c = \"computer files\",\n",
    "                                      t = \"theses\",\n",
    "                                      a = \"archives\",\n",
    "                                      z = \"reserves\")\n",
    "unique(books$sub_collection)\n",
    "\n",
    "# you can do the same to recode formats:\n",
    "books$format <- dplyr::recode(books$format,\n",
    "                              a = \"book\",\n",
    "                              e = \"serial\",\n",
    "                              w = \"microform\",\n",
    "                              s = \"e-gov doc\",\n",
    "                              o = \"map\",\n",
    "                              n = \"database\",\n",
    "                              k = \"cd-rom\",\n",
    "                              m = \"image\",\n",
    "                              \"5\" = \"kit/object\",\n",
    "                              \"4\" = \"online video\")\n",
    "\n",
    "unique(books$format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subsetting dataframes\n",
    "\n",
    "### Subsetting using brackets in Base R\n",
    "In the same way we used brackets to subset vectors, we also use them to subset dataframes. However, vectors have only one direction, but dataframes have two. Therefore we have to use two values in the brackets: the first representing the row, and the second representing the column: `[[row, column]]`. \n",
    "\n",
    "When using tibbles, single brackets will return a tibble, but double brackets will return the individual vectors or values without names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# subsetting a vector \n",
    "c(\"do\", \"re\", \"mi\", \"fa\", \"so\") [1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# subsetting a data frame:\n",
    "# pull a single variable into a tibble with names\n",
    "books[5, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# return the second value in multiple columns\n",
    "books[2, c(\"title\", \"format\", \"sub_collection\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# leave the row space blank to return all rows. \n",
    "# This will give you the titles column only\n",
    "myTitles <- books[, \"title\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the relational operators (see above) to return values based on a logical condition. Below, we use `complete.cases()` to subset out the NA values. This function evaluates each row and checks if it is an NA value. If it is not, a TRUE value is returned, and vice versa. Since the column space is blank in the brackets `[row, column]`, it will return all columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "completeCallnumbers <- books[complete.cases(books$callnumber), ]\n",
    "completeCallnumbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsetting using `filter()` in the `dplyr` package\n",
    "Subsetting using brackets is important to understand, but as with other R functions, the `dplyr` package makes it much more straightforward, using the `filter()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# filter books to return only those items where the format is books\n",
    "booksOnly <- dplyr::filter(books, format == \"book\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use multiple filter conditions, \n",
    "# e.g. books to include only books with more than zero checkouts\n",
    "bookCheckouts <- dplyr::filter(books,\n",
    "                               format == \"book\",\n",
    "                               tot_chkout > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How many books have checkouts?\n",
    "nrow(bookCheckouts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divide the number of rows of books with checkouts by the number of rows of books \n",
    "# then multiply by 100 to get the percent of books with checkouts\n",
    "nrow(bookCheckouts)/nrow(booksOnly) * 100\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the str_detect() function from the stringr package to return all \n",
    "# books with the word \"Science\" in the SUBJECT variable, published after 1999\n",
    "scienceBooks <- dplyr::filter(books,\n",
    "                              format == \"book\",\n",
    "                              stringr::str_detect(subject, \"Science\"), \n",
    "                              pubyear > 1999) \n",
    "scienceBooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**TRY IT YOURSELF**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run unique(books$format) and unique(books$sub_collection) to \n",
    "# confirm the values in each of these fields\n",
    "unique(books$format)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use filter() on books to create a data frame consisting only of format serials.\n",
    "# Assign it to a value called serials\n",
    "\n",
    "# How many rows are there?\n",
    "\n",
    "# Use sum() to compute the number of total checkouts of serials\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use filter() to create a data frame consisting of format books and sub_collection juvenile materials. \n",
    "# Assign it to a value called juv\n",
    "\n",
    "# What is the average number of checkouts tot_chkout for juvenile books?\n",
    "\n",
    "# Filter the data frame to include books with between 10 and 20 checkouts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting variables\n",
    "The `select()` function allows you to keep or remove specific variables. It also provides a convenient way to reorder variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "# specify the variables you want to keep by name\n",
    "booksTitleCheckouts <- dplyr::select(books, title, tot_chkout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify the variables you want to remove with a -\n",
    "books <- dplyr::select(books, -call_item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reorder columns, combined with everything()\n",
    "booksReordered <- dplyr::select(books, title, tot_chkout, loutdate, everything())\n",
    "booksReordered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ordering data\n",
    "The `arrange()` function in the `dplyr` package allows you to sort your data by alphabetical or numerical order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA,",
     "eval": "FALSE",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "booksTitleArrange <- dplyr::arrange(books, title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use desc() to sort a variable in descending order\n",
    "booksHighestChkout <- dplyr::arrange(books, desc(tot_chkout))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# order data based on multiple variables (e.g. sort first by checkout, then by publication year)\n",
    "booksChkoutYear <- dplyr::arrange(books, desc(tot_chkout), desc(pubyear))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating new variables\n",
    "The `mutate()` function allows you to create new variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "head(books$callnumber)\n",
    "\n",
    "# use mutate to create a new column lc_class\n",
    "# use the str_sub() function from the stringr package to extract the first character of the callnumber variable (the LC Class)\n",
    "\n",
    "booksLC <- mutate(books\n",
    "                  , lc_class = stringr::str_sub(callnumber, 1, 1))\n",
    "\n",
    "head(booksLC$lc_class)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together with %>%\n",
    "\n",
    "The [Pipe Operator](https://www.datacamp.com/community/tutorials/pipe-r-tutorial) `%>%` is loaded with the `tidyverse`. It takes the output of one statement and makes it the input of the next statement. You can think of it as \"then\" in natural language. So in the following example, the `books` tibble is first loaded, then the format is filtered to include only books, then only the title and tot_chkout columns are selected, and finally the data is rearranged from most to least checkouts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "comment": "NA",
     "id": ""
    }
   },
   "outputs": [],
   "source": [
    "myBooks <- books %>%\n",
    "  dplyr::filter(format == \"book\") %>%\n",
    "  dplyr::select(title, tot_chkout) %>%\n",
    "  dplyr::arrange(desc(tot_chkout))\n",
    "myBooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TRY IT YOURSELF**\n",
    "\n",
    "\n",
    "Experiment with different combinations of the pipe operator.\n",
    "\n",
    "1. Create a data frame with these conditions:\n",
    "    + filter to include subCollection juvenile & k-12 materials and format books\n",
    "    + select only title, call number, total checkouts, and pub year\n",
    "    + arrange by total checkouts in descending order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Create a data frame with these conditions:\n",
    "    + rename the isbn column to all caps: ISBN\n",
    "    + filter out NA values in the call number column\n",
    "    + filter to include only books published after 1990\n",
    "    + arrange from oldest to newest publication year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Help with dplyr\n",
    "\n",
    "* Read more about `dplyr` at <https://dplyr.tidyverse.org/>. \n",
    "* In your console, after loading `library(dplyr)`, run `vignette(\"dplyr\")` to read an extremely helpful explanation of how to use it. \n",
    "* See the [http://r4ds.had.co.nz/transform.html](\"Data Transformation\" chapter) in Garrett Grolemund and Hadley Wickham's book *R for Data Science.*\n",
    "* Watch this Data School video: [https://www.youtube.com/watch?v=jWjqLW-u3hc](Hands-on dplyr tutorial for faster data manipulation in R.)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}