Click-and-Download

In the simplest case, the data you need is already on the internet in a tabular format. There are a couple of strategies here:

• Use read.csv or readr::read_csv to read the data straight into R
• Use the downloader package or curl from the shell to download the file and store a local copy, then use read_csv or something similar to read the data into R
  • Even if the file disappears from the internet, you have a local copy cached

Even in this instance, files may need cleaning and transformation when you bring them into R.

Data supplied on the web

Many times, the data that you want is not already organized into one or a few tables that you can read directly into R. More frequently, you find this data is given in the form of an API. Application Programming Interfaces (APIs) are descriptions of the kind of requests that can be made of a certain piece of software, and descriptions of the kind of answers that are returned. Many sources of data - databases, websites, services - have made all (or part) of their data available via APIs over the internet. Computer programs (“clients”) can make requests of the server, and the server will respond by sending data (or an error message). This client can be many kinds of other programs or websites, including R running from your laptop.

Install and play packages

Many common web services and APIs have been “wrapped”, i.e. R functions have been written around them which send your query to the server and format the response.

Why do we want this?

• provenance
• reproducible
• updating
• ease
• scaling

Search birds by geography

The ebird website categorizes some popular locations as “Hotspots”. These are areas where there are both lots of birds and lots of birders. Once such location is at Lincoln Park Zoo in Chicago. You can see data for this site at http://ebird.org/ebird/hotspot/L1573785

At that link, you can see a page like this:

Lincoln Park Zoo

The data already look to be organized in a data frame! rebird allows us to read these data directly into R.

The ID code for Lincoln Park Zoo is L1573785

ebirdhotspot(locID = "L1573785") %>%
  as_tibble()

## # A tibble: 24 x 11
##          lng                   locName howMany               sciName
##        <dbl>                     <chr>   <int>                 <chr>
##  1 -87.63272 Lincoln Park Zoo, Chicago       2 Nycticorax nycticorax
##  2 -87.63272 Lincoln Park Zoo, Chicago      15    Larus delawarensis
##  3 -87.63272 Lincoln Park Zoo, Chicago      15       Hirundo rustica
##  4 -87.63272 Lincoln Park Zoo, Chicago       2  Poecile atricapillus
##  5 -87.63272 Lincoln Park Zoo, Chicago       1    Sitta carolinensis
##  6 -87.63272 Lincoln Park Zoo, Chicago      15      Sturnus vulgaris
##  7 -87.63272 Lincoln Park Zoo, Chicago      13   Agelaius phoeniceus
##  8 -87.63272 Lincoln Park Zoo, Chicago       8    Quiscalus quiscula
##  9 -87.63272 Lincoln Park Zoo, Chicago       3        Spinus tristis
## 10 -87.63272 Lincoln Park Zoo, Chicago      35     Passer domesticus
## # ... with 14 more rows, and 7 more variables: obsValid <lgl>,
## #   locationPrivate <lgl>, obsDt <chr>, obsReviewed <lgl>, comName <chr>,
## #   lat <dbl>, locID <chr>

We can use the function ebirdgeo to get a list for an area. (Note that South and West are negative):

chibirds <- ebirdgeo(lat = 41.8781, lng = -87.6298)
chibirds %>%
  as_tibble() %>%
  str()

## Classes 'tbl_df', 'tbl' and 'data.frame':    137 obs. of  11 variables:
##  $ lng            : num  -87.6 -87.6 -87.6 -87.6 -87.6 ...
##  $ locName        : chr  "U. of Chicago-University Ave. from 61st to 58th" "U. of Chicago-University Ave. from 61st to 58th" "U. of Chicago-University Ave. from 61st to 58th" "U. of Chicago-University Ave. from 61st to 58th" ...
##  $ howMany        : int  5 2 2 3 1 3 4 3 7 1 ...
##  $ sciName        : chr  "Corvus brachyrhynchos" "Passer domesticus" "Chaetura pelagica" "Spizella passerina" ...
##  $ obsValid       : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
##  $ locationPrivate: logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
##  $ obsDt          : chr  "2017-07-12 08:40" "2017-07-12 08:40" "2017-07-12 08:40" "2017-07-12 08:40" ...
##  $ obsReviewed    : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ comName        : chr  "American Crow" "House Sparrow" "Chimney Swift" "Chipping Sparrow" ...
##  $ lat            : num  41.8 41.8 41.8 41.8 41.8 ...
##  $ locID          : chr  "L1242933" "L1242933" "L1242933" "L1242933" ...

Note: Check the defaults on this function. e.g. radius of circle, time of year.

We can also search by “region”, which refers to short codes which serve as common shorthands for different political units. For example, France is represented by the letters FR

frenchbirds <- ebirdregion("FR")

frenchbirds %>%
  as_tibble() %>%
  str()

## Classes 'tbl_df', 'tbl' and 'data.frame':    267 obs. of  11 variables:
##  $ lng            : num  7.75 -1.17 -1.17 -1.17 -1.17 ...
##  $ locName        : chr  "Strasbourg (cité)" "Sainte-Eulalie-en-Born " "Sainte-Eulalie-en-Born " "Sainte-Eulalie-en-Born " ...
##  $ howMany        : int  1 8 1 3 5 2 3 25 1 2 ...
##  $ sciName        : chr  "Fulica atra" "Hirundo rustica" "Troglodytes troglodytes" "Motacilla alba" ...
##  $ obsValid       : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
##  $ locationPrivate: logi  FALSE TRUE TRUE TRUE TRUE TRUE ...
##  $ obsDt          : chr  "2017-07-12 12:27" "2017-07-12 07:00" "2017-07-12 07:00" "2017-07-12 07:00" ...
##  $ obsReviewed    : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ comName        : chr  "Eurasian Coot" "Barn Swallow" "Eurasian Wren" "White Wagtail" ...
##  $ lat            : num  48.6 44.3 44.3 44.3 44.3 ...
##  $ locID          : chr  "L3737750" "L6072426" "L6072426" "L6072426" ...

Find out WHEN a bird has been seen in a certain place! Choosing a name from chibirds above (the Bald Eagle):

warbler <- ebirdgeo(species = 'Setophaga coronata', lat = 41.8781, lng = -87.6298)

warbler %>%
  as_tibble() %>%
  str()

## Classes 'tbl_df', 'tbl' and 'data.frame':    0 obs. of  0 variables

rebird knows where you are:

ebirdgeo(species = 'Setophaga coronata') %>%
  as_tibble() %>%
  knitr::kable()

## Warning: As a complete lat/long pair was not provided, your location was
## determined using your computer's public-facing IP address. This will likely
## not reflect your physical location if you are using a remote server or
## proxy.

API authentication

Many APIs require you to register for access. This allows them to track which users are submitting queries and manage demand - if you submit too many queries too quickly, you might be rate-limited and your requests de-prioritized or blocked. Always check the API access policy of the web site to determine what these limits are.

There are a few things we need to do to be able to use this package to access the geonames API:

1. go to the geonames site and register an account.
2. click here to enable the free web service
3. Tell R your geonames username. You could run the line

options(geonamesUsername = "my_user_name")

in R. However this is insecure. We don’t want to risk committing this line and pushing it to our public GitHub page! Instead, you should create a file in the same place as your .Rproj file. Name this file .Rprofile, and add

options(geonamesUsername = "my_user_name")

to that file.

# in .Rprofile
options("this_is_my_key" = XXXX)

# later, in the R script:
key <- getOption("this_is_my_key")

Using Geonames

What can we do? Get access to lots of geographical information via the various “web services”

countryInfo <- GNcountryInfo()

countryInfo %>%
  as_tibble() %>%
  str()

## Classes 'tbl_df', 'tbl' and 'data.frame':    250 obs. of  17 variables:
##  $ continent    : chr  "EU" "AS" "AS" "NA" ...
##  $ capital      : chr  "Andorra la Vella" "Abu Dhabi" "Kabul" "Saint John’s" ...
##  $ languages    : chr  "ca" "ar-AE,fa,en,hi,ur" "fa-AF,ps,uz-AF,tk" "en-AG" ...
##  $ geonameId    : chr  "3041565" "290557" "1149361" "3576396" ...
##  $ south        : chr  "42.4284925987684" "22.6333293914795" "29.377472" "16.996979" ...
##  $ isoAlpha3    : chr  "AND" "ARE" "AFG" "ATG" ...
##  $ north        : chr  "42.6560438963" "26.0841598510742" "38.483418" "17.729387" ...
##  $ fipsCode     : chr  "AN" "AE" "AF" "AC" ...
##  $ population   : chr  "84000" "4975593" "29121286" "86754" ...
##  $ east         : chr  "1.78654277783198" "56.3816604614258" "74.879448" "-61.672421" ...
##  $ isoNumeric   : chr  "020" "784" "004" "028" ...
##  $ areaInSqKm   : chr  "468.0" "82880.0" "647500.0" "443.0" ...
##  $ countryCode  : chr  "AD" "AE" "AF" "AG" ...
##  $ west         : chr  "1.40718671411128" "51.5833282470703" "60.478443" "-61.906425" ...
##  $ countryName  : chr  "Principality of Andorra" "United Arab Emirates" "Islamic Republic of Afghanistan" "Antigua and Barbuda" ...
##  $ continentName: chr  "Europe" "Asia" "Asia" "North America" ...
##  $ currencyCode : chr  "EUR" "AED" "AFN" "XCD" ...

This country info dataset is very helpful for accessing the rest of the data, because it gives us the standardized codes for country and language.

Load library and set API key

Accessing data from the Manifesto Project API requires an authentication key. You can create an account and key here. Here I store my key in .Rprofile and retrieve it using mp_setapikey().

library(manifestoR)

# retrieve API key stored in .Rprofile
mp_setapikey(key = getOption("manifesto_key"))

Retrieve the database

(mpds <- mp_maindataset())

## Connecting to Manifesto Project DB API... 
## Connecting to Manifesto Project DB API... corpus version: 2016-6

## # A tibble: 4,121 x 173
##    country countryname oecdmember eumember      edate   date party
##      <dbl>       <chr>      <dbl>    <dbl>     <date>  <dbl> <dbl>
##  1      11      Sweden          0        0 1944-09-17 194409 11220
##  2      11      Sweden          0        0 1944-09-17 194409 11320
##  3      11      Sweden          0        0 1944-09-17 194409 11420
##  4      11      Sweden          0        0 1944-09-17 194409 11620
##  5      11      Sweden          0        0 1944-09-17 194409 11810
##  6      11      Sweden          0        0 1948-09-19 194809 11220
##  7      11      Sweden          0        0 1948-09-19 194809 11320
##  8      11      Sweden          0        0 1948-09-19 194809 11420
##  9      11      Sweden          0        0 1948-09-19 194809 11620
## 10      11      Sweden          0        0 1948-09-19 194809 11810
## # ... with 4,111 more rows, and 166 more variables: partyname <chr>,
## #   partyabbrev <chr>, parfam <dbl>, coderid <dbl>, manual <dbl>,
## #   coderyear <dbl>, testresult <dbl>, testeditsim <dbl>, pervote <dbl>,
## #   voteest <dbl>, presvote <dbl>, absseat <dbl>, totseats <dbl>,
## #   progtype <dbl>, datasetorigin <dbl>, corpusversion <chr>, total <dbl>,
## #   peruncod <dbl>, per101 <dbl>, per102 <dbl>, per103 <dbl>,
## #   per104 <dbl>, per105 <dbl>, per106 <dbl>, per107 <dbl>, per108 <dbl>,
## #   per109 <dbl>, per110 <dbl>, per201 <dbl>, per202 <dbl>, per203 <dbl>,
## #   per204 <dbl>, per301 <dbl>, per302 <dbl>, per303 <dbl>, per304 <dbl>,
## #   per305 <dbl>, per401 <dbl>, per402 <dbl>, per403 <dbl>, per404 <dbl>,
## #   per405 <dbl>, per406 <dbl>, per407 <dbl>, per408 <dbl>, per409 <dbl>,
## #   per410 <dbl>, per411 <dbl>, per412 <dbl>, per413 <dbl>, per414 <dbl>,
## #   per415 <dbl>, per416 <dbl>, per501 <dbl>, per502 <dbl>, per503 <dbl>,
## #   per504 <dbl>, per505 <dbl>, per506 <dbl>, per507 <dbl>, per601 <dbl>,
## #   per602 <dbl>, per603 <dbl>, per604 <dbl>, per605 <dbl>, per606 <dbl>,
## #   per607 <dbl>, per608 <dbl>, per701 <dbl>, per702 <dbl>, per703 <dbl>,
## #   per704 <dbl>, per705 <dbl>, per706 <dbl>, per1011 <dbl>,
## #   per1012 <dbl>, per1013 <dbl>, per1014 <dbl>, per1015 <dbl>,
## #   per1016 <dbl>, per1021 <dbl>, per1022 <dbl>, per1023 <dbl>,
## #   per1024 <dbl>, per1025 <dbl>, per1026 <dbl>, per1031 <dbl>,
## #   per1032 <dbl>, per1033 <dbl>, per2021 <dbl>, per2022 <dbl>,
## #   per2023 <dbl>, per2031 <dbl>, per2032 <dbl>, per2033 <dbl>,
## #   per2041 <dbl>, per3011 <dbl>, per3051 <dbl>, per3052 <dbl>,
## #   per3053 <dbl>, ...

mp_maindataset() includes a data frame describing each manifesto included in the database. You can use this database for some exploratory data analysis. For instance, how many manifestos have been published by each political party in Sweden?

mpds %>%
  filter(countryname == "Sweden") %>%
  count(partyname) %>%
  ggplot(aes(fct_reorder(partyname, n), n)) +
  geom_col() +
  labs(title = "Political manifestos published in Sweden",
       x = NULL,
       y = "Total (1948-present)") +
  coord_flip()

Or we can use scaling functions to identify each party manifesto on an ideological dimension. For example, how have the Democratic and Republican Party manifestos in the United States changed over time?

mpds %>%
  filter(party == 61320 | party == 61620) %>%
  mutate(ideo = mp_scale(.)) %>%
  select(partyname, edate, ideo) %>%
  ggplot(aes(edate, ideo, color = partyname)) +
  geom_line() +
  scale_color_manual(values = c("blue", "red")) +
  labs(title = "Ideological scaling of major US political parties",
       x = "Year",
       y = "Ideological position",
       color = NULL) +
  theme(legend.position = "bottom")

Download manifestos

mp_corpus() can be used to download the original manifestos as full text documents stored as a corpus. Once you obtain the corpus, you can perform text analysis. As an example, let’s compare the most common words in the Democratic and Republican Party manifestos from the 2012 U.S. presidential election:

# download documents
(docs <- mp_corpus(countryname == "United States" & edate > as.Date("2012-01-01")))

## Connecting to Manifesto Project DB API... 
## Connecting to Manifesto Project DB API... corpus version: 2017-1 
## Connecting to Manifesto Project DB API... 
## Connecting to Manifesto Project DB API... corpus version: 2017-1 
## Connecting to Manifesto Project DB API... corpus version: 2017-1 
## Connecting to Manifesto Project DB API... corpus version: 2017-1

## <<ManifestoCorpus>>
## Metadata:  corpus specific: 0, document level (indexed): 0
## Content:  documents: 2

# generate wordcloud of most common terms
docs %>%
  tidy() %>%
  mutate(party = factor(party, levels = c(61320, 61620),
                        labels = c("Democratic Party", "Republican Party"))) %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words) %>%
  count(party, word, sort = TRUE) %>%
  na.omit() %>%
  reshape2::acast(word ~ party, value.var = "n", fill = 0) %>%
  comparison.cloud(max.words = 200)