{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Undocument APIs\n",
    "\n",
    "[**Neal Caren**](mailto:neal.caren@gmail.com)  \n",
    "University of North Carolina, Chapel Hill"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Social scientists are often interested in collecting massive amounts of digital data from specific websites. When you are fortunate, someone else has already scrapped it for you and has released in on [GitHub](https://github.com) or as a [Kaggle dataset](https://www.kaggle.com/datasets). Sometimes the website provides an [API](https://en.wikipedia.org/wiki/Web_API), an interface for collecting the data in a systematic way that is usually reasonably well-document. The most prominent of these for social scientists is probably [Twitter](https://developer.twitter.com/en/docs.html), who have enabled hundreds of academic studies by making available information about posts and users. \n",
    "\n",
    "Other times, an API exists and is being used by the website, but information about is not publicly available. Most website searches, for example, involve using an internal web API. Your search term, along with other relevant parameters, is used to extract the corresponding results from a dataset and then displayed on a web page. While we see the data as part of a web page, the data is often transmitted in a different, more research-friend format, usually in a [JSON](https://en.wikipedia.org/wiki/JSON) format. Using an undocumented API allows you to systematically collect the data without the parsing the HTML of each page. \n",
    "\n",
    "Below, I walk through the steps I recently used  when trying to gather Fox News Opinion articles. In this case, the API doesn't return the full text of the articles, but, as is often the case, returns all the metadata about the article, including the article URL for subsequent scraping.\n",
    "\n",
    "I use Python, but the overall logic is similar for other languages. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I begin by enabling the developer toolbar on my browser. In Safari, this is under **Preferences->Advanced** and then clicking **Show Develop Menu in Menu Bar**. The process is similar in Chrome. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I visited the front page.\n",
    "![frontpage](images/frontpage.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I clicked on **Opinion**, the section I wanted to scrape.\n",
    "![opinion](images/opinion.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I scrolled down the page and found a list of articles with a **Show More** button. This type of button, which leads to the next set of results, is a key component for understanding how the webpage is structured.  \n",
    "![more.png](images/more.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, **Show More** does not load a new page but it does expand the number of results shown on the existing page. If you copy the link associated with this button (`https://www.foxnews.com/opinion#`) and paste it in a new browser window, it merely displays the first set of results, so that is a dead end for uncovering an API. \n",
    "\n",
    "To be confident that there might be a decent number of articles available, I clicked the **Show More** button several times. Each time it loaded more articles.\n",
    "![more.png](images/more2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lots of data is passed between my computer and the Foxnews website with each click. Additional information about these streams can be revealed through the **Develop** menu and then the **Show Page Resources** option.\n",
    "\n",
    "![develop.png](images/develop.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This defaults to showing the page's HTML code.\n",
    "![html.png](images/html.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To look for signs of an API, I click on the **Network** tab. Results are sometimes already listed, but I start clean by using the trash icon on the far right. With the **Network** tab visible, I then click **Show More** on the web page. Each resource exchanged between my computer and various servers are now displayed. When the action stops, I sort the list by size. \n",
    "![network.png](images/network1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I now review each of the items for something that looks like the results of search API, a plain text file with the search results. \n",
    "\n",
    "Usually, the top of the list displays the images that the page retrieved. Images usually have a .png or .jpg extension. Selecting the first item in the results, an JPG with a name full of numbers, confirms that it is a picture. \n",
    "\n",
    "![image.png](images/network_image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The name of the second item, **article-search** has more promise. The **Preview** tab shows that this is a JSON object that appears to be a list of the articles that were returned after the **Show More** button was pressed. Bingo!\n",
    "\n",
    "![image.png](images/network_search.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Header** tab reveals the specific URL that returned this JSON. \n",
    "![image.png](images/network_header.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The URL has all the signs of an undocumented API. First, it contains \"API\" as part of the string. Second, it includes a series of search parameters, such as `isCategory` and `size`.  Now I copy the URL and paste and save it as a Python string. I split the string over several lines to view it more clearly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = ('https://www.foxnews.com/api/article-search?'\n",
    "       'isCategory=true&isTag=false&isKeyword=false&'\n",
    "       'isFixed=false&isFeedUrl=false&searchSelected=opinion&'\n",
    "       'contentTypes=%7B%22interactive%22:true,%22slideshow%22:true,%22video%22:false,%22article%22:true%7D&'\n",
    "       'size=11&offset=30')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to see if Python can access the API with a straightforward command. Some APIs require confirmation that the search is originating from original websites, while others do not enforce that. Figuring out the right way to programmatically access the website is a process of trial and error. I use the `requests` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "r = requests.get(url)\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A status code of 200 means that something was returned. \n",
    "\n",
    "Since it looked like a JSON object when viewed in the browser, I take advantage of the `requests` JSON decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'category': {'name': \"Laura Ingraham's Monologue\",\n",
       "   'url': '/category/shows/ingraham-angle/transcript/lauras-monologue'},\n",
       "  'description': \"Another day, another wacky liberal steps on to the 2020 field. And although it wasn't a real surprise when Vermont socialist, Bernie Sanders decided to formally enter the race, it's always stunning to hear what the Muppet-like leftie actually believes.\",\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/20/348/196/694940094001_6004324363001_6004323407001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-20T08:39:06-05:00',\n",
       "  'publicationDate': '2019-02-20T08:39:06-05:00',\n",
       "  'title': 'Laura Ingraham: Meet the candidates – Every day another wacky liberal steps on to the 2020 field',\n",
       "  'url': '/opinion/laura-ingraham-meet-the-candidates-every-day-another-wacky-liberal-steps-on-to-the-2020-field'},\n",
       " {'category': {'name': \"Tucker Carlson's Monologue\",\n",
       "   'url': '/category/shows/tucker-carlson-tonight/transcript/tuckers-monologue'},\n",
       "  'description': 'Identity politics is a tactic designed to prevent conversations dangerous to the ruling class, obviously. It is also a way for some of the most privileged people in America to recast themselves as victims.\\xa0',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/20/348/196/694940094001_6004284786001_6004281395001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-20T07:13:09-05:00',\n",
       "  'publicationDate': '2019-02-20T07:13:09-05:00',\n",
       "  'title': \"Tucker Carlson: Identity politics is a scam and it's not so different from the one Jussie Smollett just pulled\",\n",
       "  'url': '/opinion/tucker-carlson-identity-politics-is-a-scam-and-its-not-so-different-from-the-one-jussie-smollett-just-pulled'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'As this socialist tragedy unfolds in Venezuela, my questions is this: Do the American people know how deadly socialism is, and how real the threat of it is here in the United States?',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/19/348/196/694940094001_6003795855001_6003798058001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-20T04:00:51-05:00',\n",
       "  'publicationDate': '2019-02-20T04:00:51-05:00',\n",
       "  'title': 'Charlie Kirk: Americans must wake up and fight against the socialist movement that’s already well underway',\n",
       "  'url': '/opinion/charlie-kirk-americans-must-wake-up-and-fight-against-the-socialist-movement-thats-already-well-underway'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'I lost a friend last week. She was 42 years old, and it was a drug overdose. She was beautiful, smart, kind. I gave a eulogy at her funeral. Peering at her loved ones and our mutual friends through a wall of tears, I said she had been a burst of color in a gray world. She was a real person, and now she’s gone.',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/01/16/348/196/694940094001_5989963110001_5989957792001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-20T04:00:43-05:00',\n",
       "  'publicationDate': '2019-02-20T04:00:43-05:00',\n",
       "  'title': 'The opioid epidemic keeps killing my friends',\n",
       "  'url': '/opinion/the-opioid-epidemic-keeps-killing-my-friends'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'Trump has gotten Pyongyang to come to Hanoi. Now the challenge is for Pyongyang to become Hanoi.',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/static.foxnews.com/foxnews.com/content/uploads/2019/02/348/196/NKO020619-e1549447177848.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-20T04:00:24-05:00',\n",
       "  'publicationDate': '2019-02-20T04:00:24-05:00',\n",
       "  'title': \"Trump's big challenge at the upcoming North Korea summit\",\n",
       "  'url': '/opinion/trumps-big-challenge-at-the-upcoming-north-korea-summit'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'Rep. Ilhan Omar is a leading critic of Israel in Congress and pretends this has nothing to do with her feelings about the Jewish people.',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/16/348/196/694940094001_6003003306001_6003006060001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-20T04:00:12-05:00',\n",
       "  'publicationDate': '2019-02-20T04:00:12-05:00',\n",
       "  'title': 'Dems shouldn’t support anti-Israel Rep. Ilhan Omar – She doesn’t speak for me and other American Muslims',\n",
       "  'url': '/opinion/dems-shouldnt-support-anti-israel-rep-ilhan-omar-she-doesnt-speak-for-me-and-other-american-muslims'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'Democratic socialist Bernie Sanders is running for the Democratic presidential nomination again, as the far left struggles with moderates for control of the party.',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/19/348/196/694940094001_6004224104001_6004219774001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-19T19:53:25-05:00',\n",
       "  'publicationDate': '2019-02-19T19:53:25-05:00',\n",
       "  'title': 'Bernie Sanders’ presidential candidacy will further fracture Democratic Party',\n",
       "  'url': '/opinion/bernie-sanders-presidential-candidacy-will-further-fracture-democratic-party'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'When someone invokes the ugliness of hate crimes for fraudulent ends, they disrespect every real victim of every lynching and racist attack down through the years.',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/19/348/196/694940094001_6003636660001_6003633662001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-19T13:41:44-05:00',\n",
       "  'publicationDate': '2019-02-19T13:41:44-05:00',\n",
       "  'title': 'Geraldo Rivera: If Jussie Smollett’s attack was a hoax, victims of real hate crime will pay the price',\n",
       "  'url': '/opinion/geraldo-rivera-if-jussie-smolletts-attack-was-a-hoax-victims-of-real-hate-crime-will-pay-the-price'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'Did you notice how, during the Andrew McCabe interview with \"60 Minutes,\" the former FBI acting director kept talking about “articulable facts?”',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/19/348/196/694940094001_6003910071001_6003906789001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-19T12:07:39-05:00',\n",
       "  'publicationDate': '2019-02-19T12:07:39-05:00',\n",
       "  'title': \"Dan Bongino: Like Comey and Brennan, McCabe finds impetus of Trump probe 'inarticulable'\",\n",
       "  'url': '/opinion/dan-bongino-like-comey-and-brennan-mccabe-finds-impetus-of-trump-probe-inarticulable'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': \"China and the West are locked in a struggle for control of the future of communication technology. If America doesn't take decisive action soon, we'll lose.\",\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/static.foxnews.com/foxnews.com/content/uploads/2018/09/348/196/ap18143328116561.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-19T11:58:24-05:00',\n",
       "  'publicationDate': '2019-02-19T11:58:24-05:00',\n",
       "  'title': 'Newt Gingrich:  America in race against China - and the clock - to control future of tech',\n",
       "  'url': '/opinion/newt-gingrich-america-in-race-against-china-and-the-clock-to-control-future-of-tech'},\n",
       " {'category': {'name': 'OPINION', 'url': '/category/opinion'},\n",
       "  'description': 'Many experts on the Middle East never thought they’d see the day the foreign ministers of Saudi Arabia, Bahrain, Oman and the United Arab Emirates would be sitting collegially with the prime minister of Israel.',\n",
       "  'duration': '',\n",
       "  'imageUrl': 'https://a57.foxnews.com/media2.foxnews.com/BrightCove/694940094001/2019/02/11/348/196/694940094001_6000890885001_6000881936001-vs.jpg?ve=1&tl=1',\n",
       "  'isBreaking': False,\n",
       "  'isLive': False,\n",
       "  'lastPublishedDate': '2019-02-19T10:45:37-05:00',\n",
       "  'publicationDate': '2019-02-19T10:45:37-05:00',\n",
       "  'title': 'Iran should return to ancient teachings of Islam, a religion hijacked in 1979 by a dictatorial regime',\n",
       "  'url': '/opinion/iran-should-return-to-ancient-teachings-of-islam-a-religion-hijacked-in-1979-by-a-dictatorial-regime'}]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best way to turn a JSON into usable a format is with the `pandas` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(r.json())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'name': 'Laura Ingraham's Monologue', 'url': ...</td>\n",
       "      <td>Another day, another wacky liberal steps on to...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>Laura Ingraham: Meet the candidates – Every da...</td>\n",
       "      <td>/opinion/laura-ingraham-meet-the-candidates-ev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'name': 'Tucker Carlson's Monologue', 'url': ...</td>\n",
       "      <td>Identity politics is a tactic designed to prev...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>Tucker Carlson: Identity politics is a scam an...</td>\n",
       "      <td>/opinion/tucker-carlson-identity-politics-is-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'name': 'OPINION', 'url': '/category/opinion'}</td>\n",
       "      <td>As this socialist tragedy unfolds in Venezuela...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>Charlie Kirk: Americans must wake up and fight...</td>\n",
       "      <td>/opinion/charlie-kirk-americans-must-wake-up-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'name': 'OPINION', 'url': '/category/opinion'}</td>\n",
       "      <td>I lost a friend last week. She was 42 years ol...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>The opioid epidemic keeps killing my friends</td>\n",
       "      <td>/opinion/the-opioid-epidemic-keeps-killing-my-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{'name': 'OPINION', 'url': '/category/opinion'}</td>\n",
       "      <td>Trump has gotten Pyongyang to come to Hanoi. N...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>Trump's big challenge at the upcoming North Ko...</td>\n",
       "      <td>/opinion/trumps-big-challenge-at-the-upcoming-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            category  \\\n",
       "0  {'name': 'Laura Ingraham's Monologue', 'url': ...   \n",
       "1  {'name': 'Tucker Carlson's Monologue', 'url': ...   \n",
       "2    {'name': 'OPINION', 'url': '/category/opinion'}   \n",
       "3    {'name': 'OPINION', 'url': '/category/opinion'}   \n",
       "4    {'name': 'OPINION', 'url': '/category/opinion'}   \n",
       "\n",
       "                                         description duration  \\\n",
       "0  Another day, another wacky liberal steps on to...            \n",
       "1  Identity politics is a tactic designed to prev...            \n",
       "2  As this socialist tragedy unfolds in Venezuela...            \n",
       "3  I lost a friend last week. She was 42 years ol...            \n",
       "4  Trump has gotten Pyongyang to come to Hanoi. N...            \n",
       "\n",
       "                                            imageUrl  isBreaking  isLive  \\\n",
       "0  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "\n",
       "           lastPublishedDate            publicationDate  \\\n",
       "0  2019-02-20T08:39:06-05:00  2019-02-20T08:39:06-05:00   \n",
       "1  2019-02-20T07:13:09-05:00  2019-02-20T07:13:09-05:00   \n",
       "2  2019-02-20T04:00:51-05:00  2019-02-20T04:00:51-05:00   \n",
       "3  2019-02-20T04:00:43-05:00  2019-02-20T04:00:43-05:00   \n",
       "4  2019-02-20T04:00:24-05:00  2019-02-20T04:00:24-05:00   \n",
       "\n",
       "                                               title  \\\n",
       "0  Laura Ingraham: Meet the candidates – Every da...   \n",
       "1  Tucker Carlson: Identity politics is a scam an...   \n",
       "2  Charlie Kirk: Americans must wake up and fight...   \n",
       "3       The opioid epidemic keeps killing my friends   \n",
       "4  Trump's big challenge at the upcoming North Ko...   \n",
       "\n",
       "                                                 url  \n",
       "0  /opinion/laura-ingraham-meet-the-candidates-ev...  \n",
       "1  /opinion/tucker-carlson-identity-politics-is-a...  \n",
       "2  /opinion/charlie-kirk-americans-must-wake-up-a...  \n",
       "3  /opinion/the-opioid-epidemic-keeps-killing-my-...  \n",
       "4  /opinion/trumps-big-challenge-at-the-upcoming-...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks pretty good. The JSON appears to have some nested elements in it. For example, Category contains a dictionary. These can often be flattened with `json_normalize`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Laura Ingraham's Monologue</td>\n",
       "      <td>/category/shows/ingraham-angle/transcript/laur...</td>\n",
       "      <td>Another day, another wacky liberal steps on to...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>Laura Ingraham: Meet the candidates – Every da...</td>\n",
       "      <td>/opinion/laura-ingraham-meet-the-candidates-ev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tucker Carlson's Monologue</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight/transcr...</td>\n",
       "      <td>Identity politics is a tactic designed to prev...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>Tucker Carlson: Identity politics is a scam an...</td>\n",
       "      <td>/opinion/tucker-carlson-identity-politics-is-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>As this socialist tragedy unfolds in Venezuela...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>Charlie Kirk: Americans must wake up and fight...</td>\n",
       "      <td>/opinion/charlie-kirk-americans-must-wake-up-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I lost a friend last week. She was 42 years ol...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>The opioid epidemic keeps killing my friends</td>\n",
       "      <td>/opinion/the-opioid-epidemic-keeps-killing-my-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Trump has gotten Pyongyang to come to Hanoi. N...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>Trump's big challenge at the upcoming North Ko...</td>\n",
       "      <td>/opinion/trumps-big-challenge-at-the-upcoming-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                category.name  \\\n",
       "0  Laura Ingraham's Monologue   \n",
       "1  Tucker Carlson's Monologue   \n",
       "2                     OPINION   \n",
       "3                     OPINION   \n",
       "4                     OPINION   \n",
       "\n",
       "                                        category.url  \\\n",
       "0  /category/shows/ingraham-angle/transcript/laur...   \n",
       "1  /category/shows/tucker-carlson-tonight/transcr...   \n",
       "2                                  /category/opinion   \n",
       "3                                  /category/opinion   \n",
       "4                                  /category/opinion   \n",
       "\n",
       "                                         description duration  \\\n",
       "0  Another day, another wacky liberal steps on to...            \n",
       "1  Identity politics is a tactic designed to prev...            \n",
       "2  As this socialist tragedy unfolds in Venezuela...            \n",
       "3  I lost a friend last week. She was 42 years ol...            \n",
       "4  Trump has gotten Pyongyang to come to Hanoi. N...            \n",
       "\n",
       "                                            imageUrl  isBreaking  isLive  \\\n",
       "0  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "\n",
       "           lastPublishedDate            publicationDate  \\\n",
       "0  2019-02-20T08:39:06-05:00  2019-02-20T08:39:06-05:00   \n",
       "1  2019-02-20T07:13:09-05:00  2019-02-20T07:13:09-05:00   \n",
       "2  2019-02-20T04:00:51-05:00  2019-02-20T04:00:51-05:00   \n",
       "3  2019-02-20T04:00:43-05:00  2019-02-20T04:00:43-05:00   \n",
       "4  2019-02-20T04:00:24-05:00  2019-02-20T04:00:24-05:00   \n",
       "\n",
       "                                               title  \\\n",
       "0  Laura Ingraham: Meet the candidates – Every da...   \n",
       "1  Tucker Carlson: Identity politics is a scam an...   \n",
       "2  Charlie Kirk: Americans must wake up and fight...   \n",
       "3       The opioid epidemic keeps killing my friends   \n",
       "4  Trump's big challenge at the upcoming North Ko...   \n",
       "\n",
       "                                                 url  \n",
       "0  /opinion/laura-ingraham-meet-the-candidates-ev...  \n",
       "1  /opinion/tucker-carlson-identity-politics-is-a...  \n",
       "2  /opinion/charlie-kirk-americans-must-wake-up-a...  \n",
       "3  /opinion/the-opioid-epidemic-keeps-killing-my-...  \n",
       "4  /opinion/trumps-big-challenge-at-the-upcoming-...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pandas.io.json import json_normalize\n",
    "\n",
    "df = json_normalize(r.json())\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! This process demonstrates that FoxNews has an undocumented API that can be accessed via Python. The dataset, however, only has a few cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to see if more data can be collected. I usually focus on two parameters: How do I get to the next set of results? Can I get more results with each call? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.foxnews.com/api/article-search?isCategory=true&isTag=false&isKeyword=false&isFixed=false&isFeedUrl=false&searchSelected=opinion&contentTypes=%7B%22interactive%22:true,%22slideshow%22:true,%22video%22:false,%22article%22:true%7D&size=11&offset=30\n"
     ]
    }
   ],
   "source": [
    "print(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking back at the URL, the likely suspects for manipulation are `size`, which usually determines the number of results, and `offset` which usually means \"start with the nth result\". I first check to see how many results can be returned in one call. If the answer is 10,000, I don't need to do much more.\n",
    "\n",
    "Since I'll be making many calls using the API, I write a quick function to take the URL and return a dataframe. Once I'm confident it will work, I would likely make a more robust function that allows more direct manipulation of the parameters, but I don't want to spend too much time on that if the whole API is a dead end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fox_df(url):\n",
    "    r = requests.get(url)\n",
    "    df = json_normalize(r.json())\n",
    "    print('Return a dataframe of length',len(df))\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I confirm that the function works using the original url."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 11\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Laura Ingraham's Monologue</td>\n",
       "      <td>/category/shows/ingraham-angle/transcript/laur...</td>\n",
       "      <td>Another day, another wacky liberal steps on to...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>Laura Ingraham: Meet the candidates – Every da...</td>\n",
       "      <td>/opinion/laura-ingraham-meet-the-candidates-ev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tucker Carlson's Monologue</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight/transcr...</td>\n",
       "      <td>Identity politics is a tactic designed to prev...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>Tucker Carlson: Identity politics is a scam an...</td>\n",
       "      <td>/opinion/tucker-carlson-identity-politics-is-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>As this socialist tragedy unfolds in Venezuela...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>Charlie Kirk: Americans must wake up and fight...</td>\n",
       "      <td>/opinion/charlie-kirk-americans-must-wake-up-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I lost a friend last week. She was 42 years ol...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>The opioid epidemic keeps killing my friends</td>\n",
       "      <td>/opinion/the-opioid-epidemic-keeps-killing-my-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Trump has gotten Pyongyang to come to Hanoi. N...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>Trump's big challenge at the upcoming North Ko...</td>\n",
       "      <td>/opinion/trumps-big-challenge-at-the-upcoming-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Rep. Ilhan Omar is a leading critic of Israel ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:12-05:00</td>\n",
       "      <td>2019-02-20T04:00:12-05:00</td>\n",
       "      <td>Dems shouldn’t support anti-Israel Rep. Ilhan ...</td>\n",
       "      <td>/opinion/dems-shouldnt-support-anti-israel-rep...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Democratic socialist Bernie Sanders is running...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T19:53:25-05:00</td>\n",
       "      <td>2019-02-19T19:53:25-05:00</td>\n",
       "      <td>Bernie Sanders’ presidential candidacy will fu...</td>\n",
       "      <td>/opinion/bernie-sanders-presidential-candidacy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>When someone invokes the ugliness of hate crim...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T13:41:44-05:00</td>\n",
       "      <td>2019-02-19T13:41:44-05:00</td>\n",
       "      <td>Geraldo Rivera: If Jussie Smollett’s attack wa...</td>\n",
       "      <td>/opinion/geraldo-rivera-if-jussie-smolletts-at...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Did you notice how, during the Andrew McCabe i...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T12:07:39-05:00</td>\n",
       "      <td>2019-02-19T12:07:39-05:00</td>\n",
       "      <td>Dan Bongino: Like Comey and Brennan, McCabe fi...</td>\n",
       "      <td>/opinion/dan-bongino-like-comey-and-brennan-mc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>China and the West are locked in a struggle fo...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T11:58:24-05:00</td>\n",
       "      <td>2019-02-19T11:58:24-05:00</td>\n",
       "      <td>Newt Gingrich:  America in race against China ...</td>\n",
       "      <td>/opinion/newt-gingrich-america-in-race-against...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Many experts on the Middle East never thought ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T10:45:37-05:00</td>\n",
       "      <td>2019-02-19T10:45:37-05:00</td>\n",
       "      <td>Iran should return to ancient teachings of Isl...</td>\n",
       "      <td>/opinion/iran-should-return-to-ancient-teachin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 category.name  \\\n",
       "0   Laura Ingraham's Monologue   \n",
       "1   Tucker Carlson's Monologue   \n",
       "2                      OPINION   \n",
       "3                      OPINION   \n",
       "4                      OPINION   \n",
       "5                      OPINION   \n",
       "6                      OPINION   \n",
       "7                      OPINION   \n",
       "8                      OPINION   \n",
       "9                      OPINION   \n",
       "10                     OPINION   \n",
       "\n",
       "                                         category.url  \\\n",
       "0   /category/shows/ingraham-angle/transcript/laur...   \n",
       "1   /category/shows/tucker-carlson-tonight/transcr...   \n",
       "2                                   /category/opinion   \n",
       "3                                   /category/opinion   \n",
       "4                                   /category/opinion   \n",
       "5                                   /category/opinion   \n",
       "6                                   /category/opinion   \n",
       "7                                   /category/opinion   \n",
       "8                                   /category/opinion   \n",
       "9                                   /category/opinion   \n",
       "10                                  /category/opinion   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   Another day, another wacky liberal steps on to...            \n",
       "1   Identity politics is a tactic designed to prev...            \n",
       "2   As this socialist tragedy unfolds in Venezuela...            \n",
       "3   I lost a friend last week. She was 42 years ol...            \n",
       "4   Trump has gotten Pyongyang to come to Hanoi. N...            \n",
       "5   Rep. Ilhan Omar is a leading critic of Israel ...            \n",
       "6   Democratic socialist Bernie Sanders is running...            \n",
       "7   When someone invokes the ugliness of hate crim...            \n",
       "8   Did you notice how, during the Andrew McCabe i...            \n",
       "9   China and the West are locked in a struggle fo...            \n",
       "10  Many experts on the Middle East never thought ...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "5   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "6   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "7   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "10  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2019-02-20T08:39:06-05:00  2019-02-20T08:39:06-05:00   \n",
       "1   2019-02-20T07:13:09-05:00  2019-02-20T07:13:09-05:00   \n",
       "2   2019-02-20T04:00:51-05:00  2019-02-20T04:00:51-05:00   \n",
       "3   2019-02-20T04:00:43-05:00  2019-02-20T04:00:43-05:00   \n",
       "4   2019-02-20T04:00:24-05:00  2019-02-20T04:00:24-05:00   \n",
       "5   2019-02-20T04:00:12-05:00  2019-02-20T04:00:12-05:00   \n",
       "6   2019-02-19T19:53:25-05:00  2019-02-19T19:53:25-05:00   \n",
       "7   2019-02-19T13:41:44-05:00  2019-02-19T13:41:44-05:00   \n",
       "8   2019-02-19T12:07:39-05:00  2019-02-19T12:07:39-05:00   \n",
       "9   2019-02-19T11:58:24-05:00  2019-02-19T11:58:24-05:00   \n",
       "10  2019-02-19T10:45:37-05:00  2019-02-19T10:45:37-05:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   Laura Ingraham: Meet the candidates – Every da...   \n",
       "1   Tucker Carlson: Identity politics is a scam an...   \n",
       "2   Charlie Kirk: Americans must wake up and fight...   \n",
       "3        The opioid epidemic keeps killing my friends   \n",
       "4   Trump's big challenge at the upcoming North Ko...   \n",
       "5   Dems shouldn’t support anti-Israel Rep. Ilhan ...   \n",
       "6   Bernie Sanders’ presidential candidacy will fu...   \n",
       "7   Geraldo Rivera: If Jussie Smollett’s attack wa...   \n",
       "8   Dan Bongino: Like Comey and Brennan, McCabe fi...   \n",
       "9   Newt Gingrich:  America in race against China ...   \n",
       "10  Iran should return to ancient teachings of Isl...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/laura-ingraham-meet-the-candidates-ev...  \n",
       "1   /opinion/tucker-carlson-identity-politics-is-a...  \n",
       "2   /opinion/charlie-kirk-americans-must-wake-up-a...  \n",
       "3   /opinion/the-opioid-epidemic-keeps-killing-my-...  \n",
       "4   /opinion/trumps-big-challenge-at-the-upcoming-...  \n",
       "5   /opinion/dems-shouldnt-support-anti-israel-rep...  \n",
       "6   /opinion/bernie-sanders-presidential-candidacy...  \n",
       "7   /opinion/geraldo-rivera-if-jussie-smolletts-at...  \n",
       "8   /opinion/dan-bongino-like-comey-and-brennan-mc...  \n",
       "9   /opinion/newt-gingrich-america-in-race-against...  \n",
       "10  /opinion/iran-should-return-to-ancient-teachin...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox_df(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a first attempt, I try to retrieve 100 results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 30\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Laura Ingraham's Monologue</td>\n",
       "      <td>/category/shows/ingraham-angle/transcript/laur...</td>\n",
       "      <td>Another day, another wacky liberal steps on to...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>2019-02-20T08:39:06-05:00</td>\n",
       "      <td>Laura Ingraham: Meet the candidates – Every da...</td>\n",
       "      <td>/opinion/laura-ingraham-meet-the-candidates-ev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tucker Carlson's Monologue</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight/transcr...</td>\n",
       "      <td>Identity politics is a tactic designed to prev...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>2019-02-20T07:13:09-05:00</td>\n",
       "      <td>Tucker Carlson: Identity politics is a scam an...</td>\n",
       "      <td>/opinion/tucker-carlson-identity-politics-is-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>As this socialist tragedy unfolds in Venezuela...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>2019-02-20T04:00:51-05:00</td>\n",
       "      <td>Charlie Kirk: Americans must wake up and fight...</td>\n",
       "      <td>/opinion/charlie-kirk-americans-must-wake-up-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I lost a friend last week. She was 42 years ol...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>2019-02-20T04:00:43-05:00</td>\n",
       "      <td>The opioid epidemic keeps killing my friends</td>\n",
       "      <td>/opinion/the-opioid-epidemic-keeps-killing-my-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Trump has gotten Pyongyang to come to Hanoi. N...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>2019-02-20T04:00:24-05:00</td>\n",
       "      <td>Trump's big challenge at the upcoming North Ko...</td>\n",
       "      <td>/opinion/trumps-big-challenge-at-the-upcoming-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Rep. Ilhan Omar is a leading critic of Israel ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-20T04:00:12-05:00</td>\n",
       "      <td>2019-02-20T04:00:12-05:00</td>\n",
       "      <td>Dems shouldn’t support anti-Israel Rep. Ilhan ...</td>\n",
       "      <td>/opinion/dems-shouldnt-support-anti-israel-rep...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Democratic socialist Bernie Sanders is running...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T19:53:25-05:00</td>\n",
       "      <td>2019-02-19T19:53:25-05:00</td>\n",
       "      <td>Bernie Sanders’ presidential candidacy will fu...</td>\n",
       "      <td>/opinion/bernie-sanders-presidential-candidacy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>When someone invokes the ugliness of hate crim...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T13:41:44-05:00</td>\n",
       "      <td>2019-02-19T13:41:44-05:00</td>\n",
       "      <td>Geraldo Rivera: If Jussie Smollett’s attack wa...</td>\n",
       "      <td>/opinion/geraldo-rivera-if-jussie-smolletts-at...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Did you notice how, during the Andrew McCabe i...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T12:07:39-05:00</td>\n",
       "      <td>2019-02-19T12:07:39-05:00</td>\n",
       "      <td>Dan Bongino: Like Comey and Brennan, McCabe fi...</td>\n",
       "      <td>/opinion/dan-bongino-like-comey-and-brennan-mc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>China and the West are locked in a struggle fo...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T11:58:24-05:00</td>\n",
       "      <td>2019-02-19T11:58:24-05:00</td>\n",
       "      <td>Newt Gingrich:  America in race against China ...</td>\n",
       "      <td>/opinion/newt-gingrich-america-in-race-against...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Many experts on the Middle East never thought ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T10:45:37-05:00</td>\n",
       "      <td>2019-02-19T10:45:37-05:00</td>\n",
       "      <td>Iran should return to ancient teachings of Isl...</td>\n",
       "      <td>/opinion/iran-should-return-to-ancient-teachin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I have often argued that we don't have a true ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T10:19:52-05:00</td>\n",
       "      <td>2019-02-19T10:19:52-05:00</td>\n",
       "      <td>Judge Andrew Napolitano: The need for the cons...</td>\n",
       "      <td>/opinion/judge-andrew-napolitano-the-need-for-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>In a viral exchange at a congressional hearing...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T08:51:59-05:00</td>\n",
       "      <td>2019-02-19T08:51:59-05:00</td>\n",
       "      <td>Rich Lowry: Ilhan Omar’s big lie</td>\n",
       "      <td>/opinion/rich-lowry-ilhan-omars-big-lie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Why does the left insist on inventing bigoted ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T06:35:18-05:00</td>\n",
       "      <td>2019-02-19T06:35:18-05:00</td>\n",
       "      <td>Tucker Carlson: Why the left so desperately wa...</td>\n",
       "      <td>/opinion/tucker-carlson-why-the-left-so-desper...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The new generation of Democrats in power may l...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T04:00:46-05:00</td>\n",
       "      <td>2019-02-19T04:00:46-05:00</td>\n",
       "      <td>New Dem members of Congress may look diverse b...</td>\n",
       "      <td>/opinion/new-dem-members-of-congress-may-look-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I spent much of the last few years travelling ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T04:00:42-05:00</td>\n",
       "      <td>2019-02-19T04:00:42-05:00</td>\n",
       "      <td>Trump's election was fueled by social decay an...</td>\n",
       "      <td>/opinion/trumps-election-was-fueled-by-social-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Alexandria Ocasio-Cortez has done our country ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T04:00:41-05:00</td>\n",
       "      <td>2019-02-19T04:00:41-05:00</td>\n",
       "      <td>Liz Peek: Alexandria Ocasio-Cortez’s Amazon ta...</td>\n",
       "      <td>/opinion/liz-peek-alexandria-ocasio-cortezs-am...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Pat Caddell very simply invented political con...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T04:00:27-05:00</td>\n",
       "      <td>2019-02-19T04:00:27-05:00</td>\n",
       "      <td>Doug Schoen: Remembering Pat Caddell -- a coll...</td>\n",
       "      <td>/opinion/doug-schoen-remembering-pat-caddell-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>There’s absolutely no need for the strong arm ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T04:00:14-05:00</td>\n",
       "      <td>2019-02-19T04:00:14-05:00</td>\n",
       "      <td>Virginia’s ‘Green New Deal’ exposes what Democ...</td>\n",
       "      <td>/opinion/virginias-green-new-deal-exposes-what...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The fight against drug trafficking in our coun...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-19T04:00:12-05:00</td>\n",
       "      <td>2019-02-19T04:00:12-05:00</td>\n",
       "      <td>Rep. Michael Guest: President Trump’s border w...</td>\n",
       "      <td>/opinion/rep-michael-guest-president-trumps-bo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Andrew McCabe is a good witness and he made a ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T13:16:06-05:00</td>\n",
       "      <td>2019-02-18T13:16:06-05:00</td>\n",
       "      <td>Andrew McCarthy: McCabe and ‘60 Minutes’ avoid...</td>\n",
       "      <td>/opinion/andrew-mccarthy-mccabe-and-60-minutes...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Public weariness, combined with hyperventilati...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T12:42:43-05:00</td>\n",
       "      <td>2019-02-18T12:42:43-05:00</td>\n",
       "      <td>As Democrats and Republicans barrel toward 202...</td>\n",
       "      <td>/opinion/as-democrats-and-republicans-barrel-t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Faith &amp; Values</td>\n",
       "      <td>/category/faith-values</td>\n",
       "      <td>In order to move past tragedies, we must allow...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T10:45:04-05:00</td>\n",
       "      <td>2019-02-18T10:45:04-05:00</td>\n",
       "      <td>4 ways to overcome trauma and tragedy (always ...</td>\n",
       "      <td>/opinion/4-ways-to-overcome-trauma-and-tragedy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>/category/tech/companies/amazon</td>\n",
       "      <td>Amazon’s decision last week to scrap plans for...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T08:16:20-05:00</td>\n",
       "      <td>2019-02-18T08:16:20-05:00</td>\n",
       "      <td>Michael Goodwin: Amazon's New York debacle ope...</td>\n",
       "      <td>/opinion/michael-goodwin-amazons-new-york-deba...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>The Next Revolution</td>\n",
       "      <td>/category/shows/the-next-revolution</td>\n",
       "      <td>A New York Times op-ed by Jonathan Rauch and P...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T07:42:33-05:00</td>\n",
       "      <td>2019-02-18T07:42:33-05:00</td>\n",
       "      <td>Steve Hilton: Never Trumpers just hate that Tr...</td>\n",
       "      <td>/opinion/steve-hilton-never-trumpers-just-hate...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>While John Wayne might have eschewed backup ca...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T04:00:56-05:00</td>\n",
       "      <td>2019-02-18T04:00:56-05:00</td>\n",
       "      <td>No Presidents Day car sales for me: What I lea...</td>\n",
       "      <td>/opinion/no-presidents-day-car-sales-for-me-wh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The political world barely raised its eyebrows...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T04:00:31-05:00</td>\n",
       "      <td>2019-02-18T04:00:31-05:00</td>\n",
       "      <td>Democrats ignore Bernie Sanders’ 2020 announce...</td>\n",
       "      <td>/opinion/democrats-ignore-bernie-sanders-2020-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>On President’s Day, perhaps it would be helpfu...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T04:00:24-05:00</td>\n",
       "      <td>2019-02-18T04:00:24-05:00</td>\n",
       "      <td>On Presidents Day, honor those who have served...</td>\n",
       "      <td>/opinion/on-presidents-day-honor-those-who-hav...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Andrew McCabe lied multiple times to federal i...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T04:00:21-05:00</td>\n",
       "      <td>2019-02-18T04:00:21-05:00</td>\n",
       "      <td>Jason Chaffetz: FBI’s Andrew McCabe should be ...</td>\n",
       "      <td>/opinion/jason-chaffetz-fbis-andrew-mccabe-sho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Ask any college senior where the best jobs are...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-18T04:00:13-05:00</td>\n",
       "      <td>2019-02-18T04:00:13-05:00</td>\n",
       "      <td>Don't put flyover country in a corner. It's le...</td>\n",
       "      <td>/opinion/dont-put-flyover-country-in-a-corner-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 category.name  \\\n",
       "0   Laura Ingraham's Monologue   \n",
       "1   Tucker Carlson's Monologue   \n",
       "2                      OPINION   \n",
       "3                      OPINION   \n",
       "4                      OPINION   \n",
       "5                      OPINION   \n",
       "6                      OPINION   \n",
       "7                      OPINION   \n",
       "8                      OPINION   \n",
       "9                      OPINION   \n",
       "10                     OPINION   \n",
       "11                     OPINION   \n",
       "12                     OPINION   \n",
       "13                     OPINION   \n",
       "14                     OPINION   \n",
       "15                     OPINION   \n",
       "16                     OPINION   \n",
       "17                     OPINION   \n",
       "18                     OPINION   \n",
       "19                     OPINION   \n",
       "20                     OPINION   \n",
       "21                     OPINION   \n",
       "22              Faith & Values   \n",
       "23                      Amazon   \n",
       "24         The Next Revolution   \n",
       "25                     OPINION   \n",
       "26                     OPINION   \n",
       "27                     OPINION   \n",
       "28                     OPINION   \n",
       "29                     OPINION   \n",
       "\n",
       "                                         category.url  \\\n",
       "0   /category/shows/ingraham-angle/transcript/laur...   \n",
       "1   /category/shows/tucker-carlson-tonight/transcr...   \n",
       "2                                   /category/opinion   \n",
       "3                                   /category/opinion   \n",
       "4                                   /category/opinion   \n",
       "5                                   /category/opinion   \n",
       "6                                   /category/opinion   \n",
       "7                                   /category/opinion   \n",
       "8                                   /category/opinion   \n",
       "9                                   /category/opinion   \n",
       "10                                  /category/opinion   \n",
       "11                                  /category/opinion   \n",
       "12                                  /category/opinion   \n",
       "13                                  /category/opinion   \n",
       "14                                  /category/opinion   \n",
       "15                                  /category/opinion   \n",
       "16                                  /category/opinion   \n",
       "17                                  /category/opinion   \n",
       "18                                  /category/opinion   \n",
       "19                                  /category/opinion   \n",
       "20                                  /category/opinion   \n",
       "21                                  /category/opinion   \n",
       "22                             /category/faith-values   \n",
       "23                    /category/tech/companies/amazon   \n",
       "24                /category/shows/the-next-revolution   \n",
       "25                                  /category/opinion   \n",
       "26                                  /category/opinion   \n",
       "27                                  /category/opinion   \n",
       "28                                  /category/opinion   \n",
       "29                                  /category/opinion   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   Another day, another wacky liberal steps on to...            \n",
       "1   Identity politics is a tactic designed to prev...            \n",
       "2   As this socialist tragedy unfolds in Venezuela...            \n",
       "3   I lost a friend last week. She was 42 years ol...            \n",
       "4   Trump has gotten Pyongyang to come to Hanoi. N...            \n",
       "5   Rep. Ilhan Omar is a leading critic of Israel ...            \n",
       "6   Democratic socialist Bernie Sanders is running...            \n",
       "7   When someone invokes the ugliness of hate crim...            \n",
       "8   Did you notice how, during the Andrew McCabe i...            \n",
       "9   China and the West are locked in a struggle fo...            \n",
       "10  Many experts on the Middle East never thought ...            \n",
       "11  I have often argued that we don't have a true ...            \n",
       "12  In a viral exchange at a congressional hearing...            \n",
       "13  Why does the left insist on inventing bigoted ...            \n",
       "14  The new generation of Democrats in power may l...            \n",
       "15  I spent much of the last few years travelling ...            \n",
       "16  Alexandria Ocasio-Cortez has done our country ...            \n",
       "17  Pat Caddell very simply invented political con...            \n",
       "18  There’s absolutely no need for the strong arm ...            \n",
       "19  The fight against drug trafficking in our coun...            \n",
       "20  Andrew McCabe is a good witness and he made a ...            \n",
       "21  Public weariness, combined with hyperventilati...            \n",
       "22  In order to move past tragedies, we must allow...            \n",
       "23  Amazon’s decision last week to scrap plans for...            \n",
       "24  A New York Times op-ed by Jonathan Rauch and P...            \n",
       "25  While John Wayne might have eschewed backup ca...            \n",
       "26  The political world barely raised its eyebrows...            \n",
       "27  On President’s Day, perhaps it would be helpfu...            \n",
       "28  Andrew McCabe lied multiple times to federal i...            \n",
       "29  Ask any college senior where the best jobs are...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "5   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "6   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "7   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "10  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "11  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "12  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "13  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "14  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "15  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "16  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "17  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "18  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "19  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "20  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "21  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "22  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "23  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "24  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "25  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "26  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "27  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "28  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "29  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2019-02-20T08:39:06-05:00  2019-02-20T08:39:06-05:00   \n",
       "1   2019-02-20T07:13:09-05:00  2019-02-20T07:13:09-05:00   \n",
       "2   2019-02-20T04:00:51-05:00  2019-02-20T04:00:51-05:00   \n",
       "3   2019-02-20T04:00:43-05:00  2019-02-20T04:00:43-05:00   \n",
       "4   2019-02-20T04:00:24-05:00  2019-02-20T04:00:24-05:00   \n",
       "5   2019-02-20T04:00:12-05:00  2019-02-20T04:00:12-05:00   \n",
       "6   2019-02-19T19:53:25-05:00  2019-02-19T19:53:25-05:00   \n",
       "7   2019-02-19T13:41:44-05:00  2019-02-19T13:41:44-05:00   \n",
       "8   2019-02-19T12:07:39-05:00  2019-02-19T12:07:39-05:00   \n",
       "9   2019-02-19T11:58:24-05:00  2019-02-19T11:58:24-05:00   \n",
       "10  2019-02-19T10:45:37-05:00  2019-02-19T10:45:37-05:00   \n",
       "11  2019-02-19T10:19:52-05:00  2019-02-19T10:19:52-05:00   \n",
       "12  2019-02-19T08:51:59-05:00  2019-02-19T08:51:59-05:00   \n",
       "13  2019-02-19T06:35:18-05:00  2019-02-19T06:35:18-05:00   \n",
       "14  2019-02-19T04:00:46-05:00  2019-02-19T04:00:46-05:00   \n",
       "15  2019-02-19T04:00:42-05:00  2019-02-19T04:00:42-05:00   \n",
       "16  2019-02-19T04:00:41-05:00  2019-02-19T04:00:41-05:00   \n",
       "17  2019-02-19T04:00:27-05:00  2019-02-19T04:00:27-05:00   \n",
       "18  2019-02-19T04:00:14-05:00  2019-02-19T04:00:14-05:00   \n",
       "19  2019-02-19T04:00:12-05:00  2019-02-19T04:00:12-05:00   \n",
       "20  2019-02-18T13:16:06-05:00  2019-02-18T13:16:06-05:00   \n",
       "21  2019-02-18T12:42:43-05:00  2019-02-18T12:42:43-05:00   \n",
       "22  2019-02-18T10:45:04-05:00  2019-02-18T10:45:04-05:00   \n",
       "23  2019-02-18T08:16:20-05:00  2019-02-18T08:16:20-05:00   \n",
       "24  2019-02-18T07:42:33-05:00  2019-02-18T07:42:33-05:00   \n",
       "25  2019-02-18T04:00:56-05:00  2019-02-18T04:00:56-05:00   \n",
       "26  2019-02-18T04:00:31-05:00  2019-02-18T04:00:31-05:00   \n",
       "27  2019-02-18T04:00:24-05:00  2019-02-18T04:00:24-05:00   \n",
       "28  2019-02-18T04:00:21-05:00  2019-02-18T04:00:21-05:00   \n",
       "29  2019-02-18T04:00:13-05:00  2019-02-18T04:00:13-05:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   Laura Ingraham: Meet the candidates – Every da...   \n",
       "1   Tucker Carlson: Identity politics is a scam an...   \n",
       "2   Charlie Kirk: Americans must wake up and fight...   \n",
       "3        The opioid epidemic keeps killing my friends   \n",
       "4   Trump's big challenge at the upcoming North Ko...   \n",
       "5   Dems shouldn’t support anti-Israel Rep. Ilhan ...   \n",
       "6   Bernie Sanders’ presidential candidacy will fu...   \n",
       "7   Geraldo Rivera: If Jussie Smollett’s attack wa...   \n",
       "8   Dan Bongino: Like Comey and Brennan, McCabe fi...   \n",
       "9   Newt Gingrich:  America in race against China ...   \n",
       "10  Iran should return to ancient teachings of Isl...   \n",
       "11  Judge Andrew Napolitano: The need for the cons...   \n",
       "12                   Rich Lowry: Ilhan Omar’s big lie   \n",
       "13  Tucker Carlson: Why the left so desperately wa...   \n",
       "14  New Dem members of Congress may look diverse b...   \n",
       "15  Trump's election was fueled by social decay an...   \n",
       "16  Liz Peek: Alexandria Ocasio-Cortez’s Amazon ta...   \n",
       "17  Doug Schoen: Remembering Pat Caddell -- a coll...   \n",
       "18  Virginia’s ‘Green New Deal’ exposes what Democ...   \n",
       "19  Rep. Michael Guest: President Trump’s border w...   \n",
       "20  Andrew McCarthy: McCabe and ‘60 Minutes’ avoid...   \n",
       "21  As Democrats and Republicans barrel toward 202...   \n",
       "22  4 ways to overcome trauma and tragedy (always ...   \n",
       "23  Michael Goodwin: Amazon's New York debacle ope...   \n",
       "24  Steve Hilton: Never Trumpers just hate that Tr...   \n",
       "25  No Presidents Day car sales for me: What I lea...   \n",
       "26  Democrats ignore Bernie Sanders’ 2020 announce...   \n",
       "27  On Presidents Day, honor those who have served...   \n",
       "28  Jason Chaffetz: FBI’s Andrew McCabe should be ...   \n",
       "29  Don't put flyover country in a corner. It's le...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/laura-ingraham-meet-the-candidates-ev...  \n",
       "1   /opinion/tucker-carlson-identity-politics-is-a...  \n",
       "2   /opinion/charlie-kirk-americans-must-wake-up-a...  \n",
       "3   /opinion/the-opioid-epidemic-keeps-killing-my-...  \n",
       "4   /opinion/trumps-big-challenge-at-the-upcoming-...  \n",
       "5   /opinion/dems-shouldnt-support-anti-israel-rep...  \n",
       "6   /opinion/bernie-sanders-presidential-candidacy...  \n",
       "7   /opinion/geraldo-rivera-if-jussie-smolletts-at...  \n",
       "8   /opinion/dan-bongino-like-comey-and-brennan-mc...  \n",
       "9   /opinion/newt-gingrich-america-in-race-against...  \n",
       "10  /opinion/iran-should-return-to-ancient-teachin...  \n",
       "11  /opinion/judge-andrew-napolitano-the-need-for-...  \n",
       "12            /opinion/rich-lowry-ilhan-omars-big-lie  \n",
       "13  /opinion/tucker-carlson-why-the-left-so-desper...  \n",
       "14  /opinion/new-dem-members-of-congress-may-look-...  \n",
       "15  /opinion/trumps-election-was-fueled-by-social-...  \n",
       "16  /opinion/liz-peek-alexandria-ocasio-cortezs-am...  \n",
       "17  /opinion/doug-schoen-remembering-pat-caddell-a...  \n",
       "18  /opinion/virginias-green-new-deal-exposes-what...  \n",
       "19  /opinion/rep-michael-guest-president-trumps-bo...  \n",
       "20  /opinion/andrew-mccarthy-mccabe-and-60-minutes...  \n",
       "21  /opinion/as-democrats-and-republicans-barrel-t...  \n",
       "22  /opinion/4-ways-to-overcome-trauma-and-tragedy...  \n",
       "23  /opinion/michael-goodwin-amazons-new-york-deba...  \n",
       "24  /opinion/steve-hilton-never-trumpers-just-hate...  \n",
       "25  /opinion/no-presidents-day-car-sales-for-me-wh...  \n",
       "26  /opinion/democrats-ignore-bernie-sanders-2020-...  \n",
       "27  /opinion/on-presidents-day-honor-those-who-hav...  \n",
       "28  /opinion/jason-chaffetz-fbis-andrew-mccabe-sho...  \n",
       "29  /opinion/dont-put-flyover-country-in-a-corner-...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url100 = url.replace('size=11','size=100')\n",
    "\n",
    "fox_df(url100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This does not return 100 results, but it does return 30, which appears to be the internal maximum.  Collecting a larger corpus of article metadata will need to be done in batches of 30. \n",
    "\n",
    "The next question is how far back can the API go? Here, the first suspect is the `offset` parameter. The current value of 30 is usually associated with starting with the 30th results. So an `offset` of 30 with a `size` of `10` is likely to return results 30-39.\n",
    "\n",
    "As a first pass, I set the value of `offset` to 0 to get the most recent results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 11\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Laura Ingraham's Monologue</td>\n",
       "      <td>/category/shows/ingraham-angle/transcript/laur...</td>\n",
       "      <td>Jussie Smollett could not resist the allure of...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T08:11:57-05:00</td>\n",
       "      <td>2019-02-22T08:11:57-05:00</td>\n",
       "      <td>Laura Ingraham: Jussie Smollett learned that v...</td>\n",
       "      <td>/opinion/laura-ingraham-smollett-learned-that-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tucker Carlson's Monologue</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight/transcr...</td>\n",
       "      <td>What Smollett allegedly did was not simply ill...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T07:47:57-05:00</td>\n",
       "      <td>2019-02-22T07:47:57-05:00</td>\n",
       "      <td>Tucker Carlson: Jussie Smollett pretended to b...</td>\n",
       "      <td>/opinion/tucker-carlson-jussie-smollett-preten...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sean Hannity's Monologue</td>\n",
       "      <td>/category/shows/hannity/transcript/hannitys-mo...</td>\n",
       "      <td>The Smollett case proves what I have been sayi...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T07:02:10-05:00</td>\n",
       "      <td>2019-02-22T07:02:10-05:00</td>\n",
       "      <td>Sean Hannity: Smollett case proves that journa...</td>\n",
       "      <td>/opinion/sean-hannity-smollett-case-proves-tha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tucker Carlson's Monologue</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight/transcr...</td>\n",
       "      <td>Tucker Carlson: What you should know about 'wo...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T06:54:34-05:00</td>\n",
       "      <td>2019-02-22T06:51:51-05:00</td>\n",
       "      <td>Tucker Carlson: 'Woke' billionaires love socia...</td>\n",
       "      <td>/opinion/tucker-carlson-woke-billionaires-love...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>To reduce this flow of illegal immigrants, the...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T04:00:27-05:00</td>\n",
       "      <td>2019-02-22T04:00:27-05:00</td>\n",
       "      <td>A border wall isn’t enough – Asylum laws must ...</td>\n",
       "      <td>/opinion/a-border-wall-isnt-enough-asylum-laws...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The left complains that conservatives are \"obs...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T04:00:23-05:00</td>\n",
       "      <td>2019-02-22T04:00:23-05:00</td>\n",
       "      <td>Marc Thiessen: Alexandria Ocasio-Cortez is an ...</td>\n",
       "      <td>/opinion/marc-thiessen-alexandria-ocasio-corte...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The Green New Deal would bring economic ruin a...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T04:00:15-05:00</td>\n",
       "      <td>2019-02-22T04:00:15-05:00</td>\n",
       "      <td>Green New Deal support could doom Dem presiden...</td>\n",
       "      <td>/opinion/green-new-deal-support-could-doom-dem...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>From Supreme Court Justice Brett Kavanaugh to ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-22T04:00:03-05:00</td>\n",
       "      <td>2019-02-22T04:00:03-05:00</td>\n",
       "      <td>On Smollett, Covington and Kavanaugh, anti-Tru...</td>\n",
       "      <td>/opinion/on-smollett-covington-and-kavanaugh-a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The Justice Department should file an indictme...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-21T18:59:10-05:00</td>\n",
       "      <td>2019-02-21T18:59:10-05:00</td>\n",
       "      <td>Andrew McCarthy: Indict the ‘ISIS Bride’</td>\n",
       "      <td>/opinion/andrew-mccarthy-indict-the-isis-bride</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The activist press embraced Jussie Smollett as...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-21T18:23:31-05:00</td>\n",
       "      <td>2019-02-21T18:23:31-05:00</td>\n",
       "      <td>Gutfeld on Smollett’s arrest</td>\n",
       "      <td>/opinion/gutfeld-on-smolletts-arrest</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Over half a dozen rescue workers in South Afri...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-21T16:15:22-05:00</td>\n",
       "      <td>2019-02-21T16:15:22-05:00</td>\n",
       "      <td>Amid abortion debate, miraculous rescue of bab...</td>\n",
       "      <td>/opinion/amid-abortion-debate-miraculous-rescu...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 category.name  \\\n",
       "0   Laura Ingraham's Monologue   \n",
       "1   Tucker Carlson's Monologue   \n",
       "2     Sean Hannity's Monologue   \n",
       "3   Tucker Carlson's Monologue   \n",
       "4                      OPINION   \n",
       "5                      OPINION   \n",
       "6                      OPINION   \n",
       "7                      OPINION   \n",
       "8                      OPINION   \n",
       "9                      OPINION   \n",
       "10                     OPINION   \n",
       "\n",
       "                                         category.url  \\\n",
       "0   /category/shows/ingraham-angle/transcript/laur...   \n",
       "1   /category/shows/tucker-carlson-tonight/transcr...   \n",
       "2   /category/shows/hannity/transcript/hannitys-mo...   \n",
       "3   /category/shows/tucker-carlson-tonight/transcr...   \n",
       "4                                   /category/opinion   \n",
       "5                                   /category/opinion   \n",
       "6                                   /category/opinion   \n",
       "7                                   /category/opinion   \n",
       "8                                   /category/opinion   \n",
       "9                                   /category/opinion   \n",
       "10                                  /category/opinion   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   Jussie Smollett could not resist the allure of...            \n",
       "1   What Smollett allegedly did was not simply ill...            \n",
       "2   The Smollett case proves what I have been sayi...            \n",
       "3   Tucker Carlson: What you should know about 'wo...            \n",
       "4   To reduce this flow of illegal immigrants, the...            \n",
       "5   The left complains that conservatives are \"obs...            \n",
       "6   The Green New Deal would bring economic ruin a...            \n",
       "7   From Supreme Court Justice Brett Kavanaugh to ...            \n",
       "8   The Justice Department should file an indictme...            \n",
       "9   The activist press embraced Jussie Smollett as...            \n",
       "10  Over half a dozen rescue workers in South Afri...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "5   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "6   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "7   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "10  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2019-02-22T08:11:57-05:00  2019-02-22T08:11:57-05:00   \n",
       "1   2019-02-22T07:47:57-05:00  2019-02-22T07:47:57-05:00   \n",
       "2   2019-02-22T07:02:10-05:00  2019-02-22T07:02:10-05:00   \n",
       "3   2019-02-22T06:54:34-05:00  2019-02-22T06:51:51-05:00   \n",
       "4   2019-02-22T04:00:27-05:00  2019-02-22T04:00:27-05:00   \n",
       "5   2019-02-22T04:00:23-05:00  2019-02-22T04:00:23-05:00   \n",
       "6   2019-02-22T04:00:15-05:00  2019-02-22T04:00:15-05:00   \n",
       "7   2019-02-22T04:00:03-05:00  2019-02-22T04:00:03-05:00   \n",
       "8   2019-02-21T18:59:10-05:00  2019-02-21T18:59:10-05:00   \n",
       "9   2019-02-21T18:23:31-05:00  2019-02-21T18:23:31-05:00   \n",
       "10  2019-02-21T16:15:22-05:00  2019-02-21T16:15:22-05:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   Laura Ingraham: Jussie Smollett learned that v...   \n",
       "1   Tucker Carlson: Jussie Smollett pretended to b...   \n",
       "2   Sean Hannity: Smollett case proves that journa...   \n",
       "3   Tucker Carlson: 'Woke' billionaires love socia...   \n",
       "4   A border wall isn’t enough – Asylum laws must ...   \n",
       "5   Marc Thiessen: Alexandria Ocasio-Cortez is an ...   \n",
       "6   Green New Deal support could doom Dem presiden...   \n",
       "7   On Smollett, Covington and Kavanaugh, anti-Tru...   \n",
       "8            Andrew McCarthy: Indict the ‘ISIS Bride’   \n",
       "9                        Gutfeld on Smollett’s arrest   \n",
       "10  Amid abortion debate, miraculous rescue of bab...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/laura-ingraham-smollett-learned-that-...  \n",
       "1   /opinion/tucker-carlson-jussie-smollett-preten...  \n",
       "2   /opinion/sean-hannity-smollett-case-proves-tha...  \n",
       "3   /opinion/tucker-carlson-woke-billionaires-love...  \n",
       "4   /opinion/a-border-wall-isnt-enough-asylum-laws...  \n",
       "5   /opinion/marc-thiessen-alexandria-ocasio-corte...  \n",
       "6   /opinion/green-new-deal-support-could-doom-dem...  \n",
       "7   /opinion/on-smollett-covington-and-kavanaugh-a...  \n",
       "8      /opinion/andrew-mccarthy-indict-the-isis-bride  \n",
       "9                /opinion/gutfeld-on-smolletts-arrest  \n",
       "10  /opinion/amid-abortion-debate-miraculous-rescu...  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_off = url.replace('offset=30','offset=0')\n",
    "\n",
    "fox_df(url_off)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `lastPublicationDate` value of the first row is 2019-02-08. Hopefully a larger value will be associated with articles published earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 11\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Critics of the Amazon plan can rejoice in thei...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T16:17:04-05:00</td>\n",
       "      <td>2019-02-14T16:17:04-05:00</td>\n",
       "      <td>Amazon quits New York -- 'Victory' by progress...</td>\n",
       "      <td>/opinion/amazon-quits-new-york-victory-by-prog...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Attorney General Barr is the right man for the...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T13:49:21-05:00</td>\n",
       "      <td>2019-02-14T13:49:21-05:00</td>\n",
       "      <td>William Barr is our new attorney general -- He...</td>\n",
       "      <td>/opinion/william-barr-is-our-new-attorney-gene...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>President Trump’s standing has improved over t...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T13:32:30-05:00</td>\n",
       "      <td>2019-02-14T13:32:30-05:00</td>\n",
       "      <td>Karl Rove: Trump's approval numbers are going ...</td>\n",
       "      <td>/opinion/karl-rove-trumps-approval-numbers-are...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>All signs indicate that Special Counsel Robert...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T11:47:24-05:00</td>\n",
       "      <td>2019-02-14T11:47:24-05:00</td>\n",
       "      <td>David Bossie: Mueller should end his probe and...</td>\n",
       "      <td>/opinion/david-bossie-mueller-should-end-his-p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Kamala Harris cannot erase her connections to ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T11:38:28-05:00</td>\n",
       "      <td>2019-02-14T11:37:57-05:00</td>\n",
       "      <td>Kamala Harris is a longtime ally of government...</td>\n",
       "      <td>/opinion/kamala-harris-is-a-longtime-ally-of-g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Faith &amp; Values</td>\n",
       "      <td>/category/faith-values</td>\n",
       "      <td>The heart of Christianity is love. The whole b...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T10:14:40-05:00</td>\n",
       "      <td>2019-02-14T10:14:40-05:00</td>\n",
       "      <td>Valentine’s Day: The ‘heart’ of Christianity i...</td>\n",
       "      <td>/opinion/valentines-day-the-heart-of-christian...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I have been beneath the wedding canopy with mo...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T09:32:35-05:00</td>\n",
       "      <td>2019-02-14T04:00:25-05:00</td>\n",
       "      <td>A Valentine's Day lesson: I have married over ...</td>\n",
       "      <td>/opinion/a-valentines-day-lesson-i-have-marrie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Tucker Carlson's Monologue</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight/transcr...</td>\n",
       "      <td>The Green New Deal is not about the environmen...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T08:33:26-05:00</td>\n",
       "      <td>2019-02-14T08:33:26-05:00</td>\n",
       "      <td>Tucker Carlson: Green New Deal is a religious ...</td>\n",
       "      <td>/opinion/tucker-carlson-green-new-deal-is-a-re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Laura Ingraham's Monologue</td>\n",
       "      <td>/category/shows/ingraham-angle/transcript/laur...</td>\n",
       "      <td>Oh the excitement, oh the promise, it seems li...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T08:05:54-05:00</td>\n",
       "      <td>2019-02-14T08:05:54-05:00</td>\n",
       "      <td>Laura Ingraham: California's high speed rail w...</td>\n",
       "      <td>/opinion/laura-ingraham-californias-high-speed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>It never ceases to amaze that liberals and pro...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T07:35:42-05:00</td>\n",
       "      <td>2019-02-14T04:00:57-05:00</td>\n",
       "      <td>Tammy Bruce: Elizabeth Warren, Democrats and t...</td>\n",
       "      <td>/opinion/tammy-bruce-elizabeth-warren-democrat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Sean Hannity's Monologue</td>\n",
       "      <td>/category/shows/hannity/transcript/hannitys-mo...</td>\n",
       "      <td>We know this is a president that keeps his pro...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2019-02-14T07:06:37-05:00</td>\n",
       "      <td>2019-02-14T07:06:37-05:00</td>\n",
       "      <td>Sean Hannity: Trump keeps his promises. I pred...</td>\n",
       "      <td>/opinion/sean-hannity-trump-keeps-his-promises...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 category.name  \\\n",
       "0                      OPINION   \n",
       "1                      OPINION   \n",
       "2                      OPINION   \n",
       "3                      OPINION   \n",
       "4                      OPINION   \n",
       "5               Faith & Values   \n",
       "6                      OPINION   \n",
       "7   Tucker Carlson's Monologue   \n",
       "8   Laura Ingraham's Monologue   \n",
       "9                      OPINION   \n",
       "10    Sean Hannity's Monologue   \n",
       "\n",
       "                                         category.url  \\\n",
       "0                                   /category/opinion   \n",
       "1                                   /category/opinion   \n",
       "2                                   /category/opinion   \n",
       "3                                   /category/opinion   \n",
       "4                                   /category/opinion   \n",
       "5                              /category/faith-values   \n",
       "6                                   /category/opinion   \n",
       "7   /category/shows/tucker-carlson-tonight/transcr...   \n",
       "8   /category/shows/ingraham-angle/transcript/laur...   \n",
       "9                                   /category/opinion   \n",
       "10  /category/shows/hannity/transcript/hannitys-mo...   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   Critics of the Amazon plan can rejoice in thei...            \n",
       "1   Attorney General Barr is the right man for the...            \n",
       "2   President Trump’s standing has improved over t...            \n",
       "3   All signs indicate that Special Counsel Robert...            \n",
       "4   Kamala Harris cannot erase her connections to ...            \n",
       "5   The heart of Christianity is love. The whole b...            \n",
       "6   I have been beneath the wedding canopy with mo...            \n",
       "7   The Green New Deal is not about the environmen...            \n",
       "8   Oh the excitement, oh the promise, it seems li...            \n",
       "9   It never ceases to amaze that liberals and pro...            \n",
       "10  We know this is a president that keeps his pro...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "5   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "6   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "7   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "10  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2019-02-14T16:17:04-05:00  2019-02-14T16:17:04-05:00   \n",
       "1   2019-02-14T13:49:21-05:00  2019-02-14T13:49:21-05:00   \n",
       "2   2019-02-14T13:32:30-05:00  2019-02-14T13:32:30-05:00   \n",
       "3   2019-02-14T11:47:24-05:00  2019-02-14T11:47:24-05:00   \n",
       "4   2019-02-14T11:38:28-05:00  2019-02-14T11:37:57-05:00   \n",
       "5   2019-02-14T10:14:40-05:00  2019-02-14T10:14:40-05:00   \n",
       "6   2019-02-14T09:32:35-05:00  2019-02-14T04:00:25-05:00   \n",
       "7   2019-02-14T08:33:26-05:00  2019-02-14T08:33:26-05:00   \n",
       "8   2019-02-14T08:05:54-05:00  2019-02-14T08:05:54-05:00   \n",
       "9   2019-02-14T07:35:42-05:00  2019-02-14T04:00:57-05:00   \n",
       "10  2019-02-14T07:06:37-05:00  2019-02-14T07:06:37-05:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   Amazon quits New York -- 'Victory' by progress...   \n",
       "1   William Barr is our new attorney general -- He...   \n",
       "2   Karl Rove: Trump's approval numbers are going ...   \n",
       "3   David Bossie: Mueller should end his probe and...   \n",
       "4   Kamala Harris is a longtime ally of government...   \n",
       "5   Valentine’s Day: The ‘heart’ of Christianity i...   \n",
       "6   A Valentine's Day lesson: I have married over ...   \n",
       "7   Tucker Carlson: Green New Deal is a religious ...   \n",
       "8   Laura Ingraham: California's high speed rail w...   \n",
       "9   Tammy Bruce: Elizabeth Warren, Democrats and t...   \n",
       "10  Sean Hannity: Trump keeps his promises. I pred...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/amazon-quits-new-york-victory-by-prog...  \n",
       "1   /opinion/william-barr-is-our-new-attorney-gene...  \n",
       "2   /opinion/karl-rove-trumps-approval-numbers-are...  \n",
       "3   /opinion/david-bossie-mueller-should-end-his-p...  \n",
       "4   /opinion/kamala-harris-is-a-longtime-ally-of-g...  \n",
       "5   /opinion/valentines-day-the-heart-of-christian...  \n",
       "6   /opinion/a-valentines-day-lesson-i-have-marrie...  \n",
       "7   /opinion/tucker-carlson-green-new-deal-is-a-re...  \n",
       "8   /opinion/laura-ingraham-californias-high-speed...  \n",
       "9   /opinion/tammy-bruce-elizabeth-warren-democrat...  \n",
       "10  /opinion/sean-hannity-trump-keeps-his-promises...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_off = url.replace('offset=30','offset=100')\n",
    "\n",
    "fox_df(url_off)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The article with an offset value of 100 was published on 2019-02-01, or roughly a week before our offset 0. Perfect! Increasing the offset value yields additional article metadata in chronological order. \n",
    "\n",
    "Next, how far back can we go?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 11\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The death of a president brings our nation tog...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-05T04:00:35-05:00</td>\n",
       "      <td>2018-12-05T04:00:35-05:00</td>\n",
       "      <td>Remembering George H.W. Bush: Why does it take...</td>\n",
       "      <td>/opinion/remembering-george-h-w-bush-why-does-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>House Democrats are planning to go after Presi...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-05T04:00:07-05:00</td>\n",
       "      <td>2018-12-05T04:00:07-05:00</td>\n",
       "      <td>David Bossie: House Democrats plan a witch hun...</td>\n",
       "      <td>/opinion/david-bossie-house-democrats-plan-a-w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>The struggle to juggle and balance career and ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-05T04:00:06-05:00</td>\n",
       "      <td>2018-12-05T04:00:06-05:00</td>\n",
       "      <td>Leslie Marshall: Michelle Obama told the truth...</td>\n",
       "      <td>/opinion/leslie-marshall-michelle-obama-told-t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>This morning Google told me that it would not ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-05T04:00:06-05:00</td>\n",
       "      <td>2018-12-05T04:00:06-05:00</td>\n",
       "      <td>John Stossel: Google and Facebook cross 'The C...</td>\n",
       "      <td>/opinion/john-stossel-google-and-facebook-cros...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>President George H.W. Bush called on us to be ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-05T04:00:06-05:00</td>\n",
       "      <td>2018-12-05T04:00:06-05:00</td>\n",
       "      <td>Marc Thiessen: For just a few days, let's be t...</td>\n",
       "      <td>/opinion/marc-thiessen-for-just-a-few-days-let...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>A sentencing memo that Special Counsel Robert ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-05T00:53:12-05:00</td>\n",
       "      <td>2018-12-05T00:53:12-05:00</td>\n",
       "      <td>Gregg Jarrett: Mueller strikes out trying to n...</td>\n",
       "      <td>/opinion/mueller-strikes-out-trying-to-nail-tr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Democrats need to embrace moderate policies to...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T18:42:11-05:00</td>\n",
       "      <td>2018-12-04T18:42:11-05:00</td>\n",
       "      <td>Doug Schoen: Democrats will be hurt by radical...</td>\n",
       "      <td>/opinion/doug-schoen-democrats-will-be-hurt-by...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>People come and go on our planet, and in our l...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T17:40:03-05:00</td>\n",
       "      <td>2018-12-04T17:40:03-05:00</td>\n",
       "      <td>Gutfeld on the passing of George Herbert Walke...</td>\n",
       "      <td>/opinion/gutfeld-on-the-passing-of-george-herb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>I understand that millions of Americans still ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T16:27:20-05:00</td>\n",
       "      <td>2018-12-04T16:27:20-05:00</td>\n",
       "      <td>Geraldo Rivera: Migrants at our border want a ...</td>\n",
       "      <td>/opinion/geraldo-rivera-migrants-at-our-border...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>People from around the world are recalling Pr...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T16:25:31-05:00</td>\n",
       "      <td>2018-12-04T16:19:21-05:00</td>\n",
       "      <td>George H.W. Bush's service dog 'Sully' isn't a...</td>\n",
       "      <td>/opinion/george-h-w-bushs-service-dog-sully-is...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>As I sat a few feet from President Trump while...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T13:27:48-05:00</td>\n",
       "      <td>2018-12-04T13:27:48-05:00</td>\n",
       "      <td>Michael Goodwin: Trump is prepping for all-out...</td>\n",
       "      <td>/opinion/michael-goodwin-trump-is-prepping-for...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   category.name       category.url  \\\n",
       "0        OPINION  /category/opinion   \n",
       "1        OPINION  /category/opinion   \n",
       "2        OPINION  /category/opinion   \n",
       "3        OPINION  /category/opinion   \n",
       "4        OPINION  /category/opinion   \n",
       "5        OPINION  /category/opinion   \n",
       "6        OPINION  /category/opinion   \n",
       "7        OPINION  /category/opinion   \n",
       "8        OPINION  /category/opinion   \n",
       "9        OPINION  /category/opinion   \n",
       "10       OPINION  /category/opinion   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   The death of a president brings our nation tog...            \n",
       "1   House Democrats are planning to go after Presi...            \n",
       "2   The struggle to juggle and balance career and ...            \n",
       "3   This morning Google told me that it would not ...            \n",
       "4   President George H.W. Bush called on us to be ...            \n",
       "5   A sentencing memo that Special Counsel Robert ...            \n",
       "6   Democrats need to embrace moderate policies to...            \n",
       "7   People come and go on our planet, and in our l...            \n",
       "8   I understand that millions of Americans still ...            \n",
       "9    People from around the world are recalling Pr...            \n",
       "10  As I sat a few feet from President Trump while...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "2   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "4   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "5   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "6   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "7   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "10  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2018-12-05T04:00:35-05:00  2018-12-05T04:00:35-05:00   \n",
       "1   2018-12-05T04:00:07-05:00  2018-12-05T04:00:07-05:00   \n",
       "2   2018-12-05T04:00:06-05:00  2018-12-05T04:00:06-05:00   \n",
       "3   2018-12-05T04:00:06-05:00  2018-12-05T04:00:06-05:00   \n",
       "4   2018-12-05T04:00:06-05:00  2018-12-05T04:00:06-05:00   \n",
       "5   2018-12-05T00:53:12-05:00  2018-12-05T00:53:12-05:00   \n",
       "6   2018-12-04T18:42:11-05:00  2018-12-04T18:42:11-05:00   \n",
       "7   2018-12-04T17:40:03-05:00  2018-12-04T17:40:03-05:00   \n",
       "8   2018-12-04T16:27:20-05:00  2018-12-04T16:27:20-05:00   \n",
       "9   2018-12-04T16:25:31-05:00  2018-12-04T16:19:21-05:00   \n",
       "10  2018-12-04T13:27:48-05:00  2018-12-04T13:27:48-05:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   Remembering George H.W. Bush: Why does it take...   \n",
       "1   David Bossie: House Democrats plan a witch hun...   \n",
       "2   Leslie Marshall: Michelle Obama told the truth...   \n",
       "3   John Stossel: Google and Facebook cross 'The C...   \n",
       "4   Marc Thiessen: For just a few days, let's be t...   \n",
       "5   Gregg Jarrett: Mueller strikes out trying to n...   \n",
       "6   Doug Schoen: Democrats will be hurt by radical...   \n",
       "7   Gutfeld on the passing of George Herbert Walke...   \n",
       "8   Geraldo Rivera: Migrants at our border want a ...   \n",
       "9   George H.W. Bush's service dog 'Sully' isn't a...   \n",
       "10  Michael Goodwin: Trump is prepping for all-out...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/remembering-george-h-w-bush-why-does-...  \n",
       "1   /opinion/david-bossie-house-democrats-plan-a-w...  \n",
       "2   /opinion/leslie-marshall-michelle-obama-told-t...  \n",
       "3   /opinion/john-stossel-google-and-facebook-cros...  \n",
       "4   /opinion/marc-thiessen-for-just-a-few-days-let...  \n",
       "5   /opinion/mueller-strikes-out-trying-to-nail-tr...  \n",
       "6   /opinion/doug-schoen-democrats-will-be-hurt-by...  \n",
       "7   /opinion/gutfeld-on-the-passing-of-george-herb...  \n",
       "8   /opinion/geraldo-rivera-migrants-at-our-border...  \n",
       "9   /opinion/george-h-w-bushs-service-dog-sully-is...  \n",
       "10  /opinion/michael-goodwin-trump-is-prepping-for...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_off = url.replace('offset=30','offset=1000')\n",
    "\n",
    "fox_df(url_off)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An offset of 1,000 takes us back about nine months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 11\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Americas</td>\n",
       "      <td>/category/world/world-regions/americas</td>\n",
       "      <td>Over the weekend, Venezuela succumbed to dicta...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:01:26-04:00</td>\n",
       "      <td>2017-08-02T10:00:00-04:00</td>\n",
       "      <td>The world must turn back Venezuela's growing d...</td>\n",
       "      <td>/opinion/the-world-must-turn-back-venezuelas-g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bellwether</td>\n",
       "      <td>/category/columns/bellwether</td>\n",
       "      <td>Just as President Trump prepares to sign punis...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:01:16-04:00</td>\n",
       "      <td>2017-08-02T10:11:00-04:00</td>\n",
       "      <td>Putin sends Trump a message: Don't mess with us</td>\n",
       "      <td>/opinion/putin-sends-trump-a-message-dont-mess...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Law</td>\n",
       "      <td>/category/politics/executive/law</td>\n",
       "      <td>Special Counsel Robert Mueller has lost his cr...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:01:05-04:00</td>\n",
       "      <td>2017-08-02T10:21:00-04:00</td>\n",
       "      <td>Rep. Trent Franks: Robert Mueller must resign</td>\n",
       "      <td>/opinion/rep-trent-franks-robert-mueller-must-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Health care</td>\n",
       "      <td>/category/politics/executive/health-care</td>\n",
       "      <td>Conservatives should recognize that the key to...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:00:53-04:00</td>\n",
       "      <td>2017-08-02T12:00:00-04:00</td>\n",
       "      <td>Newt Gingrich: Republicans must resist the urg...</td>\n",
       "      <td>/opinion/newt-gingrich-republicans-must-resist...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Budgets</td>\n",
       "      <td>/category/politics/executive/budgets</td>\n",
       "      <td>I follow the news closely, but until I researc...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:00:40-04:00</td>\n",
       "      <td>2017-08-02T15:30:00-04:00</td>\n",
       "      <td>John Stossel: The Trump budget</td>\n",
       "      <td>/opinion/john-stossel-the-trump-budget</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>POLITICS</td>\n",
       "      <td>/category/politics</td>\n",
       "      <td>When the New Yorker's Ryan Lizza first made pu...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:00:31-04:00</td>\n",
       "      <td>2017-08-02T11:17:00-04:00</td>\n",
       "      <td>The latest lesson in media hypocrisy courtesy ...</td>\n",
       "      <td>/opinion/the-latest-lesson-in-media-hypocrisy-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>White House</td>\n",
       "      <td>/category/politics/executive/white-house</td>\n",
       "      <td>An old story from Albany is unforgettable.</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:00:16-04:00</td>\n",
       "      <td>2017-08-02T11:41:00-04:00</td>\n",
       "      <td>Trump must learn to wield his political power ...</td>\n",
       "      <td>/opinion/trump-must-learn-to-wield-his-politic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Law</td>\n",
       "      <td>/category/politics/executive/law</td>\n",
       "      <td>There was a time, not so long ago, when candid...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T17:00:05-04:00</td>\n",
       "      <td>2017-08-02T12:17:00-04:00</td>\n",
       "      <td>Gregg Jarrett: A second special counsel must i...</td>\n",
       "      <td>/opinion/gregg-jarrett-a-second-special-counse...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>IMMIGRATION</td>\n",
       "      <td>/category/us/immigration</td>\n",
       "      <td>At the outset, let me set the record straight:...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T16:59:55-04:00</td>\n",
       "      <td>2017-08-02T12:40:00-04:00</td>\n",
       "      <td>Texas Attorney General Paxton: We must phase o...</td>\n",
       "      <td>/opinion/texas-attorney-general-paxton-we-must...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Todd Starnes</td>\n",
       "      <td>/category/columns/todds-american-dispatch</td>\n",
       "      <td>For the past two years, the Young Americans fo...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T16:59:43-04:00</td>\n",
       "      <td>2017-08-02T18:03:00-04:00</td>\n",
       "      <td>University moves 9/11 flag display to avoid tr...</td>\n",
       "      <td>/opinion/university-moves-9-11-flag-display-to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>National Anthem Protests</td>\n",
       "      <td>/category/news-events/national-anthem-protests</td>\n",
       "      <td>This weekend, the more than 100 NFL players wh...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2017-09-26T16:58:45-04:00</td>\n",
       "      <td>2017-09-26T15:42:00-04:00</td>\n",
       "      <td>Marc Thiessen: Want to protest Trump? Disrespe...</td>\n",
       "      <td>/opinion/marc-thiessen-want-to-protest-trump-d...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               category.name                                    category.url  \\\n",
       "0               The Americas          /category/world/world-regions/americas   \n",
       "1                 Bellwether                    /category/columns/bellwether   \n",
       "2                        Law                /category/politics/executive/law   \n",
       "3                Health care        /category/politics/executive/health-care   \n",
       "4                    Budgets            /category/politics/executive/budgets   \n",
       "5                   POLITICS                              /category/politics   \n",
       "6                White House        /category/politics/executive/white-house   \n",
       "7                        Law                /category/politics/executive/law   \n",
       "8                IMMIGRATION                        /category/us/immigration   \n",
       "9               Todd Starnes       /category/columns/todds-american-dispatch   \n",
       "10  National Anthem Protests  /category/news-events/national-anthem-protests   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   Over the weekend, Venezuela succumbed to dicta...            \n",
       "1   Just as President Trump prepares to sign punis...            \n",
       "2   Special Counsel Robert Mueller has lost his cr...            \n",
       "3   Conservatives should recognize that the key to...            \n",
       "4   I follow the news closely, but until I researc...            \n",
       "5   When the New Yorker's Ryan Lizza first made pu...            \n",
       "6          An old story from Albany is unforgettable.            \n",
       "7   There was a time, not so long ago, when candid...            \n",
       "8   At the outset, let me set the record straight:...            \n",
       "9   For the past two years, the Young Americans fo...            \n",
       "10  This weekend, the more than 100 NFL players wh...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "2   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "3   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "5   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "6   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "7   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "10  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2017-09-26T17:01:26-04:00  2017-08-02T10:00:00-04:00   \n",
       "1   2017-09-26T17:01:16-04:00  2017-08-02T10:11:00-04:00   \n",
       "2   2017-09-26T17:01:05-04:00  2017-08-02T10:21:00-04:00   \n",
       "3   2017-09-26T17:00:53-04:00  2017-08-02T12:00:00-04:00   \n",
       "4   2017-09-26T17:00:40-04:00  2017-08-02T15:30:00-04:00   \n",
       "5   2017-09-26T17:00:31-04:00  2017-08-02T11:17:00-04:00   \n",
       "6   2017-09-26T17:00:16-04:00  2017-08-02T11:41:00-04:00   \n",
       "7   2017-09-26T17:00:05-04:00  2017-08-02T12:17:00-04:00   \n",
       "8   2017-09-26T16:59:55-04:00  2017-08-02T12:40:00-04:00   \n",
       "9   2017-09-26T16:59:43-04:00  2017-08-02T18:03:00-04:00   \n",
       "10  2017-09-26T16:58:45-04:00  2017-09-26T15:42:00-04:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   The world must turn back Venezuela's growing d...   \n",
       "1     Putin sends Trump a message: Don't mess with us   \n",
       "2       Rep. Trent Franks: Robert Mueller must resign   \n",
       "3   Newt Gingrich: Republicans must resist the urg...   \n",
       "4                      John Stossel: The Trump budget   \n",
       "5   The latest lesson in media hypocrisy courtesy ...   \n",
       "6   Trump must learn to wield his political power ...   \n",
       "7   Gregg Jarrett: A second special counsel must i...   \n",
       "8   Texas Attorney General Paxton: We must phase o...   \n",
       "9   University moves 9/11 flag display to avoid tr...   \n",
       "10  Marc Thiessen: Want to protest Trump? Disrespe...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/the-world-must-turn-back-venezuelas-g...  \n",
       "1   /opinion/putin-sends-trump-a-message-dont-mess...  \n",
       "2   /opinion/rep-trent-franks-robert-mueller-must-...  \n",
       "3   /opinion/newt-gingrich-republicans-must-resist...  \n",
       "4              /opinion/john-stossel-the-trump-budget  \n",
       "5   /opinion/the-latest-lesson-in-media-hypocrisy-...  \n",
       "6   /opinion/trump-must-learn-to-wield-his-politic...  \n",
       "7   /opinion/gregg-jarrett-a-second-special-counse...  \n",
       "8   /opinion/texas-attorney-general-paxton-we-must...  \n",
       "9   /opinion/university-moves-9-11-flag-display-to...  \n",
       "10  /opinion/marc-thiessen-want-to-protest-trump-d...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_off = url.replace('offset=30','offset=5000')\n",
    "\n",
    "fox_df(url_off)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5,000 brings us back to 2017. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_off = url.replace('offset=30','offset=10000')\n",
    "\n",
    "fox_df(url_off)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But 10,000 fails. 😞.\n",
    "\n",
    "After attempting multiple values, it appears something around 9,950 is the maximum offset that will return results, which is approximately three years of data. This limit could be hard coded into the API or this could simply be all the data that is available on the website. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return a dataframe of length 11\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Campaigning</td>\n",
       "      <td>/category/politics/elections/campaigning</td>\n",
       "      <td>After watching Donald Trump in a town hall set...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-03T23:41:09-04:00</td>\n",
       "      <td>2016-04-04T00:00:00-04:00</td>\n",
       "      <td>After a terrible week Trump sticks with style ...</td>\n",
       "      <td>/opinion/after-a-terrible-week-trump-sticks-wi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MILITARY</td>\n",
       "      <td>/category/us/military</td>\n",
       "      <td>Since the Vietnam War, the military has mainta...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-03T21:12:03-04:00</td>\n",
       "      <td>2016-04-04T00:00:00-04:00</td>\n",
       "      <td>Latest attack on POW/MIA religious symbol is c...</td>\n",
       "      <td>/opinion/latest-attack-on-pow-mia-religious-sy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>It’s great that some NBA players have picked u...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com///static.foxnews.com/s...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T13:34:37-04:00</td>\n",
       "      <td>2011-10-18T13:37:00-04:00</td>\n",
       "      <td>Restaurants? Internships? Selling Furniture? L...</td>\n",
       "      <td>/opinion/restaurants-internships-selling-furni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Supreme Court</td>\n",
       "      <td>/category/politics/judiciary/supreme-court</td>\n",
       "      <td>Democratic presidential candidate Hillary Clin...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T13:12:37-04:00</td>\n",
       "      <td>2016-04-01T07:53:00-04:00</td>\n",
       "      <td>Hypocrisy, thy name is Hillary Clinton</td>\n",
       "      <td>/opinion/hypocrisy-thy-name-is-hillary-clinton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Youngsters at the Volunteer State’s flagship u...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T10:32:04-04:00</td>\n",
       "      <td>2016-04-01T09:45:00-04:00</td>\n",
       "      <td>Tennessee lawmakers fume over state university...</td>\n",
       "      <td>/opinion/tennessee-lawmakers-fume-over-state-u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>POLITICS</td>\n",
       "      <td>/category/politics</td>\n",
       "      <td>President Obama, our openness to refugees is n...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T09:36:30-04:00</td>\n",
       "      <td>2016-04-01T06:00:00-04:00</td>\n",
       "      <td>Rep. Blackburn: An open door for Syrian refuge...</td>\n",
       "      <td>/opinion/rep-blackburn-an-open-door-for-syrian...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>TERRORISM</td>\n",
       "      <td>/category/world/terrorism</td>\n",
       "      <td>Imagine more of the African continent engulfed...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/a57.foxnews.com/media2...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T06:00:33-04:00</td>\n",
       "      <td>2016-04-01T06:00:00-04:00</td>\n",
       "      <td>Islamist violence threatens Judeo-Christian ci...</td>\n",
       "      <td>/opinion/islamist-violence-threatens-judeo-chr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Iran</td>\n",
       "      <td>/category/world/conflicts/iran</td>\n",
       "      <td>Iran tested two ballistic missiles last fall a...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com///static.foxnews.com/s...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T05:30:01-04:00</td>\n",
       "      <td>2016-04-01T05:00:00-04:00</td>\n",
       "      <td>The real meaning of Iranian Supreme Leader Kha...</td>\n",
       "      <td>/opinion/the-real-meaning-of-iranian-supreme-l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Zika</td>\n",
       "      <td>/category/health/infectious-disease/zika</td>\n",
       "      <td>To protect Americans, all levels of government...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-04-01T05:00:19-04:00</td>\n",
       "      <td>2016-04-01T05:00:00-04:00</td>\n",
       "      <td>CDC Chief: Zika is coming. To fully protect Am...</td>\n",
       "      <td>/opinion/cdc-chief-zika-is-coming-to-fully-pro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>At 31, I closed my bedroom door when it came t...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-03-31T16:18:31-04:00</td>\n",
       "      <td>2016-03-31T16:00:00-04:00</td>\n",
       "      <td>Why I'm proud to be a prude</td>\n",
       "      <td>/opinion/why-im-proud-to-be-a-prude</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Campaigning</td>\n",
       "      <td>/category/politics/elections/campaigning</td>\n",
       "      <td>Donald Trump is the leading candidate for the ...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2016-03-31T15:22:32-04:00</td>\n",
       "      <td>2016-03-30T12:24:00-04:00</td>\n",
       "      <td>Who influences Donald Trump on foreign affairs...</td>\n",
       "      <td>/opinion/who-influences-donald-trump-on-foreig...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    category.name                                category.url  \\\n",
       "0     Campaigning    /category/politics/elections/campaigning   \n",
       "1        MILITARY                       /category/us/military   \n",
       "2         OPINION                           /category/opinion   \n",
       "3   Supreme Court  /category/politics/judiciary/supreme-court   \n",
       "4         OPINION                           /category/opinion   \n",
       "5        POLITICS                          /category/politics   \n",
       "6       TERRORISM                   /category/world/terrorism   \n",
       "7            Iran              /category/world/conflicts/iran   \n",
       "8            Zika    /category/health/infectious-disease/zika   \n",
       "9         OPINION                           /category/opinion   \n",
       "10    Campaigning    /category/politics/elections/campaigning   \n",
       "\n",
       "                                          description duration  \\\n",
       "0   After watching Donald Trump in a town hall set...            \n",
       "1   Since the Vietnam War, the military has mainta...            \n",
       "2   It’s great that some NBA players have picked u...            \n",
       "3   Democratic presidential candidate Hillary Clin...            \n",
       "4   Youngsters at the Volunteer State’s flagship u...            \n",
       "5   President Obama, our openness to refugees is n...            \n",
       "6   Imagine more of the African continent engulfed...            \n",
       "7   Iran tested two ballistic missiles last fall a...            \n",
       "8   To protect Americans, all levels of government...            \n",
       "9   At 31, I closed my bedroom door when it came t...            \n",
       "10  Donald Trump is the leading candidate for the ...            \n",
       "\n",
       "                                             imageUrl  isBreaking  isLive  \\\n",
       "0   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "2   https://a57.foxnews.com///static.foxnews.com/s...       False   False   \n",
       "3   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "4   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "5   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "6   https://a57.foxnews.com/a57.foxnews.com/media2...       False   False   \n",
       "7   https://a57.foxnews.com///static.foxnews.com/s...       False   False   \n",
       "8   https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "9   https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "10  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "\n",
       "            lastPublishedDate            publicationDate  \\\n",
       "0   2016-04-03T23:41:09-04:00  2016-04-04T00:00:00-04:00   \n",
       "1   2016-04-03T21:12:03-04:00  2016-04-04T00:00:00-04:00   \n",
       "2   2016-04-01T13:34:37-04:00  2011-10-18T13:37:00-04:00   \n",
       "3   2016-04-01T13:12:37-04:00  2016-04-01T07:53:00-04:00   \n",
       "4   2016-04-01T10:32:04-04:00  2016-04-01T09:45:00-04:00   \n",
       "5   2016-04-01T09:36:30-04:00  2016-04-01T06:00:00-04:00   \n",
       "6   2016-04-01T06:00:33-04:00  2016-04-01T06:00:00-04:00   \n",
       "7   2016-04-01T05:30:01-04:00  2016-04-01T05:00:00-04:00   \n",
       "8   2016-04-01T05:00:19-04:00  2016-04-01T05:00:00-04:00   \n",
       "9   2016-03-31T16:18:31-04:00  2016-03-31T16:00:00-04:00   \n",
       "10  2016-03-31T15:22:32-04:00  2016-03-30T12:24:00-04:00   \n",
       "\n",
       "                                                title  \\\n",
       "0   After a terrible week Trump sticks with style ...   \n",
       "1   Latest attack on POW/MIA religious symbol is c...   \n",
       "2   Restaurants? Internships? Selling Furniture? L...   \n",
       "3              Hypocrisy, thy name is Hillary Clinton   \n",
       "4   Tennessee lawmakers fume over state university...   \n",
       "5   Rep. Blackburn: An open door for Syrian refuge...   \n",
       "6   Islamist violence threatens Judeo-Christian ci...   \n",
       "7   The real meaning of Iranian Supreme Leader Kha...   \n",
       "8   CDC Chief: Zika is coming. To fully protect Am...   \n",
       "9                         Why I'm proud to be a prude   \n",
       "10  Who influences Donald Trump on foreign affairs...   \n",
       "\n",
       "                                                  url  \n",
       "0   /opinion/after-a-terrible-week-trump-sticks-wi...  \n",
       "1   /opinion/latest-attack-on-pow-mia-religious-sy...  \n",
       "2   /opinion/restaurants-internships-selling-furni...  \n",
       "3      /opinion/hypocrisy-thy-name-is-hillary-clinton  \n",
       "4   /opinion/tennessee-lawmakers-fume-over-state-u...  \n",
       "5   /opinion/rep-blackburn-an-open-door-for-syrian...  \n",
       "6   /opinion/islamist-violence-threatens-judeo-chr...  \n",
       "7   /opinion/the-real-meaning-of-iranian-supreme-l...  \n",
       "8   /opinion/cdc-chief-zika-is-coming-to-fully-pro...  \n",
       "9                 /opinion/why-im-proud-to-be-a-prude  \n",
       "10  /opinion/who-influences-donald-trump-on-foreig...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_off = url.replace('offset=30','offset=9950')\n",
    "\n",
    "fox_df(url_off)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step in collecting the article metadata via the API is to make all the necessary API calls to gather all the relevant article metadata. I slightly revise the `fox_df` function so that it takes a value of an offset. I hardcoded the URL, fixing the size to 30.  A more robust function would turn the entire set of parameters into a dictionary that was modifiable. I also took out the print statement which would clutter the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fox_df(offset):\n",
    "    url = ('https://www.foxnews.com/api/article-search?'\n",
    "           'isCategory=true&isTag=false&isKeyword=false&'\n",
    "           'isFixed=false&isFeedUrl=false&searchSelected=opinion&'\n",
    "           'contentTypes=%7B%22interactive%22:true,%22slideshow%22:true,%22video%22:false,%22article%22:true%7D&'\n",
    "           'size=30&offset=0')\n",
    "    \n",
    "    url = url.replace('offset=0', 'offset=%s' % offset)\n",
    "    \n",
    "    r = requests.get(url)\n",
    "    df = json_normalize(r.json())\n",
    "    return df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, I loop over the function. I create an empty dataframe and then append the results after each loop. I pause three seconds each pass in order to not access the web server too many times. The first time I ran this, I only retrieved a few pages of results to make sure everything worked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep # for pausing\n",
    "\n",
    "# create empty dataframe to store results\n",
    "fox_opinion_df = pd.DataFrame() \n",
    "\n",
    "# Create a loop that counts up by 30.\n",
    "for offset in range(0, 1000, 30):\n",
    "    new_df = fox_df(offset)\n",
    "    \n",
    "    # Add the new results to the existing database\n",
    "    fox_opinion_df = fox_opinion_df.append(new_df, ignore_index=True)\n",
    "    \n",
    "    # Pause for three seconds to be polite to the web server\n",
    "    sleep(3)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category.name</th>\n",
       "      <th>category.url</th>\n",
       "      <th>description</th>\n",
       "      <th>duration</th>\n",
       "      <th>imageUrl</th>\n",
       "      <th>isBreaking</th>\n",
       "      <th>isLive</th>\n",
       "      <th>lastPublishedDate</th>\n",
       "      <th>publicationDate</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1015</th>\n",
       "      <td>Tucker Carlson Tonight</td>\n",
       "      <td>/category/shows/tucker-carlson-tonight</td>\n",
       "      <td>Jerome Corsi though has not backed down.</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T08:08:54-05:00</td>\n",
       "      <td>2018-12-04T08:08:54-05:00</td>\n",
       "      <td>Tucker Carlson: The Mueller probe continues to...</td>\n",
       "      <td>/opinion/tucker-carlson-the-mueller-probe-cont...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1016</th>\n",
       "      <td>Sean Hannity's Monologue</td>\n",
       "      <td>/category/shows/hannity/transcript/hannitys-mo...</td>\n",
       "      <td>George H. W. Bush never forgot the brave men w...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T07:41:41-05:00</td>\n",
       "      <td>2018-12-04T07:41:41-05:00</td>\n",
       "      <td>Sean Hannity: George Bush was a patriot and ne...</td>\n",
       "      <td>/opinion/sean-hannity-george-bush-was-a-patrio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1017</th>\n",
       "      <td>Laura Ingraham's Monologue</td>\n",
       "      <td>/category/shows/ingraham-angle/transcript/laur...</td>\n",
       "      <td>At a time like this when a father, a grandfath...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T07:10:28-05:00</td>\n",
       "      <td>2018-12-04T07:10:28-05:00</td>\n",
       "      <td>Laura Ingraham: George Bush would have wanted ...</td>\n",
       "      <td>/opinion/laura-ingraham-george-bush-would-have...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1018</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>Christmas inspires these moments to cherish, a...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/static.foxnews.com/fox...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T04:00:57-05:00</td>\n",
       "      <td>2018-12-04T04:00:57-05:00</td>\n",
       "      <td>Sam Sorbo: The greatest Christmas gift I can t...</td>\n",
       "      <td>/opinion/sam-sorbo-the-greatest-christmas-gift...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1019</th>\n",
       "      <td>OPINION</td>\n",
       "      <td>/category/opinion</td>\n",
       "      <td>It would be a Christmas miracle if Melania Tru...</td>\n",
       "      <td></td>\n",
       "      <td>https://a57.foxnews.com/media2.foxnews.com/Bri...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2018-12-04T04:00:22-05:00</td>\n",
       "      <td>2018-12-04T04:00:22-05:00</td>\n",
       "      <td>Lauren Appell: Media scrooges need to lighten ...</td>\n",
       "      <td>/opinion/media-scrooges-need-to-lighten-up-on-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   category.name  \\\n",
       "1015      Tucker Carlson Tonight   \n",
       "1016    Sean Hannity's Monologue   \n",
       "1017  Laura Ingraham's Monologue   \n",
       "1018                     OPINION   \n",
       "1019                     OPINION   \n",
       "\n",
       "                                           category.url  \\\n",
       "1015             /category/shows/tucker-carlson-tonight   \n",
       "1016  /category/shows/hannity/transcript/hannitys-mo...   \n",
       "1017  /category/shows/ingraham-angle/transcript/laur...   \n",
       "1018                                  /category/opinion   \n",
       "1019                                  /category/opinion   \n",
       "\n",
       "                                            description duration  \\\n",
       "1015           Jerome Corsi though has not backed down.            \n",
       "1016  George H. W. Bush never forgot the brave men w...            \n",
       "1017  At a time like this when a father, a grandfath...            \n",
       "1018  Christmas inspires these moments to cherish, a...            \n",
       "1019  It would be a Christmas miracle if Melania Tru...            \n",
       "\n",
       "                                               imageUrl  isBreaking  isLive  \\\n",
       "1015  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "1016  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1017  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "1018  https://a57.foxnews.com/static.foxnews.com/fox...       False   False   \n",
       "1019  https://a57.foxnews.com/media2.foxnews.com/Bri...       False   False   \n",
       "\n",
       "              lastPublishedDate            publicationDate  \\\n",
       "1015  2018-12-04T08:08:54-05:00  2018-12-04T08:08:54-05:00   \n",
       "1016  2018-12-04T07:41:41-05:00  2018-12-04T07:41:41-05:00   \n",
       "1017  2018-12-04T07:10:28-05:00  2018-12-04T07:10:28-05:00   \n",
       "1018  2018-12-04T04:00:57-05:00  2018-12-04T04:00:57-05:00   \n",
       "1019  2018-12-04T04:00:22-05:00  2018-12-04T04:00:22-05:00   \n",
       "\n",
       "                                                  title  \\\n",
       "1015  Tucker Carlson: The Mueller probe continues to...   \n",
       "1016  Sean Hannity: George Bush was a patriot and ne...   \n",
       "1017  Laura Ingraham: George Bush would have wanted ...   \n",
       "1018  Sam Sorbo: The greatest Christmas gift I can t...   \n",
       "1019  Lauren Appell: Media scrooges need to lighten ...   \n",
       "\n",
       "                                                    url  \n",
       "1015  /opinion/tucker-carlson-the-mueller-probe-cont...  \n",
       "1016  /opinion/sean-hannity-george-bush-was-a-patrio...  \n",
       "1017  /opinion/laura-ingraham-george-bush-would-have...  \n",
       "1018  /opinion/sam-sorbo-the-greatest-christmas-gift...  \n",
       "1019  /opinion/media-scrooges-need-to-lighten-up-on-...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fox_opinion_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataframe can be stored as either as CSV file or JSON. CSV is best if you want to use it elsewhere, while a JSON usually is more robust to handling text strings which somtimes trip up CSV readers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "fox_opinion_df.to_csv('foxnews_opinion.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "fox_opinion_df.to_json('foxnews_opinion.json', orient ='records')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things aren't always this easy, but when you stumble across an undocumented API you can quickly put together a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}