{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Gathering historical data about the addition of newspaper titles to Trove\n", "\n", "The number of digitised newspapers available through Trove has increased dramatically since 2009. Understanding when newspapers were added is important for historiographical purposes, but there's no data about this available directly from Trove. This notebook uses web archives to extract lists of newspapers in Trove over time, and chart Trove's development.\n", "\n", "Trove has always provided a browseable list of digitised newspaper titles. The url and format of this list has changed over time, but it's possible to find captures of this page in the Internet Archive and extract the full list of titles. The pages are also captured in the Australian Web Archive, but the Wayback Machine has a more detailed record.\n", "\n", "The pages that I'm looking for are:\n", "\n", "* [http://trove.nla.gov.au/ndp/del/titles](https://web.archive.org/web/*/http://trove.nla.gov.au/ndp/del/titles)\n", "* [https://trove.nla.gov.au/newspaper/about](https://web.archive.org/web/*/https://trove.nla.gov.au/newspaper/about)\n", "\n", "This notebook creates the following data files:\n", "\n", "* [trove_newspaper_titles_2009_2021.csv](https://github.com/GLAM-Workbench/trove-newspapers/blob/master/trove_newspaper_titles_2009_2021.csv) – complete dataset of captures and titles\n", "* [trove_newspaper_titles_first_appearance_2009_2021.csv](https://github.com/GLAM-Workbench/trove-newspapers/blob/master/trove_newspaper_titles_first_appearance_2009_2021.csv) – filtered dataset, showing only the first appearance of each title / place / date range combination\n", "\n", "I've also created a [browseable list of titles](https://gist.github.com/wragge/7d80507c3e7957e271c572b8f664031a), showing when they first appeared in Trove." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import json\n", "import re\n", "from pathlib import Path\n", "\n", "import altair as alt\n", "import arrow\n", "import pandas as pd\n", "import requests_cache\n", "from bs4 import BeautifulSoup\n", "from requests.adapters import HTTPAdapter\n", "from requests.packages.urllib3.util.retry import Retry\n", "from surt import surt\n", "\n", "s = requests_cache.CachedSession(\"archived_titles\")\n", "retries = Retry(total=5, backoff_factor=1, status_forcelist=[502, 503, 504])\n", "s.mount(\"https://\", HTTPAdapter(max_retries=retries))\n", "s.mount(\"http://\", HTTPAdapter(max_retries=retries))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Code for harvesting web archive captures\n", "\n", "We're using the Memento protocol to get a list of captures. See the [Web Archives section](https://glam-workbench.net/web-archives/) of the GLAM Workbench for more details." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# The code in this cell is copied from notebooks in the Web Archives section of the GLAM Workbench (https://glam-workbench.net/web-archives/)\n", "# In particular see: https://glam-workbench.net/web-archives/#find-all-the-archived-versions-of-a-web-page\n", "\n", "# These are the repositories we'll be using\n", "TIMEGATES = {\n", " \"awa\": \"https://web.archive.org.au/awa/\",\n", " \"nzwa\": \"https://ndhadeliver.natlib.govt.nz/webarchive/wayback/\",\n", " \"ukwa\": \"https://www.webarchive.org.uk/wayback/en/archive/\",\n", " \"ia\": \"https://web.archive.org/web/\",\n", "}\n", "\n", "\n", "def convert_lists_to_dicts(results):\n", " \"\"\"\n", " Converts IA style timemap (a JSON array of arrays) to a list of dictionaries.\n", " Renames keys to standardise IA with other Timemaps.\n", " \"\"\"\n", " if results:\n", " keys = results[0]\n", " results_as_dicts = [dict(zip(keys, v)) for v in results[1:]]\n", " else:\n", " results_as_dicts = results\n", " for d in results_as_dicts:\n", " d[\"status\"] = d.pop(\"statuscode\")\n", " d[\"mime\"] = d.pop(\"mimetype\")\n", " d[\"url\"] = d.pop(\"original\")\n", " return results_as_dicts\n", "\n", "\n", "def get_capture_data_from_memento(url, request_type=\"head\"):\n", " \"\"\"\n", " For OpenWayback systems this can get some extra capture info to insert into Timemaps.\n", " \"\"\"\n", " if request_type == \"head\":\n", " response = s.head(url)\n", " else:\n", " response = s.get(url)\n", " headers = response.headers\n", " length = headers.get(\"x-archive-orig-content-length\")\n", " status = headers.get(\"x-archive-orig-status\")\n", " status = status.split(\" \")[0] if status else None\n", " mime = headers.get(\"x-archive-orig-content-type\")\n", " mime = mime.split(\";\")[0] if mime else None\n", " return {\"length\": length, \"status\": status, \"mime\": mime}\n", "\n", "\n", "def convert_link_to_json(results, enrich_data=False):\n", " \"\"\"\n", " Converts link formatted Timemap to JSON.\n", " \"\"\"\n", " data = []\n", " for line in results.splitlines():\n", " parts = line.split(\"; \")\n", " if len(parts) > 1:\n", " link_type = re.search(\n", " r'rel=\"(original|self|timegate|first memento|last memento|memento)\"',\n", " parts[1],\n", " ).group(1)\n", " if link_type == \"memento\":\n", " link = parts[0].strip(\"<>\")\n", " timestamp, original = re.search(r\"/(\\d{14})/(.*)$\", link).groups()\n", " capture = {\n", " \"urlkey\": surt(original),\n", " \"timestamp\": timestamp,\n", " \"url\": original,\n", " }\n", " if enrich_data:\n", " capture.update(get_capture_data_from_memento(link))\n", " print(capture)\n", " data.append(capture)\n", " return data\n", "\n", "\n", "def get_timemap_as_json(timegate, url, enrich_data=False):\n", " \"\"\"\n", " Get a Timemap then normalise results (if necessary) to return a list of dicts.\n", " \"\"\"\n", " tg_url = f\"{TIMEGATES[timegate]}timemap/json/{url}/\"\n", " response = s.get(tg_url)\n", " response_type = response.headers[\"content-type\"]\n", " if response_type == \"text/x-ndjson\":\n", " data = [json.loads(line) for line in response.text.splitlines()]\n", " elif response_type == \"application/json\":\n", " data = convert_lists_to_dicts(response.json())\n", " elif response_type in [\"application/link-format\", \"text/html;charset=utf-8\"]:\n", " data = convert_link_to_json(response.text, enrich_data=enrich_data)\n", " return data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Harvest the title data from the Internet Archive\n", "\n", "This gets the web page captures from the Internet Archive, scrapes the list of titles from the page, then does a bit of normalisation of the title data." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "titles = []\n", "\n", "# These are the pages that listed available titles.\n", "# There was a change in 2016\n", "pages = [\n", " {\"url\": \"http://trove.nla.gov.au/ndp/del/titles\", \"path\": \"/ndp/del/title/\"},\n", " {\"url\": \"https://trove.nla.gov.au/newspaper/about\", \"path\": \"/newspaper/title/\"},\n", "]\n", "\n", "for page in pages:\n", " for capture in get_timemap_as_json(\"ia\", page[\"url\"]):\n", " if capture[\"status\"] == \"200\":\n", " url = f'https://web.archive.org/web/{capture[\"timestamp\"]}id_/{capture[\"url\"]}'\n", " # print(url)\n", " capture_date = arrow.get(capture[\"timestamp\"][:8], \"YYYYMMDD\").format(\n", " \"YYYY-MM-DD\"\n", " )\n", " # print(capture_date)\n", " response = s.get(url)\n", " soup = BeautifulSoup(response.content)\n", " title_links = soup.find_all(\"a\", href=re.compile(page[\"path\"]))\n", " for title in title_links:\n", " # Get the title text\n", " full_title = title.get_text().strip()\n", "\n", " # Get the title id\n", " title_id = re.search(r\"\\/(\\d+)\\/?$\", title[\"href\"]).group(1)\n", "\n", " # Most of the code below is aimed at normalising the publication place and dates values to allow for easy grouping & deduplication\n", " brief_title = re.sub(r\"\\(.+\\)\\s*$\", \"\", full_title).strip()\n", " try:\n", " details = re.search(r\"\\((.+)\\)\\s*$\", full_title).group(1).split(\":\")\n", " except AttributeError:\n", " place = \"\"\n", " dates = \"\"\n", " else:\n", " try:\n", " place = details[0].strip()\n", " # Normalise states\n", " try:\n", " place = re.sub(\n", " r\"(, )?([A-Za-z]+)[\\.\\s]*$\",\n", " lambda match: f'{match.group(1) if match.group(1) else \"\"}{match.group(2).upper()}',\n", " place,\n", " )\n", " except AttributeError:\n", " pass\n", " # Normalise dates\n", " dates = \" - \".join(\n", " [d.strip() for d in details[1].strip().split(\"-\")]\n", " )\n", " except IndexError:\n", " place = \"\"\n", " dates = \" - \".join(\n", " [d.strip() for d in details[0].strip().split(\"-\")]\n", " )\n", " titles.append(\n", " {\n", " \"title_id\": title_id,\n", " \"full_title\": full_title,\n", " \"title\": brief_title,\n", " \"place\": place,\n", " \"dates\": dates,\n", " \"capture_date\": capture_date,\n", " \"capture_timestamp\": capture[\"timestamp\"],\n", " }\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convert the title data to a DataFrame for analysis" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame(titles)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | title_id | \n", "full_title | \n", "title | \n", "place | \n", "dates | \n", "capture_date | \n", "capture_timestamp | \n", "
---|---|---|---|---|---|---|---|
0 | \n", "34 | \n", "Advertiser (Adelaide, SA : 1889-1931) | \n", "Advertiser | \n", "Adelaide, SA | \n", "1889 - 1931 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
1 | \n", "13 | \n", "Argus (Melbourne, Vic. : 1848-1954) | \n", "Argus | \n", "Melbourne, VIC | \n", "1848 - 1954 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
2 | \n", "16 | \n", "Brisbane Courier (Qld. : 1864-1933) | \n", "Brisbane Courier | \n", "QLD | \n", "1864 - 1933 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
3 | \n", "11 | \n", "Canberra Times (ACT : 1926-1954) | \n", "Canberra Times | \n", "ACT | \n", "1926 - 1954 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
4 | \n", "24 | \n", "Colonial Times (Hobart, Tas. : 1828-1857) | \n", "Colonial Times | \n", "Hobart, TAS | \n", "1828 - 1857 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
107017 | \n", "1331 | \n", "South Australian Record and Australasian and S... | \n", "South Australian Record and Australasian and S... | \n", "London, ENGLAND | \n", "1840 - 1841 | \n", "2022-01-16 | \n", "20220116142742 | \n", "
107018 | \n", "1369 | \n", "Territory of Papua Government Gazette (Papua N... | \n", "Territory of Papua Government Gazette | \n", "Papua New GUINEA | \n", "1906 - 1942 | \n", "2022-01-16 | \n", "20220116142742 | \n", "
107019 | \n", "1371 | \n", "Territory of Papua and New Guinea Government G... | \n", "Territory of Papua and New Guinea Government G... | \n", "\n", " | 1949 - 1971 | \n", "2022-01-16 | \n", "20220116142742 | \n", "
107020 | \n", "1370 | \n", "Territory of Papua-New Guinea Government Gazet... | \n", "Territory of Papua-New Guinea Government Gazette | \n", "\n", " | 1945 - 1949 | \n", "2022-01-16 | \n", "20220116142742 | \n", "
107021 | \n", "1391 | \n", "Tribune (Philippines : 1932 - 1945) | \n", "Tribune | \n", "PHILIPPINES | \n", "1932 - 1945 | \n", "2022-01-16 | \n", "20220116142742 | \n", "
107022 rows × 7 columns
\n", "\n", " | capture_date | \n", "total | \n", "
---|---|---|
0 | \n", "2022-01-16 | \n", "1700 | \n", "
1 | \n", "2022-01-10 | \n", "1700 | \n", "
2 | \n", "2021-12-13 | \n", "1697 | \n", "
3 | \n", "2021-11-20 | \n", "1690 | \n", "
4 | \n", "2021-11-16 | \n", "1690 | \n", "
... | \n", "... | \n", "... | \n", "
115 | \n", "2010-05-01 | \n", "37 | \n", "
116 | \n", "2009-11-24 | \n", "34 | \n", "
117 | \n", "2009-11-22 | \n", "34 | \n", "
118 | \n", "2009-12-12 | \n", "34 | \n", "
119 | \n", "2009-11-12 | \n", "34 | \n", "
120 rows × 2 columns
\n", "\n", " | title_id | \n", "full_title | \n", "title | \n", "place | \n", "dates | \n", "capture_date | \n", "capture_timestamp | \n", "
---|---|---|---|---|---|---|---|
0 | \n", "34 | \n", "Advertiser (Adelaide, SA : 1889-1931) | \n", "Advertiser | \n", "Adelaide, SA | \n", "1889 - 1931 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
1 | \n", "13 | \n", "Argus (Melbourne, Vic. : 1848-1954) | \n", "Argus | \n", "Melbourne, VIC | \n", "1848 - 1954 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
2 | \n", "16 | \n", "Brisbane Courier (Qld. : 1864-1933) | \n", "Brisbane Courier | \n", "QLD | \n", "1864 - 1933 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
3 | \n", "11 | \n", "Canberra Times (ACT : 1926-1954) | \n", "Canberra Times | \n", "ACT | \n", "1926 - 1954 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
4 | \n", "24 | \n", "Colonial Times (Hobart, Tas. : 1828-1857) | \n", "Colonial Times | \n", "Hobart, TAS | \n", "1828 - 1857 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
105023 | \n", "1773 | \n", "Dawn Newsletter (Perth, WA : 1952 - 1954) | \n", "Dawn Newsletter | \n", "Perth, WA | \n", "1952 - 1954 | \n", "2022-01-10 | \n", "20220110214554 | \n", "
105112 | \n", "1388 | \n", "La Rondine (Perth, WA : 1970 - 1974; 1983 - 1984) | \n", "La Rondine | \n", "Perth, WA | \n", "1970 - 1974; 1983 - 1984 | \n", "2022-01-10 | \n", "20220110214554 | \n", "
105121 | \n", "1537 | \n", "Listening Post (Perth, WA : 1921 - 1954) | \n", "Listening Post | \n", "Perth, WA | \n", "1921 - 1954 | \n", "2022-01-10 | \n", "20220110214554 | \n", "
105274 | \n", "99 | \n", "Western Argus (Kalgoorlie, WA : 1894 - 1895) | \n", "Western Argus | \n", "Kalgoorlie, WA | \n", "1894 - 1895 | \n", "2022-01-10 | \n", "20220110214554 | \n", "
106887 | \n", "1649 | \n", "North Coolgardie Herald and Miners Daily News ... | \n", "North Coolgardie Herald and Miners Daily News | \n", "Menzies, WA | \n", "1899 - 1904 | \n", "2022-01-16 | \n", "20220116142742 | \n", "
2120 rows × 7 columns
\n", "\n", " | title_id | \n", "full_title | \n", "title | \n", "place | \n", "dates | \n", "capture_date | \n", "capture_timestamp | \n", "
---|---|---|---|---|---|---|---|
3 | \n", "11 | \n", "Canberra Times (ACT : 1926-1954) | \n", "Canberra Times | \n", "ACT | \n", "1926 - 1954 | \n", "2009-11-12 | \n", "20091112000713 | \n", "
9395 | \n", "11 | \n", "Canberra Times (ACT : 1926 - 1995) | \n", "Canberra Times | \n", "ACT | \n", "1926 - 1995 | \n", "2012-12-27 | \n", "20121227113753 | \n", "