{ "cells": [ { "cell_type": "markdown", "id": "a2cca763-3b3e-42a6-9381-c9478d574abd", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# Harvesting articles that mention \"Anzac Day\" on Anzac Day\n", "\n", "The Trove Newspaper Harvester web app and command-line tool make it easy for you to harvest the results of a single search. But if you want to harvest very large or complex searches, you might find it easier to import the `trove_newspaper_harvester` library directly and take control of the harvesting process.\n", "\n", "For example, how would you harvest all of the newspaper articles mentioning \"Anzac Day\" that were published *on* Anzac Day, 25 April? It's possible to search for results from a single day using the `date` index. So, theoretically, you could combine multiple dates using `OR` and build a very long search query by doing something like this:\n", "\n", "``` python\n", "days = []\n", "for year in range(1916, 1955):\n", " days.append(f\"date:[{year}-04-24T00:00:00Z TO {year}-04-25T00:00:00Z]\")\n", "query_string = f'\"anzac day\" AND ({\" OR \".join(days)})'\n", "```\n", "\n", "However, if you try searching in Trove using the query string generated by this code it [returns no results](https://trove.nla.gov.au/search/category/newspapers?keyword=%22anzac%20day%22%20AND%20%28date%3A%5B1916-04-24T00%3A00%3A00Z%20TO%201916-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1917-04-24T00%3A00%3A00Z%20TO%201917-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1918-04-24T00%3A00%3A00Z%20TO%201918-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1919-04-24T00%3A00%3A00Z%20TO%201919-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1920-04-24T00%3A00%3A00Z%20TO%201920-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1921-04-24T00%3A00%3A00Z%20TO%201921-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1922-04-24T00%3A00%3A00Z%20TO%201922-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1923-04-24T00%3A00%3A00Z%20TO%201923-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1924-04-24T00%3A00%3A00Z%20TO%201924-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1925-04-24T00%3A00%3A00Z%20TO%201925-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1926-04-24T00%3A00%3A00Z%20TO%201926-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1927-04-24T00%3A00%3A00Z%20TO%201927-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1928-04-24T00%3A00%3A00Z%20TO%201928-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1929-04-24T00%3A00%3A00Z%20TO%201929-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1930-04-24T00%3A00%3A00Z%20TO%201930-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1931-04-24T00%3A00%3A00Z%20TO%201931-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1932-04-24T00%3A00%3A00Z%20TO%201932-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1933-04-24T00%3A00%3A00Z%20TO%201933-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1934-04-24T00%3A00%3A00Z%20TO%201934-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1935-04-24T00%3A00%3A00Z%20TO%201935-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1936-04-24T00%3A00%3A00Z%20TO%201936-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1937-04-24T00%3A00%3A00Z%20TO%201937-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1938-04-24T00%3A00%3A00Z%20TO%201938-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1939-04-24T00%3A00%3A00Z%20TO%201939-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1940-04-24T00%3A00%3A00Z%20TO%201940-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1941-04-24T00%3A00%3A00Z%20TO%201941-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1942-04-24T00%3A00%3A00Z%20TO%201942-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1943-04-24T00%3A00%3A00Z%20TO%201943-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1944-04-24T00%3A00%3A00Z%20TO%201944-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1945-04-24T00%3A00%3A00Z%20TO%201945-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1946-04-24T00%3A00%3A00Z%20TO%201946-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1947-04-24T00%3A00%3A00Z%20TO%201947-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1948-04-24T00%3A00%3A00Z%20TO%201948-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1949-04-24T00%3A00%3A00Z%20TO%201949-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1950-04-24T00%3A00%3A00Z%20TO%201950-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1951-04-24T00%3A00%3A00Z%20TO%201951-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1952-04-24T00%3A00%3A00Z%20TO%201952-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1953-04-24T00%3A00%3A00Z%20TO%201953-04-25T00%3A00%3A00Z%5D%20OR%20date%3A%5B1954-04-24T00%3A00%3A00Z%20TO%201954-04-25T00%3A00%3A00Z%5D%29). Presumably it has hit a limit on query length. But even if you reduce the span of years you can get some odd results. It seems safer to search for each day independently, but how can you do that without manually creating lots of separate harvests?\n", "\n", "The example below does the following:\n", "\n", "- imports the `trove_newspaper_harvester` `Harvester` class and `prepare_query` function\n", "- uses `prepare_query` to create the basic set of parameters (without the date search)\n", "- loops through the desired span of years, adding the date search to the query, initialising the `Harvester`, running the harvest, and saving the results as a CSV file\n", "\n", "It also uses the `data_dir` and `harvest_dir` parameters of `Harvester` to tell it where to save the results. These options help you keep related searches together. In this instance, all the searches are saved in the `anzac-day` parent directory, with each individual search saved in a directory named by the year of the search query. So you end up with one results directory for each year in the span. The separate results files can be easily combined, as shown below." ] }, { "cell_type": "markdown", "id": "350d61ed-9536-4830-8eee-0b481ceabd10", "metadata": {}, "source": [ "## Set things up" ] }, { "cell_type": "code", "execution_count": 1, "id": "27667834-d179-4e21-8294-7dd43cc46b11", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import os\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "\n", "# importing the trove_newspaper_harvester!\n", "from trove_newspaper_harvester.core import Harvester, prepare_query" ] }, { "cell_type": "code", "execution_count": 2, "id": "73853861-8c9f-430b-8b15-73ca36aa9c30", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "%%capture\n", "# Load variables from the .env file if it exists\n", "# Use %%capture to suppress messages\n", "%load_ext dotenv\n", "%dotenv" ] }, { "cell_type": "code", "execution_count": 3, "id": "892af7ae-14fd-4f42-8d6b-0289b0576f10", "metadata": {}, "outputs": [], "source": [ "# Insert your Trove API key\n", "API_KEY = \"YOUR API KEY\"\n", "\n", "# Use api key value from environment variables if it is available\n", "if os.getenv(\"TROVE_API_KEY\"):\n", " API_KEY = os.getenv(\"TROVE_API_KEY\")" ] }, { "cell_type": "markdown", "id": "9aaf407a-b0cf-4731-ace2-f76d9b4a2999", "metadata": {}, "source": [ "## Run the harvester\n", "\n", "First of all we use `prepare_query` to create a base set of parameters. We'll feed it a search for the term \"anzac day\" and then add in the dates later." ] }, { "cell_type": "code", "execution_count": 4, "id": "24991234-a03a-40af-91c8-f107138a9c56", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "query = 'https://trove.nla.gov.au/search/category/newspapers?keyword=\"anzac day\"'\n", "query_params = prepare_query(query=query)" ] }, { "cell_type": "markdown", "id": "0d559f30-e620-4112-ba0d-20853e214aa9", "metadata": {}, "source": [ "Next we'll loop through our desired span of years, harvesting the results each Anzac Day. For demonstration purposes I'll use a short span, harvesting results for the years 1916 to 1919. But you could just as easily harvest results from 1916 to the present." ] }, { "cell_type": "code", "execution_count": null, "id": "5958b7f7-04c3-4b1f-9a82-0fb2276e41e5", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# Loop through the desired span of years\n", "# Note that the end of the range is not inclusive, so you have to set it to the value above the end you want,\n", "# so this loop will output 1916, 1917, 1918 and 1919, but not 1920.\n", "for year in range(1916, 1920):\n", " # Copy the base params\n", " params = query_params.copy()\n", " # Add the date search to the query string\n", " params[\n", " \"q\"\n", " ] = f\"{query_params['q']} date:[{year}-04-24T00:00:00Z TO {year}-04-25T00:00:00Z]\"\n", "\n", " # Initialise the harvester\n", " # The data-dir parameter sets the parent directory, in this case \"anzac-day\"\n", " # The harvest-dir parameter sets the directory, within the parent directory, where the current set of results will be saved,\n", " # in this case the results directory will be named by the year\n", " harvester = Harvester(\n", " query_params=params, key=API_KEY, data_dir=\"anzac-day\", harvest_dir=str(year)\n", " )\n", "\n", " # Harvest the results\n", " harvester.harvest()\n", "\n", " # Convert the JSON results to CSV\n", " harvester.save_csv()" ] }, { "cell_type": "markdown", "id": "8acb581f-47fa-4e42-b542-bfdc62506cc4", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "The result of this code will be a series of directories and files like this:\n", "\n", "```\n", "- anzac-day\n", " - 1916\n", " - results.csv\n", " - ro-crate-metadata.json\n", " - harvester_config.json\n", " - results.ndjson\n", " - 1917\n", " - results.csv\n", " - ro-crate-metadata.json\n", " - harvester_config.json\n", " - results.ndjson\n", " - 1918\n", " - results.csv\n", " - ro-crate-metadata.json\n", " - harvester_config.json\n", " - results.ndjson\n", " - 1919\n", " - results.csv\n", " - ro-crate-metadata.json\n", " - harvester_config.json\n", " - results.ndjson\n", "```" ] }, { "cell_type": "markdown", "id": "7da50d15-842e-412b-a457-cd4c3ef9bf0f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Combine results\n", "\n", "After harvesting the data above, the results for each year will be in a separate directory. If you want to join the result sets together, you can do something like this to create a single dataframe." ] }, { "cell_type": "code", "execution_count": 6, "id": "98a4e539-7f99-4cf9-9958-d0794f166e54", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
article_idtitledatepagenewspaper_idnewspaper_titlecategorywordsillustratededition...snippetrelevancecorrectionslast_correctedtagscommentsliststextpdfimages
0102545637CHURCH OF ENGLAND. ANZAC DAY.1916-04-252348Port Pirie Recorder and North Western Mail (SA...Article52NNaN...There will be a Celebration of Holy Communion ...217.1441200NaN000NaNNaNNaN
1102545649Advertising1916-04-252348Port Pirie Recorder and North Western Mail (SA...Advertising823NNaN...NaN0.8714900NaN000NaNNaNNaN
2102545658ANZAC DAY AT PORT PIRIE.1916-04-252348Port Pirie Recorder and North Western Mail (SA...Article66NNaN...Anzac Day will be officially commemorated in t...217.7422000NaN000NaNNaNNaN
31040069Classified Advertising1916-04-25210The Mercury (Hobart, Tas. : 1860 - 1954)Advertising3816NNaN...NaN0.4634480NaN000NaNNaNNaN
41040072ANZAC DAY.1916-04-25810The Mercury (Hobart, Tas. : 1860 - 1954)Article89NNaN...May I inquire if on Anzac Day in Hobart any se...285.0453500NaN000NaNNaNNaN
\n", "

5 rows × 24 columns

\n", "
" ], "text/plain": [ " article_id title date page newspaper_id \\\n", "0 102545637 CHURCH OF ENGLAND. ANZAC DAY. 1916-04-25 2 348 \n", "1 102545649 Advertising 1916-04-25 2 348 \n", "2 102545658 ANZAC DAY AT PORT PIRIE. 1916-04-25 2 348 \n", "3 1040069 Classified Advertising 1916-04-25 2 10 \n", "4 1040072 ANZAC DAY. 1916-04-25 8 10 \n", "\n", " newspaper_title category words \\\n", "0 Port Pirie Recorder and North Western Mail (SA... Article 52 \n", "1 Port Pirie Recorder and North Western Mail (SA... Advertising 823 \n", "2 Port Pirie Recorder and North Western Mail (SA... Article 66 \n", "3 The Mercury (Hobart, Tas. : 1860 - 1954) Advertising 3816 \n", "4 The Mercury (Hobart, Tas. : 1860 - 1954) Article 89 \n", "\n", " illustrated edition ... snippet \\\n", "0 N NaN ... There will be a Celebration of Holy Communion ... \n", "1 N NaN ... NaN \n", "2 N NaN ... Anzac Day will be officially commemorated in t... \n", "3 N NaN ... NaN \n", "4 N NaN ... May I inquire if on Anzac Day in Hobart any se... \n", "\n", " relevance corrections last_corrected tags comments lists text pdf \\\n", "0 217.144120 0 NaN 0 0 0 NaN NaN \n", "1 0.871490 0 NaN 0 0 0 NaN NaN \n", "2 217.742200 0 NaN 0 0 0 NaN NaN \n", "3 0.463448 0 NaN 0 0 0 NaN NaN \n", "4 285.045350 0 NaN 0 0 0 NaN NaN \n", "\n", " images \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN \n", "\n", "[5 rows x 24 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# A list to hold all the dataframes\n", "dfs = []\n", "\n", "# Loop through the span of years\n", "for year in range(1916, 1920):\n", " # Convert the results CSV file to a dataframe and add to the list of dfs\n", " dfs.append(pd.read_csv(Path(\"anzac-day\", str(year), \"results.csv\")))\n", "\n", "# Combine the dataframes into one\n", "df = pd.concat(dfs)\n", "\n", "# View a sample\n", "df.head()" ] }, { "cell_type": "markdown", "id": "53cd2737-5809-4428-bbee-d31710d93879", "metadata": {}, "source": [ "To make sure we have the combined results, we can look at the number of articles by each Anzac Day." ] }, { "cell_type": "code", "execution_count": 7, "id": "46f41296-5d2d-4442-962f-b09a07a69c2b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "date\n", "1917-04-25 445\n", "1918-04-25 384\n", "1919-04-25 344\n", "1916-04-25 315\n", "Name: count, dtype: int64" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"date\"].value_counts()" ] }, { "cell_type": "markdown", "id": "176f4b83-6a56-4118-a131-44d8e455b567", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "----\n", "\n", "Created by [Tim Sherratt](https://timsherratt.org) ([@wragge](https://twitter.com/wragge)) for the [GLAM Workbench](https://github.com/glam-workbench/). \n", "Support this project by [becoming a GitHub sponsor](https://github.com/sponsors/wragge?o=esb).\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "rocrate": { "author": [ { "name": "Sherratt, Tim", "orcid": "https://orcid.org/0000-0001-7956-4498" } ], "name": "Harvesting articles that mention \"Anzac Day\" on Anzac Day" } }, "nbformat": 4, "nbformat_minor": 5 }