{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Finding Open Access versions of articles in *Australian Historical Studies*\n", "\n", "Open access isn't just what historians expect from GLAM organisations, it's we do with the products of our research. [*Australian Historical Studies*](https://www.tandfonline.com/toc/rahs20/current) is one of the major journals for Australian historians. How much of it is accessible to researchers without the luxury of an institutional subscription?\n", "\n", "AHS is published by Taylor & Francis, and under their terms and conditions there are two ways articles can be made openly accessible:\n", "\n", "* The author can pay an article publishing charge (APC) to make the article open immediately upon publication. The APC is currently set at $3775. This is known as **Gold** Open Access.\n", "* The author can share the Author Accepted Manuscript (AAM) version of their article. The AAM version is the one after peer-review, but before copy-editing and typesetting. This version can be shared immediately on the author's personal website or, after an 18 month embargo, it can be uploaded to an institutional or subject repository. This is known as **Green** Open Access.\n", "\n", "If Green Open Access versions are uploaded to a recognised repository, they become findable. Tools such as the [Open Access Button](https://openaccessbutton.org/) and [Unpaywall](https://unpaywall.org/) can redirect you from a paywalled version to an open access alternative. If you use Zotero to save articles from a journal, it'll automatically look for Open Access versions via Unpaywall if no free downloads are available. Green Open Access costs nothing, but it opens your work to new audiences and new uses.\n", "\n", "So how many authors are taking advantage of Green Open Access arrangements? Let's have a look and see.\n", "\n", "## The dataset\n", "\n", "For this little experiment I'm going to look at 10 years of articles, from 2008 to 2018. I'm finishing in 2018 because it's outside of the 18 month embargo period. Everything published in 2018 or before **can be made open access**. I'm focusing on research articles, excluding editorials, reviews, and commentaries.\n", "\n", "My plan is to save details of the articles to Zotero, access the details from the Zotero API, then use the Open Access Button API to look for open access versions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Import what we need" ] }, { "cell_type": "code", "execution_count": 334, "metadata": {}, "outputs": [], "source": [ "from pyzotero import zotero\n", "import requests\n", "import time\n", "from IPython.display import JSON, display\n", "import pandas as pd\n", "import altair as alt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Get the list of articles from the Zotero API\n", "\n", "To create a list of articles to check I just went to the page for every issue from 2008 to 2018 and used Zotero to save the details of research articles (I haven't included reviews, editorials, or commentaries). You can view the [collection of 242 articles](https://www.zotero.org/groups/2589863/australian_history_journals/collections/922CMTJU) in the Zotero web interface.\n", "\n", "To access the data for each of these articles, I had to create an API token for read-only access to the collection. I can then use [PyZotero](https://pyzotero.readthedocs.io/en/latest/) to request the list of articles from the [Zotero API](https://www.zotero.org/support/dev/web_api/v3/start)." ] }, { "cell_type": "code", "execution_count": 188, "metadata": {}, "outputs": [], "source": [ "# Details of the public Zotero group into which I've captured article details\n", "# https://www.zotero.org/groups/2589863/australian_history_journals/library\n", "# This key id read-only\n", "ZOTERO_API_KEY = 'FT3a7ByHQCRUpCnEeoKlhhKy'\n", "ZOTERO_GROUP_ID = '2589863'\n", "ZOTERO_LIBRARY_TYPE = 'group'\n", "\n", "# Create the Zotero group client\n", "zot = zotero.Zotero(ZOTERO_USER_ID, ZOTERO_LIBRARY_TYPE, ZOTERO_API_KEY)\n", "zot.add_parameters(sort='title')\n", "\n", "# This is the Australian Historical Studies collection\n", "articles = zot.everything(zot.collection_items_top('922CMTJU'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Free but not open?\n", "\n", "As I was saving the articles into Zotero, I noticed that some had a green tick next to them, indicating that you could access the content without a subscription. These are described as 'free access' articles, rather then 'open access' articles which have the orange, open padlock icon. The difference is that Open Access articles are both freely available, **and** openly licensed. I don't know why the journal makes some articles 'free'. I did some checking and found that the 'free' articles don't seem to show up in the open access databases. In order to include them with the OA articles, I manually added the 'free' article link to Zotero. As you'll see below, I check for this link before searching for an OA version of each article. So the final results are a combination of the 'free' and OA articles." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Look for OA versions of the articles\n", "\n", "Now we're going to see if we can find open access versions of the articles. The code below will get the DOI for each article in our dataset and then look it up using the Open Access Button API. If it finds an OA version, it'll display the title and link, and add the link to the article's metadata." ] }, { "cell_type": "code", "execution_count": 189, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "A Case of Identity: The Artefacts of the 1770 Kamay (Botany Bay) Encounter\n", "https://www.repository.cam.ac.uk/handle/1810/293268\n", "\n", "A Historical Myth? Matthew Flinders and the Quest for a Strait\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2016.1250791?needAccess=true\n", "\n", "A Shield Loaded with History: Encounters, Objects and Exhibitions\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2017.1408663\n", "\n", "Asian Servants for the Imperial Telegraph: Imagining North Australia as an Indian Ocean Colony before 1914\n", "https://ro.uow.edu.au/cgi/viewcontent.cgi?article=4008&context=lhapapers\n", "\n", "Colonial Judiciaries, Aboriginal Protection and South Australia's Policy of Punishing ‘with Exemplary Severity’\n", "http://pdfs.semanticscholar.org/217a/cc68a93d95204f0230edb7256880d7dc92ad.pdf\n", "\n", "Galahs\n", "https://openresearch-repository.anu.edu.au/bitstream/1885/47799/4/galahs_long.pdf\n", "\n", "‘Habeas Corpus Mongols’—Chinese Litigants and the Politics of Immigration in 1888\n", "https://research-repository.griffith.edu.au/bitstream/10072/63973/1/97921_1.pdf\n", "\n", "Mediatisation and Institutions of Public Memory: Digital Storytelling and the Apology\n", "https://eprints.qut.edu.au/32980/1/c32980.pdf\n", "\n", "Neither a Discipline nor a Colony: Renaissance and Re-imagination in Economic History\n", "https://ro.uow.edu.au/cgi/viewcontent.cgi?article=4074&context=lhapapers\n", "\n", "Re-Routing Empire? Steam-Age Circulations and the Making of an Anglo Pacific, c.1850–90\n", "https://ro.uow.edu.au/cgi/viewcontent.cgi?article=3174&context=lhapapers\n", "\n", "Remembering and Fighting for Their Own: Vietnam Veterans and the Long Tan Cross\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2017.1394887?needAccess=true\n", "\n", "Rewriting Quarantine: Pacific History at Australia's Edge\n", "https://www.repository.cam.ac.uk/bitstream/1810/250281/1/Bashford%20%26%20Hobbins%202015%20Australian%20Historical%20Studies.pdf\n", "\n", "Settler Justice and Aboriginal Homicide in Late Colonial Australia\n", "https://research-repository.griffith.edu.au/bitstream/10072/41848/1/73407_1.pdf\n", "\n", "‘Such a Great Space of Water between Us’: Anzac Day in Britain, 1916–39\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2014.912667?needAccess=true\n", "\n", "The Place of Anzac in Australian Historical Consciousness\n", "https://opus.lib.uts.edu.au/bitstream/10453/88058/4/7C03BF5A-9BA9-435C-8299-DF822C4D8B10%20am.pdf\n", "\n", "The Significance of the Northern Territory in the Formulation of ‘White Australia’ Policies, 1880–1901\n", "https://research-repository.griffith.edu.au/bitstream/10072/390800/2/Fong79965-Accepted.pdf\n", "\n", "Transnational Histories of Penal Transportation: Punishment, Labour and Governance in the British Imperial World, 1788–1939\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2016.1203962?needAccess=true\n", "\n", "Vida Lahey's Progressive Activism for Children's Art Education\n", "https://research-repository.griffith.edu.au/bitstream/10072/36552/1/66889_1.pdf\n", "\n", "Visiting the Neighbours: The Political Meanings of Australian Travel to Cold War Asia\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2013.817450\n", "\n", "White Men in Quarantine: Disease, Race, Commerce and Mobility in the Pacific, 1872\n", "https://www.tandfonline.com/doi/pdf/10.1080/1031461X.2017.1293704?needAccess=true\n" ] } ], "source": [ "# Open Access button API endpoint\n", "OA_API_URL = 'https://api.openaccessbutton.org/find'\n", "\n", "oa_articles = []\n", "for article in articles:\n", " # Some articles have been made 'free' by the journal, though they're not open access\n", " # These aren't included in the OA Button db, so I've added the PDF links to their Zotero records.\n", " # Here we'll check to see if the article has one of these links.\n", " for child in zot.children(article['key']):\n", " if child['data']['title'] == 'Free access PDF':\n", " article['data']['oadoi'] = child['data']['url']\n", " article['data']['oa_type'] = 'free access'\n", " break\n", " # If there's not free access, we'll see if there's an OA version\n", " if not article['data'].get('oadoi'):\n", " # Search the OA db using the DOI\n", " response = requests.get(OA_API_URL, params={'id': article['data']['DOI']})\n", " data = response.json()\n", " # Try the title if we couldn't find it by DOI\n", " # if not data['found']:\n", " # response = requests.get(OA_API_URL, params={'title': article['data']['title']})\n", " # data = response.json()\n", " # Is there an OA version?\n", " if data['found']:\n", " article['data']['oadoi'] = data['found']['oadoi']\n", " article['data']['licence'] = data['metadata'].get('licence', '')\n", " time.sleep(1)\n", " if article['data'].get('oadoi'):\n", " print(f'\\n{article[\"data\"][\"title\"]}')\n", " print(article['data']['oadoi'])\n", " oa_articles.append(article['data']) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convert to a dataframe\n", "\n", "Now we'll convert the list of articles into a dataframe for further exploration." ] }, { "cell_type": "code", "execution_count": 238, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
keyversionitemTypetitlecreatorsabstractNotepublicationTitlevolumeissuepages...rightsextratagscollectionsrelationsdateAddeddateModifiedoadoilicenceoa_type
04CTNPIKI26journalArticleA Case of Identity: The Artefacts of the 1770 ...[{'creatorType': 'author', 'firstName': 'Nicho...Collections of Indigenous artefacts made durin...Australian Historical Studies4914-27...Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30Zhttps://www.repository.cam.ac.uk/handle/1810/2...NaN
154PJ4CIG6journalArticleA Disenfranchised Grief: Post-war Death and Me...[{'creatorType': 'author', 'firstName': 'Marin...The 1918 Armistice signalled the end of the Fi...Australian Historical Studies40179-95...Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN
26XPGZ57R11journalArticleA Dog in the Manger: White Australia and its V...[{'creatorType': 'author', 'firstName': 'Russe...Between the world wars Australia was commonly ...Australian Historical Studies432157-173...Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN
3C6XRZRJ311journalArticle‘A Halo of Protection’: Colonial Protectors an...[{'creatorType': 'author', 'firstName': 'Amand...Scholarship on Australia's colonial protectora...Australian Historical Studies433396-411...Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN
4LTEDXQLF24journalArticleA Historical Myth? Matthew Flinders and the Qu...[{'creatorType': 'author', 'firstName': 'Kenne...This article takes issue with a recent argumen...Australian Historical Studies48152-67...Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30Zhttps://www.tandfonline.com/doi/pdf/10.1080/10...pdNaN
\n", "

5 rows × 35 columns

\n", "
" ], "text/plain": [ " key version itemType \\\n", "0 4CTNPIKI 26 journalArticle \n", "1 54PJ4CIG 6 journalArticle \n", "2 6XPGZ57R 11 journalArticle \n", "3 C6XRZRJ3 11 journalArticle \n", "4 LTEDXQLF 24 journalArticle \n", "\n", " title \\\n", "0 A Case of Identity: The Artefacts of the 1770 ... \n", "1 A Disenfranchised Grief: Post-war Death and Me... \n", "2 A Dog in the Manger: White Australia and its V... \n", "3 ‘A Halo of Protection’: Colonial Protectors an... \n", "4 A Historical Myth? Matthew Flinders and the Qu... \n", "\n", " creators \\\n", "0 [{'creatorType': 'author', 'firstName': 'Nicho... \n", "1 [{'creatorType': 'author', 'firstName': 'Marin... \n", "2 [{'creatorType': 'author', 'firstName': 'Russe... \n", "3 [{'creatorType': 'author', 'firstName': 'Amand... \n", "4 [{'creatorType': 'author', 'firstName': 'Kenne... \n", "\n", " abstractNote \\\n", "0 Collections of Indigenous artefacts made durin... \n", "1 The 1918 Armistice signalled the end of the Fi... \n", "2 Between the world wars Australia was commonly ... \n", "3 Scholarship on Australia's colonial protectora... \n", "4 This article takes issue with a recent argumen... \n", "\n", " publicationTitle volume issue pages ... rights \\\n", "0 Australian Historical Studies 49 1 4-27 ... \n", "1 Australian Historical Studies 40 1 79-95 ... \n", "2 Australian Historical Studies 43 2 157-173 ... \n", "3 Australian Historical Studies 43 3 396-411 ... \n", "4 Australian Historical Studies 48 1 52-67 ... \n", "\n", " extra tags collections \\\n", "0 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "1 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "2 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "3 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "4 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "\n", " relations dateAdded dateModified \\\n", "0 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "1 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "2 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "3 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "4 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "\n", " oadoi licence oa_type \n", "0 https://www.repository.cam.ac.uk/handle/1810/2... NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 https://www.tandfonline.com/doi/pdf/10.1080/10... pd NaN \n", "\n", "[5 rows x 35 columns]" ] }, "execution_count": 238, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame(oa_articles)\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 239, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(242, 35)" ] }, "execution_count": 239, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Check the total number of articles\n", "df.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## All articles with Open Access (or free access) versions\n", "\n", "Let's look at all the articles that have an OA link." ] }, { "cell_type": "code", "execution_count": 226, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(20, 35)" ] }, "execution_count": 226, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# How many articles have OA versions?\n", "df.loc[df['oadoi'].notnull()].shape" ] }, { "cell_type": "code", "execution_count": 233, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitledateoadoi
010.1080/1031461X.2017.1414862A Case of Identity: The Artefacts of the 1770 ...January 2, 2018https://www.repository.cam.ac.uk/handle/1810/2...
410.1080/1031461X.2016.1250791A Historical Myth? Matthew Flinders and the Qu...January 2, 2017https://www.tandfonline.com/doi/pdf/10.1080/10...
610.1080/1031461X.2017.1408663A Shield Loaded with History: Encounters, Obje...January 2, 2018https://www.tandfonline.com/doi/pdf/10.1080/10...
2610.1080/1031461X.2017.1279196Asian Servants for the Imperial Telegraph: Ima...April 3, 2017https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...
5710.1080/1031461X.2010.493947Colonial Judiciaries, Aboriginal Protection an...September 1, 2010http://pdfs.semanticscholar.org/217a/cc68a93d9...
8310.1080/10314610903067094GalahsSeptember 1, 2009https://openresearch-repository.anu.edu.au/bit...
8610.1080/1031461X.2014.911759‘Habeas Corpus Mongols’—Chinese Litigants and ...May 4, 2014https://research-repository.griffith.edu.au/bi...
12210.1080/10314611003716861Mediatisation and Institutions of Public Memor...June 1, 2010https://eprints.qut.edu.au/32980/1/c32980.pdf
12810.1080/1031461X.2017.1279197Neither a Discipline nor a Colony: Renaissance...April 3, 2017https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...
15110.1080/1031461X.2015.1071416Re-Routing Empire? Steam-Age Circulations and ...September 2, 2015https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...
15410.1080/1031461X.2017.1394887Remembering and Fighting for Their Own: Vietna...January 2, 2018https://www.tandfonline.com/doi/pdf/10.1080/10...
15910.1080/1031461X.2015.1071860Rewriting Quarantine: Pacific History at Austr...September 2, 2015https://www.repository.cam.ac.uk/bitstream/181...
16510.1080/1031461X.2011.560610Settler Justice and Aboriginal Homicide in Lat...June 1, 2011https://research-repository.griffith.edu.au/bi...
17110.1080/1031461X.2014.912667‘Such a Great Space of Water between Us’: Anza...May 4, 2014https://www.tandfonline.com/doi/pdf/10.1080/10...
20210.1080/1031461X.2016.1250790The Place of Anzac in Australian Historical Co...January 2, 2017https://opus.lib.uts.edu.au/bitstream/10453/88...
20710.1080/1031461X.2018.1515963The Significance of the Northern Territory in ...October 2, 2018https://research-repository.griffith.edu.au/bi...
21710.1080/1031461X.2016.1203962Transnational Histories of Penal Transportatio...September 1, 2016https://www.tandfonline.com/doi/pdf/10.1080/10...
22410.1080/1031461X.2010.493945Vida Lahey's Progressive Activism for Children...September 1, 2010https://research-repository.griffith.edu.au/bi...
22510.1080/1031461X.2013.817450Visiting the Neighbours: The Political Meaning...September 1, 2013https://www.tandfonline.com/doi/pdf/10.1080/10...
23310.1080/1031461X.2017.1293704White Men in Quarantine: Disease, Race, Commer...April 3, 2017https://www.tandfonline.com/doi/pdf/10.1080/10...
\n", "
" ], "text/plain": [ " DOI \\\n", "0 10.1080/1031461X.2017.1414862 \n", "4 10.1080/1031461X.2016.1250791 \n", "6 10.1080/1031461X.2017.1408663 \n", "26 10.1080/1031461X.2017.1279196 \n", "57 10.1080/1031461X.2010.493947 \n", "83 10.1080/10314610903067094 \n", "86 10.1080/1031461X.2014.911759 \n", "122 10.1080/10314611003716861 \n", "128 10.1080/1031461X.2017.1279197 \n", "151 10.1080/1031461X.2015.1071416 \n", "154 10.1080/1031461X.2017.1394887 \n", "159 10.1080/1031461X.2015.1071860 \n", "165 10.1080/1031461X.2011.560610 \n", "171 10.1080/1031461X.2014.912667 \n", "202 10.1080/1031461X.2016.1250790 \n", "207 10.1080/1031461X.2018.1515963 \n", "217 10.1080/1031461X.2016.1203962 \n", "224 10.1080/1031461X.2010.493945 \n", "225 10.1080/1031461X.2013.817450 \n", "233 10.1080/1031461X.2017.1293704 \n", "\n", " title date \\\n", "0 A Case of Identity: The Artefacts of the 1770 ... January 2, 2018 \n", "4 A Historical Myth? Matthew Flinders and the Qu... January 2, 2017 \n", "6 A Shield Loaded with History: Encounters, Obje... January 2, 2018 \n", "26 Asian Servants for the Imperial Telegraph: Ima... April 3, 2017 \n", "57 Colonial Judiciaries, Aboriginal Protection an... September 1, 2010 \n", "83 Galahs September 1, 2009 \n", "86 ‘Habeas Corpus Mongols’—Chinese Litigants and ... May 4, 2014 \n", "122 Mediatisation and Institutions of Public Memor... June 1, 2010 \n", "128 Neither a Discipline nor a Colony: Renaissance... April 3, 2017 \n", "151 Re-Routing Empire? Steam-Age Circulations and ... September 2, 2015 \n", "154 Remembering and Fighting for Their Own: Vietna... January 2, 2018 \n", "159 Rewriting Quarantine: Pacific History at Austr... September 2, 2015 \n", "165 Settler Justice and Aboriginal Homicide in Lat... June 1, 2011 \n", "171 ‘Such a Great Space of Water between Us’: Anza... May 4, 2014 \n", "202 The Place of Anzac in Australian Historical Co... January 2, 2017 \n", "207 The Significance of the Northern Territory in ... October 2, 2018 \n", "217 Transnational Histories of Penal Transportatio... September 1, 2016 \n", "224 Vida Lahey's Progressive Activism for Children... September 1, 2010 \n", "225 Visiting the Neighbours: The Political Meaning... September 1, 2013 \n", "233 White Men in Quarantine: Disease, Race, Commer... April 3, 2017 \n", "\n", " oadoi \n", "0 https://www.repository.cam.ac.uk/handle/1810/2... \n", "4 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "6 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "26 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "57 http://pdfs.semanticscholar.org/217a/cc68a93d9... \n", "83 https://openresearch-repository.anu.edu.au/bit... \n", "86 https://research-repository.griffith.edu.au/bi... \n", "122 https://eprints.qut.edu.au/32980/1/c32980.pdf \n", "128 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "151 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "154 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "159 https://www.repository.cam.ac.uk/bitstream/181... \n", "165 https://research-repository.griffith.edu.au/bi... \n", "171 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "202 https://opus.lib.uts.edu.au/bitstream/10453/88... \n", "207 https://research-repository.griffith.edu.au/bi... \n", "217 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "224 https://research-repository.griffith.edu.au/bi... \n", "225 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "233 https://www.tandfonline.com/doi/pdf/10.1080/10... " ] }, "execution_count": 233, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display them\n", "df.loc[df['oadoi'].notnull()][['DOI', 'title', 'date', 'oadoi']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How many articles have OA versions available?" ] }, { "cell_type": "code", "execution_count": 297, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8.26% of articles are freely available\n" ] } ], "source": [ "print(f'{df.loc[df[\"oadoi\"].notnull()].shape[0] / df.shape[0]:.2%} of articles are freely available')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Gold open access articles\n", "\n", "The Gold OA articles have links that go back to the Taylor & Francis site, but are not the 'free access' articles I identified manually." ] }, { "cell_type": "code", "execution_count": 243, "metadata": {}, "outputs": [], "source": [ "# Set OA type to gold\n", "df.loc[(df['oadoi'].fillna('').str.contains('tandfonline')) & (df['oa_type'] != 'free access'),'oa_type'] = 'gold'" ] }, { "cell_type": "code", "execution_count": 292, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(5, 36)" ] }, "execution_count": 292, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Number of articles\n", "df.loc[df['oa_type'] == 'gold'].shape" ] }, { "cell_type": "code", "execution_count": 244, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitledateoadoi
410.1080/1031461X.2016.1250791A Historical Myth? Matthew Flinders and the Qu...January 2, 2017https://www.tandfonline.com/doi/pdf/10.1080/10...
15410.1080/1031461X.2017.1394887Remembering and Fighting for Their Own: Vietna...January 2, 2018https://www.tandfonline.com/doi/pdf/10.1080/10...
17110.1080/1031461X.2014.912667‘Such a Great Space of Water between Us’: Anza...May 4, 2014https://www.tandfonline.com/doi/pdf/10.1080/10...
21710.1080/1031461X.2016.1203962Transnational Histories of Penal Transportatio...September 1, 2016https://www.tandfonline.com/doi/pdf/10.1080/10...
23310.1080/1031461X.2017.1293704White Men in Quarantine: Disease, Race, Commer...April 3, 2017https://www.tandfonline.com/doi/pdf/10.1080/10...
\n", "
" ], "text/plain": [ " DOI \\\n", "4 10.1080/1031461X.2016.1250791 \n", "154 10.1080/1031461X.2017.1394887 \n", "171 10.1080/1031461X.2014.912667 \n", "217 10.1080/1031461X.2016.1203962 \n", "233 10.1080/1031461X.2017.1293704 \n", "\n", " title date \\\n", "4 A Historical Myth? Matthew Flinders and the Qu... January 2, 2017 \n", "154 Remembering and Fighting for Their Own: Vietna... January 2, 2018 \n", "171 ‘Such a Great Space of Water between Us’: Anza... May 4, 2014 \n", "217 Transnational Histories of Penal Transportatio... September 1, 2016 \n", "233 White Men in Quarantine: Disease, Race, Commer... April 3, 2017 \n", "\n", " oadoi \n", "4 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "154 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "171 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "217 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "233 https://www.tandfonline.com/doi/pdf/10.1080/10... " ] }, "execution_count": 244, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[df['oa_type'] == 'gold'][['DOI', 'title', 'date', 'oadoi']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Green Open Access articles" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the OA url doesn't include 'tandfonline' and it's not 'free access', then it looks like it's Green Open Access." ] }, { "cell_type": "code", "execution_count": 246, "metadata": {}, "outputs": [], "source": [ "# Set oa_type to 'green'\n", "df.loc[(df['oadoi'].notnull()) & (~df['oadoi'].fillna('').str.contains('tandfonline')) & (df['oa_type'] != 'free access'), 'oa_type'] = 'green'" ] }, { "cell_type": "code", "execution_count": 291, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(13, 36)" ] }, "execution_count": 291, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Number of articles\n", "df.loc[df['oa_type'] == 'green'].shape" ] }, { "cell_type": "code", "execution_count": 247, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitledateoadoi
010.1080/1031461X.2017.1414862A Case of Identity: The Artefacts of the 1770 ...January 2, 2018https://www.repository.cam.ac.uk/handle/1810/2...
2610.1080/1031461X.2017.1279196Asian Servants for the Imperial Telegraph: Ima...April 3, 2017https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...
5710.1080/1031461X.2010.493947Colonial Judiciaries, Aboriginal Protection an...September 1, 2010http://pdfs.semanticscholar.org/217a/cc68a93d9...
8310.1080/10314610903067094GalahsSeptember 1, 2009https://openresearch-repository.anu.edu.au/bit...
8610.1080/1031461X.2014.911759‘Habeas Corpus Mongols’—Chinese Litigants and ...May 4, 2014https://research-repository.griffith.edu.au/bi...
12210.1080/10314611003716861Mediatisation and Institutions of Public Memor...June 1, 2010https://eprints.qut.edu.au/32980/1/c32980.pdf
12810.1080/1031461X.2017.1279197Neither a Discipline nor a Colony: Renaissance...April 3, 2017https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...
15110.1080/1031461X.2015.1071416Re-Routing Empire? Steam-Age Circulations and ...September 2, 2015https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...
15910.1080/1031461X.2015.1071860Rewriting Quarantine: Pacific History at Austr...September 2, 2015https://www.repository.cam.ac.uk/bitstream/181...
16510.1080/1031461X.2011.560610Settler Justice and Aboriginal Homicide in Lat...June 1, 2011https://research-repository.griffith.edu.au/bi...
20210.1080/1031461X.2016.1250790The Place of Anzac in Australian Historical Co...January 2, 2017https://opus.lib.uts.edu.au/bitstream/10453/88...
20710.1080/1031461X.2018.1515963The Significance of the Northern Territory in ...October 2, 2018https://research-repository.griffith.edu.au/bi...
22410.1080/1031461X.2010.493945Vida Lahey's Progressive Activism for Children...September 1, 2010https://research-repository.griffith.edu.au/bi...
\n", "
" ], "text/plain": [ " DOI \\\n", "0 10.1080/1031461X.2017.1414862 \n", "26 10.1080/1031461X.2017.1279196 \n", "57 10.1080/1031461X.2010.493947 \n", "83 10.1080/10314610903067094 \n", "86 10.1080/1031461X.2014.911759 \n", "122 10.1080/10314611003716861 \n", "128 10.1080/1031461X.2017.1279197 \n", "151 10.1080/1031461X.2015.1071416 \n", "159 10.1080/1031461X.2015.1071860 \n", "165 10.1080/1031461X.2011.560610 \n", "202 10.1080/1031461X.2016.1250790 \n", "207 10.1080/1031461X.2018.1515963 \n", "224 10.1080/1031461X.2010.493945 \n", "\n", " title date \\\n", "0 A Case of Identity: The Artefacts of the 1770 ... January 2, 2018 \n", "26 Asian Servants for the Imperial Telegraph: Ima... April 3, 2017 \n", "57 Colonial Judiciaries, Aboriginal Protection an... September 1, 2010 \n", "83 Galahs September 1, 2009 \n", "86 ‘Habeas Corpus Mongols’—Chinese Litigants and ... May 4, 2014 \n", "122 Mediatisation and Institutions of Public Memor... June 1, 2010 \n", "128 Neither a Discipline nor a Colony: Renaissance... April 3, 2017 \n", "151 Re-Routing Empire? Steam-Age Circulations and ... September 2, 2015 \n", "159 Rewriting Quarantine: Pacific History at Austr... September 2, 2015 \n", "165 Settler Justice and Aboriginal Homicide in Lat... June 1, 2011 \n", "202 The Place of Anzac in Australian Historical Co... January 2, 2017 \n", "207 The Significance of the Northern Territory in ... October 2, 2018 \n", "224 Vida Lahey's Progressive Activism for Children... September 1, 2010 \n", "\n", " oadoi \n", "0 https://www.repository.cam.ac.uk/handle/1810/2... \n", "26 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "57 http://pdfs.semanticscholar.org/217a/cc68a93d9... \n", "83 https://openresearch-repository.anu.edu.au/bit... \n", "86 https://research-repository.griffith.edu.au/bi... \n", "122 https://eprints.qut.edu.au/32980/1/c32980.pdf \n", "128 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "151 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "159 https://www.repository.cam.ac.uk/bitstream/181... \n", "165 https://research-repository.griffith.edu.au/bi... \n", "202 https://opus.lib.uts.edu.au/bitstream/10453/88... \n", "207 https://research-repository.griffith.edu.au/bi... \n", "224 https://research-repository.griffith.edu.au/bi... " ] }, "execution_count": 247, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[df['oa_type'] == 'green'][['DOI', 'title', 'date', 'oadoi']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 'Free access' articles" ] }, { "cell_type": "code", "execution_count": 293, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitledateoadoi
610.1080/1031461X.2017.1408663A Shield Loaded with History: Encounters, Obje...January 2, 2018https://www.tandfonline.com/doi/pdf/10.1080/10...
22510.1080/1031461X.2013.817450Visiting the Neighbours: The Political Meaning...September 1, 2013https://www.tandfonline.com/doi/pdf/10.1080/10...
\n", "
" ], "text/plain": [ " DOI \\\n", "6 10.1080/1031461X.2017.1408663 \n", "225 10.1080/1031461X.2013.817450 \n", "\n", " title date \\\n", "6 A Shield Loaded with History: Encounters, Obje... January 2, 2018 \n", "225 Visiting the Neighbours: The Political Meaning... September 1, 2013 \n", "\n", " oadoi \n", "6 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "225 https://www.tandfonline.com/doi/pdf/10.1080/10... " ] }, "execution_count": 293, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[df['oa_type'] == 'free access'][['DOI', 'title', 'date', 'oadoi']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Articles over time\n", "\n", "Let's see how the number of articles varies over time. First we'll extract the `year` from the date string." ] }, { "cell_type": "code", "execution_count": 254, "metadata": {}, "outputs": [], "source": [ "# Add a year column by extracting the year from the date column\n", "df['year'] = df['date'].str.extract(r'(\\d{4})$').astype(int)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's plot the results." ] }, { "cell_type": "code", "execution_count": 294, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 294, "metadata": {}, "output_type": "execute_result" } ], "source": [ "alt.Chart(df.fillna('$$$')).mark_bar().encode(\n", " x=alt.X('year:O', title='Year'),\n", " y=alt.Y('count():Q', title='Number of articles', axis=alt.Axis(tickMinStep=1)),\n", " color=alt.Color('oa_type:N', scale=alt.Scale(range=['lightgrey', 'blue', 'gold', 'green']), legend=alt.Legend(title='OA type')),\n", " tooltip=[alt.Tooltip('count():Q', title='Number of articles'), alt.Tooltip('oa_type', title='OA type')]\n", ").properties(width=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## OA articles by repository\n", "\n", "We can extract the domain from the OA url to see where the articles come from." ] }, { "cell_type": "code", "execution_count": 266, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "www.tandfonline.com 7\n", "research-repository.griffith.edu.au 4\n", "ro.uow.edu.au 3\n", "www.repository.cam.ac.uk 2\n", "pdfs.semanticscholar.org 1\n", "opus.lib.uts.edu.au 1\n", "openresearch-repository.anu.edu.au 1\n", "eprints.qut.edu.au 1\n", "dtype: int64" ] }, "execution_count": 266, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[df['oadoi'].notnull()]['oadoi'].str.extract(r'^https*://(.*?)/').value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why so few?\n", "\n", "Only 8.26% of research articles published in *Australian Historical Studies* between 2008 and 2018 are available in an open access version. That's pretty disappointing. Remember too that the embargo period for AHS is 18 months, so everything published up to the end of 2018 **could** now be open access. So why aren't they? There's a few possible reasons why they're not showing up.\n", "\n", "* Perhaps the repositories aren't being properly indexed by the Open Access Button / Unpaywall services. The articles might be available, but missing from our results.\n", "* There might be records for the articles in repositories, but either the AAM version hasn't been uploaded, or the embargo settings are wrong.\n", "* Records might not have been added to a repository at all.\n", "\n", "One way we might explore this further is to look at another index of content from Australian university repositories – Trove. From Trove we can find how many of the articles are listed in repositories, and do a bit of cross-checking with the Open Access sources.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Search for the articles in Trove\n", "\n", "Trove harvests records from all Australian university repositories. In some cases the records will include DOIs, but not always. We'll search for the title of each article first in the `article` or journal zone (this is now the Research category in the web interface). If that doesn't work we'll try searching for the DOI. There might be multiple records for each article, either because they're held by multiple repositories, or they've been indexed into something like Informit, which also supplies data to Trove. To try and be as thorough as possible, we'll look for repository links in all the matching records. We can get rid of any duplicates later." ] }, { "cell_type": "code", "execution_count": 390, "metadata": {}, "outputs": [], "source": [ "TROVE_API_KEY = 'YOUR TROVE API KEY'\n", "\n", "# Trove search parameters\n", "params = {\n", " 'key': TROVE_API_KEY,\n", " 'encoding': 'json',\n", " 'zone': 'article',\n", " 'format': 'Article',\n", " 'include': 'workVersions,links'\n", "}\n", "\n", "def check_link(link):\n", " '''\n", " Filter out links that go back to the journal site (rather than to a repository).\n", " '''\n", " if not 'tandfonline' in link and not 'doi.org' in link:\n", " return True\n", " \n", "def add_repo(link, article, work):\n", " '''\n", " Add a repository link to the data set.\n", " '''\n", " # Add the basic details into a dictionary.\n", " repo = {'DOI': article['data']['DOI'], 'title': article['data']['title'], 'trove_url': work.get('troveUrl')}\n", " \n", " # In most cases link will be a dictionary with a linktype attribute.\n", " # However, sometimes in version records it can just be a string.\n", " # Here we'll handle either case.\n", " if 'linktype' in link:\n", " # Standardise urls so we can remove duplicates later\n", " url = link['value'].replace('http:', 'https:')\n", " repo['link_type'] = link['linktype']\n", " repo['repo_url'] = url\n", " else:\n", " repo['link_type'] = 'unknown'\n", " repo['repo_url'] = link\n", " repositories.append(repo)\n", " \n", "def query_api(params):\n", " # Query the API\n", " response = requests.get('https://api.trove.nla.gov.au/v2/result', params=params)\n", " data = response.json()\n", " \n", " # How many matches?\n", " total_results = int(data[\"response\"][\"zone\"][0][\"records\"][\"total\"])\n", " \n", " return total_results, data\n", " \n", "repositories = [] \n", "\n", "for article in articles:\n", " \n", " # Set the q parameter to the title of the article (note use of 'title:' to search the title field.)\n", " params['q'] = f'title:\"{article[\"data\"][\"title\"]}\"'\n", " total_results, data = query_api(params)\n", "\n", " if total_results == 0:\n", " \n", " # Try searching for the DOI\n", " params['q'] = f'\"{article[\"data\"][\"DOI\"]}\"'\n", " total_results, data = query_api(params)\n", " \n", " # In some cases the title is not very specific and returns lots of results (eg 'Galahs').\n", " # Let's try limiting the results further by adding an 'creator:' parameter\n", " if total_results > 4:\n", " \n", " # Add the first author's surname to the query\n", " params['q'] = f'title:\"{article[\"data\"][\"title\"]}\" creator:{article[\"data\"][\"creators\"][0][\"lastName\"]}'\n", " total_results, data = query_api(params)\n", " \n", " # If there's still too many results, we'll just flag the title to look at later\n", " if total_results > 4:\n", " print(f'Too many choices! - {article[\"data\"][\"title\"]}')\n", " else:\n", " # Get a list of the matching works, if any\n", " try:\n", " works = data['response']['zone'][0]['records']['work']\n", " except KeyError:\n", " pass\n", " else:\n", " \n", " # Loop through the works\n", " for work in works:\n", " \n", " # Repository links can be at the aggregated 'work' level, or in individual version records.\n", " # We'll try and get them all and remove any duplicates later.\n", " # First check links at the work level\n", " for link in work.get('identifier', []):\n", " if check_link(link['value']) is True:\n", " add_repo(link, article, work)\n", " \n", " # Then loop through each version\n", " for version in work['version']:\n", " for link in version.get('identifier', []):\n", " if check_link(link['value']) is True:\n", " add_repo(link, article, work)\n", " \n", " # Version metadata can be nested under 'metadata', so check there as well\n", " if 'metadata' in version:\n", " for link in version['metadata'].get('identifier', []):\n", " if check_link(link['value']) is True:\n", " add_repo(link, article, work)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convert to a dataframe and remove any duplicates" ] }, { "cell_type": "code", "execution_count": 391, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitletrove_urllink_typerepo_url
010.1080/10314610802663035A Disenfranchised Grief: Post-war Death and Me...https://trove.nla.gov.au/work/66604896restrictedhttps://hdl.handle.net/1959.9/471872
210.1080/1031461X.2011.640695A Dog in the Manger: White Australia and its V...https://trove.nla.gov.au/work/169309446restrictedhttps://researchonline.jcu.edu.au/22338/
410.1080/1031461X.2012.706621‘A Halo of Protection’: Colonial Protectors an...https://trove.nla.gov.au/work/173739658notonlinehttps://hdl.handle.net/2440/74425
610.1080/1031461X.2012.760636A House Committee on Un-Australian Activities?...https://trove.nla.gov.au/work/181447421restrictedhttps://vuir.vu.edu.au/24499/
810.1080/1031461X.2014.996574‘Accurate to the Point of Mania’: Eyewitness T...https://trove.nla.gov.au/work/201475554notonlinehttps://hdl.handle.net/1885/63982
\n", "
" ], "text/plain": [ " DOI \\\n", "0 10.1080/10314610802663035 \n", "2 10.1080/1031461X.2011.640695 \n", "4 10.1080/1031461X.2012.706621 \n", "6 10.1080/1031461X.2012.760636 \n", "8 10.1080/1031461X.2014.996574 \n", "\n", " title \\\n", "0 A Disenfranchised Grief: Post-war Death and Me... \n", "2 A Dog in the Manger: White Australia and its V... \n", "4 ‘A Halo of Protection’: Colonial Protectors an... \n", "6 A House Committee on Un-Australian Activities?... \n", "8 ‘Accurate to the Point of Mania’: Eyewitness T... \n", "\n", " trove_url link_type \\\n", "0 https://trove.nla.gov.au/work/66604896 restricted \n", "2 https://trove.nla.gov.au/work/169309446 restricted \n", "4 https://trove.nla.gov.au/work/173739658 notonline \n", "6 https://trove.nla.gov.au/work/181447421 restricted \n", "8 https://trove.nla.gov.au/work/201475554 notonline \n", "\n", " repo_url \n", "0 https://hdl.handle.net/1959.9/471872 \n", "2 https://researchonline.jcu.edu.au/22338/ \n", "4 https://hdl.handle.net/2440/74425 \n", "6 https://vuir.vu.edu.au/24499/ \n", "8 https://hdl.handle.net/1885/63982 " ] }, "execution_count": 391, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_repos = pd.DataFrame(repositories)\n", "df_repos.drop_duplicates(inplace=True)\n", "df_repos.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How many links do we have?" ] }, { "cell_type": "code", "execution_count": 392, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(161, 5)" ] }, "execution_count": 392, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_repos.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, it's possible that we might have mutiple links for a single article. Let's look at how many unique DOIs there are in this dataset." ] }, { "cell_type": "code", "execution_count": 393, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(128,)" ] }, "execution_count": 393, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_repos['DOI'].unique().shape" ] }, { "cell_type": "code", "execution_count": 394, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "52.89% of article have records in university repositories (according to Trove)\n" ] } ], "source": [ "print(f'{df_repos[\"DOI\"].unique().shape[0] / df.shape[0]:.2%} of article have records in university repositories (according to Trove)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So about half of the articles have records in repositories. Or to put it another way – about 40% of articles are listed in repositores, but don't provide AAM versions for download." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Types of links\n", "\n", "When it harvests records from repositories, Trove tries to figure out how accessible things actually are. It assigns a `linktype` based on this assessment – `fulltext`, `restricted`, or `notonline`. As you might have guessed, `fulltext` indicates that an item is available for download or viewing online." ] }, { "cell_type": "code", "execution_count": 395, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "notonline 102\n", "restricted 32\n", "fulltext 26\n", "thumbnail 1\n", "Name: link_type, dtype: int64" ] }, "execution_count": 395, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_repos['link_type'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, there may be duplicates. Let's see how many unique DOIs have 'fulltext' links." ] }, { "cell_type": "code", "execution_count": 396, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(16,)" ] }, "execution_count": 396, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_repos.loc[df_repos['link_type'] == 'fulltext']['DOI'].unique().shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So Trove seems to think there are 16 articles that are freely available online. How does this compare to the list of Open Access articles we've already found? Let's combine our two datasets to find out." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Merging datasets\n", "\n", "Here we'll combine the datasets using the `DOI` field to link them." ] }, { "cell_type": "code", "execution_count": 397, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitletrove_urllink_typerepo_urlkeyversionitemTypecreatorsabstractNote...extratagscollectionsrelationsdateAddeddateModifiedoadoilicenceoa_typeyear
010.1080/10314610802663035A Disenfranchised Grief: Post-war Death and Me...https://trove.nla.gov.au/work/66604896restrictedhttps://hdl.handle.net/1959.9/47187254PJ4CIG6journalArticle[{'creatorType': 'author', 'firstName': 'Marin...The 1918 Armistice signalled the end of the Fi......Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN2009
110.1080/1031461X.2011.640695A Dog in the Manger: White Australia and its V...https://trove.nla.gov.au/work/169309446restrictedhttps://researchonline.jcu.edu.au/22338/6XPGZ57R11journalArticle[{'creatorType': 'author', 'firstName': 'Russe...Between the world wars Australia was commonly ......Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN2012
210.1080/1031461X.2012.706621‘A Halo of Protection’: Colonial Protectors an...https://trove.nla.gov.au/work/173739658notonlinehttps://hdl.handle.net/2440/74425C6XRZRJ311journalArticle[{'creatorType': 'author', 'firstName': 'Amand...Scholarship on Australia's colonial protectora......Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN2012
310.1080/1031461X.2012.760636A House Committee on Un-Australian Activities?...https://trove.nla.gov.au/work/181447421restrictedhttps://vuir.vu.edu.au/24499/QWZDSVVC14journalArticle[{'creatorType': 'author', 'firstName': 'Lachl...Legislation introduced by Prime Minister Rober......Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN2013
410.1080/1031461X.2014.996574‘Accurate to the Point of Mania’: Eyewitness T...https://trove.nla.gov.au/work/201475554notonlinehttps://hdl.handle.net/1885/639827WTYHC3419journalArticle[{'creatorType': 'author', 'firstName': 'Marga...The collection of official war art housed in t......Publisher: Routledge\\n_eprint: https://doi.org...[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30ZNaNNaNNaN2015
\n", "

5 rows × 39 columns

\n", "
" ], "text/plain": [ " DOI \\\n", "0 10.1080/10314610802663035 \n", "1 10.1080/1031461X.2011.640695 \n", "2 10.1080/1031461X.2012.706621 \n", "3 10.1080/1031461X.2012.760636 \n", "4 10.1080/1031461X.2014.996574 \n", "\n", " title \\\n", "0 A Disenfranchised Grief: Post-war Death and Me... \n", "1 A Dog in the Manger: White Australia and its V... \n", "2 ‘A Halo of Protection’: Colonial Protectors an... \n", "3 A House Committee on Un-Australian Activities?... \n", "4 ‘Accurate to the Point of Mania’: Eyewitness T... \n", "\n", " trove_url link_type \\\n", "0 https://trove.nla.gov.au/work/66604896 restricted \n", "1 https://trove.nla.gov.au/work/169309446 restricted \n", "2 https://trove.nla.gov.au/work/173739658 notonline \n", "3 https://trove.nla.gov.au/work/181447421 restricted \n", "4 https://trove.nla.gov.au/work/201475554 notonline \n", "\n", " repo_url key version \\\n", "0 https://hdl.handle.net/1959.9/471872 54PJ4CIG 6 \n", "1 https://researchonline.jcu.edu.au/22338/ 6XPGZ57R 11 \n", "2 https://hdl.handle.net/2440/74425 C6XRZRJ3 11 \n", "3 https://vuir.vu.edu.au/24499/ QWZDSVVC 14 \n", "4 https://hdl.handle.net/1885/63982 7WTYHC34 19 \n", "\n", " itemType creators \\\n", "0 journalArticle [{'creatorType': 'author', 'firstName': 'Marin... \n", "1 journalArticle [{'creatorType': 'author', 'firstName': 'Russe... \n", "2 journalArticle [{'creatorType': 'author', 'firstName': 'Amand... \n", "3 journalArticle [{'creatorType': 'author', 'firstName': 'Lachl... \n", "4 journalArticle [{'creatorType': 'author', 'firstName': 'Marga... \n", "\n", " abstractNote ... \\\n", "0 The 1918 Armistice signalled the end of the Fi... ... \n", "1 Between the world wars Australia was commonly ... ... \n", "2 Scholarship on Australia's colonial protectora... ... \n", "3 Legislation introduced by Prime Minister Rober... ... \n", "4 The collection of official war art housed in t... ... \n", "\n", " extra tags collections \\\n", "0 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "1 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "2 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "3 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "4 Publisher: Routledge\\n_eprint: https://doi.org... [] [922CMTJU] \n", "\n", " relations dateAdded dateModified oadoi licence oa_type \\\n", "0 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z NaN NaN NaN \n", "1 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z NaN NaN NaN \n", "2 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z NaN NaN NaN \n", "3 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z NaN NaN NaN \n", "4 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z NaN NaN NaN \n", "\n", " year \n", "0 2009 \n", "1 2012 \n", "2 2012 \n", "3 2013 \n", "4 2015 \n", "\n", "[5 rows x 39 columns]" ] }, "execution_count": 397, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all = pd.merge(df_repos, df, how='outer', on=['DOI', 'title'])\n", "df_all.head()" ] }, { "cell_type": "code", "execution_count": 398, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(275, 39)" ] }, "execution_count": 398, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Articles in repositories\n", "\n", "Now we have a dataset of articles that are in university repositories, as reported by Trove. Let's analyse them as we did above.\n", "\n", "First we'll look at the number of articles per year that have records in repositories." ] }, { "cell_type": "code", "execution_count": 399, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 399, "metadata": {}, "output_type": "execute_result" } ], "source": [ "alt.Chart(df_all.drop_duplicates(subset='DOI').fillna('-')).mark_bar().encode(\n", " x=alt.X('year:O', title='Year'),\n", " y=alt.Y('count():Q', title='Number of articles', axis=alt.Axis(tickMinStep=1)),\n", " color=alt.Color('link_type:N', scale=alt.Scale(range=['lightgrey', 'blue', 'red', 'orange']), legend=alt.Legend(title='Link type')),\n", " tooltip=[alt.Tooltip('count()', title='Number of articles'), alt.Tooltip('link_type', title='Link type')]\n", ").properties(width=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Oddly, there seems to have been a drop in the proportion of articles being added to repositories.\n", "\n", "Let's look at the breakdown by repository." ] }, { "cell_type": "code", "execution_count": 400, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "hdl.handle.net 68\n", "espace.library.uq.edu.au 12\n", "research-repository.uwa.edu.au 10\n", "researchers.mq.edu.au 9\n", "ro.uow.edu.au 6\n", "ecite.utas.edu.au 5\n", "researchonline.federation.edu.au 4\n", "vuir.vu.edu.au 2\n", "researchoutputs.unisa.edu.au 2\n", "researchonline.jcu.edu.au 2\n", "researchbank.rmit.edu.au 2\n", "eprints.usq.edu.au 2\n", "researchrepository.murdoch.edu.au 1\n", "hdl.cqu.edu.au 1\n", "handle.uws.edu.au:8081 1\n", "handle.unsw.edu.au 1\n", "dtype: int64" ] }, "execution_count": 400, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.drop_duplicates(subset='DOI')['repo_url'].str.extract(r'^https*://(.*?)/').value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's see if we can disambiguate those `handle.net` links. The code below tries to resolve the handle links, grabbing the address at the end of the redirects." ] }, { "cell_type": "code", "execution_count": 401, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://hdl.handle.net/1959.8/115505\n", "https://hdl.handle.net/1959.8/151729\n" ] } ], "source": [ "def get_redirected_url(url):\n", " if not pd.isna(url) and 'handle.net' in url:\n", " try:\n", " response = requests.get(url, timeout=60)\n", " except requests.exceptions.Timeout:\n", " print(url)\n", " else:\n", " return response.url\n", " return url\n", "\n", "df_all['redirected_url'] = df_all['repo_url'].apply(get_redirected_url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see above, two handle addresses failed to resolve. If you click on them you'll see they go to UNISA. I'm not sure what the problem is. \n", "\n", "Let's breakdown the repository details by the redirected urls." ] }, { "cell_type": "code", "execution_count": 402, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "openresearch-repository.anu.edu.au 19\n", "arrow.latrobe.edu.au:8080 15\n", "espace.library.uq.edu.au 12\n", "dro.deakin.edu.au 10\n", "research-repository.uwa.edu.au 10\n", "researchers.mq.edu.au 9\n", "rune.une.edu.au 7\n", "ro.uow.edu.au 6\n", "ecite.utas.edu.au 5\n", "research-repository.griffith.edu.au 5\n", "digital.library.adelaide.edu.au 4\n", "researchonline.federation.edu.au 4\n", "ogma.newcastle.edu.au:443 3\n", "eprints.usq.edu.au 2\n", "vuir.vu.edu.au 2\n", "researchbank.rmit.edu.au 2\n", "researchbank.swinburne.edu.au 2\n", "researchonline.jcu.edu.au 2\n", "researchoutputs.unisa.edu.au 2\n", "minerva-access.unimelb.edu.au 1\n", "hdl.cqu.edu.au 1\n", "handle.uws.edu.au:8081 1\n", "handle.unsw.edu.au 1\n", "researchdirect.westernsydney.edu.au 1\n", "researchrepository.murdoch.edu.au 1\n", "opus.lib.uts.edu.au 1\n", "dtype: int64" ] }, "execution_count": 402, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.drop_duplicates(subset='DOI')['redirected_url'].str.extract(r'^https*://(.*?)/').value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you compare this to the breakdown of OA articles by repository you'll see, for example, that the ANU repository has records for 19 articles, but only one of these makes a Green OA version available for download." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Are 'fulltext' articles really open?\n", "\n", "Now let's compare what we found in Trove, with the links we found using the OA Button API.\n", "\n", "First of all let's look for articles with 'fulltext' links for which we've already found an OA version." ] }, { "cell_type": "code", "execution_count": 403, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitledateoadoiredirected_urllink_type
5610.1080/1031461X.2014.911759‘Habeas Corpus Mongols’—Chinese Litigants and ...May 4, 2014https://research-repository.griffith.edu.au/bi...https://research-repository.griffith.edu.au/ha...fulltext
7810.1080/10314611003716861Mediatisation and Institutions of Public Memor...June 1, 2010https://eprints.qut.edu.au/32980/1/c32980.pdfhttps://eprints.usq.edu.au/6715/fulltext
10910.1080/1031461X.2011.560610Settler Justice and Aboriginal Homicide in Lat...June 1, 2011https://research-repository.griffith.edu.au/bi...https://research-repository.griffith.edu.au/ha...fulltext
14110.1080/1031461X.2018.1515963The Significance of the Northern Territory in ...October 2, 2018https://research-repository.griffith.edu.au/bi...https://research-repository.griffith.edu.au/ha...fulltext
15010.1080/1031461X.2010.493945Vida Lahey's Progressive Activism for Children...September 1, 2010https://research-repository.griffith.edu.au/bi...https://research-repository.griffith.edu.au/ha...fulltext
\n", "
" ], "text/plain": [ " DOI \\\n", "56 10.1080/1031461X.2014.911759 \n", "78 10.1080/10314611003716861 \n", "109 10.1080/1031461X.2011.560610 \n", "141 10.1080/1031461X.2018.1515963 \n", "150 10.1080/1031461X.2010.493945 \n", "\n", " title date \\\n", "56 ‘Habeas Corpus Mongols’—Chinese Litigants and ... May 4, 2014 \n", "78 Mediatisation and Institutions of Public Memor... June 1, 2010 \n", "109 Settler Justice and Aboriginal Homicide in Lat... June 1, 2011 \n", "141 The Significance of the Northern Territory in ... October 2, 2018 \n", "150 Vida Lahey's Progressive Activism for Children... September 1, 2010 \n", "\n", " oadoi \\\n", "56 https://research-repository.griffith.edu.au/bi... \n", "78 https://eprints.qut.edu.au/32980/1/c32980.pdf \n", "109 https://research-repository.griffith.edu.au/bi... \n", "141 https://research-repository.griffith.edu.au/bi... \n", "150 https://research-repository.griffith.edu.au/bi... \n", "\n", " redirected_url link_type \n", "56 https://research-repository.griffith.edu.au/ha... fulltext \n", "78 https://eprints.usq.edu.au/6715/ fulltext \n", "109 https://research-repository.griffith.edu.au/ha... fulltext \n", "141 https://research-repository.griffith.edu.au/ha... fulltext \n", "150 https://research-repository.griffith.edu.au/ha... fulltext " ] }, "execution_count": 403, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.loc[(df_all['link_type'] == 'fulltext') & (df_all['oadoi'].notnull())][['DOI', 'title', 'date', 'oadoi', 'redirected_url', 'link_type']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So the four Green OA articles in the Griffith repository also show up as 'fulltext' links in Trove. Yay! That's how things are meant to work.\n", "\n", "The other article seems a bit odd because the repository link goes to USQ, while the OA link goes to QUT. But this is actually ok. What's happening here is that there are Green OA versions of this article in two repositories. The Open Access Button API only gives us one of them. You can check that they're both there, however, by using the Unpaywall API directly. Look under `oa_locations` in the results below." ] }, { "cell_type": "code", "execution_count": 404, "metadata": {}, "outputs": [ { "data": { "application/json": { "best_oa_location": { "endpoint_id": "278d6ce3836860ab32c", "evidence": "oa repository (via OAI-PMH doi match)", "host_type": "repository", "is_best": true, "license": null, "oa_date": null, "pmh_id": "oai:eprints.qut.edu.au:32980", "repository_institution": null, "updated": "2020-10-18T13:45:47.024459", "url": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "url_for_landing_page": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "url_for_pdf": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "version": "publishedVersion" }, "data_standard": 2, "doi": "10.1080/10314611003716861", "doi_url": "https://doi.org/10.1080/10314611003716861", "first_oa_location": { "endpoint_id": "278d6ce3836860ab32c", "evidence": "oa repository (via OAI-PMH doi match)", "host_type": "repository", "is_best": true, "license": null, "oa_date": null, "pmh_id": "oai:eprints.qut.edu.au:32980", "repository_institution": null, "updated": "2020-10-18T13:45:47.024459", "url": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "url_for_landing_page": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "url_for_pdf": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "version": "publishedVersion" }, "genre": "journal-article", "has_repository_copy": true, "is_oa": true, "is_paratext": false, "journal_is_in_doaj": false, "journal_is_oa": false, "journal_issn_l": "1031-461X", "journal_issns": "1031-461X,1940-5049", "journal_name": "Australian Historical Studies", "oa_locations": [ { "endpoint_id": "278d6ce3836860ab32c", "evidence": "oa repository (via OAI-PMH doi match)", "host_type": "repository", "is_best": true, "license": null, "oa_date": null, "pmh_id": "oai:eprints.qut.edu.au:32980", "repository_institution": null, "updated": "2020-10-18T13:45:47.024459", "url": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "url_for_landing_page": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "url_for_pdf": "https://eprints.qut.edu.au/32980/1/c32980.pdf", "version": "publishedVersion" }, { "endpoint_id": "10e9952acfae17459d7", "evidence": "oa repository (via OAI-PMH title and first author match)", "host_type": "repository", "is_best": false, "license": null, "oa_date": null, "pmh_id": "oai:eprints.usq.edu.au:6715", "repository_institution": "University of Southern Queensland - University of Southern Queensland ePrints", "updated": "2020-09-22T11:56:32.862976", "url": "http://eprints.usq.edu.au/6715/3/Burgess_Klaebe_McWilliam_AHS_2010_AV.pdf", "url_for_landing_page": "http://eprints.usq.edu.au/6715/3/Burgess_Klaebe_McWilliam_AHS_2010_AV.pdf", "url_for_pdf": "http://eprints.usq.edu.au/6715/3/Burgess_Klaebe_McWilliam_AHS_2010_AV.pdf", "version": "acceptedVersion" } ], "oa_status": "green", "published_date": "2010-06-01", "publisher": "Informa UK Limited", "title": "Mediatisation and Institutions of Public Memory: Digital Storytelling and the Apology", "updated": "2020-10-13T01:16:44.029643", "year": 2010, "z_authors": [ { "family": "Burgess", "given": "Jean", "sequence": "first" }, { "family": "Klaebe", "given": "Helen", "sequence": "additional" }, { "family": "McWilliam", "given": "Kelly", "sequence": "additional" } ] }, "text/plain": [ "" ] }, "metadata": { "application/json": { "expanded": false, "root": "root" } }, "output_type": "display_data" } ], "source": [ "response = requests.get('https://api.unpaywall.org/v2/10.1080/10314611003716861?email=tim@discontents.com.au')\n", "display(JSON(response.json()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But why aren't both locations of this article showing up in our dataset. Let's look for the DOI." ] }, { "cell_type": "code", "execution_count": 405, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitletrove_urllink_typerepo_urlkeyversionitemTypecreatorsabstractNote...tagscollectionsrelationsdateAddeddateModifiedoadoilicenceoa_typeyearredirected_url
7810.1080/10314611003716861Mediatisation and Institutions of Public Memor...https://trove.nla.gov.au/work/37861673fulltexthttps://eprints.usq.edu.au/6715/6PXHXET77journalArticle[{'creatorType': 'author', 'firstName': 'Jean'...Institutions of public memory are increasingly......[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30Zhttps://eprints.qut.edu.au/32980/1/c32980.pdfgreen2010https://eprints.usq.edu.au/6715/
7910.1080/10314611003716861Mediatisation and Institutions of Public Memor...https://trove.nla.gov.au/work/37861673notonlinehttps://eprints.qut.edu.au/32980/6PXHXET77journalArticle[{'creatorType': 'author', 'firstName': 'Jean'...Institutions of public memory are increasingly......[][922CMTJU]{}2020-10-20T02:22:30Z2020-10-20T02:22:30Zhttps://eprints.qut.edu.au/32980/1/c32980.pdfgreen2010https://eprints.qut.edu.au/32980/
\n", "

2 rows × 40 columns

\n", "
" ], "text/plain": [ " DOI \\\n", "78 10.1080/10314611003716861 \n", "79 10.1080/10314611003716861 \n", "\n", " title \\\n", "78 Mediatisation and Institutions of Public Memor... \n", "79 Mediatisation and Institutions of Public Memor... \n", "\n", " trove_url link_type \\\n", "78 https://trove.nla.gov.au/work/37861673 fulltext \n", "79 https://trove.nla.gov.au/work/37861673 notonline \n", "\n", " repo_url key version itemType \\\n", "78 https://eprints.usq.edu.au/6715/ 6PXHXET7 7 journalArticle \n", "79 https://eprints.qut.edu.au/32980/ 6PXHXET7 7 journalArticle \n", "\n", " creators \\\n", "78 [{'creatorType': 'author', 'firstName': 'Jean'... \n", "79 [{'creatorType': 'author', 'firstName': 'Jean'... \n", "\n", " abstractNote ... tags collections \\\n", "78 Institutions of public memory are increasingly... ... [] [922CMTJU] \n", "79 Institutions of public memory are increasingly... ... [] [922CMTJU] \n", "\n", " relations dateAdded dateModified \\\n", "78 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "79 {} 2020-10-20T02:22:30Z 2020-10-20T02:22:30Z \n", "\n", " oadoi licence oa_type year \\\n", "78 https://eprints.qut.edu.au/32980/1/c32980.pdf green 2010 \n", "79 https://eprints.qut.edu.au/32980/1/c32980.pdf green 2010 \n", "\n", " redirected_url \n", "78 https://eprints.usq.edu.au/6715/ \n", "79 https://eprints.qut.edu.au/32980/ \n", "\n", "[2 rows x 40 columns]" ] }, "execution_count": 405, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.loc[df_all['DOI'] == '10.1080/10314611003716861']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ah, so they are both there, but look at the `link_type` values. One is 'fulltext', but the other is 'notonline' even though it points to a Green OA version. As I noted above, Trove makes an assessment of the online status of the article based on the available metadata – obviously something's going wrong here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How many more OA versions aren't labelled as 'fulltext' in Trove?" ] }, { "cell_type": "code", "execution_count": 406, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitleoadoiredirected_urllink_typeoa_type
1610.1080/1031461X.2017.1279196Asian Servants for the Imperial Telegraph: Ima...https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...https://ro.uow.edu.au/lhapapers/2997notonlinegreen
3610.1080/1031461X.2010.493947Colonial Judiciaries, Aboriginal Protection an...http://pdfs.semanticscholar.org/217a/cc68a93d9...https://digital.library.adelaide.edu.au/dspace...notonlinegreen
5210.1080/10314610903067094Galahshttps://openresearch-repository.anu.edu.au/bit...https://openresearch-repository.anu.edu.au/han...notonlinegreen
7910.1080/10314611003716861Mediatisation and Institutions of Public Memor...https://eprints.qut.edu.au/32980/1/c32980.pdfhttps://eprints.qut.edu.au/32980/notonlinegreen
8410.1080/1031461X.2017.1279197Neither a Discipline nor a Colony: Renaissance...https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...https://ro.uow.edu.au/lhapapers/3062notonlinegreen
10010.1080/1031461X.2015.1071416Re-Routing Empire? Steam-Age Circulations and ...https://ro.uow.edu.au/cgi/viewcontent.cgi?arti...https://ro.uow.edu.au/lhapapers/2166notonlinegreen
10310.1080/1031461X.2017.1394887Remembering and Fighting for Their Own: Vietna...https://www.tandfonline.com/doi/pdf/10.1080/10...http://dro.deakin.edu.au/view/DU:30106429notonlinegold
11510.1080/1031461X.2014.912667‘Such a Great Space of Water between Us’: Anza...https://www.tandfonline.com/doi/pdf/10.1080/10...https://openresearch-repository.anu.edu.au/han...notonlinegold
14010.1080/1031461X.2016.1250790The Place of Anzac in Australian Historical Co...https://opus.lib.uts.edu.au/bitstream/10453/88...https://opus.lib.uts.edu.au/handle/10453/88058notonlinegreen
\n", "
" ], "text/plain": [ " DOI \\\n", "16 10.1080/1031461X.2017.1279196 \n", "36 10.1080/1031461X.2010.493947 \n", "52 10.1080/10314610903067094 \n", "79 10.1080/10314611003716861 \n", "84 10.1080/1031461X.2017.1279197 \n", "100 10.1080/1031461X.2015.1071416 \n", "103 10.1080/1031461X.2017.1394887 \n", "115 10.1080/1031461X.2014.912667 \n", "140 10.1080/1031461X.2016.1250790 \n", "\n", " title \\\n", "16 Asian Servants for the Imperial Telegraph: Ima... \n", "36 Colonial Judiciaries, Aboriginal Protection an... \n", "52 Galahs \n", "79 Mediatisation and Institutions of Public Memor... \n", "84 Neither a Discipline nor a Colony: Renaissance... \n", "100 Re-Routing Empire? Steam-Age Circulations and ... \n", "103 Remembering and Fighting for Their Own: Vietna... \n", "115 ‘Such a Great Space of Water between Us’: Anza... \n", "140 The Place of Anzac in Australian Historical Co... \n", "\n", " oadoi \\\n", "16 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "36 http://pdfs.semanticscholar.org/217a/cc68a93d9... \n", "52 https://openresearch-repository.anu.edu.au/bit... \n", "79 https://eprints.qut.edu.au/32980/1/c32980.pdf \n", "84 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "100 https://ro.uow.edu.au/cgi/viewcontent.cgi?arti... \n", "103 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "115 https://www.tandfonline.com/doi/pdf/10.1080/10... \n", "140 https://opus.lib.uts.edu.au/bitstream/10453/88... \n", "\n", " redirected_url link_type oa_type \n", "16 https://ro.uow.edu.au/lhapapers/2997 notonline green \n", "36 https://digital.library.adelaide.edu.au/dspace... notonline green \n", "52 https://openresearch-repository.anu.edu.au/han... notonline green \n", "79 https://eprints.qut.edu.au/32980/ notonline green \n", "84 https://ro.uow.edu.au/lhapapers/3062 notonline green \n", "100 https://ro.uow.edu.au/lhapapers/2166 notonline green \n", "103 http://dro.deakin.edu.au/view/DU:30106429 notonline gold \n", "115 https://openresearch-repository.anu.edu.au/han... notonline gold \n", "140 https://opus.lib.uts.edu.au/handle/10453/88058 notonline green " ] }, "execution_count": 406, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.loc[(df_all['link_type'].notnull()) & (df_all['link_type'] != 'fulltext') & (df_all['oadoi'].notnull())].drop_duplicates(subset='DOI')[['DOI', 'title', 'oadoi', 'redirected_url', 'link_type', 'oa_type']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The repository managers might like to look at these to see why the `link_type` values are not being set correctly in Trove. What's most disappointing here is that in two cases authors have gone to the trouble of making their articles Gold Open Access, but anyone searching for them in Trove will be told that they're not online, even though there's a link to them!\n", "\n", "Now let's look at 'fulltext' links for which we haven't found an OA version." ] }, { "cell_type": "code", "execution_count": 407, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DOItitleoadoiredirected_urllink_typeoa_type
810.1080/1031461X.2017.1338740‘Annamese Coolies’ at Australian Ports: Charti...NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
1110.1080/1031461X.2011.640692Antipodean Eco-nazis? The Organic Gardening an...NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
1310.1080/1031461X.2014.994539Anzac for Sale: Consumer Culture, Regulation a...NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
1910.1080/1031461X.2014.877779Australia and Anti-SlaveryNaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
2210.1080/1031461X.2015.1044757Australia's Women Surfers: History, Methodolog...NaNhttps://espace.library.uq.edu.au/view/UQ:368053fulltextNaN
4410.1080/1031461X.2014.946524Emotional Encounters: Indigenous Peoples in th...NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
8310.1080/1031461X.2018.1449872National Shame/National Treasure: Narrating Ho...NaNhttps://handle.unsw.edu.au/1959.4/unsworks_51159fulltextNaN
8910.1080/10314610701837250Obliterating History?NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
12110.1080/1031461X.2013.877503The Bloody Skirt of Settlement: Arthur Vogan a...NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
12810.1080/1031461X.2017.1313875The Indigenous Franchise and AssimilationNaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
15210.1080/10314610802263331‘Wanton With Plenty’ Questioning Ethno-histori...NaNhttps://research-repository.uwa.edu.au/en/publ...fulltextNaN
\n", "
" ], "text/plain": [ " DOI \\\n", "8 10.1080/1031461X.2017.1338740 \n", "11 10.1080/1031461X.2011.640692 \n", "13 10.1080/1031461X.2014.994539 \n", "19 10.1080/1031461X.2014.877779 \n", "22 10.1080/1031461X.2015.1044757 \n", "44 10.1080/1031461X.2014.946524 \n", "83 10.1080/1031461X.2018.1449872 \n", "89 10.1080/10314610701837250 \n", "121 10.1080/1031461X.2013.877503 \n", "128 10.1080/1031461X.2017.1313875 \n", "152 10.1080/10314610802263331 \n", "\n", " title oadoi \\\n", "8 ‘Annamese Coolies’ at Australian Ports: Charti... NaN \n", "11 Antipodean Eco-nazis? The Organic Gardening an... NaN \n", "13 Anzac for Sale: Consumer Culture, Regulation a... NaN \n", "19 Australia and Anti-Slavery NaN \n", "22 Australia's Women Surfers: History, Methodolog... NaN \n", "44 Emotional Encounters: Indigenous Peoples in th... NaN \n", "83 National Shame/National Treasure: Narrating Ho... NaN \n", "89 Obliterating History? NaN \n", "121 The Bloody Skirt of Settlement: Arthur Vogan a... NaN \n", "128 The Indigenous Franchise and Assimilation NaN \n", "152 ‘Wanton With Plenty’ Questioning Ethno-histori... NaN \n", "\n", " redirected_url link_type oa_type \n", "8 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "11 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "13 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "19 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "22 https://espace.library.uq.edu.au/view/UQ:368053 fulltext NaN \n", "44 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "83 https://handle.unsw.edu.au/1959.4/unsworks_51159 fulltext NaN \n", "89 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "121 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "128 https://research-repository.uwa.edu.au/en/publ... fulltext NaN \n", "152 https://research-repository.uwa.edu.au/en/publ... fulltext NaN " ] }, "execution_count": 407, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_all.loc[(df_all['link_type'] == 'fulltext') & (df_all['oadoi'].isnull())].drop_duplicates(subset='DOI')[['DOI', 'title', 'oadoi', 'redirected_url', 'link_type', 'oa_type']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Nine of these links are from UWA. It looks like they might not be getting indexed properly by Unpaywall. Let's display the links so we can click on them and see what's really going on." ] }, { "cell_type": "code", "execution_count": 408, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "10.1080/10314610701837250\n", "Obliterating History?\n", " * https://research-repository.uwa.edu.au/en/publications/526f9c94-1337-4944-8235-e5480a15dd1b\n", " * https://research-repository.uwa.edu.au/en/publications/obliterating-history-the-transformation-of-inner-city-industrial-suburbs(526f9c94-1337-4944-8235-e5480a15dd1b).html\n", "\n", "10.1080/10314610802263331\n", "‘Wanton With Plenty’ Questioning Ethno-historical Constructions of Sexual Savagery in Aboriginal Societies, 1788–1803\n", " * https://research-repository.uwa.edu.au/en/publications/7d37441e-48ba-4931-af28-42a1acaf6c33\n", " * https://research-repository.uwa.edu.au/en/publications/wanton-with-plenty-questioning-ethnohistorical-constructions-of-sexual-savagery-in-aboriginal-societies-17881803(7d37441e-48ba-4931-af28-42a1acaf6c33).html\n", "\n", "10.1080/1031461X.2011.640692\n", "Antipodean Eco-nazis? The Organic Gardening and Farming Movement and Far-right Ecology in Postwar Australia\n", " * https://research-repository.uwa.edu.au/en/publications/antipodean-econazis-the-organic-gardening-and-farming-movement-and-farright-ecology-in-postwar-australia(4cdfaeec-1c96-4960-93a3-a3ebcdcac4de).html\n", " * https://research-repository.uwa.edu.au/en/publications/4cdfaeec-1c96-4960-93a3-a3ebcdcac4de\n", "\n", "10.1080/1031461X.2013.877503\n", "The Bloody Skirt of Settlement: Arthur Vogan and Anti-Slavery in 1890s Australia\n", " * https://research-repository.uwa.edu.au/en/publications/7f0ea44a-ef69-4c16-9cd9-ccb21870e846\n", " * https://research-repository.uwa.edu.au/en/publications/the-bloody-skirt-of-settlement-arthur-vogan-and-antislavery-in-1890s-australia(7f0ea44a-ef69-4c16-9cd9-ccb21870e846).html\n", "\n", "10.1080/1031461X.2014.877779\n", "Australia and Anti-Slavery\n", " * https://research-repository.uwa.edu.au/en/publications/20509505-bbe8-4eae-8707-26645b9971c8\n", " * https://research-repository.uwa.edu.au/en/publications/australia-and-antislavery(20509505-bbe8-4eae-8707-26645b9971c8).html\n", "\n", "10.1080/1031461X.2014.946524\n", "Emotional Encounters: Indigenous Peoples in the Dutch East India Company's Interactions with the South Lands\n", " * https://research-repository.uwa.edu.au/en/publications/emotional-encounters-indigenous-peoples-in-the-dutch-east-india-companys-interactions-with-the-south-lands(4448d7a1-5dff-4aad-a73e-0c72f1de180d).html\n", " * https://research-repository.uwa.edu.au/en/publications/4448d7a1-5dff-4aad-a73e-0c72f1de180d\n", "\n", "10.1080/1031461X.2014.994539\n", "Anzac for Sale: Consumer Culture, Regulation and the Shaping of a Legend, 1915–21\n", " * https://research-repository.uwa.edu.au/en/publications/3d8a2558-b325-4bb0-817a-c8ff7c1a5a53\n", " * https://research-repository.uwa.edu.au/en/publications/anzac-for-sale-consumer-culture-regulation-and-the-shaping-of-a-legend-191521(3d8a2558-b325-4bb0-817a-c8ff7c1a5a53).html\n", "\n", "10.1080/1031461X.2015.1044757\n", "Australia's Women Surfers: History, Methodology and the Digital Humanities\n", " * https://espace.library.uq.edu.au/view/UQ:368053\n", " * https://espace.library.uq.edu.au/view/UQ:368053/UQ368053_OA.pdf\n", "\n", "10.1080/1031461X.2017.1313875\n", "The Indigenous Franchise and Assimilation\n", " * https://research-repository.uwa.edu.au/en/publications/the-indigenous-franchise-and-assimilation(3eca97f2-44d2-4b4a-b6f4-b298a89a1eb4).html\n", " * https://research-repository.uwa.edu.au/en/publications/3eca97f2-44d2-4b4a-b6f4-b298a89a1eb4\n", "\n", "10.1080/1031461X.2017.1338740\n", "‘Annamese Coolies’ at Australian Ports: Charting Colonial Geographies of Emotion, and Settler Memory, from French Vietnam to New Caledonia via Interwar Australia\n", " * https://research-repository.uwa.edu.au/en/publications/annamese-coolies-at-australian-ports(412bb598-bfa6-4aae-8ffb-8ca2633cdbfb).html\n", " * https://research-repository.uwa.edu.au/en/publications/412bb598-bfa6-4aae-8ffb-8ca2633cdbfb\n", "\n", "10.1080/1031461X.2018.1449872\n", "National Shame/National Treasure: Narrating Homeless Veterans in Australia 1915–1930s\n", " * https://handle.unsw.edu.au/1959.4/unsworks_51159\n" ] } ], "source": [ "for article, links in df_all.loc[(df_all['link_type'] == 'fulltext') & (df_all['oadoi'].isnull())].groupby(by=['DOI', 'title']):\n", " print(f'\\n{article[0]}')\n", " print(f'{article[1]}')\n", " for link in links['repo_url'].to_list():\n", " print(f' * {link}')\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you click on the UWA repository links you'll see that there's no link to an OA version of the articles. So they're not 'fulltext' links at all. \n", "\n", "However, the [UNSW repository link](https://handle.unsw.edu.au/1959.4/unsworks_51159) doesn indeed lead to a Green OA version! After I shared this information on Twitter, Richard Orr from Unpaywall checked out the link and [explained the problem](https://twitter.com/unpaywall_dev/status/1318963084392804354). Fiona Bradley from UNSW also [noted that they're shifting to new repository software](https://twitter.com/Fiona_Bradley/status/1319036507244560385), so hopefully that'll fix things.\n", "\n", "The [UQ repository link](https://espace.library.uq.edu.au/view/UQ:368053) also leads to a Green OA version. I didn't pick this one up in my first pass, but I've now asked Richard about it as well.\n", "\n", "So what's clear is that we can't rely on the `link_type` value in Trove, and there may be some repositories whose OA data is not being captured by Unpaywall.\n", "\n", "The only way to really check this thoroughly would be to work through all the other Trove repository links to see if there are more Green OA versions hiding there. After a bit of semi-random clicking I did find one example in NOVA, the University of Newcastle repository: http://hdl.handle.net/1959.13/805160. Again, Richard Orr checked this out and has [implemented a fix in Unpaywall](https://twitter.com/unpaywall_dev/status/1318961609771986944) so the Green OA version should now be indexed. I checked the other two links to NOVA in the data from Trove, but unfortunately those records don't have Green OA versions.\n", "\n", "It's possible there are more Green OA versions that I haven't found. I suppose I could try loading all the repository links and looking for some soert of download link. But I think I'll leave that for another day..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusions\n", "\n", "Adding in the 3 extra Green OA versions I found exploring the data from Trove, we have a total of 23 articles out of 242 that are available via open access. **That's just 9.5%!**\n", "\n", "Although some repositories aren't being indexed properly, the major reason why we can't find OA versions for 90% of the articles is just that the authors haven't made them available through an open repository. This is both disappointing and hopeful. Disappointing in that so much significant scholarship remains locked up behind a paywall. But hopeful in that the solution is pretty straightforward. If you're the author of one of these articles:\n", "\n", "* If you're at a university, talk to your local librarians or repository managers about how you can upload an AAM version of your article.\n", "* If you're not at a university, you can use [Share Your Paper](https://shareyourpaper.org/) to upload it to Zenodo. \n", "\n", "If you want to get access to one of these articles:\n", "\n", "* Use the [Open Access Button](https://openaccessbutton.org/) to send a request to the author to make it available.\n", "\n", "If you're about to publish an article in *Australian Historical Studies* or *History Australia* (or many other journals – check their OA policies using [Sherpa Romeo](http://v2.sherpa.ac.uk/romeo/)):\n", "\n", "* Upload the AAM version to a repository immediately upon publication, and set the embargo period for 18 months.\n", "* Share your AAM version immediately upon publication through your own personal website.\n", "\n", "If you're thinking about publishing an article:\n", "\n", "* Check out the [Directory of Open Access Journals](https://doaj.org/) for a full open access alternative!\n", "* Share the pre-print version of your article (that's the version *before* you've submitted it to a journal) in a repository like [Humanities Commons](https://hcommons.org/). Yes, it's a bit scary but you might get some useful feedback or find new connections." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }