This is an experimental interface to the NSW Post Office Directories, which have been digitised from the collections of the State Library of NSW and are now available on Trove.
Searching for entries in the directories on Trove is difficult, because Trove's results are grouped by 'article', which is not very helpful for publications like these where information is organised by row. This interface searches for individual lines of text, rather than pages or articles. So it points you straight to entries of interest. Once you've found something, you can view the entry within the context of the complete page, or click back to Trove to explore further.
You can currently search across 15 volumes from 1886 to 1908.\",\n", " \"databases\": {\n", " \"post-office-directories\": {\n", " \"title\": \"New South Wales Post Office Directories, 1886 to 1908\",\n", " \"source\": \"Trove\",\n", " \"source_url\": \"https://nla.gov.au/nla.obj-518308191\",\n", " \"tables\": {},\n", " }\n", " },\n", "}" ] }, { "cell_type": "markdown", "id": "1c1c10d8-25cd-4cb7-b904-fe9a85cba5fa", "metadata": {}, "source": [ "Now we'll process the data and add it to an SQLite database." ] }, { "cell_type": "code", "execution_count": null, "id": "360267d7-8329-4632-8664-971b7e043bef", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "nbval-skip" ] }, "outputs": [], "source": [ "# Create the database\n", "# Change the name as apporpriate!\n", "db = Database(Path(db_path, \"post-office-directories.db\"))\n", "\n", "# Create database tables for pages and volumes\n", "page_table = db[\"pages\"]\n", "vols_table = db[\"volumes\"]\n", "\n", "# Loop through the directories of each volume created by the harvesting process (above)\n", "# Use natsorted so that they're processed in date order\n", "vols = natsorted(\n", " [\n", " d\n", " for d in Path(\"directories\").glob(\n", " \"the-new-south-wales-post-office-directory-collection*\"\n", " )\n", " if d.is_dir()\n", " ]\n", ")\n", "\n", "for vol in vols:\n", " print(vol)\n", "\n", " # In the case of the PO Directories each volume has a different year (or year range) in the title.\n", " # Here we're extracting the year and id, but extracting the year in this way might not work for other titles.\n", " # The year is used as the title of the table in Datasette\n", " year, obj_id = re.search(r\"-([0-9\\-]+?)-(nla.obj-\\d+)\", vol.name).groups()\n", "\n", " # Add a record for this volume to the database\n", " vols_table.insert({\"vol_id\": obj_id, \"year\": year}, pk=\"vol_id\")\n", "\n", " # Get the embedded JSON data for this volume and extract a list of pages\n", " work_data = get_work_metadata(obj_id)\n", " work_pages = work_data[\"children\"][\"page\"]\n", "\n", " # Get the title from the work metadata\n", " title = work_data[\"title\"]\n", "\n", " # Update the metadata file with details of this volume\n", " metadata[\"databases\"][\"post-office-directories\"][\"tables\"][year] = {\n", " \"title\": f\"{title} {year}\",\n", " \"source\": \"Trove\",\n", " \"source_url\": f\"https://nla.gov.au/{obj_id}\",\n", " \"searchmode\": \"raw\",\n", " }\n", "\n", " # Create a table for this volume. For the PO directories I'm using the year as the table name.\n", " # If year isn't available, some other way of naming the table would be necessary, such as the full title.\n", " vol_table = db[year]\n", "\n", " # Loop through all the text page by page for this volume\n", " pages = natsorted([p for p in vol.glob(\"*\") if p.is_file()], alg=ns.PATH)\n", " for page in pages:\n", " lines = []\n", "\n", " # Insert details about this page into the pages table\n", " page_num = int(re.search(r\"-page(\\d+)$\", page.name).group(1))\n", " page_table.insert(\n", " {\n", " \"vol_id\": obj_id,\n", " \"page_id\": work_pages[page_num][\"pid\"],\n", " \"page\": page_num,\n", " },\n", " pk=(\"vol_id\", \"page\"),\n", " foreign_keys=[(\"vol_id\", \"volumes\")],\n", " )\n", "\n", " # Open the text file and loop through the lines\n", " with page.open(\"r\") as txt:\n", " for num, line in enumerate(txt):\n", " # Get rid of blank lines\n", " line = line.replace(\"\\n\", \"\")\n", " # If line is not blank, add details to a list of lines from this page\n", " if line:\n", " lines.append(\n", " {\n", " \"page\": page_num,\n", " \"line\": num + 1,\n", " \"text\": line,\n", " \"page_id\": work_pages[page_num][\"pid\"],\n", " }\n", " )\n", " # Insert all the lines from this page into the db\n", " vol_table.insert_all(\n", " lines, pk=(\"page\", \"line\"), foreign_keys=[(\"page_id\", \"pages\", \"page_id\")]\n", " )\n", " # Add a full text index on the line text\n", " vol_table.enable_fts([\"text\"])\n", "\n", "# Save the updated metadata file\n", "with open(Path(db_path, \"metadata.json\"), \"w\") as md_file:\n", " json.dump(metadata, md_file, indent=4)" ] }, { "cell_type": "markdown", "id": "c2080f86-6a82-43b2-9e6f-29f7297abc9f", "metadata": {}, "source": [ "## Opening the db in Datasette" ] }, { "cell_type": "markdown", "id": "57ef2e22-fa52-48b2-9ddc-237ea2d0a150", "metadata": {}, "source": [ "You should now have an SQLite database containing the indexed text of all the volumes in the collection. You can explore the database using Datasette. In a Python environment you'll need to install:\n", "\n", "* [Datasette](https://datasette.io/)\n", "* [datasette-search-all](https://github.com/simonw/datasette-search-all)\n", "* [datasette-template-sql](https://datasette.io/plugins/datasette-template-sql)\n", "\n", "Then, from within the directory containing the database, run:\n", "\n", "```shell\n", "datasette post-office-directories.db -m metadata.json\n", "```\n", "\n", "To share your database publicly, look at [Datasette's publishing options](https://docs.datasette.io/en/stable/publish.html)." ] }, { "cell_type": "markdown", "id": "eab2e43c-71d4-43b7-a9a2-ac6b7eacf95c", "metadata": {}, "source": [ "----\n", "\n", "Created by [Tim Sherratt](https://timsherratt.org/) for the [GLAM Workbench](https://glam-workbench.net/) as part of the [Everyday Heritage](https://everydayheritage.au/) project." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "rocrate": { "author": [ { "mainEntityOfPage": "https://timsherratt.au", "name": "Sherratt, Tim", "orcid": "https://orcid.org/0000-0001-7956-4498" } ], "category": "Harvesting text", "description": "The code here was used to create the NSW Post Office Directories search interface which helps you search across 54 volumes from 1886 to 1950. The same code, with minor modifications, could be used to index any publication where it would be useful to search by line (rather than Trove's default 'article') – for example, lists, directories and gazetteers – turning them into searchable databases.", "mainEntityOfPage": "https://glam-workbench.net/trove-journals/create-text-db-indexed-by-line/", "name": "Create a database to search across each line of text in a series of volumes", "position": 6, "workExample": [ { "name": "NSW Post Office Directories, 1886 to 1950", "url": "https://glam-workbench.net/nsw-post-office-directories/" }, { "name": "Sydney Telephone Directories, 1926 to 1954", "url": "https://glam-workbench.net/trove-journals/sydney-telephone-directories/" } ] } }, "nbformat": 4, "nbformat_minor": 5 }