{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a citable text corpus from OCRE\n",
    "\n",
    "This notebook shows you how to load OCRE data from a CEX file over the internet, and build a corpus of text citable by CTS URN.  It uses version `1.7.0` of the `nomisma` library. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Configure Jupyter notebook\n",
    "\n",
    "First configure the Jupyter notebook. In addition to the `nomisma` library, we'll need the `cite` and `ohco2` libraries from the CITE architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// 1. Add maven repository where we can find our libraries\n",
    "val myBT = coursierapi.MavenRepository.of(\"https://dl.bintray.com/neelsmith/maven\")\n",
    "interp.repositories() ++= Seq(myBT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// 2. Make libraries available with `$ivy` imports:\n",
    "import $ivy.`edu.holycross.shot::nomisma:1.7.0`\n",
    "import $ivy.`edu.holycross.shot::ohco2:10.16.0`\n",
    "import $ivy.`edu.holycross.shot.cite::xcite:4.1.1`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the full OCRE data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import edu.holycross.shot.nomisma._\n",
    "val ocreCex = \"https://raw.githubusercontent.com/neelsmith/nomisma/master/cex/ocre-cite-ids.cex\"\n",
    "val ocre = OcreSource.fromUrl(ocreCex)\n",
    "\n",
    "// Sanity check:\n",
    "require(ocre.size > 50000) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TL;DR\n",
    "\n",
    "You can build an OHCO2 corpus with the `corpus` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import edu.holycross.shot.ohco2._\n",
    "import edu.holycross.shot.cite._\n",
    "\n",
    "val corpus: Corpus = ocre.corpus\n",
    "println(\"Citable nodes of text in corpus: \" + corpus.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works for individual issues\n",
    "\n",
    "The `OcreIssue` class includes a `textNodes` function that creates a Vector of 0-2 `CitableNode`s.  There will be two text nodes if the issue has both an obverse and reverse legend.  Let's examine the CTS URNS of an issue that has both obverse and reverse legends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val issueId = \"3.com.43\"\n",
    "val randomIssue = ocre.issue(issueId).get\n",
    "\n",
    "println(\"In issue \" + issueId + \", made \" + randomIssue.textNodes.size + \" text nodes\")\n",
    "\n",
    "for (n <- randomIssue.textNodes) {\n",
    "    println(\"\\nReference: \" + n.urn)\n",
    "    println(\"Text content: \" + n.text)\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's parse the components of the URN.\n",
    "\n",
    "It belongs to the CTS namespace `hcnum`, and a text group `issues`.  \n",
    "\n",
    "Within that group, its document identifier is `ric`, and the specific version identifier is `raw`. When we process the corpus (e.g., to generate a fully expanded version of abbreviated terms), we will use a different version identifier, but the rest of the URN will be the same.\n",
    "\n",
    "The passage component is directly adapted from the nomisma.org identifier:  `3.com.43` identifies RIC volume 3, Commodus, issue 43.  The final piece of the passage component distinguishes obverse text from reverse text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works: building a corpus\n",
    "\n",
    "The `corpus` function in `Ocre` creates 0-2 `CitableNode`s for each issue and compiles them into a text `Corpus`.  \n",
    "\n",
    "As in any CTS environment, we can then select texts identified at any level of the passage and work hierarchies.\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val commodus43 = corpus.nodes.filter(_.urn <=  CtsUrn(\"urn:cts:hcnum:issues.ric.raw:3.com.43\"))\n",
    "println(\"**OBV** \" + commodus43.map(_.text).mkString(\" **REV** \"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val allCommodus = corpus.nodes.filter(_.urn <= CtsUrn(\"urn:cts:hcnum:issues.ric.raw:3.com\"))\n",
    "println(\"All legends in coins of Commodus: \" + allCommodus.size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val allRIC3 = corpus.nodes.filter(_.urn <= CtsUrn(\"urn:cts:hcnum:issues.ric.raw:3\"))\n",
    "println(\"All legends in RIC 3: \" + allRIC3.size)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Scala (2.12)",
   "language": "scala",
   "name": "scala212"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "text/x-scala",
   "name": "scala",
   "nbconvert_exporter": "script",
   "version": "2.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}