{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "# Tutorial\n", "\n", "This notebook gets you started with using\n", "[Text-Fabric](https://annotation.github.io/text-fabric/) for coding in the Old-Babylonian Letter corpus (cuneiform).\n", "\n", "Familiarity with the underlying\n", "[data model](https://annotation.github.io/text-fabric/tf/about/datamodel.html)\n", "is recommended." ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "## Installing Text-Fabric\n", "\n", "See [here](https://annotation.github.io/text-fabric/tf/about/install.html)" ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "## Tip\n", "If you start computing with this tutorial, first copy its parent directory to somewhere else, outside your copy of the repository\n", "If you pull changes from the repository later, your work will not be overwritten.\n", "Where you put your tutorial directory is up to you.\n", "It will work from any directory." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Old Babylonian data\n", "\n", "Text-Fabric will fetch the data set for you from the newest GitHub release binaries.\n", "\n", "The data will be stored in the `text-fabric-data` in your home directory." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Features\n", "The data of the corpus is organized in features.\n", "They are *columns* of data.\n", "Think of the corpus as a gigantic spreadsheet, where row 1 corresponds to the\n", "first sign, row 2 to the second sign, and so on, for all 200,000 signs.\n", "\n", "The information which reading each sign has, constitutes a column in that spreadsheet.\n", "The Old Babylonian corpus contains nearly 60 columns, not only for the signs, but also for thousands of other\n", "textual objects, such as clusters, lines, columns, faces, documents.\n", "\n", "Instead of putting that information in one big table, the data is organized in separate columns.\n", "We call those columns **features**." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:16.202764Z", "start_time": "2018-05-18T09:17:16.197546Z" } }, "outputs": [], "source": [ "import os\n", "import collections" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Incantation\n", "\n", "The simplest way to get going is by this *incantation*:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:17.537171Z", "start_time": "2018-05-18T09:17:17.517809Z" } }, "outputs": [], "source": [ "from tf.app import use" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the very last version, use `hot`.\n", "\n", "For the latest release, use `latest`.\n", "\n", "If you have cloned the repos (TF app and data), use `clone`.\n", "\n", "If you do not want/need to upgrade, leave out the checkout specifiers." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "**Locating corpus resources ...**" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "app: ~/text-fabric-data/github/Nino-cunei/oldbabylonian/app" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "data: ~/text-fabric-data/github/Nino-cunei/oldbabylonian/tf/1.0.6" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " TF: TF API 12.1.3, Nino-cunei/oldbabylonian/app v3, Search Reference
\n", " Data: Nino-cunei - oldbabylonian 1.0.6, Character table, Feature docs
\n", "
Node types\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "
Name# of nodes# slots / node% coverage
document1285158.15100
face283471.71100
line273757.42100
word765052.64100
cluster234491.7821
sign2032191.00100
\n", " Sets: no custom sets
\n", " Features:
\n", "
Old Babylonian Letters 1900-1600: Cuneiform tablets\n", "
\n", "\n", "
\n", "
\n", "ARK\n", "
\n", "
str
\n", "\n", " persistent identifier of type ARK from metadata field \"UCLA Library ARK\"\n", "\n", "
\n", "\n", "
\n", "
\n", "after\n", "
\n", "
str
\n", "\n", " what comes after a sign or word (- or space)\n", "\n", "
\n", "\n", "
\n", "
\n", "afterr\n", "
\n", "
str
\n", "\n", " what comes after a sign or word (- or space); between adjacent signs a ␣ is inserted\n", "\n", "
\n", "\n", "
\n", "
\n", "afteru\n", "
\n", "
str
\n", "\n", " what comes after a sign when represented as unicode (space)\n", "\n", "
\n", "\n", "
\n", "
\n", "atf\n", "
\n", "
str
\n", "\n", " full atf of a sign (without cluster chars) or word (including cluster chars)\n", "\n", "
\n", "\n", "
\n", "
\n", "atfpost\n", "
\n", "
str
\n", "\n", " atf of cluster closings at sign\n", "\n", "
\n", "\n", "
\n", "
\n", "atfpre\n", "
\n", "
str
\n", "\n", " atf of cluster openings at sign\n", "\n", "
\n", "\n", "
\n", "
\n", "author\n", "
\n", "
str
\n", "\n", " author from metadata field \"Author(s)\"\n", "\n", "
\n", "\n", "
\n", "
\n", "col\n", "
\n", "
int
\n", "\n", " ATF column number\n", "\n", "
\n", "\n", "
\n", "
\n", "collated\n", "
\n", "
int
\n", "\n", " whether a sign is collated (*)\n", "\n", "
\n", "\n", "
\n", "
\n", "collection\n", "
\n", "
str
\n", "\n", " collection of a document\n", "\n", "
\n", "\n", "
\n", "
\n", "comment\n", "
\n", "
str
\n", "\n", " $ comment to line or inline comment to slot ($ and $)\n", "\n", "
\n", "\n", "
\n", "
\n", "damage\n", "
\n", "
int
\n", "\n", " whether a sign is damaged\n", "\n", "
\n", "\n", "
\n", "
\n", "det\n", "
\n", "
int
\n", "\n", " whether a sign is a determinative gloss - between braces { }\n", "\n", "
\n", "\n", "
\n", "
\n", "docnote\n", "
\n", "
str
\n", "\n", " additional remarks in the document identification\n", "\n", "
\n", "\n", "
\n", "
\n", "docnumber\n", "
\n", "
str
\n", "\n", " number of a document within a collection-volume\n", "\n", "
\n", "\n", "
\n", "
\n", "excavation\n", "
\n", "
str
\n", "\n", " excavation number from metadata field \"Excavation no.\"\n", "\n", "
\n", "\n", "
\n", "
\n", "excised\n", "
\n", "
int
\n", "\n", " whether a sign is excised - between double angle brackets << >>\n", "\n", "
\n", "\n", "
\n", "
\n", "face\n", "
\n", "
str
\n", "\n", " full name of a face including the enclosing object\n", "\n", "
\n", "\n", "
\n", "
\n", "flags\n", "
\n", "
str
\n", "\n", " sequence of flags after a sign\n", "\n", "
\n", "\n", "
\n", "
\n", "fraction\n", "
\n", "
str
\n", "\n", " fraction of a numeral\n", "\n", "
\n", "\n", "
\n", "
\n", "genre\n", "
\n", "
str
\n", "\n", " genre from metadata field \"Genre\"\n", "\n", "
\n", "\n", "
\n", "
\n", "grapheme\n", "
\n", "
str
\n", "\n", " grapheme of a sign\n", "\n", "
\n", "\n", "
\n", "
\n", "graphemer\n", "
\n", "
str
\n", "\n", " grapheme of a sign using non-ascii characters\n", "\n", "
\n", "\n", "
\n", "
\n", "graphemeu\n", "
\n", "
str
\n", "\n", " grapheme of a sign using cuneiform unicode characters\n", "\n", "
\n", "\n", "
\n", "
\n", "lang\n", "
\n", "
str
\n", "\n", " language of a document\n", "\n", "
\n", "\n", "
\n", "
\n", "langalt\n", "
\n", "
int
\n", "\n", " 1 if a sign is in the alternate language (i.e. Sumerian) - between underscores _ _\n", "\n", "
\n", "\n", "
\n", "
\n", "ln\n", "
\n", "
int
\n", "\n", " ATF line number of a numbered line, without prime\n", "\n", "
\n", "\n", "
\n", "
\n", "lnc\n", "
\n", "
str
\n", "\n", " ATF line identification of a comment line ($)\n", "\n", "
\n", "\n", "
\n", "
\n", "lnno\n", "
\n", "
str
\n", "\n", " ATF line number, may be $ or #, with prime; column number prepended\n", "\n", "
\n", "\n", "
\n", "
\n", "material\n", "
\n", "
str
\n", "\n", " material indication from metadata field \"Material\"\n", "\n", "
\n", "\n", "
\n", "
\n", "missing\n", "
\n", "
int
\n", "\n", " whether a sign is missing - between square brackets [ ]\n", "\n", "
\n", "\n", "
\n", "
\n", "museumcode\n", "
\n", "
str
\n", "\n", " museum code from metadata field \"Museum no.\"\n", "\n", "
\n", "\n", "
\n", "
\n", "museumname\n", "
\n", "
str
\n", "\n", " museum name from metadata field \"Collection\"\n", "\n", "
\n", "\n", "
\n", "
\n", "object\n", "
\n", "
str
\n", "\n", " name of an object of a document\n", "\n", "
\n", "\n", "
\n", "
\n", "operator\n", "
\n", "
str
\n", "\n", " the ! or x in a !() or x() construction\n", "\n", "
\n", "\n", "
\n", "
\n", "operatorr\n", "
\n", "
str
\n", "\n", " the ! or x in a !() or x() construction, represented as =, ␣\n", "\n", "
\n", "\n", "
\n", "
\n", "operatoru\n", "
\n", "
str
\n", "\n", " the ! or x in a !() or x() construction, represented as =, ␣\n", "\n", "
\n", "\n", "
\n", "
\n", "otype\n", "
\n", "
str
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "period\n", "
\n", "
str
\n", "\n", " period indication from metadata field \"Period\"\n", "\n", "
\n", "\n", "
\n", "
\n", "pnumber\n", "
\n", "
str
\n", "\n", " P number of a document\n", "\n", "
\n", "\n", "
\n", "
\n", "primecol\n", "
\n", "
int
\n", "\n", " whether a prime is present on a column number\n", "\n", "
\n", "\n", "
\n", "
\n", "primeln\n", "
\n", "
int
\n", "\n", " whether a prime is present on a line number\n", "\n", "
\n", "\n", "
\n", "
\n", "pubdate\n", "
\n", "
str
\n", "\n", " publication date from metadata field \"Publication date\"\n", "\n", "
\n", "\n", "
\n", "
\n", "question\n", "
\n", "
int
\n", "\n", " whether a sign has the question flag (?)\n", "\n", "
\n", "\n", "
\n", "
\n", "reading\n", "
\n", "
str
\n", "\n", " reading of a sign\n", "\n", "
\n", "\n", "
\n", "
\n", "readingr\n", "
\n", "
str
\n", "\n", " reading of a sign using non-ascii characters\n", "\n", "
\n", "\n", "
\n", "
\n", "readingu\n", "
\n", "
str
\n", "\n", " reading of a sign using cuneiform unicode characters\n", "\n", "
\n", "\n", "
\n", "
\n", "remarkable\n", "
\n", "
int
\n", "\n", " whether a sign is remarkable (!)\n", "\n", "
\n", "\n", "
\n", "
\n", "remarks\n", "
\n", "
str
\n", "\n", " # comment to line\n", "\n", "
\n", "\n", "
\n", "
\n", "repeat\n", "
\n", "
int
\n", "\n", " repeat of a numeral; the value n (unknown) is represented as -1\n", "\n", "
\n", "\n", "
\n", "
\n", "srcLn\n", "
\n", "
str
\n", "\n", " full line in source file\n", "\n", "
\n", "\n", "
\n", "
\n", "srcLnNum\n", "
\n", "
int
\n", "\n", " line number in source file\n", "\n", "
\n", "\n", "
\n", "
\n", "srcfile\n", "
\n", "
str
\n", "\n", " source file name of a document\n", "\n", "
\n", "\n", "
\n", "
\n", "subgenre\n", "
\n", "
str
\n", "\n", " genre from metadata field \"Sub-genre\"\n", "\n", "
\n", "\n", "
\n", "
\n", "supplied\n", "
\n", "
int
\n", "\n", " whether a sign is supplied - between angle brackets < >\n", "\n", "
\n", "\n", "
\n", "
\n", "sym\n", "
\n", "
str
\n", "\n", " essential part of a sign or of a word\n", "\n", "
\n", "\n", "
\n", "
\n", "symr\n", "
\n", "
str
\n", "\n", " essential part of a sign or of a word using non-ascii characters\n", "\n", "
\n", "\n", "
\n", "
\n", "symu\n", "
\n", "
str
\n", "\n", " essential part of a sign or of a word using cuneiform unicode characters\n", "\n", "
\n", "\n", "
\n", "
\n", "trans\n", "
\n", "
int
\n", "\n", " whether a line has a translation\n", "\n", "
\n", "\n", "
\n", "
\n", "transcriber\n", "
\n", "
str
\n", "\n", " person who did the encoding into ATF from metadata field \"ATF source\"\n", "\n", "
\n", "\n", "
\n", "
\n", "translation@ll\n", "
\n", "
str
\n", "\n", " translation of line in language en = English\n", "\n", "
\n", "\n", "
\n", "
\n", "type\n", "
\n", "
str
\n", "\n", " name of a type of cluster or kind of sign\n", "\n", "
\n", "\n", "
\n", "
\n", "uncertain\n", "
\n", "
int
\n", "\n", " whether a sign is uncertain - between brackets ( )\n", "\n", "
\n", "\n", "
\n", "
\n", "volume\n", "
\n", "
int
\n", "\n", " volume of a document within a collection\n", "\n", "
\n", "\n", "
\n", "
\n", "oslots\n", "
\n", "
none
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "\n", " Settings:
specified
  1. apiVersion: 3
  2. appName: Nino-cunei/oldbabylonian
  3. appPath:/Users/me/text-fabric-data/github/Nino-cunei/oldbabylonian/app
  4. commit: 02ef17b79197de5431aaec71a8244b7551d77cbc
  5. css:.pnum {
    font-family: sans-serif;
    font-size: small;
    font-weight: bold;
    color: #444444;
    }
    .op {
    padding: 0.5em 0.1em 0.1em 0.1em;
    margin: 0.8em 0.1em 0.1em 0.1em;
    font-family: monospace;
    font-size: x-large;
    font-weight: bold;
    }
    .period {
    font-family: monospace;
    font-size: medium;
    font-weight: bold;
    color: #0000bb;
    }
    .comment {
    color: #7777dd;
    font-family: monospace;
    font-size: small;
    }
    .operator {
    color: #ff77ff;
    font-size: large;
    }
    /* LANGUAGE: superscript and subscript */

    /* cluster */
    .det {
    vertical-align: super;
    }
    /* cluster */
    .langalt {
    vertical-align: sub;
    }
    /* REDACTIONAL: line over or under */

    /* flag */
    .collated {
    font-weight: bold;
    text-decoration: underline;
    }
    /* cluster */
    .excised {
    color: #dd0000;
    text-decoration: line-through;
    }
    /* cluster */
    .supplied {
    color: #0000ff;
    text-decoration: overline;
    }
    /* flag */
    .remarkable {
    font-weight: bold;
    text-decoration: overline;
    }

    /* UNSURE: italic*/

    /* cluster */
    .uncertain {
    font-style: italic
    }
    /* flag */
    .question {
    font-weight: bold;
    font-style: italic
    }

    /* BROKEN: text-shadow */

    /* cluster */
    .missing {
    color: #999999;
    text-shadow: #bbbbbb 1px 1px;
    }
    /* flag */
    .damage {
    font-weight: bold;
    color: #999999;
    text-shadow: #bbbbbb 1px 1px;
    }
    .empty {
    color: #ff0000;
    }

  6. dataDisplay:
    • showVerseInTuple: True
    • textFormats:
      • layout-orig-rich:
        • method: layoutRich
        • style: trans
      • layout-orig-unicode:
        • method: layoutUnicode
        • style: orig
      • text-orig-full: {style: source}
      • text-orig-plain: {style: trans}
      • text-orig-rich: {style: trans}
      • text-orig-unicode: {style: orig}
  7. docs:
    • charText: mapping from readings to UNICODE
    • charUrl:https://nbviewer.jupyter.org/github/Nino-cunei/tfFromAtf/blob/master/programs/mapReadings.ipynb
    • docPage: about
    • featureBase:https://github.com/Nino-cunei/tfFromAtf/blob/master/docs/transcription{docExt}
    • featurePage: ''
  8. interfaceDefaults: {lineNumbers: 0}
  9. isCompatible: True
  10. local: local
  11. localDir:/Users/me/text-fabric-data/github/Nino-cunei/oldbabylonian/_temp
  12. provenanceSpec:
    • corpus: Old Babylonian Letters 1900-1600: Cuneiform tablets
    • doi: 10.5281/zenodo.2579207
    • org: Nino-cunei
    • relative: /tf
    • repo: oldbabylonian
    • version: 1.0.6
    • webBase: https://cdli.ucla.edu
    • webHint: Show this document on CDLI
    • webUrl:{webBase}/search/search_results.php?SearchMode=Text&ObjectID=<1>
  13. release: no value
  14. typeDisplay:
    • cluster:
      • label: {type}
      • stretch: 0
    • document:
      • featuresBare: collection volume docnumber docnote
      • lineNumber: srcLnNum
    • face:
      • featuresBare: object
      • lineNumber: srcLnNum
    • line:
      • features: remarks translation@en
      • lineNumber: srcLnNum
    • sign:
    • \n", " features:\n", " collated remarkable question damage det uncertain missing excised supplied langalt comment remarks repeat fraction operator grapheme\n", "
    • word:
      • base: True
      • label: True
      • wrap: 0
  15. writing: akk
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
TF API: names N F E L T S C TF Fs Fall Es Eall Cs Call directly usable
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A = use(\"Nino-cunei/oldbabylonian\", hoist=globals())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see which features have been loaded, and if you click on a feature name, you find its documentation.\n", "If you hover over a name, you see where the feature is located on your system." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## API\n", "\n", "The result of the incantation is that we have a bunch of special variables at our disposal\n", "that give us access to the text and data of the corpus.\n", "\n", "At this point it is helpful to throw a quick glance at the text-fabric API documentation\n", "(see the links under **API Members** above).\n", "\n", "The most essential thing for now is that we can use `F` to access the data in the features\n", "we've loaded.\n", "But there is more, such as `N`, which helps us to walk over the text, as we see in a minute.\n", "\n", "The **API members** above show you exactly which new names have been inserted in your namespace.\n", "If you click on these names, you go to the API documentation for them." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Search\n", "Text-Fabric contains a flexible search engine, that does not only work for the data,\n", "of this corpus, but also for other corpora and data that you add to corpora.\n", "\n", "**Search is the quickest way to come up-to-speed with your data, without too much programming.**\n", "\n", "Jump to the dedicated [search](search.ipynb) search tutorial first, to whet your appetite.\n", "\n", "The real power of search lies in the fact that it is integrated in a programming environment.\n", "You can use programming to:\n", "\n", "* compose dynamic queries\n", "* process query results\n", "\n", "Therefore, the rest of this tutorial is still important when you want to tap that power.\n", "If you continue here, you learn all the basics of data-navigation with Text-Fabric." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Counting\n", "\n", "In order to get acquainted with the data, we start with the simple task of counting.\n", "\n", "## Count all nodes\n", "We use the\n", "[`N.walk()` generator](https://annotation.github.io/text-fabric/tf/core/nodes.html#tf.core.nodes.Nodes.walk)\n", "to walk through the nodes.\n", "\n", "We compared the TF data to a gigantic spreadsheet, where the rows correspond to the signs.\n", "In Text-Fabric, we call the rows `slots`, because they are the textual positions that can be filled with signs.\n", "\n", "We also mentioned that there are also other textual objects.\n", "They are the clusters, lines, faces and documents.\n", "They also correspond to rows in the big spreadsheet.\n", "\n", "In Text-Fabric we call all these rows *nodes*, and the `N()` generator\n", "carries us through those nodes in the textual order.\n", "\n", "Just one extra thing: the `info` statements generate timed messages.\n", "If you use them instead of `print` you'll get a sense of the amount of time that\n", "the various processing steps typically need." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:43.894153Z", "start_time": "2018-05-18T09:17:43.597128Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s Counting nodes ...\n", " 0.03s 334667 nodes\n" ] } ], "source": [ "A.indent(reset=True)\n", "A.info(\"Counting nodes ...\")\n", "\n", "i = 0\n", "for n in N.walk():\n", " i += 1\n", "\n", "A.info(\"{} nodes\".format(i))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here you see it: over 300,000 nodes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What are those nodes?\n", "Every node has a type, like sign, or line, face.\n", "But what exactly are they?\n", "\n", "Text-Fabric has two special features, `otype` and `oslots`, that must occur in every Text-Fabric data set.\n", "`otype` tells you for each node its type, and you can ask for the number of `slot`s in the text.\n", "\n", "Here we go!" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:47.820323Z", "start_time": "2018-05-18T09:17:47.812328Z" } }, "outputs": [ { "data": { "text/plain": [ "'sign'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.otype.slotType" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:48.549430Z", "start_time": "2018-05-18T09:17:48.543371Z" } }, "outputs": [ { "data": { "text/plain": [ "203219" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.otype.maxSlot" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:49.251302Z", "start_time": "2018-05-18T09:17:49.244467Z" } }, "outputs": [ { "data": { "text/plain": [ "334667" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.otype.maxNode" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:49.922863Z", "start_time": "2018-05-18T09:17:49.916078Z" } }, "outputs": [ { "data": { "text/plain": [ "('document', 'face', 'line', 'word', 'cluster', 'sign')" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.otype.all" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:51.782779Z", "start_time": "2018-05-18T09:17:51.774167Z" } }, "outputs": [ { "data": { "text/plain": [ "(('document', 158.14708171206226, 226669, 227953),\n", " ('face', 71.70748059280169, 227954, 230787),\n", " ('line', 7.423525114155251, 230788, 258162),\n", " ('word', 2.6436180641788116, 258163, 334667),\n", " ('cluster', 1.782122905027933, 203220, 226668),\n", " ('sign', 1, 1, 203219))" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "C.levels.data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is interesting: above you see all the textual objects, with the average size of their objects,\n", "the node where they start, and the node where they end." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Count individual object types\n", "This is an intuitive way to count the number of nodes in each type.\n", "Note in passing, how we use the `indent` in conjunction with `info` to produce neat timed\n", "and indented progress messages." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:17:57.806821Z", "start_time": "2018-05-18T09:17:57.558523Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s counting objects ...\n", " | 0.00s 1285 documents\n", " | 0.00s 2834 faces\n", " | 0.00s 27375 lines\n", " | 0.01s 76505 words\n", " | 0.00s 23449 clusters\n", " | 0.02s 203219 signs\n", " 0.04s Done\n" ] } ], "source": [ "A.indent(reset=True)\n", "A.info(\"counting objects ...\")\n", "\n", "for otype in F.otype.all:\n", " i = 0\n", "\n", " A.indent(level=1, reset=True)\n", "\n", " for n in F.otype.s(otype):\n", " i += 1\n", "\n", " A.info(\"{:>7} {}s\".format(i, otype))\n", "\n", "A.indent(level=0)\n", "A.info(\"Done\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Viewing textual objects\n", "\n", "You can use the A API (the extra power) to display cuneiform text.\n", "\n", "See the [display](display.ipynb) tutorial." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Feature statistics\n", "\n", "`F`\n", "gives access to all features.\n", "Every feature has a method\n", "`freqList()`\n", "to generate a frequency list of its values, higher frequencies first.\n", "Here are the repeats of numerals (the `-1` comes from a `n(rrr)`:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:18.039544Z", "start_time": "2018-05-18T09:18:17.784073Z" } }, "outputs": [ { "data": { "text/plain": [ "((1, 877),\n", " (2, 398),\n", " (5, 246),\n", " (3, 239),\n", " (4, 152),\n", " (6, 67),\n", " (8, 40),\n", " (7, 26),\n", " (9, 15),\n", " (-1, 3))" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.repeat.freqList()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Signs have types and clusters have types. We can count them separately:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:18.039544Z", "start_time": "2018-05-18T09:18:17.784073Z" } }, "outputs": [ { "data": { "text/plain": [ "(('langalt', 7600),\n", " ('missing', 7572),\n", " ('det', 6794),\n", " ('uncertain', 1183),\n", " ('supplied', 231),\n", " ('excised', 69))" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.type.freqList(\"cluster\")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:18.039544Z", "start_time": "2018-05-18T09:18:17.784073Z" } }, "outputs": [ { "data": { "text/plain": [ "(('reading', 188292),\n", " ('unknown', 8761),\n", " ('numeral', 2184),\n", " ('ellipsis', 1617),\n", " ('grapheme', 1272),\n", " ('commentline', 969),\n", " ('complex', 122),\n", " ('comment', 2))" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.type.freqList(\"sign\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, the flags:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:18.039544Z", "start_time": "2018-05-18T09:18:17.784073Z" } }, "outputs": [ { "data": { "text/plain": [ "(('#', 9830),\n", " ('?', 421),\n", " ('#?', 131),\n", " ('!', 91),\n", " ('*', 9),\n", " ('?#', 7),\n", " ('#!', 5),\n", " ('!*', 2),\n", " ('#*', 1),\n", " ('?!*', 1))" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.flags.freqList()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Word matters\n", "\n", "## Top 20 frequent words\n", "\n", "We represent words by their essential symbols, collected in the feature *sym* (which also exists for signs)." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 7089 x\n", " 4071 a-na\n", " 2517 u3\n", " 2361 sza\n", " 1645 um-ma\n", " 1440 i-na\n", " 1365 ...\n", " 1140 qi2-bi2-ma\n", " 1075 la\n", " 796 u2-ul\n", " 776 d⁼utu\n", " 626 d⁼marduk\n", " 585 ku3-babbar\n", " 551 ki-ma\n", " 540 asz-szum\n", " 407 hi-a\n", " 388 lu\n", " 381 1(disz)\n", " 363 ki-a-am\n", " 341 szum-ma\n" ] } ], "source": [ "for (w, amount) in F.sym.freqList(\"word\")[0:20]:\n", " print(f\"{amount:>5} {w}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Word distribution\n", "\n", "Let's do a bit more fancy word stuff.\n", "\n", "### Hapaxes\n", "\n", "A hapax can be found by picking the words with frequency 1\n", "\n", "We print 20 hapaxes." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\"...-BU-szu\"\n", "\"...-BU-um\"\n", "\"...-DI\"\n", "\"...-IG-ti-sza\"\n", "\"...-SZI\"\n", "\"...-ZU\"\n", "\"...-a-tim\"\n", "\"...-ab-ba-lam\"\n", "\"...-am-ma\"\n", "\"...-an\"\n", "\"...-ar\"\n", "\"...-ar-ra\"\n", "\"...-ba-lam\"\n", "\"...-da-an-ni\"\n", "\"...-d⁼en-lil2\"\n", "\"...-d⁼la-ga-ma-al\"\n", "\"...-ha-ar\"\n", "\"...-hu\"\n", "\"...-im\"\n", "\"...-ir\"\n" ] } ], "source": [ "for w in [w for (w, amount) in F.sym.freqList(\"word\") if amount == 1][0:20]:\n", " print(f'\"{w}\"')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Small occurrence base\n", "\n", "The occurrence base of a word are the documents in which occurs.\n", "\n", "We compute the occurrence base of each word." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "occurrenceBase = collections.defaultdict(set)\n", "\n", "for w in F.otype.s(\"word\"):\n", " pNum = T.sectionFromNode(w)[0]\n", " occurrenceBase[F.sym.v(w)].add(pNum)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An overview of how many words have how big occurrence bases:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "base size 1 : 11957 words\n", "base size 2 : 1817 words\n", "base size 3 : 745 words\n", "base size 4 : 367 words\n", "base size 5 : 229 words\n", "base size 6 : 150 words\n", "base size 7 : 128 words\n", "base size 9 : 75 words\n", "base size 8 : 74 words\n", "base size 10 : 64 words\n", "...\n", "base size 459 : 1 words\n", "base size 624 : 1 words\n", "base size 626 : 1 words\n", "base size 649 : 1 words\n", "base size 736 : 1 words\n", "base size 967 : 1 words\n", "base size 1031 : 1 words\n", "base size 1119 : 1 words\n", "base size 1169 : 1 words\n", "base size 1255 : 1 words\n" ] } ], "source": [ "occurrenceSize = collections.Counter()\n", "\n", "for (w, pNums) in occurrenceBase.items():\n", " occurrenceSize[len(pNums)] += 1\n", "\n", "occurrenceSize = sorted(\n", " occurrenceSize.items(),\n", " key=lambda x: (-x[1], x[0]),\n", ")\n", "\n", "for (size, amount) in occurrenceSize[0:10]:\n", " print(f\"base size {size:>4} : {amount:>5} words\")\n", "print(\"...\")\n", "for (size, amount) in occurrenceSize[-10:]:\n", " print(f\"base size {size:>4} : {amount:>5} words\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's give the predicate *private* to those words whose occurrence base is a single document." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "11957" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "privates = {w for (w, base) in occurrenceBase.items() if len(base) == 1}\n", "len(privates)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Peculiarity of documents\n", "\n", "As a final exercise with words, lets make a list of all documents, and show their\n", "\n", "* total number of words\n", "* number of private words\n", "* the percentage of private words: a measure of the peculiarity of the document" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:52.143337Z", "start_time": "2018-05-18T09:18:52.130385Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found 7 empty documents\n", "Found 30 ordinary documents (i.e. without private words)\n" ] } ], "source": [ "docList = []\n", "\n", "empty = set()\n", "ordinary = set()\n", "\n", "for d in F.otype.s(\"document\"):\n", " pNum = T.documentName(d)\n", " words = {F.sym.v(w) for w in L.d(d, otype=\"word\")}\n", " a = len(words)\n", " if not a:\n", " empty.add(pNum)\n", " continue\n", " o = len({w for w in words if w in privates})\n", " if not o:\n", " ordinary.add(pNum)\n", " continue\n", " p = 100 * o / a\n", " docList.append((pNum, a, o, p))\n", "\n", "docList = sorted(docList, key=lambda e: (-e[3], -e[1], e[0]))\n", "\n", "print(f\"Found {len(empty):>4} empty documents\")\n", "print(f\"Found {len(ordinary):>4} ordinary documents (i.e. without private words)\")" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:52.143337Z", "start_time": "2018-05-18T09:18:52.130385Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "document #all #own %own\n", "-----------------------------------\n", "P292935 20 11 55.0%\n", "P510702 39 21 53.8%\n", "P386445 40 20 50.0%\n", "P510808 28 14 50.0%\n", "P313381 24 12 50.0%\n", "P510849 16 8 50.0%\n", "P510657 4 2 50.0%\n", "P292931 48 23 47.9%\n", "P305788 17 8 47.1%\n", "P313326 45 21 46.7%\n", "P292992 28 13 46.4%\n", "P292810 26 12 46.2%\n", "P305774 13 6 46.2%\n", "P510641 13 6 46.2%\n", "P292984 24 11 45.8%\n", "P481778 24 11 45.8%\n", "P373043 104 47 45.2%\n", "P491917 29 13 44.8%\n", "P355749 226 100 44.2%\n", "P365938 25 11 44.0%\n", "...\n", "P510540 21 1 4.8%\n", "P372962 22 1 4.5%\n", "P275094 24 1 4.2%\n", "P372895 24 1 4.2%\n", "P385995 24 1 4.2%\n", "P510732 24 1 4.2%\n", "P386005 27 1 3.7%\n", "P510730 27 1 3.7%\n", "P372914 28 1 3.6%\n", "P372900 30 1 3.3%\n", "P372929 30 1 3.3%\n", "P510528 30 1 3.3%\n", "P413618 31 1 3.2%\n", "P510661 64 2 3.1%\n", "P510535 32 1 3.1%\n", "P365118 34 1 2.9%\n", "P510663 34 1 2.9%\n", "P510817 36 1 2.8%\n", "P275114 40 1 2.5%\n", "P372420 61 1 1.6%\n" ] } ], "source": [ "print(\n", " \"{:<20}{:>5}{:>5}{:>5}\\n{}\".format(\n", " \"document\",\n", " \"#all\",\n", " \"#own\",\n", " \"%own\",\n", " \"-\" * 35,\n", " )\n", ")\n", "\n", "for x in docList[0:20]:\n", " print(\"{:<20} {:>4} {:>4} {:>4.1f}%\".format(*x))\n", "print(\"...\")\n", "for x in docList[-20:]:\n", " print(\"{:<20} {:>4} {:>4} {:>4.1f}%\".format(*x))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Locality API\n", "We travel upwards and downwards, forwards and backwards through the nodes.\n", "The Locality-API (`L`) provides functions: `u()` for going up, and `d()` for going down,\n", "`n()` for going to next nodes and `p()` for going to previous nodes.\n", "\n", "These directions are indirect notions: nodes are just numbers, but by means of the\n", "`oslots` feature they are linked to slots. One node *contains* an other node, if the one is linked to a set of slots that contains the set of slots that the other is linked to.\n", "And one if next or previous to an other, if its slots follow or precede the slots of the other one.\n", "\n", "`L.u(node)` **Up** is going to nodes that embed `node`.\n", "\n", "`L.d(node)` **Down** is the opposite direction, to those that are contained in `node`.\n", "\n", "`L.n(node)` **Next** are the next *adjacent* nodes, i.e. nodes whose first slot comes immediately after the last slot of `node`.\n", "\n", "`L.p(node)` **Previous** are the previous *adjacent* nodes, i.e. nodes whose last slot comes immediately before the first slot of `node`.\n", "\n", "All these functions yield nodes of all possible node types.\n", "By passing an optional parameter, you can restrict the results to nodes of that type.\n", "\n", "The result are ordered according to the order of things in the text.\n", "\n", "The functions return always a tuple, even if there is just one node in the result.\n", "\n", "## Going up\n", "We go from the first word to the document it contains.\n", "Note the `[0]` at the end. You expect one document, yet `L` returns a tuple.\n", "To get the only element of that tuple, you need to do that `[0]`.\n", "\n", "If you are like me, you keep forgetting it, and that will lead to weird error messages later on." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:55.410034Z", "start_time": "2018-05-18T09:18:55.404051Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "226669\n" ] } ], "source": [ "firstDoc = L.u(1, otype=\"document\")[0]\n", "print(firstDoc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And let's see all the containing objects of sign 3:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:56.772513Z", "start_time": "2018-05-18T09:18:56.766324Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sign 3 is contained in document 226669\n", "sign 3 is contained in face 227954\n", "sign 3 is contained in line 230788\n", "sign 3 is contained in word 258164\n", "sign 3 is contained in cluster 203222\n" ] } ], "source": [ "s = 3\n", "for otype in F.otype.all:\n", " if otype == F.otype.slotType:\n", " continue\n", " up = L.u(s, otype=otype)\n", " upNode = \"x\" if len(up) == 0 else up[0]\n", " print(\"sign {} is contained in {} {}\".format(s, otype, upNode))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Going next\n", "Let's go to the next nodes of the first document." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:18:58.821681Z", "start_time": "2018-05-18T09:18:58.814893Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 348: sign first slot=348 , last slot=348 \n", " 258314: word first slot=348 , last slot=349 \n", " 230824: line first slot=348 , last slot=355 \n", " 227956: face first slot=348 , last slot=482 \n", " 226670: document first slot=348 , last slot=500 \n" ] } ], "source": [ "afterFirstDoc = L.n(firstDoc)\n", "for n in afterFirstDoc:\n", " print(\n", " \"{:>7}: {:<13} first slot={:<6}, last slot={:<6}\".format(\n", " n,\n", " F.otype.v(n),\n", " E.oslots.s(n)[0],\n", " E.oslots.s(n)[-1],\n", " )\n", " )\n", "secondDoc = L.n(firstDoc, otype=\"document\")[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Going previous\n", "\n", "And let's see what is right before the second document." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:00.163973Z", "start_time": "2018-05-18T09:19:00.154857Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 226669: document first slot=1 , last slot=347 \n", " 227955: face first slot=164 , last slot=347 \n", " 230823: line first slot=330 , last slot=347 \n", " 203293: cluster first slot=345 , last slot=347 \n", " 258313: word first slot=347 , last slot=347 \n", " 347: sign first slot=347 , last slot=347 \n" ] } ], "source": [ "for n in L.p(secondDoc):\n", " print(\n", " \"{:>7}: {:<13} first slot={:<6}, last slot={:<6}\".format(\n", " n,\n", " F.otype.v(n),\n", " E.oslots.s(n)[0],\n", " E.oslots.s(n)[-1],\n", " )\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Going down" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We go to the faces of the first document, and just count them." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:02.530705Z", "start_time": "2018-05-18T09:19:02.475279Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2\n" ] } ], "source": [ "faces = L.d(firstDoc, otype=\"face\")\n", "print(len(faces))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The first line\n", "We pick two nodes and explore what is above and below them:\n", "the first line and the first word." ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:04.024679Z", "start_time": "2018-05-18T09:19:03.995207Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Node 258163\n", " | UP\n", " | | 203220 cluster\n", " | | 230788 line\n", " | | 227954 face\n", " | | 226669 document\n", " | DOWN\n", " | | 203220 cluster\n", " | | 1 sign\n", " | | 2 sign\n", "Node 230788\n", " | UP\n", " | | 227954 face\n", " | | 226669 document\n", " | DOWN\n", " | | 258163 word\n", " | | 203220 cluster\n", " | | 1 sign\n", " | | 2 sign\n", " | | 258164 word\n", " | | 203221 cluster\n", " | | 203222 cluster\n", " | | 3 sign\n", " | | 4 sign\n", " | | 5 sign\n", " | | 203223 cluster\n", " | | 6 sign\n", " | | 7 sign\n", "Done\n" ] } ], "source": [ "for n in [\n", " F.otype.s(\"word\")[0],\n", " F.otype.s(\"line\")[0],\n", "]:\n", " A.indent(level=0)\n", " A.info(\"Node {}\".format(n), tm=False)\n", " A.indent(level=1)\n", " A.info(\"UP\", tm=False)\n", " A.indent(level=2)\n", " A.info(\"\\n\".join([\"{:<15} {}\".format(u, F.otype.v(u)) for u in L.u(n)]), tm=False)\n", " A.indent(level=1)\n", " A.info(\"DOWN\", tm=False)\n", " A.indent(level=2)\n", " A.info(\"\\n\".join([\"{:<15} {}\".format(u, F.otype.v(u)) for u in L.d(n)]), tm=False)\n", "A.indent(level=0)\n", "A.info(\"Done\", tm=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Text API\n", "\n", "So far, we have mainly seen nodes and their numbers, and the names of node types.\n", "You would almost forget that we are dealing with text.\n", "So let's try to see some text.\n", "\n", "In the same way as `F` gives access to feature data,\n", "`T` gives access to the text.\n", "That is also feature data, but you can tell Text-Fabric which features are specifically\n", "carrying the text, and in return Text-Fabric offers you\n", "a Text API: `T`.\n", "\n", "## Formats\n", "Cuneiform text can be represented in a number of ways:\n", "\n", "* original ATF, with bracketings and flags\n", "* essential symbols: readings and graphemes, repeats and fractions (of numerals), no flags, no clusterings\n", "* unicode symbols\n", "\n", "If you wonder where the information about text formats is stored:\n", "not in the program text-fabric, but in the data set.\n", "It has a feature `otext`, which specifies the formats and which features\n", "must be used to produce them. `otext` is the third special feature in a TF data set,\n", "next to `otype` and `oslots`.\n", "It is an optional feature.\n", "If it is absent, there will be no `T` API.\n", "\n", "Here is a list of all available formats in this data set." ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:05.606582Z", "start_time": "2018-05-18T09:19:05.593486Z" } }, "outputs": [ { "data": { "text/plain": [ "['layout-orig-rich',\n", " 'layout-orig-unicode',\n", " 'text-orig-full',\n", " 'text-orig-plain',\n", " 'text-orig-rich',\n", " 'text-orig-unicode']" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sorted(T.formats)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the formats\n", "\n", "The ` T.text()` function is central to get text representations of nodes. Its most basic usage is\n", "\n", "```python\n", "T.text(nodes, fmt=fmt)\n", "```\n", "where `nodes` is a list or iterable of nodes, usually word nodes, and `fmt` is the name of a format.\n", "If you leave out `fmt`, the default `text-orig-full` is chosen.\n", "\n", "The result is the text in that format for all nodes specified:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'a-na d⁼suen-i-din-namqi2-bi2-maum-'" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "T.text([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], fmt=\"text-orig-plain\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is also another usage of this function:\n", "\n", "```python\n", "T.text(node, fmt=fmt)\n", "```\n", "\n", "where `node` is a single node.\n", "In this case, the default format is `ntype-orig-full` where `ntype` is the type of `node`.\n", "\n", "If the format is defined in the corpus, it will be used. Otherwise, the word nodes contained in `node` will be looked up\n", "and represented with the default format `text-orig-full`.\n", "\n", "In this way we can sensibly represent a lot of different nodes, such as documents, faces, lines, clusters, words and signs.\n", "\n", "We compose a set of example nodes and run `T.text` on them:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[1, 258163, 203220, 230788, 227954, 226669]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exampleNodes = [\n", " F.otype.s(\"sign\")[0],\n", " F.otype.s(\"word\")[0],\n", " F.otype.s(\"cluster\")[0],\n", " F.otype.s(\"line\")[0],\n", " F.otype.s(\"face\")[0],\n", " F.otype.s(\"document\")[0],\n", "]\n", "exampleNodes" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This is sign 1:\n", "[a-\n", "\n", "This is word 258163:\n", "[a-na] \n", "\n", "This is cluster 203220:\n", "[a-na] \n", "\n", "This is line 230788:\n", "[a-na] _{d}suen_-i-[din-nam]\n", "\n", "This is face 227954:\n", "[a-na] _{d}suen_-i-[din-nam]qi2-bi2-[ma]um-ma _{d}en-lil2_-sza-du-u2-ni-ma_{d}utu_ u3 _{d}[marduk]_ a-na da-ri-a-[tim]li-ba-al-li-t,u2-u2-ka{disz}sze-ep-_{d}suen a2-gal2 [dumu] um-mi-a-mesz_ki-a-am u2-lam-mi-da-an-ni um-[ma] szu-u2-[ma]{disz}sa-am-su-ba-ah-li sza-pi2-ir ma-[tim]2(esze3) _a-sza3_ s,i-[bi]-it {disz}[ku]-un-zu-lum _sza3-gud__a-sza3 a-gar3_ na-ag-[ma-lum] _uru_ x x x{ki}sza _{d}utu_-ha-zi-[ir] isz-tu _mu 7(disz) kam_ id-di-nu-szumu3 i-na _uru_ x-szum{ki} sza-ak-nu id-di-a-am-ma2(esze3) _a-sza3 szuku_ i-li-ib-bu s,i-bi-it _nagar-mesz__a-sza3 a-gar3 uru_ ra-bu-um x [...]x x x x x x [...]$ rest broken\n", "\n", "This is document 226669:\n", "[a-na] _{d}suen_-i-[din-nam]qi2-bi2-[ma]um-ma _{d}en-lil2_-sza-du-u2-ni-ma_{d}utu_ u3 _{d}[marduk]_ a-na da-ri-a-[tim]li-ba-al-li-t,u2-u2-ka{disz}sze-ep-_{d}suen a2-gal2 [dumu] um-mi-a-mesz_ki-a-am u2-lam-mi-da-an-ni um-[ma] szu-u2-[ma]{disz}sa-am-su-ba-ah-li sza-pi2-ir ma-[tim]2(esze3) _a-sza3_ s,i-[bi]-it {disz}[ku]-un-zu-lum _sza3-gud__a-sza3 a-gar3_ na-ag-[ma-lum] _uru_ x x x{ki}sza _{d}utu_-ha-zi-[ir] isz-tu _mu 7(disz) kam_ id-di-nu-szumu3 i-na _uru_ x-szum{ki} sza-ak-nu id-di-a-am-ma2(esze3) _a-sza3 szuku_ i-li-ib-bu s,i-bi-it _nagar-mesz__a-sza3 a-gar3 uru_ ra-bu-um x [...]x x x x x x [...]$ rest broken$ beginning broken[x x] x x [...][x x] x [...][x x] x s,i-bi-it _gir3-se3#-ga#_[x x] x x x-ir ub-lamin-na-me-er-max _[a-sza3_ s,i]-bi-it ku-un-zu-lum_a-[sza3 a-gar3_ na-ag]-ma-lum _uru gan2_ x x{ki}a-[na] sa-[am-su]-ba-ah-[la] x x li ig bumi-im-ma _a-sza3_ s,i-bi-[it] _nagar-mesz_u2-ul na-di-isz-szuma-na ki-ma i-[na] _dub e2-gal_-limsza _{d}utu_-ha-zi-ir ub-lam in-na-am-ruasz-t,u3-ra-am _dub_ [usz]-ta-bi-la-kuma-wa-tim sza ma-ah-hi-rum u2-lam-ma-du-u2-maasz-szum _e2-gal_-lim la lum-mu-[di ...]_dub_-pi2 u2-sza-ab-ba-x [...]x ti u2-ul a-ga-am-ma-[x ...]u2-ul a-sza-ap-pa-[x ...][at-ta] ki-ma ti-du-u2 wa-ar-ka-az-zu pu-ru-us [x x ...]\n", "\n" ] } ], "source": [ "for n in exampleNodes:\n", " print(f\"This is {F.otype.v(n)} {n}:\")\n", " print(T.text(n))\n", " print(\"\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the formats\n", "Now let's use those formats to print out the first line in this corpus.\n", "\n", "Note that only the formats starting with `text-` are usable for this.\n", "\n", "For the `layout-` formats, see [display](display.ipynb)." ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:10.077589Z", "start_time": "2018-05-18T09:19:10.070503Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "text-orig-full:\n", "\t[a-na] _{d}suen_-i-[din-nam]qi2-bi2-[ma]um-\n", "text-orig-plain:\n", "\ta-na d⁼suen-i-din-namqi2-bi2-maum-\n", "text-orig-rich:\n", "\ta-na d⁼suen-i-din-namqi₂-bi₂-maum-\n", "text-orig-unicode:\n", "\t𒀀𒈾 𒀭𒂗𒍪𒄿𒁷𒉆𒆠𒉈𒈠𒌝\n" ] } ], "source": [ "for fmt in sorted(T.formats):\n", " if fmt.startswith(\"text-\"):\n", " print(\"{}:\\n\\t{}\".format(fmt, T.text(range(1, 12), fmt=fmt)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we do not specify a format, the **default** format is used (`text-orig-full`)." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:13.490426Z", "start_time": "2018-05-18T09:19:13.486053Z" } }, "outputs": [ { "data": { "text/plain": [ "'[a-na] _{d}suen_-i-[din-nam]qi2-bi2-[ma]um-'" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "T.text(range(1, 12))" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'[a-na] _{d}suen_-i-[din-nam]'" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "firstLine = F.otype.s(\"line\")[0]\n", "T.text(firstLine)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'𒀀𒈾 𒀭𒂗𒍪𒄿𒁷𒉆'" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "T.text(firstLine, fmt=\"text-orig-unicode\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The important things to remember are:\n", "\n", "* you can supply a list of slot nodes and get them represented in all formats\n", "* you can get non-slot nodes `n` in default format by `T.text(n)`\n", "* you can get non-slot nodes `n` in other formats by `T.text(n, fmt=fmt, descend=True)`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Whole text in all formats in just 2 seconds\n", "Part of the pleasure of working with computers is that they can crunch massive amounts of data.\n", "The text of the Old Babylonian Letters is a piece of cake.\n", "\n", "It takes just ten seconds to have that cake and eat it.\n", "In nearly a dozen formats." ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:27.839331Z", "start_time": "2018-05-18T09:19:18.526400Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s writing plain text of all letters in all text formats\n", " 1.70s done 4 formats\n", "text-orig-full\n", "[a-na] _{d}suen_-i-[din-nam]\n", "qi2-bi2-[ma]\n", "um-ma _{d}en-lil2_-sza-du-u2-ni-ma\n", "_{d}utu_ u3 _{d}[marduk]_ a-na da-ri-a-[tim]\n", "li-ba-al-li-t,u2-u2-ka\n", "\n", "text-orig-plain\n", "a-na d⁼suen-i-din-nam\n", "qi2-bi2-ma\n", "um-ma d⁼en-lil2-sza-du-u2-ni-ma\n", "d⁼utu u3 d⁼marduk a-na da-ri-a-tim\n", "li-ba-al-li-t,u2-u2-ka\n", "\n", "text-orig-rich\n", "a-na d⁼suen-i-din-nam\n", "qi₂-bi₂-ma\n", "um-ma d⁼en-lil₂-ša-du-u₂-ni-ma\n", "d⁼utu u₃ d⁼marduk a-na da-ri-a-tim\n", "li-ba-al-li-ṭu₂-u₂-ka\n", "\n", "text-orig-unicode\n", "𒀀𒈾 𒀭𒂗𒍪𒄿𒁷𒉆\n", "𒆠𒉈𒈠\n", "𒌝𒈠 𒀭𒂗𒆤𒊭𒁺𒌑𒉌𒈠\n", "𒀭𒌓 𒅇 𒀭𒀫𒌓 𒀀𒈾 𒁕𒊑𒀀𒁴\n", "𒇷𒁀𒀠𒇷𒌅𒌑𒅗\n", "\n" ] } ], "source": [ "A.indent(reset=True)\n", "A.info(\"writing plain text of all letters in all text formats\")\n", "\n", "text = collections.defaultdict(list)\n", "\n", "for ln in F.otype.s(\"line\"):\n", " for fmt in sorted(T.formats):\n", " if fmt.startswith(\"text-\"):\n", " text[fmt].append(T.text(ln, fmt=fmt, descend=True))\n", "\n", "A.info(\"done {} formats\".format(len(text)))\n", "\n", "for fmt in sorted(text):\n", " print(\"{}\\n{}\\n\".format(fmt, \"\\n\".join(text[fmt][0:5])))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The full plain text\n", "We write all formats to file, in your `Downloads` folder." ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:34.250294Z", "start_time": "2018-05-18T09:19:34.156658Z" } }, "outputs": [], "source": [ "for fmt in T.formats:\n", " if fmt.startswith(\"text-\"):\n", " with open(os.path.expanduser(f\"~/Downloads/{fmt}.txt\"), \"w\") as f:\n", " f.write(\"\\n\".join(text[fmt]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sections\n", "\n", "A section in the letter corpus is a document, a face or a line.\n", "Knowledge of sections is not baked into Text-Fabric.\n", "The config feature `otext.tf` may specify three section levels, and tell\n", "what the corresponding node types and features are.\n", "\n", "From that knowledge it can construct mappings from nodes to sections, e.g. from line\n", "nodes to tuples of the form:\n", "\n", " (p-number, face specifier, line number)\n", "\n", "You can get the section of a node as a tuple of relevant document, face, and line nodes.\n", "Or you can get it as a passage label, a string.\n", "\n", "You can ask for the passage corresponding to the first slot of a node, or the one corresponding to the last slot.\n", "\n", "If you are dealing with document and face nodes, you can ask to fill out the line and face parts as well.\n", "\n", "Here are examples of getting the section that corresponds to a node and vice versa.\n", "\n", "**NB:** `sectionFromNode` always delivers a verse specification, either from the\n", "first slot belonging to that node, or, if `lastSlot`, from the last slot\n", "belonging to that node." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "someNodes = (\n", " F.otype.s(\"sign\")[100000],\n", " F.otype.s(\"word\")[10000],\n", " F.otype.s(\"cluster\")[5000],\n", " F.otype.s(\"line\")[15000],\n", " F.otype.s(\"face\")[1000],\n", " F.otype.s(\"document\")[500],\n", ")" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "ExecuteTime": { "end_time": "2018-05-18T09:19:43.056511Z", "start_time": "2018-05-18T09:19:43.043552Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 100001 sign - P313335 obverse:8 P313335 obverse:8 ((227310, 229370, 244327), (227310, 229370, 244327))\n", " 268163 word - P510665 obverse:9 P510665 obverse:9 ((226821, 228295, 234114), (226821, 228295, 234114))\n", " 208220 cluster - P510766 obverse:9 P510766 obverse:9 ((226925, 228516, 236231), (226925, 228516, 236231))\n", " 245788 line - P313410 obverse:12' P313410 obverse:12' ((227376, 229516, 245788), (227376, 229516, 245788))\n", " 228954 face - P292765 reverse P292765 reverse:12 ((227126, 228954), (227126, 228954, 240157))\n", " 227169 document - P382526 P382526 left:2 ((227169,), (227169, 229057, 241107))\n" ] } ], "source": [ "for n in someNodes:\n", " nType = F.otype.v(n)\n", " d = f\"{n:>7} {nType}\"\n", " first = A.sectionStrFromNode(n)\n", " last = A.sectionStrFromNode(n, lastSlot=True, fillup=True)\n", " tup = (\n", " T.sectionTuple(n),\n", " T.sectionTuple(n, lastSlot=True, fillup=True),\n", " )\n", " print(f\"{d:<16} - {first:<18} {last:<18} {tup}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Clean caches\n", "\n", "Text-Fabric pre-computes data for you, so that it can be loaded faster.\n", "If the original data is updated, Text-Fabric detects it, and will recompute that data.\n", "\n", "But there are cases, when the algorithms of Text-Fabric have changed, without any changes in the data, that you might\n", "want to clear the cache of precomputed results.\n", "\n", "There are two ways to do that:\n", "\n", "* Locate the `.tf` directory of your dataset, and remove all `.tfx` files in it.\n", " This might be a bit awkward to do, because the `.tf` directory is hidden on Unix-like systems.\n", "* Call `TF.clearCache()`, which does exactly the same.\n", "\n", "It is not handy to execute the following cell all the time, that's why I have commented it out.\n", "So if you really want to clear the cache, remove the comment sign below." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "# TF.clearCache()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Next steps\n", "\n", "By now you have an impression how to compute around in the corpus.\n", "While this is still the beginning, I hope you already sense the power of unlimited programmatic access\n", "to all the bits and bytes in the data set.\n", "\n", "Here are a few directions for unleashing that power.\n", "\n", "* **[display](display.ipynb)** become an expert in creating pretty displays of your text structures\n", "* **[search](search.ipynb)** turbo charge your hand-coding with search templates\n", "* **[exportExcel](exportExcel.ipynb)** make tailor-made spreadsheets out of your results\n", "* **[share](share.ipynb)** draw in other people's data and let them use yours\n", "* **[similarLines](similarLines.ipynb)** spot the similarities between lines\n", "\n", "---\n", "\n", "See the [cookbook](cookbook) for recipes for small, concrete tasks.\n", "\n", "CC-BY Dirk Roorda" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.0" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": "block", "toc_window_display": false }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }