{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "You might want to consider the [start](search.ipynb) of this tutorial.\n", "\n", "Short introductions to other TF datasets:\n", "\n", "* [Dead Sea Scrolls](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/dss.ipynb),\n", "* [Old Babylonian Letters](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/oldbabylonian.ipynb),\n", "or the\n", "* [Quran](https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/lorentz2020/quran.ipynb)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Annotation outside TF\n", "\n", "Task:\n", "\n", "* prepare a text file based on TF data.\n", "* annotate the text file by assigning values to pieces of text\n", "* generate TF features based on these annotations\n", "\n", "We use a device in Text-Fabric that has been developed for this kind of round-trip:\n", "the [Recorder](https://annotation.github.io/text-fabric/tf/convert/recorder.html)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Incantation\n", "\n", "The ins and outs of installing Text-Fabric, getting the corpus, and initializing a notebook are\n", "explained in the [start tutorial](start.ipynb)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2018-05-24T10:06:39.818664Z", "start_time": "2018-05-24T10:06:39.796588Z" } }, "outputs": [], "source": [ "from tf.app import use\n", "from tf.convert.recorder import Recorder" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "**Locating corpus resources ...**" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "app: ~/text-fabric-data/github/ETCBC/bhsa/app" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "data: ~/text-fabric-data/github/ETCBC/bhsa/tf/2021" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "data: ~/text-fabric-data/github/ETCBC/phono/tf/2021" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "data: ~/text-fabric-data/github/ETCBC/parallels/tf/2021" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " Text-Fabric: Text-Fabric API 12.0.4, ETCBC/bhsa/app v3, Search Reference
\n", " Data: ETCBC - bhsa 2021, Character table, Feature docs
\n", "
Node types\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "
Name# of nodes# slots/node% coverage
book3910938.21100
chapter929459.19100
lex923046.22100
verse2321318.38100
half_verse451799.44100
sentence637176.70100
sentence_atom645146.61100
clause881314.84100
clause_atom907044.70100
phrase2532031.68100
phrase_atom2675321.59100
subphrase1138501.4238
word4265901.00100
\n", " Sets: no custom sets
\n", " Features:
\n", "
Parallel Passages\n", "
\n", "\n", "
\n", "
\n", "crossref\n", "
\n", "
int
\n", "\n", " 🆗 links between similar passages\n", "\n", "
\n", "\n", "
\n", "
\n", "\n", "
BHSA = Biblia Hebraica Stuttgartensia Amstelodamensis\n", "
\n", "\n", "
\n", "
\n", "book\n", "
\n", "
str
\n", "\n", " ✅ book name in Latin (Genesis; Numeri; Reges1; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "book@ll\n", "
\n", "
str
\n", "\n", " ✅ book name in amharic (ኣማርኛ)\n", "\n", "
\n", "\n", "
\n", "
\n", "chapter\n", "
\n", "
int
\n", "\n", " ✅ chapter number (1; 2; 3; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "code\n", "
\n", "
int
\n", "\n", " ✅ identifier of a clause atom relationship (0; 74; 367; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "det\n", "
\n", "
str
\n", "\n", " ✅ determinedness of phrase(atom) (det; und; NA.)\n", "\n", "
\n", "\n", "
\n", "
\n", "domain\n", "
\n", "
str
\n", "\n", " ✅ text type of clause (? (Unknown); N (narrative); D (discursive); Q (Quotation).)\n", "\n", "
\n", "\n", "
\n", "
\n", "freq_lex\n", "
\n", "
int
\n", "\n", " ✅ frequency of lexemes\n", "\n", "
\n", "\n", "
\n", "
\n", "function\n", "
\n", "
str
\n", "\n", " ✅ syntactic function of phrase (Cmpl; Objc; Pred; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "g_cons\n", "
\n", "
str
\n", "\n", " ✅ word consonantal-transliterated (B R>CJT BR> >LHJM ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "g_cons_utf8\n", "
\n", "
str
\n", "\n", " ✅ word consonantal-Hebrew (ב ראשׁית ברא אלהים)\n", "\n", "
\n", "\n", "
\n", "
\n", "g_lex\n", "
\n", "
str
\n", "\n", " ✅ lexeme pointed-transliterated (B.:- R;>CIJT B.@R@> >:ELOH ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "g_lex_utf8\n", "
\n", "
str
\n", "\n", " ✅ lexeme pointed-Hebrew (בְּ רֵאשִׁית בָּרָא אֱלֹה)\n", "\n", "
\n", "\n", "
\n", "
\n", "g_word\n", "
\n", "
str
\n", "\n", " ✅ word pointed-transliterated (B.:- R;>CI73JT B.@R@74> >:ELOHI92JM)\n", "\n", "
\n", "\n", "
\n", "
\n", "g_word_utf8\n", "
\n", "
str
\n", "\n", " ✅ word pointed-Hebrew (בְּ רֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים)\n", "\n", "
\n", "\n", "
\n", "
\n", "gloss\n", "
\n", "
str
\n", "\n", " 🆗 english translation of lexeme (beginning create god(s))\n", "\n", "
\n", "\n", "
\n", "
\n", "gn\n", "
\n", "
str
\n", "\n", " ✅ grammatical gender (m; f; NA; unknown.)\n", "\n", "
\n", "\n", "
\n", "
\n", "label\n", "
\n", "
str
\n", "\n", " ✅ (half-)verse label (half verses: A; B; C; verses: GEN 01,02)\n", "\n", "
\n", "\n", "
\n", "
\n", "language\n", "
\n", "
str
\n", "\n", " ✅ of word or lexeme (Hebrew; Aramaic.)\n", "\n", "
\n", "\n", "
\n", "
\n", "lex\n", "
\n", "
str
\n", "\n", " ✅ lexeme consonantal-transliterated (B R>CJT/ BR>[ >LHJM/)\n", "\n", "
\n", "\n", "
\n", "
\n", "lex_utf8\n", "
\n", "
str
\n", "\n", " ✅ lexeme consonantal-Hebrew (ב ראשׁית֜ ברא אלהים֜)\n", "\n", "
\n", "\n", "
\n", "
\n", "ls\n", "
\n", "
str
\n", "\n", " ✅ lexical set, subclassification of part-of-speech (card; ques; mult)\n", "\n", "
\n", "\n", "
\n", "
\n", "nametype\n", "
\n", "
str
\n", "\n", " ⚠️ named entity type (pers; mens; gens; topo; ppde.)\n", "\n", "
\n", "\n", "
\n", "
\n", "nme\n", "
\n", "
str
\n", "\n", " ✅ nominal ending consonantal-transliterated (absent; n/a; JM, ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "nu\n", "
\n", "
str
\n", "\n", " ✅ grammatical number (sg; du; pl; NA; unknown.)\n", "\n", "
\n", "\n", "
\n", "
\n", "number\n", "
\n", "
int
\n", "\n", " ✅ sequence number of an object within its context\n", "\n", "
\n", "\n", "
\n", "
\n", "otype\n", "
\n", "
str
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "pargr\n", "
\n", "
str
\n", "\n", " 🆗 hierarchical paragraph number (1; 1.2; 1.2.3.4; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "pdp\n", "
\n", "
str
\n", "\n", " ✅ phrase dependent part-of-speech (art; verb; subs; nmpr, ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "pfm\n", "
\n", "
str
\n", "\n", " ✅ preformative consonantal-transliterated (absent; n/a; J, ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "prs\n", "
\n", "
str
\n", "\n", " ✅ pronominal suffix consonantal-transliterated (absent; n/a; W; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "prs_gn\n", "
\n", "
str
\n", "\n", " ✅ pronominal suffix gender (m; f; NA; unknown.)\n", "\n", "
\n", "\n", "
\n", "
\n", "prs_nu\n", "
\n", "
str
\n", "\n", " ✅ pronominal suffix number (sg; du; pl; NA; unknown.)\n", "\n", "
\n", "\n", "
\n", "
\n", "prs_ps\n", "
\n", "
str
\n", "\n", " ✅ pronominal suffix person (p1; p2; p3; NA; unknown.)\n", "\n", "
\n", "\n", "
\n", "
\n", "ps\n", "
\n", "
str
\n", "\n", " ✅ grammatical person (p1; p2; p3; NA; unknown.)\n", "\n", "
\n", "\n", "
\n", "
\n", "qere\n", "
\n", "
str
\n", "\n", " ✅ word pointed-transliterated masoretic reading correction\n", "\n", "
\n", "\n", "
\n", "
\n", "qere_trailer\n", "
\n", "
str
\n", "\n", " ✅ interword material -pointed-transliterated (Masoretic correction)\n", "\n", "
\n", "\n", "
\n", "
\n", "qere_trailer_utf8\n", "
\n", "
str
\n", "\n", " ✅ interword material -pointed-transliterated (Masoretic correction)\n", "\n", "
\n", "\n", "
\n", "
\n", "qere_utf8\n", "
\n", "
str
\n", "\n", " ✅ word pointed-Hebrew masoretic reading correction\n", "\n", "
\n", "\n", "
\n", "
\n", "rank_lex\n", "
\n", "
int
\n", "\n", " ✅ ranking of lexemes based on freqnuecy\n", "\n", "
\n", "\n", "
\n", "
\n", "rela\n", "
\n", "
str
\n", "\n", " ✅ linguistic relation between clause/(sub)phrase(atom) (ADJ; MOD; ATR; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "sp\n", "
\n", "
str
\n", "\n", " ✅ part-of-speech (art; verb; subs; nmpr, ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "st\n", "
\n", "
str
\n", "\n", " ✅ state of a noun (a (absolute); c (construct); e (emphatic).)\n", "\n", "
\n", "\n", "
\n", "
\n", "tab\n", "
\n", "
int
\n", "\n", " ✅ clause atom: its level in the linguistic embedding\n", "\n", "
\n", "\n", "
\n", "
\n", "trailer\n", "
\n", "
str
\n", "\n", " ✅ interword material pointed-transliterated (& 00 05 00_P ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "trailer_utf8\n", "
\n", "
str
\n", "\n", " ✅ interword material pointed-Hebrew (־ ׃)\n", "\n", "
\n", "\n", "
\n", "
\n", "txt\n", "
\n", "
str
\n", "\n", " ✅ text type of clause and surrounding (repetion of ? N D Q as in feature domain)\n", "\n", "
\n", "\n", "
\n", "
\n", "typ\n", "
\n", "
str
\n", "\n", " ✅ clause/phrase(atom) type (VP; NP; Ellp; Ptcp; WayX)\n", "\n", "
\n", "\n", "
\n", "
\n", "uvf\n", "
\n", "
str
\n", "\n", " ✅ univalent final consonant consonantal-transliterated (absent; N; J; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "vbe\n", "
\n", "
str
\n", "\n", " ✅ verbal ending consonantal-transliterated (n/a; W; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "vbs\n", "
\n", "
str
\n", "\n", " ✅ root formation consonantal-transliterated (absent; n/a; H; ...)\n", "\n", "
\n", "\n", "
\n", "
\n", "verse\n", "
\n", "
int
\n", "\n", " ✅ verse number\n", "\n", "
\n", "\n", "
\n", "
\n", "voc_lex\n", "
\n", "
str
\n", "\n", " ✅ vocalized lexeme pointed-transliterated (B.: R;>CIJT BR> >:ELOHIJM)\n", "\n", "
\n", "\n", "
\n", "
\n", "voc_lex_utf8\n", "
\n", "
str
\n", "\n", " ✅ vocalized lexeme pointed-Hebrew (בְּ רֵאשִׁית ברא אֱלֹהִים)\n", "\n", "
\n", "\n", "
\n", "
\n", "vs\n", "
\n", "
str
\n", "\n", " ✅ verbal stem (qal; piel; hif; apel; pael)\n", "\n", "
\n", "\n", "
\n", "
\n", "vt\n", "
\n", "
str
\n", "\n", " ✅ verbal tense (perf; impv; wayq; infc)\n", "\n", "
\n", "\n", "
\n", "
\n", "mother\n", "
\n", "
none
\n", "\n", " ✅ linguistic dependency between textual objects\n", "\n", "
\n", "\n", "
\n", "
\n", "oslots\n", "
\n", "
none
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "\n", "
Phonetic Transcriptions\n", "
\n", "\n", "
\n", "
\n", "phono\n", "
\n", "
str
\n", "\n", " 🆗 phonological transcription (bᵊ rēšˌîṯ bārˈā ʔᵉlōhˈîm)\n", "\n", "
\n", "\n", "
\n", "
\n", "phono_trailer\n", "
\n", "
str
\n", "\n", " 🆗 interword material in phonological transcription\n", "\n", "
\n", "\n", "
\n", "
\n", "\n", " Settings:
specified
  1. apiVersion: 3
  2. appName: ETCBC/bhsa
  3. appPath: /Users/me/text-fabric-data/github/ETCBC/bhsa/app
  4. commit: gd905e3fb6e80d0fa537600337614adc2af157309
  5. css: ''
  6. dataDisplay:
    • exampleSectionHtml:<code>Genesis 1:1</code> (use <a href=\"https://github.com/{org}/{repo}/blob/master/tf/{version}/book%40en.tf\" target=\"_blank\">English book names</a>)
    • excludedFeatures:
      • g_uvf_utf8
      • g_vbs
      • kq_hybrid
      • languageISO
      • g_nme
      • lex0
      • is_root
      • g_vbs_utf8
      • g_uvf
      • dist
      • root
      • suffix_person
      • g_vbe
      • dist_unit
      • suffix_number
      • distributional_parent
      • kq_hybrid_utf8
      • crossrefSET
      • instruction
      • g_prs
      • lexeme_count
      • rank_occ
      • g_pfm_utf8
      • freq_occ
      • crossrefLCS
      • functional_parent
      • g_pfm
      • g_nme_utf8
      • g_vbe_utf8
      • kind
      • g_prs_utf8
      • suffix_gender
      • mother_object_type
    • noneValues:
      • none
      • unknown
      • no value
      • NA
  7. docs:
    • docBase: {docRoot}/{repo}
    • docExt: ''
    • docPage: ''
    • docRoot: https://{org}.github.io
    • featurePage: 0_home
  8. interfaceDefaults: {}
  9. isCompatible: True
  10. local: local
  11. localDir: /Users/me/text-fabric-data/github/ETCBC/bhsa/_temp
  12. provenanceSpec:
    • corpus: BHSA = Biblia Hebraica Stuttgartensia Amstelodamensis
    • doi: 10.5281/zenodo.1007624
    • moduleSpecs:
      • :
        • backend: no value
        • corpus: Phonetic Transcriptions
        • docUrl:https://nbviewer.jupyter.org/github/etcbc/phono/blob/master/programs/phono.ipynb
        • doi: 10.5281/zenodo.1007636
        • org: ETCBC
        • relative: /tf
        • repo: phono
      • :
        • backend: no value
        • corpus: Parallel Passages
        • docUrl:https://nbviewer.jupyter.org/github/ETCBC/parallels/blob/master/programs/parallels.ipynb
        • doi: 10.5281/zenodo.1007642
        • org: ETCBC
        • relative: /tf
        • repo: parallels
    • org: ETCBC
    • relative: /tf
    • repo: bhsa
    • version: 2021
    • webBase: https://shebanq.ancient-data.org/hebrew
    • webHint: Show this on SHEBANQ
    • webLang: la
    • webLexId: True
    • webUrl:{webBase}/text?book=<1>&chapter=<2>&verse=<3>&version={version}&mr=m&qw=q&tp=txt_p&tr=hb&wget=v&qget=v&nget=vt
    • webUrlLex: {webBase}/word?version={version}&id=<lid>
  13. release: v1.8
  14. typeDisplay:
    • clause:
      • label: {typ} {rela}
      • style: ''
    • clause_atom:
      • hidden: True
      • label: {code}
      • level: 1
      • style: ''
    • half_verse:
      • hidden: True
      • label: {label}
      • style: ''
      • verselike: True
    • lex:
      • featuresBare: gloss
      • label: {voc_lex_utf8}
      • lexOcc: word
      • style: orig
      • template: {voc_lex_utf8}
    • phrase:
      • label: {typ} {function}
      • style: ''
    • phrase_atom:
      • hidden: True
      • label: {typ} {rela}
      • level: 1
      • style: ''
    • sentence:
      • label: {number}
      • style: ''
    • sentence_atom:
      • hidden: True
      • label: {number}
      • level: 1
      • style: ''
    • subphrase:
      • hidden: True
      • label: {number}
      • style: ''
    • word:
      • features: pdp vs vt
      • featuresBare: lex:gloss
  15. writing: hbo
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Text-Fabric API: names N F E L T S C TF Fs Fall Es Eall Cs Call directly usable
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A = use(\"ETCBC/bhsa\", hoist=globals())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We work with Genesis 1 (in fact, only the first 10 clauses)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "gen1 = T.nodeFromSection((\"Genesis\", 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We prepare our portion of text for annotation outside TF.\n", "\n", "What needs to happen is, that we produce a text file and that we remember the positions of the relevant\n", "nodes in that text file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Recorder is a new thing in TF (in development) that lets you create a string from nodes,\n", "where the positions of the nodes in that string are remembered.\n", "You may add all kinds of material in between the texts of the nodes.\n", "And it is up to you how you represent the nodes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We start a recorder." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "rec = Recorder()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can add strings to the recorder, and we can tell nodes to start and to stop.\n", "\n", "We add clause atoms and phrase atoms to the recorder." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "LIMIT = 10\n", "\n", "for (i, cla) in enumerate(L.d(gen1, otype=\"clause_atom\")):\n", " if i >= LIMIT: # only first ten clause atoms\n", " break\n", "\n", " # we want a label in front of each clause atom\n", " label = \"{} {}:{}\".format(*T.sectionFromNode(cla))\n", " rec.add(f\"{label}@{i} \")\n", "\n", " # we start a clause atom node:\n", " # until we end this node, all text that we add counts as material for this clause atom\n", " rec.start(cla)\n", "\n", " for pa in L.d(cla, otype=\"phrase_atom\"):\n", " # we start a phrase node\n", " # until we end this node, all text that we add also counts as material for this phrase atom\n", " rec.start(pa)\n", "\n", " # we add text, it belongs to the current clause atom and to the current phrase atom\n", " rec.add(T.text(pa, fmt=\"text-trans-plain\"))\n", "\n", " # we end the phrase atom\n", " rec.end(pa)\n", "\n", " # we end the clause atom\n", " rec.end(cla)\n", "\n", " # very clause atom on its own line\n", " # this return character does not belong to any node\n", " rec.add(\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can print the recorded text." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genesis 1:1@0 BR>CJT BR> >LHJM >T HCMJM W>T H>RY00 \n", "Genesis 1:2@1 WH>RY HJTH THW WBHW \n", "Genesis 1:2@2 WXCK LHJM MRXPT MR >LHJM \n", "Genesis 1:3@5 JHJ >WR \n", "Genesis 1:3@6 WJHJ&>WR00 \n", "Genesis 1:4@7 WJR> >LHJM >T&H>WR \n", "Genesis 1:4@8 KJ&VWB \n", "Genesis 1:4@9 WJBDL >LHJM BJN H>WR WBJN HXCK00 \n", "\n" ] } ], "source": [ "print(rec.text())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can print the recorded node positions." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pos 14: frozenset({904776, 515690})\n", "pos 15: frozenset({904776, 515690})\n", "pos 16: frozenset({904776, 515690})\n", "pos 17: frozenset({904776, 515690})\n", "pos 18: frozenset({904776, 515690})\n", "pos 19: frozenset({904776, 515690})\n", "pos 20: frozenset({904776, 515690})\n", "pos 21: frozenset({904777, 515690})\n", "pos 22: frozenset({904777, 515690})\n", "pos 23: frozenset({904777, 515690})\n", "pos 24: frozenset({904777, 515690})\n", "pos 25: frozenset({515690, 904778})\n", "pos 26: frozenset({515690, 904778})\n", "pos 27: frozenset({515690, 904778})\n", "pos 28: frozenset({515690, 904778})\n", "pos 29: frozenset({515690, 904778})\n", "pos 30: frozenset({515690, 904778})\n", "pos 31: frozenset({515690, 904779})\n", "pos 32: frozenset({515690, 904779})\n", "pos 33: frozenset({515690, 904779})\n", "pos 34: frozenset({515690, 904779})\n", "pos 35: frozenset({515690, 904779})\n", "pos 36: frozenset({515690, 904779})\n", "pos 37: frozenset({515690, 904779})\n", "pos 38: frozenset({515690, 904779})\n", "pos 39: frozenset({515690, 904779})\n", "pos 40: frozenset({515690, 904779})\n", "pos 41: frozenset({515690, 904779})\n", "pos 42: frozenset({515690, 904779})\n", "pos 43: frozenset({515690, 904779})\n", "pos 44: frozenset({515690, 904779})\n", "pos 45: frozenset({515690, 904779})\n", "pos 46: frozenset({515690, 904779})\n", "pos 47: frozenset({515690, 904779})\n", "pos 48: frozenset({515690, 904779})\n", "pos 49: frozenset({515690, 904779})\n", "pos 50: frozenset({515690, 904779})\n", "pos 66: frozenset({515691, 904780})\n", "pos 67: frozenset({515691, 904781})\n", "pos 68: frozenset({515691, 904781})\n", "pos 69: frozenset({515691, 904781})\n", "pos 70: frozenset({515691, 904781})\n", "pos 71: frozenset({515691, 904781})\n", "pos 72: frozenset({515691, 904782})\n", "pos 73: frozenset({515691, 904782})\n", "pos 74: frozenset({515691, 904782})\n", "pos 75: frozenset({515691, 904782})\n", "pos 76: frozenset({515691, 904782})\n", "pos 77: frozenset({515691, 904783})\n", "pos 78: frozenset({515691, 904783})\n", "pos 79: frozenset({515691, 904783})\n", "pos 80: frozenset({515691, 904783})\n", "pos 81: frozenset({515691, 904783})\n", "pos 82: frozenset({515691, 904783})\n", "pos 83: frozenset({515691, 904783})\n", "pos 84: frozenset({515691, 904783})\n", "pos 85: frozenset({515691, 904783})\n", "pos 101: frozenset({904784, 515692})\n", "pos 102: frozenset({904785, 515692})\n", "pos 103: frozenset({904785, 515692})\n", "pos 104: frozenset({904785, 515692})\n", "pos 105: frozenset({904785, 515692})\n", "pos 106: frozenset({904786, 515692})\n", "pos 107: frozenset({904786, 515692})\n", "pos 108: frozenset({904786, 515692})\n", "pos 109: frozenset({904786, 515692})\n", "pos 110: frozenset({904786, 515692})\n", "pos 111: frozenset({904786, 515692})\n", "pos 112: frozenset({904786, 515692})\n", "pos 113: frozenset({904786, 515692})\n", "pos 114: frozenset({904786, 515692})\n", "pos 115: frozenset({904786, 515692})\n", "pos 116: frozenset({904786, 515692})\n", "pos 117: frozenset({904786, 515692})\n", "pos 133: frozenset({904787, 515693})\n", "pos 134: frozenset({904788, 515693})\n", "pos 135: frozenset({904788, 515693})\n", "pos 136: frozenset({904788, 515693})\n", "pos 137: frozenset({904788, 515693})\n", "pos 138: frozenset({904788, 515693})\n", "pos 139: frozenset({904788, 515693})\n", "pos 140: frozenset({904788, 515693})\n", "pos 141: frozenset({904788, 515693})\n", "pos 142: frozenset({904788, 515693})\n", "pos 143: frozenset({904788, 515693})\n", "pos 144: frozenset({515693, 904789})\n", "pos 145: frozenset({515693, 904789})\n", "pos 146: frozenset({515693, 904789})\n", "pos 147: frozenset({515693, 904789})\n", "pos 148: frozenset({515693, 904789})\n", "pos 149: frozenset({515693, 904789})\n", "pos 150: frozenset({515693, 904790})\n", "pos 151: frozenset({515693, 904790})\n", "pos 152: frozenset({515693, 904790})\n", "pos 153: frozenset({515693, 904790})\n", "pos 154: frozenset({515693, 904790})\n", "pos 155: frozenset({515693, 904790})\n", "pos 156: frozenset({515693, 904790})\n", "pos 157: frozenset({515693, 904790})\n", "pos 158: frozenset({515693, 904790})\n", "pos 159: frozenset({515693, 904790})\n", "pos 160: frozenset({515693, 904790})\n", "pos 161: frozenset({515693, 904790})\n", "pos 162: frozenset({515693, 904790})\n", "pos 163: frozenset({515693, 904790})\n", "pos 179: frozenset({515694, 904791})\n", "pos 180: frozenset({904792, 515694})\n", "pos 181: frozenset({904792, 515694})\n", "pos 182: frozenset({904792, 515694})\n", "pos 183: frozenset({904792, 515694})\n", "pos 184: frozenset({904792, 515694})\n", "pos 185: frozenset({904793, 515694})\n", "pos 186: frozenset({904793, 515694})\n", "pos 187: frozenset({904793, 515694})\n", "pos 188: frozenset({904793, 515694})\n", "pos 189: frozenset({904793, 515694})\n", "pos 190: frozenset({904793, 515694})\n", "pos 206: frozenset({904794, 515695})\n", "pos 207: frozenset({904794, 515695})\n", "pos 208: frozenset({904794, 515695})\n", "pos 209: frozenset({904794, 515695})\n", "pos 210: frozenset({904795, 515695})\n", "pos 211: frozenset({904795, 515695})\n", "pos 212: frozenset({904795, 515695})\n", "pos 213: frozenset({904795, 515695})\n", "pos 229: frozenset({515696, 904796})\n", "pos 230: frozenset({515696, 904797})\n", "pos 231: frozenset({515696, 904797})\n", "pos 232: frozenset({515696, 904797})\n", "pos 233: frozenset({515696, 904797})\n", "pos 234: frozenset({515696, 904798})\n", "pos 235: frozenset({515696, 904798})\n", "pos 236: frozenset({515696, 904798})\n", "pos 237: frozenset({515696, 904798})\n", "pos 238: frozenset({515696, 904798})\n", "pos 239: frozenset({515696, 904798})\n", "pos 255: frozenset({515697, 904799})\n", "pos 256: frozenset({904800, 515697})\n", "pos 257: frozenset({904800, 515697})\n", "pos 258: frozenset({904800, 515697})\n", "pos 259: frozenset({904800, 515697})\n", "pos 260: frozenset({515697, 904801})\n", "pos 261: frozenset({515697, 904801})\n", "pos 262: frozenset({515697, 904801})\n", "pos 263: frozenset({515697, 904801})\n", "pos 264: frozenset({515697, 904801})\n", "pos 265: frozenset({515697, 904801})\n", "pos 266: frozenset({515697, 904802})\n", "pos 267: frozenset({515697, 904802})\n", "pos 268: frozenset({515697, 904802})\n", "pos 269: frozenset({515697, 904802})\n", "pos 270: frozenset({515697, 904802})\n", "pos 271: frozenset({515697, 904802})\n", "pos 272: frozenset({515697, 904802})\n", "pos 273: frozenset({515697, 904802})\n", "pos 289: frozenset({515698, 904803})\n", "pos 290: frozenset({515698, 904803})\n", "pos 291: frozenset({515698, 904803})\n", "pos 292: frozenset({515698, 904804})\n", "pos 293: frozenset({515698, 904804})\n", "pos 294: frozenset({515698, 904804})\n", "pos 295: frozenset({515698, 904804})\n", "pos 311: frozenset({515699, 904805})\n", "pos 312: frozenset({515699, 904806})\n", "pos 313: frozenset({515699, 904806})\n", "pos 314: frozenset({515699, 904806})\n", "pos 315: frozenset({515699, 904806})\n", "pos 316: frozenset({515699, 904806})\n", "pos 317: frozenset({515699, 904807})\n", "pos 318: frozenset({515699, 904807})\n", "pos 319: frozenset({515699, 904807})\n", "pos 320: frozenset({515699, 904807})\n", "pos 321: frozenset({515699, 904807})\n", "pos 322: frozenset({515699, 904807})\n", "pos 323: frozenset({904808, 515699})\n", "pos 324: frozenset({904808, 515699})\n", "pos 325: frozenset({904808, 515699})\n", "pos 326: frozenset({904808, 515699})\n", "pos 327: frozenset({904808, 515699})\n", "pos 328: frozenset({904808, 515699})\n", "pos 329: frozenset({904808, 515699})\n", "pos 330: frozenset({904808, 515699})\n", "pos 331: frozenset({904808, 515699})\n", "pos 332: frozenset({904808, 515699})\n", "pos 333: frozenset({904808, 515699})\n", "pos 334: frozenset({904808, 515699})\n", "pos 335: frozenset({904808, 515699})\n", "pos 336: frozenset({904808, 515699})\n", "pos 337: frozenset({904808, 515699})\n", "pos 338: frozenset({904808, 515699})\n", "pos 339: frozenset({904808, 515699})\n", "pos 340: frozenset({904808, 515699})\n", "pos 341: frozenset({904808, 515699})\n", "pos 342: frozenset({904808, 515699})\n", "pos 343: frozenset({904808, 515699})\n" ] } ], "source": [ "print(\"\\n\".join(f\"pos {i}: {p}\" for (i, p) in enumerate(rec.positions()) if p))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can write the recorded text and the positions to two files:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "lines_to_next_cell": 2 }, "outputs": [], "source": [ "rec.write(\"data/gen1.txt\")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genesis 1:1@0 BR>CJT BR> >LHJM >T HCMJM W>T H>RY00 \n", "Genesis 1:2@1 WH>RY HJTH THW WBHW \n", "Genesis 1:2@2 WXCK LHJM MRXPT MR >LHJM \n", "Genesis 1:3@5 JHJ >WR \n", "Genesis 1:3@6 WJHJ&>WR00 \n", "Genesis 1:4@7 WJR> >LHJM >T&H>WR \n", "Genesis 1:4@8 KJ&VWB \n", "Genesis 1:4@9 WJBDL >LHJM BJN H>WR WBJN HXCK00 \n" ] } ], "source": [ "!head -n 10 data/gen1.txt" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "904776\t515690\n", "904776\t515690\n", "904776\t515690\n", "904776\t515690\n", "904776\t515690\n", "904776\t515690\n", "904776\t515690\n", "904777\t515690\n", "904777\t515690\n", "904777\t515690\n", "904777\t515690\n", "515690\t904778\n", "515690\t904778\n", "515690\t904778\n", "515690\t904778\n", "515690\t904778\n" ] } ], "source": [ "!head -n 30 data/gen1.txt.pos" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we produce a (fake) annotation file, based on the text.\n", "\n", "The file is tab delimited, the columns are:\n", "\n", "* start character position\n", "* end character position\n", "* feature 1 value\n", "* feature 2 value\n", "* etc" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 2 }, "source": [ "We annotate as follows:\n", "\n", "* every word that starts with a `B` gets `bword=1`\n", "* every word that ends with a `T` gets `tword=1`\n", "\n", "Then we want every phrase with a b-word to get `bword=1` and likewise\n", "every clause with a b-word to get `bword=1`,\n", "and the same for `tword`." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def annotate(fileName):\n", " annotations = {}\n", "\n", " with open(fileName) as fh:\n", " pos = 0\n", " for line in fh:\n", " words = line.split(\" \")\n", "\n", " for word in words[0:2]:\n", " lWord = len(word)\n", " pos += lWord + 1\n", " for word in words[2:]:\n", " word = word.rstrip(\"\\n\")\n", " lWord = len(word)\n", " start = pos\n", " end = pos + lWord - 1\n", " pos += lWord + 1\n", " if lWord:\n", " if word[0] == \"B\":\n", " annotations.setdefault((start, end), {})[\"bword\"] = 1\n", " if word[-1] == \"T\":\n", " annotations.setdefault((start, end), {})[\"tword\"] = 1\n", "\n", " with open(f\"{fileName}.ann\", \"w\") as fh:\n", " fh.write(\"start\\tend\\tbword\\ttword\\n\")\n", " for ((start, end), features) in annotations.items():\n", " row = \"\\t\".join(\n", " str(a)\n", " for a in (\n", " start,\n", " end,\n", " features.get(\"bword\", \"\"),\n", " features.get(\"tword\", \"\"),\n", " )\n", " )\n", " fh.write(f\"{row}\\n\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "annotate(\"data/gen1.txt\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the annotation file." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "start\tend\tbword\ttword\n", "14\t19\t1\t1\n", "21\t23\t1\t\n", "31\t32\t\t1\n", "40\t42\t\t1\n", "144\t148\t\t1\n", "323\t325\t1\t\n" ] } ], "source": [ "!cat data/gen1.txt.ann" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we want to feed back these annotations as TF features on `phrase_atom` and `clause_atom` nodes.\n", "\n", "Our recorder knows how to do that." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "features = rec.makeFeatures(\"data/gen1.txt.ann\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's see." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{904776: '1', 515690: '1', 904777: '1', 904808: '1', 515699: '1'}" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"bword\"]" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{904776: '1', 515690: '1', 904779: '1', 904789: '1', 515693: '1'}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"tword\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bword phrase_atom 904776: BR>CJT \n", "bword clause_atom 515690: BR>CJT BR> >LHJM >T HCMJM W>T H>RY00 \n", "bword phrase_atom 904777: BR> \n", "bword phrase_atom 904808: BJN H>WR WBJN HXCK00 \n", "bword clause_atom 515699: WJBDL >LHJM BJN H>WR WBJN HXCK00 \n", "tword phrase_atom 904776: BR>CJT \n", "tword clause_atom 515690: BR>CJT BR> >LHJM >T HCMJM W>T H>RY00 \n", "tword phrase_atom 904779: >T HCMJM W>T H>RY00 \n", "tword phrase_atom 904789: MRXPT \n", "tword clause_atom 515693: WRWX >LHJM MRXPT = LIMIT:\n", " break\n", " label = \"{} {}:{}\".format(*T.sectionFromNode(cla))\n", " rec.add(f\"{label}@{i} \")\n", "\n", " for w in L.d(cla, otype=\"word\"):\n", " rec.start(w)\n", " rec.add(T.text(w, fmt=\"text-trans-plain\"))\n", " rec.end(w)\n", "\n", " rec.add(\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It gives the same text:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genesis 1:1@0 BR>CJT BR> >LHJM >T HCMJM W>T H>RY00 \n", "Genesis 1:2@1 WH>RY HJTH THW WBHW \n", "Genesis 1:2@2 WXCK LHJM MRXPT MR >LHJM \n", "Genesis 1:3@5 JHJ >WR \n", "Genesis 1:3@6 WJHJ&>WR00 \n", "Genesis 1:4@7 WJR> >LHJM >T&H>WR \n", "Genesis 1:4@8 KJ&VWB \n", "Genesis 1:4@9 WJBDL >LHJM BJN H>WR WBJN HXCK00 \n", "\n" ] } ], "source": [ "print(rec.text())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "but the node positions are different:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pos 14: frozenset({1})\n", "pos 15: frozenset({2})\n", "pos 16: frozenset({2})\n", "pos 17: frozenset({2})\n", "pos 18: frozenset({2})\n", "pos 19: frozenset({2})\n", "pos 20: frozenset({2})\n", "pos 21: frozenset({3})\n", "pos 22: frozenset({3})\n", "pos 23: frozenset({3})\n", "pos 24: frozenset({3})\n", "pos 25: frozenset({4})\n", "pos 26: frozenset({4})\n", "pos 27: frozenset({4})\n", "pos 28: frozenset({4})\n", "pos 29: frozenset({4})\n", "pos 30: frozenset({4})\n", "pos 31: frozenset({5})\n", "pos 32: frozenset({5})\n", "pos 33: frozenset({5})\n", "pos 34: frozenset({6})\n", "pos 35: frozenset({7})\n", "pos 36: frozenset({7})\n", "pos 37: frozenset({7})\n", "pos 38: frozenset({7})\n", "pos 39: frozenset({7})\n", "pos 40: frozenset({8})\n", "pos 41: frozenset({9})\n", "pos 42: frozenset({9})\n", "pos 43: frozenset({9})\n", "pos 44: frozenset({10})\n", "pos 45: frozenset({11})\n", "pos 46: frozenset({11})\n", "pos 47: frozenset({11})\n", "pos 48: frozenset({11})\n", "pos 49: frozenset({11})\n", "pos 50: frozenset({11})\n", "pos 66: frozenset({12})\n", "pos 67: frozenset({13})\n", "pos 68: frozenset({14})\n", "pos 69: frozenset({14})\n", "pos 70: frozenset({14})\n", "pos 71: frozenset({14})\n", "pos 72: frozenset({15})\n", "pos 73: frozenset({15})\n", "pos 74: frozenset({15})\n", "pos 75: frozenset({15})\n", "pos 76: frozenset({15})\n", "pos 77: frozenset({16})\n", "pos 78: frozenset({16})\n", "pos 79: frozenset({16})\n", "pos 80: frozenset({16})\n", "pos 81: frozenset({17})\n", "pos 82: frozenset({18})\n", "pos 83: frozenset({18})\n", "pos 84: frozenset({18})\n", "pos 85: frozenset({18})\n", "pos 101: frozenset({19})\n", "pos 102: frozenset({20})\n", "pos 103: frozenset({20})\n", "pos 104: frozenset({20})\n", "pos 105: frozenset({20})\n", "pos 106: frozenset({21})\n", "pos 107: frozenset({21})\n", "pos 108: frozenset({21})\n", "pos 109: frozenset({22})\n", "pos 110: frozenset({22})\n", "pos 111: frozenset({22})\n", "pos 112: frozenset({22})\n", "pos 113: frozenset({23})\n", "pos 114: frozenset({23})\n", "pos 115: frozenset({23})\n", "pos 116: frozenset({23})\n", "pos 117: frozenset({23})\n", "pos 133: frozenset({24})\n", "pos 134: frozenset({25})\n", "pos 135: frozenset({25})\n", "pos 136: frozenset({25})\n", "pos 137: frozenset({25})\n", "pos 138: frozenset({26})\n", "pos 139: frozenset({26})\n", "pos 140: frozenset({26})\n", "pos 141: frozenset({26})\n", "pos 142: frozenset({26})\n", "pos 143: frozenset({26})\n", "pos 144: frozenset({27})\n", "pos 145: frozenset({27})\n", "pos 146: frozenset({27})\n", "pos 147: frozenset({27})\n", "pos 148: frozenset({27})\n", "pos 149: frozenset({27})\n", "pos 150: frozenset({28})\n", "pos 151: frozenset({28})\n", "pos 152: frozenset({28})\n", "pos 153: frozenset({29})\n", "pos 154: frozenset({29})\n", "pos 155: frozenset({29})\n", "pos 156: frozenset({29})\n", "pos 157: frozenset({30})\n", "pos 158: frozenset({31})\n", "pos 159: frozenset({31})\n", "pos 160: frozenset({31})\n", "pos 161: frozenset({31})\n", "pos 162: frozenset({31})\n", "pos 163: frozenset({31})\n", "pos 179: frozenset({32})\n", "pos 180: frozenset({33})\n", "pos 181: frozenset({33})\n", "pos 182: frozenset({33})\n", "pos 183: frozenset({33})\n", "pos 184: frozenset({33})\n", "pos 185: frozenset({34})\n", "pos 186: frozenset({34})\n", "pos 187: frozenset({34})\n", "pos 188: frozenset({34})\n", "pos 189: frozenset({34})\n", "pos 190: frozenset({34})\n", "pos 206: frozenset({35})\n", "pos 207: frozenset({35})\n", "pos 208: frozenset({35})\n", "pos 209: frozenset({35})\n", "pos 210: frozenset({36})\n", "pos 211: frozenset({36})\n", "pos 212: frozenset({36})\n", "pos 213: frozenset({36})\n", "pos 229: frozenset({37})\n", "pos 230: frozenset({38})\n", "pos 231: frozenset({38})\n", "pos 232: frozenset({38})\n", "pos 233: frozenset({38})\n", "pos 234: frozenset({39})\n", "pos 235: frozenset({39})\n", "pos 236: frozenset({39})\n", "pos 237: frozenset({39})\n", "pos 238: frozenset({39})\n", "pos 239: frozenset({39})\n", "pos 255: frozenset({40})\n", "pos 256: frozenset({41})\n", "pos 257: frozenset({41})\n", "pos 258: frozenset({41})\n", "pos 259: frozenset({41})\n", "pos 260: frozenset({42})\n", "pos 261: frozenset({42})\n", "pos 262: frozenset({42})\n", "pos 263: frozenset({42})\n", "pos 264: frozenset({42})\n", "pos 265: frozenset({42})\n", "pos 266: frozenset({43})\n", "pos 267: frozenset({43})\n", "pos 268: frozenset({43})\n", "pos 269: frozenset({44})\n", "pos 270: frozenset({45})\n", "pos 271: frozenset({45})\n", "pos 272: frozenset({45})\n", "pos 273: frozenset({45})\n", "pos 289: frozenset({46})\n", "pos 290: frozenset({46})\n", "pos 291: frozenset({46})\n", "pos 292: frozenset({47})\n", "pos 293: frozenset({47})\n", "pos 294: frozenset({47})\n", "pos 295: frozenset({47})\n", "pos 311: frozenset({48})\n", "pos 312: frozenset({49})\n", "pos 313: frozenset({49})\n", "pos 314: frozenset({49})\n", "pos 315: frozenset({49})\n", "pos 316: frozenset({49})\n", "pos 317: frozenset({50})\n", "pos 318: frozenset({50})\n", "pos 319: frozenset({50})\n", "pos 320: frozenset({50})\n", "pos 321: frozenset({50})\n", "pos 322: frozenset({50})\n", "pos 323: frozenset({51})\n", "pos 324: frozenset({51})\n", "pos 325: frozenset({51})\n", "pos 326: frozenset({51})\n", "pos 327: frozenset({52})\n", "pos 328: frozenset({53})\n", "pos 329: frozenset({53})\n", "pos 330: frozenset({53})\n", "pos 331: frozenset({53})\n", "pos 332: frozenset({54})\n", "pos 333: frozenset({55})\n", "pos 334: frozenset({55})\n", "pos 335: frozenset({55})\n", "pos 336: frozenset({55})\n", "pos 337: frozenset({56})\n", "pos 338: frozenset({57})\n", "pos 339: frozenset({57})\n", "pos 340: frozenset({57})\n", "pos 341: frozenset({57})\n", "pos 342: frozenset({57})\n", "pos 343: frozenset({57})\n" ] } ], "source": [ "print(\"\\n\".join(f\"pos {i}: {p}\" for (i, p) in enumerate(rec.positions()) if p))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have produced the same text,\n", "so we can use the earlier annotation file to create word features." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "features = rec.makeFeatures(\"data/gen1.txt.ann\")" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{1: '1', 2: '1', 3: '1', 51: '1'}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"bword\"]" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{1: '1', 2: '1', 5: '1', 8: '1', 9: '1', 27: '1'}" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"tword\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bword word 1: B\n", "bword word 2: R>CJT \n", "bword word 3: BR> \n", "bword word 51: BJN \n", "tword word 1: B\n", "tword word 2: R>CJT \n", "tword word 5: >T \n", "tword word 8: W\n", "tword word 9: >T \n", "tword word 27: MRXPT \n" ] } ], "source": [ "for feat in (\"bword\", \"tword\"):\n", " for n in features[feat]:\n", " print(f'{feat} {F.otype.v(n)} {n}: {T.text(n, fmt=\"text-trans-plain\")}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Explanation:\n", "\n", "The annotator just looked at the string `BR>CJT` without knowing that it is two words." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "start\tend\tbword\ttword\n", "14\t19\t1\t1\n", "21\t23\t1\t\n", "31\t32\t\t1\n", "40\t42\t\t1\n", "144\t148\t\t1\n", "323\t325\t1\t\n" ] } ], "source": [ "!cat data/gen1.txt.ann" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So it has annotated pos 14-19 as a `bword` and as a `tword`.\n", "\n", "But TF knows that 14-19 are slots 1 and 2, so when the annotations are applied,\n", "slots 1 and 2 are both set to `bwords` and `twords`.\n", "\n", "We can remedy the situation by producing an other text to the annotator, one where\n", "slots are always separated by a space.\n", "\n", "Lets do that by always adding a space, so real words are separated by two spaces." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "rec = Recorder()\n", "LIMIT = 10\n", "\n", "for (i, cla) in enumerate(L.d(gen1, otype=\"clause_atom\")):\n", " if i >= LIMIT:\n", " break\n", " label = \"{} {}:{}\".format(*T.sectionFromNode(cla))\n", " rec.add(f\"{label}@{i} \")\n", "\n", " for w in L.d(cla, otype=\"word\"):\n", " rec.start(w)\n", " rec.add(T.text(w, fmt=\"text-trans-plain\") + \" \")\n", " rec.end(w)\n", "\n", " rec.add(\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the text" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genesis 1:1@0 B R>CJT BR> >LHJM >T H CMJM W >T H >RY00 \n", "Genesis 1:2@1 W H >RY HJTH THW W BHW \n", "Genesis 1:2@2 W XCK LHJM MRXPT MR >LHJM \n", "Genesis 1:3@5 JHJ >WR \n", "Genesis 1:3@6 W JHJ& >WR00 \n", "Genesis 1:4@7 W JR> >LHJM >T& H >WR \n", "Genesis 1:4@8 KJ& VWB \n", "Genesis 1:4@9 W JBDL >LHJM BJN H >WR W BJN H XCK00 \n", "\n" ] } ], "source": [ "print(rec.text())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We write the text to file." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "rec.write(\"data/gen1wx.txt\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We run our annotator again, because we have a different text:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "annotate(\"data/gen1wx.txt\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the new annotation file." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "start\tend\tbword\ttword\n", "14\t14\t1\t\n", "16\t20\t\t1\n", "23\t25\t1\t\n", "35\t36\t\t1\n", "49\t50\t\t1\n", "99\t101\t1\t\n", "170\t174\t\t1\n", "373\t375\t1\t\n", "387\t389\t1\t\n" ] } ], "source": [ "!cat data/gen1wx.txt.ann" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The features are no surprise:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "features = rec.makeFeatures(\"data/gen1wx.txt.ann\")" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{1: '1', 3: '1', 18: '1', 51: '1', 55: '1'}" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"bword\"]" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{2: '1', 5: '1', 9: '1', 27: '1'}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"tword\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bword word 1: B\n", "bword word 3: BR> \n", "bword word 18: BHW \n", "bword word 51: BJN \n", "bword word 55: BJN \n", "tword word 2: R>CJT \n", "tword word 5: >T \n", "tword word 9: >T \n", "tword word 27: MRXPT \n" ] } ], "source": [ "for feat in (\"bword\", \"tword\"):\n", " for n in features[feat]:\n", " print(f'{feat} {F.otype.v(n)} {n}: {T.text(n, fmt=\"text-trans-plain\")}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# All steps\n", "\n", "* **[start](start.ipynb)** your first step in mastering the bible computationally\n", "* **[display](display.ipynb)** become an expert in creating pretty displays of your text structures\n", "* **[search](search.ipynb)** turbo charge your hand-coding with search templates\n", "* **[export Excel](exportExcel.ipynb)** make tailor-made spreadsheets out of your results\n", "* **[share](share.ipynb)** draw in other people's data and let them use yours\n", "* **[export](export.ipynb)** export your dataset as an Emdros database\n", "* **annotate** annotate plain text by means of other tools and import the annotations as TF features\n", "* **[volumes](volumes.ipynb)** work with selected books only\n", "* **[trees](trees.ipynb)** work with the BHSA data as syntax trees\n", "\n", "CC-BY Dirk Roorda" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.1" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }