{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "\n", "# Import BHSA data into Pandas\n", "\n", "This notebook contains the Pandas instructions to load the\n", "[Pandas export](export.ipynb) export of the BHSA.\n", "\n", "We then perform some simple information extracting on the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# How to get the Pandas file\n", "\n", "The direct download link is \n", "[data-2021.pd](https://github.com/ETCBC/bhsa/releases/download/v1.8/data-2021.pd)\n", "\n", "The pandas file is over 50 MB, a bit too large for GitHub without large file support.\n", "So I attached it to the\n", "[latest release](https://github.com/ETCBC/bhsa/releases/tag/v1.8).\n", "\n", "## Reproduction\n", "\n", "If you want to do it yourself,\n", "\n", "* clone this repo\n", "* find the [export](export.ipynb) notebook\n", "* run it in Jupyterlab\n", "* pick up the newly generated file from the `/pandas` subdirectory." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import os\n", "import pandas as pd # pip3 install pandas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# File locations\n", "\n", "We set up some variables for the location of the Pandas file and a location\n", "where we will save the full text of this corpus." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "VERSION = \"2021\"\n", "PANDAS_DIR = os.path.abspath(\"../pandas\")\n", "TEXT_DIR = os.path.abspath(os.path.expanduser(\"~/Downloads/text\"))\n", "TABLE_FILE_PD = f\"{PANDAS_DIR}/data-{VERSION}.pd\"\n", "TABLE_FILE_TXT = f\"{TEXT_DIR}/data-{VERSION}.txt\"\n", "\n", "if not os.path.exists(TEXT_DIR):\n", " os.makedirs(TEXT_DIR)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load the dataframe" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Done. Size=104171832\n" ] } ], "source": [ "frame = pd.read_parquet(TABLE_FILE_PD, engine=\"pyarrow\")\n", "print(\"Done. Size={}\".format(frame.size))" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1446831, 72)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "frame.shape" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ndotypeg_consg_cons_utf8g_lexg_lex_utf8g_wordg_word_utf8lexlex_utf8...tabtxttypuvfvbevbsversevoc_lexvsvt
0426591book...<NA><NA>
1426630chapter...<NA><NA>
21414389verse...<NA>1
31172308sentence...<NA><NA>
41236025sentence_atom...<NA><NA>
5427559clause...<NA>?xQtX<NA>
6515690clause_atom...<NA>xQtX<NA>
7606394half_verse...<NA><NA>
8651573phrase...<NA>PP<NA>
9904776phrase_atom...<NA>PP<NA>
101437602lexBב...<NA><NA>B.:
111wordBבB.:-בְּB.:-בְּBב...<NA>absentn/an/a<NA>B.:NANA
121437603lexR>CJT/ראשׁית֜...<NA><NA>R;>CIJT
132wordR>CJTראשׁיתR;>CIJTרֵאשִׁיתR;>CI73JTרֵאשִׁ֖יתR>CJT/ראשׁית...<NA>absentn/an/a<NA>R;>CIJTNANA
141437604lexBR>[ברא...<NA><NA>BR>
15651574phrase...<NA>VP<NA>
16904777phrase_atom...<NA>VP<NA>
173wordBR>בראB.@R@>בָּרָאB.@R@74>בָּרָ֣אBR>[ברא...<NA>absentabsent<NA>BR>qalperf
181437605lex>LHJM/אלהים֜...<NA><NA>>:ELOHIJM
19651575phrase...<NA>NP<NA>
20904778phrase_atom...<NA>NP<NA>
214word>LHJMאלהים>:ELOHאֱלֹה>:ELOHI92JMאֱלֹהִ֑ים>LHJM/אלהים...<NA>absentn/an/a<NA>>:ELOHIJMNANA
22606395half_verse...<NA><NA>
23651576phrase...<NA>PP<NA>
24904779phrase_atom...<NA>PP<NA>
251300539subphrase...<NA><NA>
261437606lex>Tאת...<NA><NA>>;T
275word>Tאת>;Tאֵת>;71Tאֵ֥ת>Tאת...<NA>absentn/an/a<NA>>;TNANA
281437607lexHה...<NA><NA>HA
296wordHהHA-הַHA-הַHה...<NA>absentn/an/a<NA>HANANA
\n", "

30 rows × 72 columns

\n", "
" ], "text/plain": [ " nd otype g_cons g_cons_utf8 g_lex g_lex_utf8 \\\n", "0 426591 book \n", "1 426630 chapter \n", "2 1414389 verse \n", "3 1172308 sentence \n", "4 1236025 sentence_atom \n", "5 427559 clause \n", "6 515690 clause_atom \n", "7 606394 half_verse \n", "8 651573 phrase \n", "9 904776 phrase_atom \n", "10 1437602 lex \n", "11 1 word B ב B.:- בְּ \n", "12 1437603 lex \n", "13 2 word R>CJT ראשׁית R;>CIJT רֵאשִׁית \n", "14 1437604 lex \n", "15 651574 phrase \n", "16 904777 phrase_atom \n", "17 3 word BR> ברא B.@R@> בָּרָא \n", "18 1437605 lex \n", "19 651575 phrase \n", "20 904778 phrase_atom \n", "21 4 word >LHJM אלהים >:ELOH אֱלֹה \n", "22 606395 half_verse \n", "23 651576 phrase \n", "24 904779 phrase_atom \n", "25 1300539 subphrase \n", "26 1437606 lex \n", "27 5 word >T את >;T אֵת \n", "28 1437607 lex \n", "29 6 word H ה HA- הַ \n", "\n", " g_word g_word_utf8 lex lex_utf8 ... tab txt typ uvf \\\n", "0 ... \n", "1 ... \n", "2 ... \n", "3 ... \n", "4 ... \n", "5 ... ? xQtX \n", "6 ... xQtX \n", "7 ... \n", "8 ... PP \n", "9 ... PP \n", "10 B ב ... \n", "11 B.:- בְּ B ב ... absent \n", "12 R>CJT/ ראשׁית֜ ... \n", "13 R;>CI73JT רֵאשִׁ֖ית R>CJT/ ראשׁית ... absent \n", "14 BR>[ ברא ... \n", "15 ... VP \n", "16 ... VP \n", "17 B.@R@74> בָּרָ֣א BR>[ ברא ... absent \n", "18 >LHJM/ אלהים֜ ... \n", "19 ... NP \n", "20 ... NP \n", "21 >:ELOHI92JM אֱלֹהִ֑ים >LHJM/ אלהים ... absent \n", "22 ... \n", "23 ... PP \n", "24 ... PP \n", "25 ... \n", "26 >T את ... \n", "27 >;71T אֵ֥ת >T את ... absent \n", "28 H ה ... \n", "29 HA- הַ H ה ... absent \n", "\n", " vbe vbs verse voc_lex vs vt \n", "0 \n", "1 \n", "2 1 \n", "3 \n", "4 \n", "5 \n", "6 \n", "7 \n", "8 \n", "9 \n", "10 B.: \n", "11 n/a n/a B.: NA NA \n", "12 R;>CIJT \n", "13 n/a n/a R;>CIJT NA NA \n", "14 BR> \n", "15 \n", "16 \n", "17 absent BR> qal perf \n", "18 >:ELOHIJM \n", "19 \n", "20 \n", "21 n/a n/a >:ELOHIJM NA NA \n", "22 \n", "23 \n", "24 \n", "25 \n", "26 >;T \n", "27 n/a n/a >;T NA NA \n", "28 HA \n", "29 n/a n/a HA NA NA \n", "\n", "[30 rows x 72 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "frame.head(30)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['nd',\n", " 'otype',\n", " 'g_cons',\n", " 'g_cons_utf8',\n", " 'g_lex',\n", " 'g_lex_utf8',\n", " 'g_word',\n", " 'g_word_utf8',\n", " 'lex',\n", " 'lex_utf8',\n", " 'phono',\n", " 'phono_trailer',\n", " 'qere',\n", " 'qere_trailer',\n", " 'qere_trailer_utf8',\n", " 'qere_utf8',\n", " 'trailer',\n", " 'trailer_utf8',\n", " 'voc_lex_utf8',\n", " 'in_book',\n", " 'in_chapter',\n", " 'in_verse',\n", " 'in_lex',\n", " 'in_half_verse',\n", " 'in_sentence',\n", " 'in_sentence_atom',\n", " 'in_clause',\n", " 'in_clause_atom',\n", " 'in_phrase',\n", " 'in_phrase_atom',\n", " 'in_subphrase',\n", " 'in_word',\n", " 'crossref',\n", " 'mother',\n", " 'book',\n", " 'chapter',\n", " 'code',\n", " 'det',\n", " 'domain',\n", " 'freq_lex',\n", " 'function',\n", " 'gloss',\n", " 'gn',\n", " 'label',\n", " 'language',\n", " 'ls',\n", " 'nametype',\n", " 'nme',\n", " 'nu',\n", " 'number',\n", " 'pargr',\n", " 'pdp',\n", " 'pfm',\n", " 'prs',\n", " 'prs_gn',\n", " 'prs_nu',\n", " 'prs_ps',\n", " 'ps',\n", " 'rank_lex',\n", " 'rela',\n", " 'sp',\n", " 'st',\n", " 'tab',\n", " 'txt',\n", " 'typ',\n", " 'uvf',\n", " 'vbe',\n", " 'vbs',\n", " 'verse',\n", " 'voc_lex',\n", " 'vs',\n", " 'vt']" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "columnList = frame.columns.values.tolist()\n", "columnList" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Books" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us extract some data.\n", "First a list of the book names." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genesis Exodus Leviticus Numeri Deuteronomium Josua Judices Samuel_I Samuel_II Reges_I Reges_II Jesaia Jeremia Ezechiel Hosea Joel Amos Obadia Jona Micha Nahum Habakuk Zephania Haggai Sacharia Maleachi Psalmi Iob Proverbia Ruth Canticum Ecclesiastes Threni Esther Daniel Esra Nehemia Chronica_I Chronica_II\n" ] } ], "source": [ "books = frame[frame.otype == \"book\"].book\n", "print(\" \".join(str(x) for x in books))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Text\n", "\n", "Now the complete text of the whole bible." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "words = frame.loc[frame.otype == \"word\"]\n", "text = words.g_word_utf8 + words.trailer_utf8\n", "\n", "with open(TABLE_FILE_TXT, \"w\") as pt:\n", " pt.write((\"\".join(text)).replace(\"\\u05C3\", \"\\u05C3\\n\").replace(\"\\\\n\", \"\\n\"))\n", " pt.write(\"\\n\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃\n", " וְהָאָ֗רֶץ הָיְתָ֥ה תֹ֨הוּ֙ וָבֹ֔הוּ וְחֹ֖שֶׁךְ עַל־פְּנֵ֣י תְהֹ֑ום וְר֣וּחַ אֱלֹהִ֔ים מְרַחֶ֖פֶת עַל־פְּנֵ֥י הַמָּֽיִם׃\n", " וַיֹּ֥אמֶר אֱלֹהִ֖ים יְהִ֣י אֹ֑ור וַֽיְהִי־אֹֽור׃\n", " וַיַּ֧רְא אֱלֹהִ֛ים אֶת־הָאֹ֖ור כִּי־טֹ֑וב וַיַּבְדֵּ֣ל אֱלֹהִ֔ים בֵּ֥ין הָאֹ֖ור וּבֵ֥ין הַחֹֽשֶׁךְ׃\n", " וַיִּקְרָ֨א אֱלֹהִ֤ים׀ לָאֹור֙ יֹ֔ום וְלַחֹ֖שֶׁךְ קָ֣רָא לָ֑יְלָה וַֽיְהִי־עֶ֥רֶב וַֽיְהִי־בֹ֖קֶר יֹ֥ום אֶחָֽד׃\n", " פ וַיֹּ֣אמֶר אֱלֹהִ֔ים יְהִ֥י רָקִ֖יעַ בְּתֹ֣וךְ הַמָּ֑יִם וִיהִ֣י מַבְדִּ֔יל בֵּ֥ין מַ֖יִם לָמָֽיִם׃\n", " וַיַּ֣עַשׂ אֱלֹהִים֮ אֶת־הָרָקִיעַ֒ וַיַּבְדֵּ֗ל בֵּ֤ין הַמַּ֨יִם֙ אֲשֶׁר֙ מִתַּ֣חַת לָרָקִ֔יעַ וּבֵ֣ין הַמַּ֔יִם אֲשֶׁ֖ר מֵעַ֣ל לָרָקִ֑יעַ וַֽיְהִי־כֵֽן׃\n", " וַיִּקְרָ֧א אֱלֹהִ֛ים לָֽרָקִ֖יעַ שָׁמָ֑יִם וַֽיְהִי־עֶ֥רֶב וַֽיְהִי־בֹ֖קֶר יֹ֥ום שֵׁנִֽי׃\n", " פ וַיֹּ֣אמֶר אֱלֹהִ֗ים יִקָּו֨וּ הַמַּ֜יִם מִתַּ֤חַת הַשָּׁמַ֨יִם֙ אֶל־מָקֹ֣ום אֶחָ֔ד וְתֵרָאֶ֖ה הַיַּבָּשָׁ֑ה וַֽיְהִי־כֵֽן׃\n", " וַיִּקְרָ֨א אֱלֹהִ֤ים׀ לַיַּבָּשָׁה֙ אֶ֔רֶץ וּלְמִקְוֵ֥ה הַמַּ֖יִם קָרָ֣א יַמִּ֑ים וַיַּ֥רְא אֱלֹהִ֖ים כִּי־טֹֽוב׃\n" ] } ], "source": [ "!head {TABLE_FILE_TXT}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Drill down to a passage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us get the words from the first verse.\n", "\n", "How do we know the node of the first verse? See the end of the\n", "[bigTable](bigTable.ipynb) notebook." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "v1 = 1414389" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Is this really a verse?" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'verse'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "element = frame[frame.nd == v1].otype.iloc[0]\n", "element" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n", "Length: 11, dtype: Int64\n" ] } ], "source": [ "wordIds = frame[(frame.otype == \"word\") & (frame.in_verse == v1)].nd\n", "print(wordIds.values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now the *text* of the first verse." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃\n", " \n" ] } ], "source": [ "words = frame[(frame.otype == \"word\") & (frame.in_verse == v1)]\n", "text = words.g_word_utf8 + words.trailer_utf8\n", "print((\"\".join(text)).replace(\"\\u05C3\", \"\\u05C3\\n\").replace(\"\\\\n\", \"\\n\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us get the words and text of an arbitrary passage, say Psalmi 131:2\n", "\n", "First the id of the chunk (i.e. the Text-Fabric node number):" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1431812\n" ] } ], "source": [ "verse_id = frame[\n", " (frame.otype == \"verse\")\n", " & (frame.book == \"Psalmi\")\n", " & (frame.chapter == 131)\n", " & (frame.verse == 2)\n", "].nd.iloc[0]\n", "print(verse_id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now the word ids of that verse:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "[333425, 333426, 333427, 333428, 333429, 333430, 333431, 333432, 333433,\n", " 333434, 333435, 333436, 333437, 333438, 333439]\n", "Length: 15, dtype: Int64\n" ] } ], "source": [ "words = frame[(frame.otype == \"word\") & (frame.in_verse == verse_id)]\n", "print(words.nd.values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And, finally, the text of those words." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "אִם־לֹ֤א שִׁוִּ֨יתִי׀ וְדֹומַ֗מְתִּי נַ֫פְשִׁ֥י כְּ֭גָמֻל עֲלֵ֣י אִמֹּ֑ו כַּגָּמֻ֖ל עָלַ֣י נַפְשִֽׁי׃\n", " \n" ] } ], "source": [ "text = words.g_word_utf8 + words.trailer_utf8\n", "print((\"\".join(text)).replace(\"\\u05C3\", \"\\u05C3\\n\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let us organize this in two functions: one that returns the verse object given a passage, and one that prints the texts of the words in a given object." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "def object2text(nd):\n", " otype = frame[frame.nd == nd].otype.iloc[0]\n", " inelement = \"in_\" + otype\n", " words = frame[(frame.otype == \"word\") & (frame[inelement] == nd)]\n", " text = words.g_word_utf8 + words.trailer_utf8\n", " return (\"\".join(text)).replace(\"\\u05C3\", \"\\u05C3\\n\").replace(\"\\\\n\", \"\\n\")\n", "\n", "\n", "def verse2object(book, chapter, verse):\n", " return frame[\n", " (frame.otype == \"verse\")\n", " & (frame.book == book)\n", " & (frame.chapter == chapter)\n", " & (frame.verse == verse)\n", " ].nd.iloc[0]\n", "\n", "\n", "def verse2text(book, chapter, verse):\n", " return object2text(verse2object(book, chapter, verse))\n", "\n", "\n", "def chapter2object(book, chapter):\n", " return frame[\n", " (frame.otype == \"chapter\") & (frame.book == book) & (frame.chapter == chapter)\n", " ].nd.iloc[0]\n", "\n", "\n", "def chapter2text(book, chapter):\n", " return object2text(chapter2object(book, chapter))" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "אִם־לֹ֤א שִׁוִּ֨יתִי׀ וְדֹומַ֗מְתִּי נַ֫פְשִׁ֥י כְּ֭גָמֻל עֲלֵ֣י אִמֹּ֑ו כַּגָּמֻ֖ל עָלַ֣י נַפְשִֽׁי׃\n", " \n" ] } ], "source": [ "print(verse2text(\"Psalmi\", 131, 2))" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "שִׁ֥יר הַֽמַּֽעֲלֹ֗ות לְדָ֫וִ֥ד יְהוָ֤ה׀ לֹא־גָבַ֣הּ לִ֭בִּי וְלֹא־רָמ֣וּ עֵינַ֑י וְלֹֽא־הִלַּ֓כְתִּי׀ בִּגְדֹלֹ֖ות וּבְנִפְלָאֹ֣ות מִמֶּֽנִּי׃\n", " אִם־לֹ֤א שִׁוִּ֨יתִי׀ וְדֹומַ֗מְתִּי נַ֫פְשִׁ֥י כְּ֭גָמֻל עֲלֵ֣י אִמֹּ֑ו כַּגָּמֻ֖ל עָלַ֣י נַפְשִֽׁי׃\n", " יַחֵ֣ל יִ֝שְׂרָאֵל אֶל־יְהוָ֑ה מֵֽ֝עַתָּ֗ה וְעַד־עֹולָֽם׃\n", " \n" ] } ], "source": [ "print(chapter2text(\"Psalmi\", 131))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Bi-grams\n", "\n", "We make a column of verse-bound bi-grams of lexemes. The two lexemes are separated by an underscore `_`." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "vsNext = frame[frame.otype == \"word\"].in_verse\n", "vsPrev = frame[frame.otype == \"word\"].in_verse.shift(1)\n", "lex = frame[frame.otype == \"word\"].lex_utf8\n", "lexNext = frame[frame.otype == \"word\"].lex_utf8.shift(1)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "lastInVs = vsPrev != vsNext\n", "lexNext[lastInVs] = \"\"" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "bigram = [\"{}_{}\".format(*p) for p in zip(lex, lexNext)]" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['אשׁה_אם',\n", " 'מן_אשׁה',\n", " 'ארץ_מן',\n", " 'מצרים_ארץ',\n", " 'ו_',\n", " 'היה_ו',\n", " 'ב_היה',\n", " 'ה_ב',\n", " 'עת_ה',\n", " 'ה_עת',\n", " 'היא_ה',\n", " 'ו_היא',\n", " 'אמר_ו',\n", " 'אבימלך_אמר',\n", " 'ו_אבימלך',\n", " 'פיכל_ו',\n", " 'שׂר_פיכל',\n", " 'צבא_שׂר',\n", " 'אל_צבא',\n", " 'אברהם_אל',\n", " 'ל_אברהם',\n", " 'אמר_ל',\n", " 'אלהים_אמר',\n", " 'עם_אלהים',\n", " 'ב_עם',\n", " 'כל_ב',\n", " 'אשׁר_כל',\n", " 'אתה_אשׁר',\n", " 'עשׂה_אתה',\n", " 'ו_']" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bigram[10_000:10_030]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.1" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }