"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"A = use(\n",
" \"CLARIAH/wp6-missieven:clone\",\n",
" checkout=\"clone\",\n",
" mod=\"CLARIAH/wp6-missieven/voc-missives/migrated/tf:clone\",\n",
" hoist=globals(),\n",
" version=\"1.0\",\n",
" silent=\"verbose\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that you can click the triangle before *CLARIAH/wp6-missieven/voc-missives/migrated/tf*,\n",
"to see which features are used.\n",
"You can then click further on the triangle before the feature data type, to see more information\n",
"about that feature, including the fact that it is an upgraded feature.\n",
"\n",
"```\n",
"creator: Sophie Arnoult\n",
"dateWritten: 2022-10-11T10:42:45Z\n",
"upgraded: ‼️ from version 0.8.1 to 1.0\n",
"writtenBy: Text-Fabric\n",
"```\n",
"\n",
"We are going to do a bit of research into the upgraded features."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(('e_n12_2_632', 8),\n",
" ('e_n13_15_2306', 8),\n",
" ('e_n7_8_809', 8),\n",
" ('e_n13_15_1302', 7),\n",
" ('e_n7_8_1080', 7),\n",
" ('e_t10_15_108', 7),\n",
" ('e_t10_15_273', 7),\n",
" ('e_n10_11_715', 6),\n",
" ('e_n12_14_130', 6),\n",
" ('e_n12_2_383', 6),\n",
" ('e_n12_2_578', 6),\n",
" ('e_n13_15_154', 6),\n",
" ('e_n13_15_1582', 6),\n",
" ('e_n13_15_1894', 6),\n",
" ('e_n13_15_285', 6),\n",
" ('e_n5_28_103', 6),\n",
" ('e_n5_28_34', 6),\n",
" ('e_n5_28_675', 6),\n",
" ('e_n5_7_515', 6),\n",
" ('e_n8_6_710', 6))"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.entityId.freqList()[0:20]"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"24500"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(F.entityId.freqList())"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(('LOC', 12790),\n",
" ('PER', 10393),\n",
" ('LOCderiv', 4279),\n",
" ('ORG', 3841),\n",
" ('SHP', 2922),\n",
" ('GPE', 1153),\n",
" ('RELderiv', 261),\n",
" ('ORGpart', 58),\n",
" ('LOCpart', 45),\n",
" ('RELpart', 28),\n",
" ('REL', 19))"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.entityKind.freqList()"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 1.96s 32249 results\n"
]
}
],
"source": [
"query = \"\"\"\n",
"word entityId entityKind*\n",
"\"\"\"\n",
"results = A.search(query)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"line 1"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
PIETER
entityId=e_t1_49_0entityKind=PER
DE
entityId=e_t1_49_0entityKind=PER
CARPENTIER,
entityId=e_t1_49_0entityKind=PER
JACOB
entityId=e_t1_49_1entityKind=PER
DEDEL,
entityId=e_t1_49_1entityKind=PER
CORNELIS
entityId=e_t1_49_2entityKind=PER
REYERSZ.
entityId=e_t1_49_2entityKind=PER
line 2"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
ABRAHAM
entityId=e_t1_49_3entityKind=PER
VAN
entityId=e_t1_49_3entityKind=PER
UFFELEN,
entityId=e_t1_49_3entityKind=PER
JAKATRA
entityId=e_t1_49_4entityKind=LOC
line 3"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Westcuste
entityId=e_t1_49_5entityKind=LOC
van
entityId=e_t1_49_5entityKind=LOC
Sumatra
entityId=e_t1_49_5entityKind=LOC
line 4"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Maleis
entityId=e_n1_49_0entityKind=LOCderiv
Atjeh.
entityId=e_n1_49_1entityKind=LOC
line 5"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Atchinder
entityId=e_t1_49_6entityKind=LOCderiv
line 6"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Atchin
entityId=e_t1_49_7entityKind=LOC
line 7"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Casembroot
entityId=e_t1_49_8entityKind=PER
line 8"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Engels
entityId=e_t1_49_9entityKind=LOCderiv
d’Unite
entityId=e_t1_49_10entityKind=SHP
line 9"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
France
entityId=e_t1_49_11entityKind=LOCderiv
Montmorenci
entityId=e_t1_49_12entityKind=SHP
De
entityId=e_n1_49_2entityKind=PER
Montmorency
entityId=e_n1_49_2entityKind=PER
de
entityId=e_n1_49_3entityKind=PER
Beaulieu (
entityId=e_n1_49_3entityKind=PER
line 10"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Atjeh,
entityId=e_n1_49_4entityKind=LOC
Maleise
entityId=e_n1_49_5entityKind=LOCderiv
Data: WP6-MISSIEVEN, Character table, Feature docs
Features:
\n",
"CLARIAH/wp6-missieven/voc-missives/export/tf
\n",
" \n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" identifier of a named entity
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" kind of a named entity
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
General Missives Dutch East India Company 1600-1800
\n",
" \n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" authors of the letter, surnames only
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" authors of the letter, full names
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" column number of a column in a row in a table
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" day part of the date of the letter
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word is the denominator in fraction, e.g. 4 in 1/4
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" whether a word is emphasized by typography
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" a folio reference
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word belongs to footnote text
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word is the numerator in fraction, e.g. 1 in 1/4
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word belongs to original text
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word is a numerical fraction, e.g. 1/4
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word belongs to the text of reference
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word belongs to the text of editorial remarks
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word has special typography possibly with OCR mistakes as well
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word has subscript typography possibly indicating the denominator of a fraction
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" whether a word has superscript typography possibly indicating the numerator of a fraction
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" whether a word is underlined by typography
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" footnote mark (not necessarily the same as shown on the printed page
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" month part of the date of the letter
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" number of a volume, letter, page, para, line, table
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
"
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" number of the first page of this letter in this volume
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" place from where the letter was sent
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" punctuation and/or whitespace following a wordup to the next word
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" punctuation and/or whitespace following a word,up to the next word, footnote text only
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" punctuation and/or whitespace following a word,up to the next word, original text only
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" punctuation and/or whitespace following a word,up to the next word, remark text only
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" the date the letter was sent
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" row number of a row of column in a table
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" ('sequence number of this letter among the letters of the same author in this volume',)
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" status of the letter, e.g. secret, copy
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" title of the letter
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" transcription of a word
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" transcription of a word, only for footnote text
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" transcription of a word, only for original text
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" transcription of a word, only for remark text
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" volume number
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
str
\n",
"\n",
"
\n",
" the page-specific part of web links for page nodes
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" column offset of a column in a row in a table
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
int
\n",
"\n",
"
\n",
" year part of the date of the letter
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
none
\n",
"\n",
"
\n",
" edge between a word and the footnotes associated with it
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
\n",
"
\n",
"
none
\n",
"\n",
"
\n",
"
\n",
" \n",
" \n",
"\n",
"
\n",
"\n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"A = use(\n",
" \"CLARIAH/wp6-missieven:clone\",\n",
" checkout=\"clone\",\n",
" mod=\"CLARIAH/wp6-missieven/voc-missives/export/tf:clone\",\n",
" hoist=globals(),\n",
" version=\"1.0\",\n",
" silent=\"verbose\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 2.46s 29159 results\n"
]
},
{
"data": {
"text/plain": [
"Counter({1: 2208,\n",
" 2: 1799,\n",
" 3: 2297,\n",
" 4: 1987,\n",
" 5: 1858,\n",
" 6: 4295,\n",
" 7: 3068,\n",
" 8: 1251,\n",
" 9: 2825,\n",
" 10: 1745,\n",
" 11: 2455,\n",
" 12: 1350,\n",
" 13: 977,\n",
" 14: 1044})"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results = A.search(query)\n",
"\n",
"eDist = collections.Counter()\n",
"\n",
"for (vol, word) in results:\n",
" eDist[F.n.v(vol)] += 1\n",
" \n",
"eDist"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Clearly, there have been additional changes leading to a very different version 1.0 than 0.8.1,\n",
"so at this point in time the migrated features (from 0.8.1 to 1.0) are practically obsolete."
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"---\n",
"\n",
"# Contents\n",
"\n",
"* **[start](start.ipynb)** start computing with this corpus\n",
"* **[search](search.ipynb)** turbo charge your hand-coding with search templates\n",
"* **[compute](compute.ipynb)** sink down a level and compute it yourself\n",
"* **[exportExcel](exportExcel.ipynb)** make tailor-made spreadsheets out of your results\n",
"* **[annotate](annotate.ipynb)** export text, annotate with BRAT, import annotations\n",
"* **[share](share.ipynb)** draw in other people's data and let them use yours\n",
"* **[entities](entities.ipynb)** use results of third-party NER (named entity recognition)\n",
"* **porting** port features made against an older version to a newer version\n",
"* **[volumes](volumes.ipynb)** work with selected volumes only\n",
"\n",
"CC-BY Dirk Roorda"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.7"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
![](\"images/tf.png\")
\n",
"![](\"images/huc.png\")
\n",
"![](\"images/logo.png\"){kind=logo}
\n",
"\n",
"---\n",
"\n",
"To get started: consult [start](start.ipynb)\n",
"\n",
"---\n",
"\n",
"# Porting annotations\n",
"\n",
"In the [entities](entities.ipynb) notebook we saw how we could use third-party features\n",
"with our corpus. There we promised to show how we can upgrade such features so that they\n",
"also work against newer versions of the corpus.\n",
"\n",
"Text-Fabric has machinery to help with that. It turns out that we have\n",
"to make a mapping between the slots of both versions, and then Text-Fabric can do the rest.\n",
"\n",
"With that mapping in hand, we can port *all* features, past, present and future, \n",
"automatically from the older version to the newer, and vice versa.\n",
"\n",
"But: there will be imperfections, unavoidably."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2018-05-24T10:06:39.818664Z",
"start_time": "2018-05-24T10:06:39.796588Z"
}
},
"outputs": [],
"source": [
"import collections\n",
"\n",
"from tf.app import use"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Make a slot mapping\n",
"\n",
"In this notebook we map the *slot* nodes from version 0.8.1 (source version) to 1.0 (target version).\n",
"\n",
"Basically this means that we map all slots from the source version to corresponding slots in the target version.\n",
"\n",
"Some slots have an empty text (most of them contain some punctuation).\n",
"\n",
"We do not want to be fussy about those slots.\n",
"We map them unto corresponding empty slots if possible, otherwise we map them onto the nearest\n",
"non-empty slot.\n",
"\n",
"After establishing the slot mapping, we extend the mapping to all nodes in a generic way.\n",
"The code for this is already in the TF library.\n",
"\n",
"Note that in this stage we do not need the entity features at all. All we do is\n",
"to compare two version of the base corpus."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from tf.dataset import Versions\n",
"\n",
"va = \"0.8.1\"\n",
"vb = \"1.0\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We load the data for both versions.\n",
"\n",
"This time, we work in our GitHub clone, because we want to make the resulting\n",
"map available to everyone, after pushing the clone to GitHub."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"A = {}"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
"