"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"A = use(\n",
" f\"CLARIAH/wp6-missieven\",\n",
" hoist=globals(),\n",
" mod=(\n",
" f\"CLARIAH/wp6-missieven/exercises/entities/tf\",\n",
" f\"CLARIAH/wp6-missieven/exercises/numerics/tf\",\n",
" ),\n",
" version=VERSION,\n",
" silent=False,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Above you see a new sections in the feature list that you can expand to see\n",
"which features that module contributed.\n",
"\n",
"Now, suppose did not know much about these feature, then we would like to do a few basic checks.\n",
"\n",
"A good start it to do inspect a frequency list of the values of the new features,\n",
"and then to perform a query looking for the nodes that have these features.\n",
"\n",
"We do that for the entity features and for the number feature."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Entities"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(('T11', 6),\n",
" ('T2', 5),\n",
" ('T13', 3),\n",
" ('T16', 3),\n",
" ('T8', 3),\n",
" ('T9', 3),\n",
" ('T10', 2),\n",
" ('T15', 2),\n",
" ('T17', 2),\n",
" ('T3', 2),\n",
" ('T5', 2),\n",
" ('T1', 1),\n",
" ('T12', 1),\n",
" ('T4', 1),\n",
" ('T6', 1),\n",
" ('T7', 1))"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.entityId.freqList()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(('Person', 18), ('GPE', 15), ('Organization', 5))"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.entityKind.freqList()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(('Ternate', 5), ('Amboina', 2))"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.entityComment.freqList()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's query all words that have an entity notation:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 4.40s 23 results\n"
]
}
],
"source": [
"query = \"\"\"\n",
"word entityId entityKind* entityComment*\n",
"\"\"\"\n",
"results = A.search(query)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we query all word where the `entityId` is present.\n",
"We also mention the `entityKind` and `entityComment` features, but with a `*` behind them.\n",
"That is a criterion that is always True, so these mentions do not alter the result list.\n",
"But now these features do occur in the query, and when we show results, these features will be shown."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"line 1"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T11entityKind=Person
orancay
entityId=T11entityKind=Person
line 2"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T11entityKind=Person
bij
entityId=T11entityKind=Person
line 3"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T13entityKind=Person
arancay
entityId=T12entityKind=Person
Nera,
entityId=T1entityKind=Person
sabandaer
entityId=T13entityKind=Person
line 4"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T11entityKind=Person
orancaye
entityId=T11entityKind=Person
line 5"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T15entityKind=GPE
Hollanders
entityId=T15entityKind=GPE
line 6"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T16entityKind=Person
Verhoeven
entityId=T16entityKind=Person
line 7"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T17entityKind=GPE
Ambojna
entityId=T17entityKind=GPE
Ternnate
entityId=T2entityKind=GPE
Coninck
entityId=T3entityKind=Person
van
entityId=T3entityKind=Person
Spagnien
entityId=T4entityKind=GPE
line 8"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T2entityKind=GPE
Heeren
entityId=T5entityKind=Organization
Staeten
entityId=T5entityKind=Organization
Coninck
entityId=T6entityKind=Person
ditto
entityId=T2entityKind=GPE
line 9"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T2entityKind=GPE
plaetse
entityId=T2entityKind=GPE
line 10"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T8entityKind=Organization
Coninck
entityId=T7entityKind=Person
Heeren
entityId=T8entityKind=Organization
Staeten,
entityId=T8entityKind=Organization
line 11"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T9entityKind=GPE
Banda
entityId=T9entityKind=GPE
line 12"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
entityId=T10entityKind=GPE
Engelsche
entityId=T10entityKind=GPE
result 1"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
115J,
number=1151transo=115J
result 2"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
loopende),
transo=loopende
deminueert
transo=deminueert
d’advance
transo=d’advance
result 3"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
9400J
number=94001transo=9400J
waerschouwingh
transo=waerschouwingh
toecomende
transo=toecomende
result 4"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
coopmanschappen,
transo=coopmanschappen
result 5"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
row
cell
Eerstelijck:
transo=Eerstelijck
cell
cargasoen,
transo=cargasoen
bestaende
transo=bestaende
lijnwaten
transo=lijnwaten
diamanten,
transo=diamanten
gescheept,
transo=gescheept
Masilipatnam
transo=Masilipatnam
aengelandt ;
transo=aengelandt
result 6"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
row
cell
2062J
number=20621transo=2062J
cell
capitael,
transo=capitael
result 7"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
vercocht,
transo=vercocht
result 8"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
d’affgedrongen
transo=d’affgedrongen
result 9"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
12818J,
number=128181transo=12818J
overgaen.
transo=overgaen
result 10"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"line
Masilipatnam
transo=Masilipatnam
124J
number=1241transo=124J
besarsteen,
transo=besarsteen
![](\"images/tf.png\")
\n",
"![](\"images/huc.png\")
\n",
"![](\"images/logo.png\"){kind=logo}
\n",
"\n",
"---\n",
"\n",
"To get started: consult [start](start.ipynb)\n",
"\n",
"---\n",
"\n",
"# Sharing data features\n",
"\n",
"## Explore additional data\n",
"\n",
"Once you analyse a corpus, it is likely that you produce data that others can reuse.\n",
"Maybe you have defined a set of proper name occurrences, or special numerals, or you have computed part-of-speech assignments.\n",
"\n",
"It is possible to turn these insights into *new features*, i.e. new `.tf` files with values assigned to specific nodes.\n",
"\n",
"## Make your own data\n",
"\n",
"New data is a product of your own methods and computations in the first place.\n",
"But how do you turn that data into new TF features?\n",
"It turns out that the last step is not that difficult.\n",
"\n",
"If you can shape your data as a mapping (dictionary) from node numbers (integers) to values\n",
"(strings or integers), then TF can turn that data into a feature file for you with one command.\n",
"\n",
"## Share your new data\n",
"You can then easily share your new features on GitHub, so that your colleagues everywhere\n",
"can try it out for themselves.\n",
"\n",
"You can add such data on the fly, by passing a `mod={org}/{repo}/{path}` parameter,\n",
"or a bunch of them separated by commas.\n",
"\n",
"If the data is there, it will be auto-downloaded and stored on your machine.\n",
"\n",
"Let's do it.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2018-05-24T10:06:39.818664Z",
"start_time": "2018-05-24T10:06:39.796588Z"
}
},
"outputs": [],
"source": [
"import re\n",
"import collections\n",
"import os\n",
"\n",
"from tf.app import use"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"TF-app: ~/text-fabric-data/github/CLARIAH/wp6-missieven/app"
],
"text/plain": [
"