{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction\n",
"Udapi is an API and framework for processing [Universal Dependencies](http://universaldependencies.org/). In this tutorial, we will focus on the Python version of Udapi. Perl and Java versions are [available](http://udapi.github.io/) as well, but they are missing some of the features.\n",
"\n",
"Udapi can be used from the shell (e.g. Bash), using the wrapper script `udapy`. It can be also used as a library, from Python, IPython or Jupyter notebooks. We will show both of these ways bellow.\n",
"\n",
"This tutorial uses Details sections for extra info (if you want to know more or if you run into problems). You need to click on it to show its content.\n",
"Details
\n",
"It is a substitute for footnotes. The content may be long and showing it in the main text may be distracting.\n",
" \n",
"\n",
"### Install (upgrade) Udapi\n",
"First, make sure you have the newest version of Udapi. If you have already installed Udapi [using git clone](https://github.com/udapi/udapi-python#install-udapi-for-developers), just run `git pull`. If you have not installed Udapi yet, run\n",
"Details
\n",
"\n",
" - The command below installs Udapi from GitHub (from the master branch). With
pip3 install --user --upgrade udapi
, you can install the last version released on PyPI (possibly older).\n",
" - The exclamation mark (!) in Jupyter or IPython means that the following command will be executed by the system shell (e.g. Bash).\n",
"
\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip3 install --user --upgrade git+https://github.com/udapi/udapi-python.git\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, make sure you can run the command-line interface `udapy`, e.g. by printing the help message."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"usage: udapy [optional_arguments] scenario\r\n",
"\r\n",
"udapy - Python interface to Udapi - API for Universal Dependencies\r\n",
"\r\n",
"Examples of usage:\r\n",
" udapy -s read.Sentences udpipe.En < in.txt > out.conllu\r\n",
" udapy -T < sample.conllu | less -R\r\n",
" udapy -HAM ud.MarkBugs < sample.conllu > bugs.html\r\n",
"\r\n",
"positional arguments:\r\n",
" scenario A sequence of blocks and their parameters.\r\n",
"\r\n",
"optional arguments:\r\n",
" -h, --help show this help message and exit\r\n",
" -q, --quiet Warning, info and debug messages are suppressed. Only fatal errors are reported.\r\n",
" -v, --verbose Warning, info and debug messages are printed to the STDERR.\r\n",
" -s, --save Add write.Conllu to the end of the scenario\r\n",
" -T, --save_text_mode_trees\r\n",
" Add write.TextModeTrees color=1 to the end of the scenario\r\n",
" -H, --save_html Add write.TextModeTreesHtml color=1 to the end of the scenario\r\n",
" -A, --save_all_attributes\r\n",
" Add attributes=form,lemma,upos,xpos,feats,deprel,misc (to be used after -T and -H)\r\n",
" -C, --save_comments Add print_comments=1 (to be used after -T and -H)\r\n",
" -M, --marked_only Add marked_only=1 to the end of the scenario (to be used after -T and -H)\r\n",
" -N, --no_color Add color=0 to the end of the scenario, this overrides color=1 of -T and -H\r\n",
"\r\n",
"See http://udapi.github.io\r\n"
]
}
],
"source": [
"!udapy -h"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Details: If the previous command fails with \"udapy: command not found\"
\n",
"This means that Udapi is not properly installed. When installing Udapi with pip3 --user
, it is installed into ~/.local/lib/python3.6/site-packages/udapi/
(or similar depending on your Python version) and the wrapper into ~/.local/bin
. Thus you need to\n",
"\n",
"export PATH=\"$HOME/.local/bin/:$PATH\"\n",
"
\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Browse CoNLL-U files\n",
"### Get sample UD data\n",
"\n",
"Download and extract [ud20sample.tgz](http://ufal.mff.cuni.cz/~popel/udapi/ud20sample.tgz). There are just 100 sentences for each of the 70 treebanks (`sample.conllu`), plus 4 bigger files (`train.conllu` and `dev.conllu`) for German, English, French and Czech. For full UD ([2.0](https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1983) or [newer](https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3424)), go to [Lindat](https://lindat.cz)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2020-12-01 07:53:37-- http://ufal.mff.cuni.cz/~popel/udapi/ud20sample.tgz\n",
"Resolving ufal.mff.cuni.cz (ufal.mff.cuni.cz)... 195.113.20.52\n",
"Connecting to ufal.mff.cuni.cz (ufal.mff.cuni.cz)|195.113.20.52|:80... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 4670982 (4,5M) [application/x-gzip]\n",
"Saving to: ‘ud20sample.tgz.1’\n",
"\n",
"ud20sample.tgz.1 100%[===================>] 4,45M 1,49MB/s in 3,0s \n",
"\n",
"2020-12-01 07:53:40 (1,49 MB/s) - ‘ud20sample.tgz.1’ saved [4670982/4670982]\n",
"\n",
"/home/martin/udapi/python/notebook/sample\n"
]
}
],
"source": [
"!wget http://ufal.mff.cuni.cz/~popel/udapi/ud20sample.tgz\n",
"!tar -xf ud20sample.tgz\n",
"%cd sample"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's choose one of the sample files and see the raw [CoNLL-U format](https://universaldependencies.org/format.html).\n",
"Details: executing from Bash, IPython, Jupyter
\n",
"\n",
"- If you see \"No such file or directory\" error, make sure you executed the previous cell. Note that the
cd
command is not prefixed by an exclamation mark because that would run in a sub-shell, which \"forgets\" the changed directory when finished. It is prefixed by a percent sign, which marks it as IPython magic.\n",
" cat
is another IPython magic command, this time an alias for the shell command of the same name (so you can prefix cat
with an exclamation mark, if you prefer), which prints a given file. With automagic
on, you can use it without the percent sign.\n",
"- In this tutorial, we use
| head
to show just the first 10 lines of the output (preventing thus big ipynb file size). You can ignore the \"cat: write error: Broken pipe\" warning.\n",
" - When using Jupyter, you can omit the
| head
because long outputs are automatically wrapped in a text box with a scrollbar.\n",
" - When running this from IPython or Bash, you can use a pager:
less UD_Ancient_Greek/sample.conllu
\n",
"
\n",
" \n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# newdoc id = tlg0008.tlg001.perseus-grc1.13.tb.xml\r\n",
"# sent_id = tlg0008.tlg001.perseus-grc1.13.tb.xml@1144\r\n",
"# text = ἐρᾷ μὲν ἁγνὸς οὐρανὸς τρῶσαι χθόνα, ἔρως δὲ γαῖαν λαμβάνει γάμου τυχεῖν·\r\n",
"1\tἐρᾷ\tἐράω\tVERB\tv3spia---\tMood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin|Voice=Act\t0\troot\t_\t_\r\n",
"2\tμὲν\tμέν\tADV\td--------\t_\t1\tadvmod\t_\t_\r\n",
"3\tἁγνὸς\tἁγνός\tADJ\ta-s---mn-\tCase=Nom|Gender=Masc|Number=Sing\t4\tnmod\t_\t_\r\n",
"4\tοὐρανὸς\tοὐρανός\tNOUN\tn-s---mn-\tCase=Nom|Gender=Masc|Number=Sing\t1\tnsubj\t_\t_\r\n",
"5\tτρῶσαι\tτιτρώσκω\tVERB\tv--ana---\tTense=Past|VerbForm=Inf|Voice=Act\t1\txcomp\t_\t_\r\n",
"6\tχθόνα\tχθών\tNOUN\tn-s---fa-\tCase=Acc|Gender=Fem|Number=Sing\t5\tobj\t_\tSpaceAfter=No\r\n",
"7\t,\t,\tPUNCT\tu--------\t_\t1\tpunct\t_\t_\r\n",
"cat: write error: Broken pipe\r\n"
]
}
],
"source": [
"cat UD_Ancient_Greek/sample.conllu | head"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Browse conllu files with `udapy -T`\n",
"While the CoNLL-U format was designed with readibility (by both machines and humans) on mind, it may be still a bit difficult to read and interpret by humans. Let's visualize the dependency tree structure using ASCII-art by piping the conllu file into `udapy -T`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2020-12-01 08:00:33,276 [ INFO] execute - No reader specified, using read.Conllu\n",
"2020-12-01 08:00:33,276 [ INFO] execute - ---- ROUND ----\n",
"2020-12-01 08:00:33,276 [ INFO] execute - Executing block Conllu\n",
"2020-12-01 08:00:33,305 [ INFO] execute - Executing block TextModeTrees\n",
"docname = tlg0008.tlg001.perseus-grc1.13.tb.xml\n",
"loaded_from = -\n",
"# sent_id = tlg0008.tlg001.perseus-grc1.13.tb.xml@1144\n",
"# text = ἐρᾷ μὲν ἁγνὸς οὐρανὸς τρῶσαι χθόνα, ἔρως δὲ γαῖαν λαμβάνει γάμου τυχεῖν·\n",
"─┮\n",
" ╰─┮ \u001b[33mἐρᾷ\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mroot\u001b[0m\n",
" ┡─╼ \u001b[33mμὲν\u001b[0m \u001b[31mADV\u001b[0m \u001b[34madvmod\u001b[0m\n",
" │ ╭─╼ \u001b[33mἁγνὸς\u001b[0m \u001b[31mADJ\u001b[0m \u001b[34mnmod\u001b[0m\n",
" ┡─┶ \u001b[33mοὐρανὸς\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mnsubj\u001b[0m\n",
" ┡─┮ \u001b[33mτρῶσαι\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mxcomp\u001b[0m\n",
" │ ╰─╼ \u001b[33mχθόνα\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobj\u001b[0m\n",
" ┡─╼ \u001b[33m,\u001b[0m \u001b[31mPUNCT\u001b[0m \u001b[34mpunct\u001b[0m\n",
" │ ╭─╼ \u001b[33mἔρως\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mnsubj\u001b[0m\n",
" ┡─╼ \u001b[33mδὲ\u001b[0m \u001b[31mCCONJ\u001b[0m \u001b[34mcc\u001b[0m │\n",
" │ ┢─╼ \u001b[33mγαῖαν\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobj\u001b[0m\n",
" ┡───────────────┾ \u001b[33mλαμβάνει\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mconj\u001b[0m\n",
" │ │ ╭─╼ \u001b[33mγάμου\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobj\u001b[0m\n",
" │ ╰─┶ \u001b[33mτυχεῖν\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mxcomp\u001b[0m\n",
" ╰─╼ \u001b[33m·\u001b[0m \u001b[31mPUNCT\u001b[0m \u001b[34mpunct\u001b[0m\n",
"\n"
]
}
],
"source": [
"cat UD_Ancient_Greek/sample.conllu | udapy -T | head -n 20"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Details:
\n",
"\n",
"- You may be used to see dependency trees where the root node is on the top and words are ordered horizontally (left to right). Here, the root is on left and words are ordered vertically (top to bottom).\n",
"
- The colors are implemented using the colorama package and ANSI escape codes. When running this from IPython or Bash and using
less
, you need to instruct it to display the colors with -R
:\n",
"\n",
"cat UD_Ancient_Greek/sample.conllu | udapy -T | less -R\n",
"
\n",
" - You can also use
udapy -T -N
to disable the colors.\n",
" udapy -q
suppresses all Udapi messages (warnings, info, debug) printed on the standard error output, so only fatal errors are printed. By default only debug messages are suppresses, but these can be printed with udapy -v
.\n",
"- But you already know this because you have read
udapy -h
, am I right?\n",
"
\n",
" \n",
"\n",
"`udapy -T` is a shortcut for `udapy write.TextModeTrees color=1`, where `write.TextModeTrees` is a so-called *block* (a basic Udapi processing unit) and `color=1` is its parameter. See [the documentation](https://udapi.readthedocs.io/en/latest/udapi.block.write.html#module-udapi.block.write.textmodetrees) (or even [the source code](https://github.com/udapi/udapi-python/blob/master/udapi/block/write/textmodetrees.py) of `write.TextModeTrees` to learn about further parameters. Now, let's print also the LEMMA and MISC columns and display the columns vertically aligned using parameters `layout=align attributes=form,lemma,upos,deprel,misc`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"docname = tlg0008.tlg001.perseus-grc1.13.tb.xml\r\n",
"loaded_from = -\r\n",
"# sent_id = tlg0008.tlg001.perseus-grc1.13.tb.xml@1144\r\n",
"# text = ἐρᾷ μὲν ἁγνὸς οὐρανὸς τρῶσαι χθόνα, ἔρως δὲ γαῖαν λαμβάνει γάμου τυχεῖν·\r\n",
"─┮ \r\n",
" ╰─┮ \u001b[33mἐρᾷ\u001b[0m \u001b[36mἐράω\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mroot\u001b[0m _\u001b[0m\r\n",
" ┡─╼ \u001b[33mμὲν\u001b[0m \u001b[36mμέν\u001b[0m \u001b[31mADV\u001b[0m \u001b[34madvmod\u001b[0m _\u001b[0m\r\n",
" │ ╭─╼ \u001b[33mἁγνὸς\u001b[0m \u001b[36mἁγνός\u001b[0m \u001b[31mADJ\u001b[0m \u001b[34mnmod\u001b[0m _\u001b[0m\r\n",
" ┡─┶ \u001b[33mοὐρανὸς\u001b[0m \u001b[36mοὐρανός\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mnsubj\u001b[0m _\u001b[0m\r\n",
" ┡─┮ \u001b[33mτρῶσαι\u001b[0m \u001b[36mτιτρώσκω\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mxcomp\u001b[0m _\u001b[0m\r\n",
" │ ╰─╼ \u001b[33mχθόνα\u001b[0m \u001b[36mχθών\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobj\u001b[0m SpaceAfter=No\u001b[0m\r\n",
" ┡─╼ \u001b[33m,\u001b[0m \u001b[36m,\u001b[0m \u001b[31mPUNCT\u001b[0m \u001b[34mpunct\u001b[0m _\u001b[0m\r\n",
" │ ╭─╼ \u001b[33mἔρως\u001b[0m \u001b[36mἔρως\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mnsubj\u001b[0m _\u001b[0m\r\n",
" ┡─╼ │ \u001b[33mδὲ\u001b[0m \u001b[36mδέ\u001b[0m \u001b[31mCCONJ\u001b[0m \u001b[34mcc\u001b[0m _\u001b[0m\r\n",
" │ ┢─╼ \u001b[33mγαῖαν\u001b[0m \u001b[36mγαῖα\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobj\u001b[0m _\u001b[0m\r\n",
" ┡───┾ \u001b[33mλαμβάνει\u001b[0m \u001b[36mλαμβάνω\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mconj\u001b[0m _\u001b[0m\r\n",
" │ │ ╭─╼ \u001b[33mγάμου\u001b[0m \u001b[36mγάμος\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobj\u001b[0m _\u001b[0m\r\n",
" │ ╰─┶ \u001b[33mτυχεῖν\u001b[0m \u001b[36mτυγχάνω\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mxcomp\u001b[0m SpaceAfter=No\u001b[0m\r\n",
" ╰─╼ \u001b[33m·\u001b[0m \u001b[36m·\u001b[0m \u001b[31mPUNCT\u001b[0m \u001b[34mpunct\u001b[0m _\u001b[0m\r\n",
"\r\n"
]
}
],
"source": [
"cat UD_Ancient_Greek/sample.conllu | udapy -q write.TextModeTrees color=1 layout=align attributes=form,lemma,upos,deprel,misc | head -n 20"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Browse conllu files from IPython/Jupyter\n",
"So far, we were using Udapi only via its command-line interface `udapy`, which is handy, but not very Pythonic. So let's now use Udapi as a library and load the English conllu sample file into a document `doc` and visualize the sixth tree (i.e. `doc[5]` in zero-based indexing)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0006\n",
"# text = The third was being run by the head of an investment firm.\n",
"─┮\n",
" │ ╭─╼ \u001b[33mThe\u001b[0m \u001b[31mDET\u001b[0m \u001b[34mdet\u001b[0m\n",
" │ ╭─┶ \u001b[33mthird\u001b[0m \u001b[31mADJ\u001b[0m \u001b[34mnsubj:pass\u001b[0m\n",
" │ ┢─╼ \u001b[33mwas\u001b[0m \u001b[31mAUX\u001b[0m \u001b[34maux\u001b[0m\n",
" │ ┢─╼ \u001b[33mbeing\u001b[0m \u001b[31mAUX\u001b[0m \u001b[34maux:pass\u001b[0m\n",
" ╰─┾ \u001b[33mrun\u001b[0m \u001b[31mVERB\u001b[0m \u001b[34mroot\u001b[0m\n",
" │ ╭─╼ \u001b[33mby\u001b[0m \u001b[31mADP\u001b[0m \u001b[34mcase\u001b[0m\n",
" │ ┢─╼ \u001b[33mthe\u001b[0m \u001b[31mDET\u001b[0m \u001b[34mdet\u001b[0m\n",
" ┡─┾ \u001b[33mhead\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mobl\u001b[0m\n",
" │ │ ╭─╼ \u001b[33mof\u001b[0m \u001b[31mADP\u001b[0m \u001b[34mcase\u001b[0m\n",
" │ │ ┢─╼ \u001b[33man\u001b[0m \u001b[31mDET\u001b[0m \u001b[34mdet\u001b[0m\n",
" │ │ ┢─╼ \u001b[33minvestment\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mcompound\u001b[0m\n",
" │ ╰─┶ \u001b[33mfirm\u001b[0m \u001b[31mNOUN\u001b[0m \u001b[34mnmod\u001b[0m\n",
" ╰─╼ \u001b[33m.\u001b[0m \u001b[31mPUNCT\u001b[0m \u001b[34mpunct\u001b[0m\n",
"\n"
]
}
],
"source": [
"import udapi\n",
"doc = udapi.Document(\"UD_English/sample.conllu\")\n",
"doc[5].draw()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Details:
\n",
"\n",
"doc = udapi.Document(filename)
is a shortcut for\n",
"\n",
"import udapi.core.document\n",
"doc = udapi.core.document.Document(filename)\n",
"
\n",
"- We can print the whole document using
doc.draw()
.\n",
" doc.draw(**kwargs)
is a shortcut for creating a write.TextModeTrees
block and applying it on the document:\n",
"\n",
"import udapi.block.write.textmodetrees\n",
"block = udapi.block.write.textmodetrees.TextModeTrees(**kwargs)\n",
"block.run(doc)\n",
"
\n",
"
\n",
" \n",
"\n",
"The `draw()` method takes the same parameters as the `write.TextModeTrees` block, so we can for example display only the node ID (aka `ord`, i.e. word-order index), form and [universal (morpho-syntactic) features](https://universaldependencies.org/u/feat/index.html).\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0006\n",
"# text = The third was being run by the head of an investment firm.\n",
"─┮ \n",
" │ ╭─╼ \u001b[32m1\u001b[0m \u001b[33mThe\u001b[0m Definite=Def|PronType=Art\u001b[0m\n",
" │ ╭─┶ \u001b[32m2\u001b[0m \u001b[33mthird\u001b[0m Degree=Pos|NumType=Ord\u001b[0m\n",
" │ ┢─╼ \u001b[32m3\u001b[0m \u001b[33mwas\u001b[0m Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin\u001b[0m\n",
" │ ┢─╼ \u001b[32m4\u001b[0m \u001b[33mbeing\u001b[0m VerbForm=Ger\u001b[0m\n",
" ╰─┾ \u001b[32m5\u001b[0m \u001b[33mrun\u001b[0m Tense=Past|VerbForm=Part|Voice=Pass\u001b[0m\n",
" │ ╭─╼ \u001b[32m6\u001b[0m \u001b[33mby\u001b[0m _\u001b[0m\n",
" │ ┢─╼ \u001b[32m7\u001b[0m \u001b[33mthe\u001b[0m Definite=Def|PronType=Art\u001b[0m\n",
" ┡─┾ \u001b[32m8\u001b[0m \u001b[33mhead\u001b[0m Number=Sing\u001b[0m\n",
" │ │ ╭─╼ \u001b[32m9\u001b[0m \u001b[33mof\u001b[0m _\u001b[0m\n",
" │ │ ┢─╼ \u001b[32m10\u001b[0m \u001b[33man\u001b[0m Definite=Ind|PronType=Art\u001b[0m\n",
" │ │ ┢─╼ \u001b[32m11\u001b[0m \u001b[33minvestment\u001b[0m Number=Sing\u001b[0m\n",
" │ ╰─┶ \u001b[32m12\u001b[0m \u001b[33mfirm\u001b[0m Number=Sing\u001b[0m\n",
" ╰─╼ \u001b[32m13\u001b[0m \u001b[33m.\u001b[0m _\u001b[0m\n",
"\n"
]
}
],
"source": [
"doc[5].draw(layout=\"align\", attributes=\"ord,form,feats\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Document representation in Udapi\n",
"\n",
"Udapi [document](https://github.com/udapi/udapi-python/blob/master/udapi/core/document.py) consists of a sequence of so-called *bundles*, mirroring a sequence of sentences in a typical natural language text.\n",
"\n",
"A [bundle](https://github.com/udapi/udapi-python/blob/master/udapi/core/bundle.py) corresponds to a sentence,\n",
"possibly in multiple versions or with different representations, such as sentence-tuples from parallel corpora, or paraphrases in the same language or alternative analyses (e.g. parses produced by different parsers). If there are more trees in a bundle, they must be distinguished by a so-called *zone* (a label which contains the language code).\n",
"\n",
"Each tree is represented by a special (artificial) [root](https://github.com/udapi/udapi-python/blob/master/udapi/core/root.py) node, which is added to the top of a CoNLL-U tree in the Udapi model. The root node bears the ID of a given tree/sentence (`sent_id`) and its word order (`ord`) is 0. Technically, Root is subclass of Node, with some extra methods.\n",
"\n",
"The [Node](https://github.com/udapi/udapi-python/blob/master/udapi/core/node.py) class corresponds to a node\n",
"of a dependency tree. It provides access to all the CoNLL-U-defined attributes (`ord`, `form`, `lemma`, `upos`, `xpos`, `feats`, `deprel`, `deps`, `misc`). There are methods for tree traversal (`parent`, `root`, `children`, `descendants`); word-order traversal (`next_node`, `prev_node`); tree manipulation (`parent` setter) including word-order changes (`shift_after_node(x)`, `shift_before_subtree(x)`, etc.); and utility methods: `is_descendant_of(x)`, `is_nonprojective()`, `precedes(x)`, `is_leaf()`, `is_root()`, `get_attrs([])`, `compute_text()`, `draw()`.\n",
"\n",
"## Exercise 1: Count prepositions and postpositions\n",
"[Prepositions and postpositions](https://en.wikipedia.org/wiki/Preposition_and_postposition) are together called *adpositions* and assigned the [ADP](https://universaldependencies.org/u/pos/ADP.html) universal part-of-speech tag (`upos`) in UD. Some languages (e.g. English) use mostly prepositions, others mostly postpositions.\n",
"* Do you know any English postpositions?\n",
"* Guess the typical adposition type (i.e. whether a given language uses more prepositions or postpositions) for at least 10 languages of your choice (from those in UD2.0).\n",
"* Complete the following code and find out how many prepositions and postpositions are in `UD_English/sample.conllu` (which has been loaded into `doc`)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"prepositions, postpositions = 0, 0\n",
"# Iterate over all nodes in the document (in all trees)\n",
"for node in doc.nodes:\n",
" if node.upos == \"ADP\":\n",
" # TODO: fix this code to actually distinguish prepositions and postpositions\n",
" prepositions += 1\n",
"# Print the results\n",
"prepositions, postpositions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you don't know how to proceed click on the following hints.\n",
"Hint 1:
\n",
"In some dependency grammars, adpositions govern noun (i.e. adposition is the *parent* of a given noun node). In other dependency grammars, adpositions depend on nouns (i.e. noun is the *parent* of a given adposition). Find out which style is being used by UD. Check the UD documentation or inspect some of the tree visualizations and guess.\n",
" \n",
"Hint 2:
\n",
"See the Node documentation and find out how to obtain dependency parent and dependency children. Note that these are properties of a given node, rather than methods, so you should not write parentheses () after the property name.\n",
" \n",
"Hint 3:
\n",
"doc.nodes
iterates over all nodes in the document sorted by the word order, but this would be cumbersome to exploit. Find a method of Node
to detect the relative word order of two nodes (within the same tree/sentence).\n",
" \n",
"Hint 4:
\n",
"Use node.parent
and node.precedes(another_node)
.\n",
"The latter is a shortcut for node.ord < another_node.ord
.\n",
" \n",
"Solution:
\n",
"\n",
"for node in doc.nodes:\n",
" if node.upos == \"ADP\":\n",
" if node.precedes(node.parent):\n",
" prepositions += 1\n",
" else:\n",
" postpositions += 1\n",
"
\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 2: Explore English postpositions\n",
"The previous exercise indicates there are 7 occurrences of postpositions in the English sample. Find these 7 occurrences and visualize them using `node.draw()`. Count which adpositions (`lemma`) with which dependency relations (`deprel`) are responsible for these occurrences. Recompute these statistics on the bigger English training data. Can you explain these occurrences? What are the reasons? Is any occurrence an annotation error?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# For the statistics, you may find useful: count[\"any string\"] += 1\n",
"import collections\n",
"count = collections.Counter()\n",
"big_doc = udapi.Document(\"UD_English/train.conllu\")\n",
"\n",
"for node in doc.nodes:\n",
" # TODO detect postposition\n",
" pass\n",
"\n",
"# Print the statistics\n",
"count.most_common()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Solution 1:
\n",
"\n",
"for node in doc.nodes:\n",
" if node.upos == \"ADP\" and node.parent.precedes(node):\n",
" node.parent.draw()\n",
" count[node.lemma + \" \" + node.deprel] += 1\n",
"
\n",
" \n",
"Hint 1:
\n",
"We can see there are many particles of phrase verbs, e.g. \"busted up\".\n",
"These seem to be correctly annotated as ADP
according to the UD guidelines.\n",
"Let's filter out those cases and focus on the rest and let's switch to the big train data.\n",
" \n",
"Solution 2:
\n",
"\n",
"count = collections.Counter()\n",
"for node in big_doc.nodes:\n",
" if node.upos == \"ADP\" and node.parent.precedes(node) and node.parent.upos != \"VERB\":\n",
" count[node.lemma + \" \" + node.deprel] += 1\n",
"count.most_common()\n",
"
\n",
"Alternatively to node.parent.upos != \"VERB\"
,\n",
"you could also filter out node.deprel != \"compound:prt\"
,\n",
"or directly focus on node.deprel == \"case\"
\n",
" \n",
"Partial answer:
\n",
"Most of the occurrences are actually annotated correctly,\n",
"although they are not typically considered as postpositions.\n",
"For example, node.deprel == \"fixed\"
is being used for multi-word adpositions,\n",
"such as \"because of\", where \"of\" depends on \"because\" from technical (and consistency) reasons,\n",
"but the whole multi-word adpositions precedes its governing nound, so it is actually a multi-word preposition.\n",
"\n",
"What about the remaining occurrences, after filtering out node.deprel not in {\"compound:prt\", \"fixed\"}
?\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the next tutorial, 02-blocks.ipynb (not finished yet), we will explore several useful Udapi blocks, some of which may be handy when working further on Exercise 2 or similar tasks."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}