{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Chapter 18 - Data Formats III (XML)\n",
    "\n",
    "In this chapter, we will learn how to work with one of the most popular structured data formats: [XML](http://www.w3schools.com/xml/). XML is used a lot in Natural Language Processing (NLP), and it is important that you know how to work with it. In theory, you could load XML data just as you read in a text file, but the structure is much too complicated to extract information manually. Therefore, we use an existing library. \n",
    "\n",
    "### At the end of this chapter, you will be able to\n",
    "* read an XML file using `etree.parse`\n",
    "* read XML from string using `etree.fromstring`\n",
    "* convert an XML element to a string using `etree.tostring`\n",
    "* use the following methods and attributes of an XML element (of type `lxml.etree._Element`):\n",
    "    * to access elements: methods `find`, `findall`, and `getchildren`\n",
    "    * to access attributes: method `get`\n",
    "    * to access element information: attributes `tag` and `text`  \n",
    "\n",
    "* [not needed for assignment] create your own XML and write it to a file\n",
    "\n",
    "### If you want to learn more about this chapter, you might find the following links useful:\n",
    "* [XML](http://www.w3schools.com/xml/)\n",
    "* [detailled XML introduction](http://www.dfki.de/~uschaefer/esslli09/xmlquerylang.pdf)\n",
    "* [NAF XML](http://www.newsreader-project.eu/files/2013/01/techreport.pdf)\n",
    "* [Xpath](http://www.w3schools.com/xml/xpath_syntax.asp)\n",
    "* Other structured data formats: [JSON-LD](http://json-ld.org/), [MicroData](https://www.w3.org/TR/microdata/), [RDF](https://www.w3.org/RDF/)\n",
    "\n",
    "If you have **questions** about this chapter, please contact us at cltl.python.course@gmail.com."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 1. Introduction to XML\n",
    "\n",
    "Natural language processing (NLP) is all about text data. More specifically, we usually want to annotate (manually or automatically) textual data with information about:\n",
    "\n",
    "* [part of speech](https://en.wikipedia.org/wiki/Part_of_speech)\n",
    "* [word senses](https://en.wikipedia.org/wiki/Word_sense)\n",
    "* [syntactic information (in particulay dependencies)](https://en.wikipedia.org/wiki/Dependency_grammar)\n",
    "* [entities](https://en.wikipedia.org/wiki/Entity_linking)\n",
    "* [semantic role labelling](https://en.wikipedia.org/wiki/Semantic_role_labeling)\n",
    "* Events\n",
    "* many many many more.....\n",
    "\n",
    "How should we represent such annotated data? This is what you get from using NLTK tools:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import nltk"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "text = nltk.word_tokenize(\"Tom Cruise is an actor.\")\n",
    "text_pos_tagged = nltk.pos_tag(text)\n",
    "print(text_pos_tagged)\n",
    "print(type(text_pos_tagged), type(text_pos_tagged[0]))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "In this example, we see that the format is a list of [tuples](https://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences).  The first element of each tuple is the word and the second element is the part of speech tag. Great, so far, this works.  \n",
    "\n",
    "However, we usually want to store more information. For instance, we may want to indicate that *Tom Cruise* is a named entity. We could also represent syntactic information about this sentence. Now we start to run into difficulties because some annotations are for single words and some are for combinations of words. In addition, we have more than one annotation per token. Data structures such as CSV and TSV are not great at **representing** linguistic information. So is there a format that is better at it? The answer is yes and the format is XML. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 2. Terminology\n",
    "Let's look at an example (the line numbers are there for explanation purposes). On purpose, we start with a non-linguistic, hopefully intuitive example. In the folder `../Data/xml_data` this XML is stored as the file `course.xml`. You can inspect this file using a text editor (e.g. [Atom](https://atom.io/), [BBEdit](https://www.barebones.com/products/bbedit/download.html) or [Notepad++](https://notepad-plus-plus.org))."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "```xml\n",
    "1.  <Course>\n",
    "2.      <person role=\"coordinator\">Van der Vliet</person>\n",
    "3.      <person role=\"instructor\">Van Miltenburg</person>\n",
    "4.      <person role=\"instructor\">Van Son</person>\n",
    "5.      <person role=\"instructor\">Postma</person>\n",
    "7.      <person role=\"student\">Baloche</person>\n",
    "8.      <person role=\"student\">De Boer</person>\n",
    "9.      <animal role=\"student\">Rubber duck</animal>\n",
    "10.     <person role=\"student\">Van Doorn</person>\n",
    "11.     <person role=\"student\">De Jager</person>\n",
    "12.     <person role=\"student\">King</person>\n",
    "13.     <person role=\"student\">Kingham</person>\n",
    "14.     <person role=\"student\">Mózes</person>\n",
    "15.     <person role=\"student\">Rübsaam</person>\n",
    "16.     <person role=\"student\">Torsi</person>\n",
    "17.     <person role=\"student\">Witteman</person>\n",
    "18.     <person role=\"student\">Wouterse</person>\n",
    "19.     <person/>\n",
    "20. </Course>\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.1 Elements\n",
    "Line 1 to 19 all show examples of [XML elements](http://www.w3schools.com/xml/xml_elements.asp). Each XML element contains a **starting tag** (e.g. ```<person>```) and an **end tag** (e.g. ```</person>```). An element can contain:\n",
    "* **text** *Van der Vliet* on line 2\n",
    "* **attributes**: *role* attribute in lines 2 to 18\n",
    "* **elements**: elements can contain other elements, e.g. *person* elements inside the *Course* element. The terminology to talk about this is as follows. In this example, we call `person` the `child` of `Course` and `Course` the `parent` of `person`.\n",
    "\n",
    "Please note that on line 19 the **starting tag** and **end tag** are combined. This happens when an element has no children and/or no text. The syntax for an element is then **``` <START_TAG/>```**."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.2 Root element\n",
    "A special element is the **root** element. In our example, `Course` is our [root element](https://en.wikipedia.org/wiki/Root_element). The element starts at line 1 (```<Course>```) and ends at line 19 (```</Course>```). Notice the difference between the begin tag (no '/') and the end tag (with '/'). A root element is special in that it is the only element, which is the sole parent element to all the other elements."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.3 Attributes\n",
    "Elements can contain [attributes](http://www.w3schools.com/xml/xml_attributes.asp), which contain information about the element. In this case, this information is the `role` a person has in the course. All attributes are located in the start tag of an XML element."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3. Working with XML in Python\n",
    "Now that we know the basics of XML, we want to be able to access it in Python. In order to work with XML, we will use the [**lxml**](http://lxml.de/) library."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from lxml import etree"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will focus on the following methods/attributes:\n",
    "* **to parse the XML from file or string**: the methods `etree.parse()` and `etree.fromstring()`\n",
    "* **to access the root element**: the methods `getroot()`\n",
    "* **to access elements**: the methods `find()`, `findall()`, and `getchildren()`\n",
    "* **to access attributes**: the method `get()`\n",
    "* **to access element information**: the attributes `tag` and `text`"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.1 Parsing XML from file or string\n",
    "\n",
    "The **`etree.fromstring()`** is used to parse XML from a string:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "xml_string = \"\"\"\n",
    "<Course>\n",
    "    <person role=\"coordinator\">Van der Vliet</person>\n",
    "    <person role=\"instructor\">Van Miltenburg</person>\n",
    "    <person role=\"instructor\">Van Son</person>\n",
    "    <person role=\"instructor\">Marten Postma</person>\n",
    "    <person role=\"student\">Baloche</person>\n",
    "    <person role=\"student\">De Boer</person>\n",
    "    <animal role=\"student\">Rubber duck</animal>\n",
    "    <person role=\"student\">Van Doorn</person>\n",
    "    <person role=\"student\">De Jager</person>\n",
    "    <person role=\"student\">King</person>\n",
    "    <person role=\"student\">Kingham</person>\n",
    "    <person role=\"student\">Mózes</person>\n",
    "    <person role=\"student\">Rübsaam</person>\n",
    "    <person role=\"student\">Torsi</person>\n",
    "    <person role=\"student\">Witteman</person>\n",
    "    <person role=\"student\">Wouterse</person>\n",
    "    <person/>\n",
    "</Course>\n",
    "\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml_string)\n",
    "print(type(tree))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# printing the tree only shows a reference to the tree, but not the tree itself \n",
    "# To access information, you will have to use the methods we introduce below.\n",
    "print(tree)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The **`etree.parse()`** method is used to load XML files on your computer:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "tree = etree.parse('../Data/xml_data/course.xml')\n",
    "print(tree)\n",
    "print(type(tree))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see, `etree.parse()` returns an `ElementTree`, whereas `etree.fromstring()` returns an `Element`. One of the important differences is that the `ElementTree` class serialises as a complete document, as opposed to a single `Element`. This includes top-level processing instructions and comments, as well as a DOCTYPE and other DTD content in the document. For now, it's not too important that you know what these are; just remember that there is a difference btween `ElementTree` and `Element`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.1 Accessing root element\n",
    "\n",
    "While `etree.fromstring()` gives you the root element right away, `etree.parse()` does not. In order to access the root element of `ElementTree`, we first need to use the **`getroot()`** method. Note that this does not show the XML element itself, but only a reference. In order to show the element itself, we can use the **`etree.dump()`** method.\n",
    "\n",
    "**Hint:** etree.dump is helpful for inspecting (parts of) an xml structure you have loaded from a file. You will see examples of this later. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "root = tree.getroot()\n",
    "print('root', type(root), root)\n",
    "print()\n",
    "print('etree.dump example')\n",
    "etree.dump(root, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "As with any python object, we can use the built-in function **`dir()`** to list all methods of an element (which has the type **`lxml.etree._Element`**) , some of which will be illustrated below."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "print(type(root))\n",
    "dir(root)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.2 Accessing elements\n",
    "There are several ways of accessing XML elements. The **`find()`** method returns the *first* matching child."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "first_person_el = root.find('person')\n",
    "# Printing the element itself again only shows a reference\n",
    "print(first_person_el)\n",
    "#instead, we use etree.dump:\n",
    "etree.dump(first_person_el, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "In order to get a list of all person children, we can use the **`findall()`** method.\n",
    "Notice that this does not return the `animal` since we are looking for `person` elements."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "all_person_els = root.findall('person')\n",
    "# Check how many we found\n",
    "print(len(all_person_els))\n",
    "all_person_els"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Sometimes, we simple want all the children, while ignoring the start tags. This can be achieved using the **`getchildren()`** method. The list created below will contain all elements under root (including the animal element)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "all_child_els = root.getchildren()\n",
    "print(len(all_child_els))\n",
    "all_child_els"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.3 Accessing element information\n",
    "We will now show how to access the attributes, text, and tag of an element."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The **`get()`** method is used to access the attribute of an element.\n",
    "**Attention**: If an attribute does not exist, it will return `None`. Hence, there will not be an error."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "first_person_el = root.find('person')\n",
    "role_first_person_el = first_person_el.get('role')\n",
    "attribute_not_found = first_person_el.get('blabla')\n",
    "print('role first person element:', role_first_person_el)\n",
    "print('value if not found:', attribute_not_found)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The **text** of an element is found in the attribute **`text`**:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "print(first_person_el.text)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The **tag** of an element is found in the attribute **`tag`**:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "print(first_person_el.tag)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 4 How to deal with more than one layer\n",
    "In our previous example, we had an XML with only one nested layer (**person**). However, XML can deal with many more. \n",
    "\n",
    "\n",
    "Let's look at such an example and think about how you would access the first **`target`** element, i.e. \n",
    "```xml\n",
    "\n",
    "<target id=\"t1\" />\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "```xml\n",
    "\n",
    "<NAF xml:lang=\"en\" version=\"v3\">\n",
    "    <terms>\n",
    "        <term id=\"t1\" type=\"open\" lemma=\"Tom\" pos=\"N\" morphofeat=\"NNP\">\n",
    "        <term id=\"t2\" type=\"open\" lemma=\"Cruise\" pos=\"N\" morphofeat=\"NNP\">\n",
    "        <term id=\"t3\" type=\"open\" lemma=\"be\" pos=\"V\" morphofeat=\"VBZ\">\n",
    "        <term id=\"t4\" type=\"open\" lemma=\"an\" pos=\"R\" morphofeat=\"DT\">\n",
    "        <term id=\"t5\" type=\"open\" lemma=\"actor\" pos=\"N\" morphofeat=\"NN\">\n",
    "    </terms>\n",
    "    <entities>\n",
    "        <entity id=\"e3\" type=\"PERSON\">\n",
    "              <references>\n",
    "                  <span>\n",
    "                      <target id=\"t1\" />\n",
    "                      <target id=\"t2\" />\n",
    "                  </span>\n",
    "              </references>\n",
    "        </entity>\n",
    "    </entities>\n",
    "</NAF>\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Again, we use `etree.fromstring()` to load XML from a string:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "naf_string = \"\"\"\n",
    "<NAF xml:lang=\"en\" version=\"v3\">\n",
    "    <text>\n",
    "        <wf id=\"w1\" offset=\"0\" length=\"3\" sent=\"1\" para=\"1\">tom</wf>\n",
    "        <wf id=\"w2\" offset=\"4\" length=\"6\" sent=\"1\" para=\"1\">cruise</wf>\n",
    "        <wf id=\"w3\" offset=\"11\" length=\"2\" sent=\"1\" para=\"1\">is</wf>\n",
    "        <wf id=\"w4\" offset=\"14\" length=\"2\" sent=\"1\" para=\"1\">an</wf>\n",
    "        <wf id=\"w5\" offset=\"17\" length=\"5\" sent=\"1\" para=\"1\">actor</wf>\n",
    "    </text>\n",
    "    <terms>\n",
    "        <term id=\"t1\" type=\"open\" lemma=\"Tom\" pos=\"N\" morphofeat=\"NNP\"/>\n",
    "        <term id=\"t2\" type=\"open\" lemma=\"Cruise\" pos=\"N\" morphofeat=\"NNP\"/>\n",
    "        <term id=\"t3\" type=\"open\" lemma=\"be\" pos=\"V\" morphofeat=\"VBZ\"/>\n",
    "        <term id=\"t4\" type=\"open\" lemma=\"an\" pos=\"R\" morphofeat=\"DT\"/>\n",
    "        <term id=\"t5\" type=\"open\" lemma=\"actor\" pos=\"N\" morphofeat=\"NN\"/>\n",
    "    </terms> \n",
    "    <entities>\n",
    "        <entity id=\"e3\" type=\"PERSON\">\n",
    "              <references>\n",
    "                  <span>\n",
    "                      <target id=\"t1\" />\n",
    "                      <target id=\"t2\" />\n",
    "                  </span>\n",
    "              </references>\n",
    "        </entity>\n",
    "    </entities>\n",
    "</NAF>\n",
    "\"\"\"\n",
    "\n",
    "naf = etree.fromstring(naf_string)\n",
    "print(type(naf))\n",
    "etree.dump(naf, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Please note that the structure is as follows:\n",
    "* the **`NAF`** element is the parent of the elements **`text`**, **`terms`**, and **`entities`**\n",
    "* the **`wf`** elements are children of the **`text`** element, which provides us information about the position of words in the text, e.g. that *tom* is the first word in the text (**`id=\"w1`\"**) and in the first sentence (**sent=\"1\"**)\n",
    "* the **`term`** elements are children of the **`term`** elements, which provide us information about lemmatization and part of speech\n",
    "* the **`entity`** element is a child of the **`entities`** element. We learn from the **`entity`** element that the terms **`t1`** and **`t2`** (e.g. Tom Cruise) form an entity of type **`person`**."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "One way of accessing the first **`target`** element is by going one level at a time:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "entities_el = naf.find('entities')\n",
    "entity_el = entities_el.find('entity')\n",
    "references_el = entity_el.find('references')\n",
    "span_el = references_el.find('span')\n",
    "target_el = span_el.find('target')\n",
    "etree.dump(target_el, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Is there a better way? The answer is yes! The following way is an easier way to find our `target` element:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "target_el = naf.find('entities/entity/references/span/target')\n",
    "etree.dump(target_el, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can also use **`findall()`** to find *all* `target` elements:\n",
    "\n",
    "(Note that **findall()** returns a list of xml elements. We can use a loop to iterate over them and print them individually.)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "\n",
    "for target_el in naf.findall('entities/entity/references/span/target'):\n",
    "    etree.dump(target_el, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 5. Extracting infromation from XML\n",
    "\n",
    "\n",
    "Often, we want to extract information from an XML file, so we can analyze and possibly manipulate it in python. It can be very useful to use python containers for this. In the following example, we want to collect all tokens (i.e. words as they appear in text) of a part of speech.\n",
    "\n",
    "To do this, we have to extract infromation from the token layer and combine it with infromation from the term layer."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "path_to_tokens = 'text/wf'\n",
    "path_to_terms = 'terms/term'\n",
    "\n",
    "\n",
    "# We define dictionaries to map identifiers to tokens ('word forms') and term tags including pos information.\n",
    "# We can use the identifiers to map the tokens to the pos tags in the next step\n",
    "tokens = naf.findall(path_to_tokens)\n",
    "terms = naf.findall(path_to_terms)\n",
    "\n",
    "id_token_dict = dict()\n",
    "id_pos_dict = dict()\n",
    "\n",
    "# map ids to tokens\n",
    "for token in tokens:\n",
    "    id_token_dict[token.get('id')] = token.text\n",
    "#map ids to terms\n",
    "for term in terms:\n",
    "    id_pos_dict[term.get('id')] = term.get('pos')\n",
    " \n",
    " #use ids to map tokens to pos tags\n",
    "for token_id, token in id_token_dict.items():\n",
    "    # term identifiers start with a t, token identifiers with a w. The numbers correspond. \n",
    "    term_id = token_id.replace('w', 't')\n",
    "    pos = id_pos_dict[term_id]\n",
    "    print(token, pos)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 6. EXTRA: Creating your own XML\n",
    "\n",
    "Please note that this section is optional, meaning that you don't need to understand this section in order to complete the assignment. \n",
    "\n",
    "There are three main steps:\n",
    "* **Step a:** Create an XML object with a root element\n",
    "* **Step b:** Creating child elements and adding them\n",
    "* **Step c:** Writing to a file"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Step a: Create an XML object with a root element\n",
    "You create a new XML object by:\n",
    "* creating the **`root`** element -> using **`etree.Element`** \n",
    "* creating the main XML object -> using **`etree.ElementTree`**\n",
    "\n",
    "You do not have to fully understand how this works. Please make sure you can reuse this code snippet when you create your own XML."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "our_root = etree.Element('Course')\n",
    "our_tree = etree.ElementTree(our_root)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can inspect what we have created by using the `etree.dump()` method. As you can see, we only have the root node `Course` currently in our document."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "etree.dump(our_root, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you see, we created an XML object, containing only the root element **Course**."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Step b: Creating child elements and adding them\n",
    "There are two ways to add child elements to the root element. The first is to create an element using the **`etree.Element()`** method and using `append()` to add it to the root:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# Define tag, attributes and text of the new element\n",
    "tag = 'person' # what the start and end tag will be \n",
    "attributes = {'role': 'student'} # dictionary of attributes, can be more than one\n",
    "name_student = 'Lee' # the text of the elements\n",
    "\n",
    "# Create new Element\n",
    "new_person_element = etree.Element(tag, attrib=attributes)\n",
    "new_person_element.text = name_student\n",
    "\n",
    "# Add to root\n",
    "our_root.append(new_person_element)\n",
    "\n",
    "# Inspect the current XML\n",
    "etree.dump(our_root, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "However, this is so common that there is a shorter and much more efficient way to do this: by using **`etree.SubElement()`**. It accepts the same arguments as the `etree.Element()` method, but additionally requires the parent as first argument:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# Define tag, attributes and text of the new element\n",
    "tag = 'person' \n",
    "attributes = {'role': 'student'} \n",
    "name_student = 'Pitt' \n",
    "\n",
    "# Add to root\n",
    "another_person_element = etree.SubElement(our_root, tag, attrib=attributes) # parent is our_root\n",
    "another_person_element.text = name_student\n",
    "\n",
    "# Inspect the current XML\n",
    "etree.dump(our_root, pretty_print=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we have seen before, XML can have multiple nested layers. Creating these works the same way as adding child elements to the root, but now we specify one of the other elements as the parent (in this case, `new_person_element`)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# Define tag, attributes and text of the new element\n",
    "tag = 'pet'\n",
    "attributes = {'role': 'joy'}\n",
    "name_pet = 'Romeo'\n",
    "\n",
    "# Add to new_person_element\n",
    "new_pet_element = etree.SubElement(new_person_element, tag, attrib=attributes) # parent is new_person_element\n",
    "new_pet_element.text = name_pet\n",
    "\n",
    "# Inspect the current XML\n",
    "etree.dump(our_root, pretty_print=True) "
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Step c: Writing to a file\n",
    "This is how we can write our selfmade XML to a file. Please inspect `../Data/xml_data/selfmade.xml` using a text editor to check if it worked."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "with open('../Data/xml_data/selfmade.xml', 'wb') as outfile:\n",
    "    our_tree.write(outfile,\n",
    "                   pretty_print=True,\n",
    "                   xml_declaration=True,\n",
    "                   encoding='utf-8')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Exercises"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Exercise 1:\n",
    "\n",
    "Have another look at the XML below. Then print the following information:\n",
    "* the names of all students\n",
    "* the names of all instructors whose name starts with 'Van'\n",
    "* all names containing a space\n",
    "* the role of 'Rubber duck'"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "xml_string = \"\"\"\n",
    "<Course>\n",
    "    <person role=\"coordinator\">Van der Vliet</person>\n",
    "    <person role=\"instructor\">Van Miltenburg</person>\n",
    "    <person role=\"instructor\">Van Son</person>\n",
    "    <person role=\"instructor\">Marten Postma</person>\n",
    "    <person role=\"student\">Baloche</person>\n",
    "    <person role=\"student\">De Boer</person>\n",
    "    <animal role=\"student\">Rubber duck</animal>\n",
    "    <person role=\"student\">Van Doorn</person>\n",
    "    <person role=\"student\">De Jager</person>\n",
    "    <person role=\"student\">King</person>\n",
    "    <person role=\"student\">Kingham</person>\n",
    "    <person role=\"student\">Mózes</person>\n",
    "    <person role=\"student\">Rübsaam</person>\n",
    "    <person role=\"student\">Torsi</person>\n",
    "    <person role=\"student\">Witteman</person>\n",
    "    <person role=\"student\">Wouterse</person>\n",
    "    <person/>\n",
    "</Course>\n",
    "\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml_string)\n",
    "print(type(tree))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Exercise 2:\n",
    "In the folder `../Data/xml_data` there is an XML file called `framenet.xml`, which is a simplified version of the data provided by the [FrameNet project](https://framenet.icsi.berkeley.edu/fndrupal/).\n",
    "\n",
    "FrameNet is a lexical database describing **semantic frames**, which are representations of events or situations and the participants in it. For example, cooking typically involves a person doing the cooking (`Cook`), the food that is to be cooked (`Food`), something to hold the food while cooking (`Container`) and a source of heat (`Heating_instrument`). In FrameNet, this is represented as a frame called `Apply_heat`. The `Cook`, `Food`, `Heating_instrument` and `Container` are called **frame elements (FEs)**. Words that evoke this frame, such as *fry*, *bake*, *boil*, and *broil*, are called **lexical units (LUs)** of the `Apply_heat` frame. FrameNet also contains relations between frames. For example, `Apply_heat` has relations with the `Absorb_heat`, `Cooking_creation` and `Intentionally_affect` frames. In FrameNet, frame descriptions are stored in XML format.\n",
    "\n",
    "`framenet.xml` contains the information about the frame `Waking_up`. Parse the XML file and print the following:\n",
    "* the name of the frame\n",
    "* the names of all lexical units\n",
    "* the definitions of all lexical units\n",
    "* the related frames with their type of relation to `Waking_up` (e.g. `Event` with the `Inherits from` relation)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Exercise 3:\n",
    "\n",
    "Something were we collect information from multiple files. Not created yet. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.10 64-bit ('nlp': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "interpreter": {
   "hash": "6106edc083458b68f61c14c570e0f5152b4e1e25a61780539c3fe413e38ae5e6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}