{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The focus of the second lab are details of the [structured prediction recipe](../chapters/structured_prediction.ipynb) and [tokenization](../chapters/tokenization.ipynb).\n",
    "\n",
    "We will go through the notes of each of these sections to highlight issues, explain code, and point to exercise which you will work on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Structured prediction\n",
    "\n",
    "Solve [exercises](../exercises/structured_prediction.ipynb) in structured prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "\n",
    "## Warm-up questions\n",
    "As we've seen, tokenization is an important step in the NLP pipeline, and it is not as trivial as one might imagine.\n",
    "\n",
    "- Can you think of 3 domains where one might expect some issues with simple tokenization we considered in the lecture?\n",
    "\n",
    "- Why isn't it possible to solve tokenization just by remembering words? What about if we limit ourselves to English only (an estimate says [English has 1,025,109.8 words](http://www.languagemonitor.com/number-of-words/number-of-words-in-the-english-language-1008879/))?\n",
    "\n",
    "- In the lecture we went through four iterations of building a simple tokenizer in the lecture:\n",
    "  - splitting by blank space\n",
    "  - splitting by any whitespace character\n",
    "  - tokenization through definition of tokens as sequences of alphanumeric characters and some punctuation\n",
    "  - tokenization by addition of words to our tokenizer\n",
    "- What are the shortcomings of each one of these approaches?\n",
    "\n",
    "Food for thought: we're doing tokenization to split a string into a series of tokens and process them accordingly. After it we're left with an array of symbols whwich we will map into a specific representation, and continue with our machine learning model. Should all tokens be included into further processing? Do all of them contribute to the performance of the model? Are there different tokens which denote the same underlying word? What should we do with them? What about punctuation, is punctuation always important? When it is, when it is not?\n",
    "\n",
    "Next, solve [tokenization exercises](../exercises/tokenization.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}