{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%html\n", "\n", "
\n", "" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "%%capture\n", "%load_ext autoreload\n", "%autoreload 2\n", "import sys\n", "sys.path.append(\"../statnlpbook/\")\n", "\n", "#util.execute_notebook('relation_extraction.ipynb')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "\n", "$$\n", "\\newcommand{\\Xs}{\\mathcal{X}}\n", "\\newcommand{\\Ys}{\\mathcal{Y}}\n", "\\newcommand{\\y}{\\mathbf{y}}\n", "\\newcommand{\\balpha}{\\boldsymbol{\\alpha}}\n", "\\newcommand{\\bbeta}{\\boldsymbol{\\beta}}\n", "\\newcommand{\\aligns}{\\mathbf{a}}\n", "\\newcommand{\\align}{a}\n", "\\newcommand{\\source}{\\mathbf{s}}\n", "\\newcommand{\\target}{\\mathbf{t}}\n", "\\newcommand{\\ssource}{s}\n", "\\newcommand{\\starget}{t}\n", "\\newcommand{\\repr}{\\mathbf{f}}\n", "\\newcommand{\\repry}{\\mathbf{g}}\n", "\\newcommand{\\x}{\\mathbf{x}}\n", "\\newcommand{\\prob}{p}\n", "\\newcommand{\\a}{\\alpha}\n", "\\newcommand{\\b}{\\beta}\n", "\\newcommand{\\vocab}{V}\n", "\\newcommand{\\params}{\\boldsymbol{\\theta}}\n", "\\newcommand{\\param}{\\theta}\n", "\\DeclareMathOperator{\\perplexity}{PP}\n", "\\DeclareMathOperator{\\argmax}{argmax}\n", "\\DeclareMathOperator{\\argmin}{argmin}\n", "\\newcommand{\\train}{\\mathcal{D}}\n", "\\newcommand{\\counts}[2]{\\#_{#1}(#2) }\n", "\\newcommand{\\length}[1]{\\text{length}(#1) }\n", "\\newcommand{\\indi}{\\mathbb{I}}\n", "$$" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "%load_ext tikzmagic" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Question Answering" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Flavours of question answering\n", "* Information retrieval\n", "* Machine reading comprehension\n", "* Question answering from structured knowledge\n", "* Executable semantic parsing\n", "* Relation extraction via reading comprehension" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Question:\n", "\n", ">Which university did Turing go to?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Answer:\n", "\n", ">Princeton" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "[Passage](https://en.wikipedia.org/wiki/Alan_Turing):\n", "\n", "> In 1938, he obtained his PhD from the Department of Mathematics at Princeton University." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Knowledge base:\n", "\n", "https://www.wikidata.org/wiki/Q7251#P69" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "\n", "## [tinyurl.com/diku-nlp-qa](https://tinyurl.com/diku-nlp-qa)\n", "\n", "([Responses](https://docs.google.com/forms/d/17j4Msqa_L_so14KGZEdGClZaAbRsvn0rQedeze8kaRM/edit#responses))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Flavours of Question answering (QA)\n", "\n", "Factoid questions:\n", "\n", "* Information retrieval (IR)-based QA on **unstructured** data (text)\n", "* Knowledge-based QA on **structured** data (DB/KB)\n", "\n", "Non-factoid questions:\n", "\n", "* \"How\" questions\n", "> How do I delete my Instagram account?\n", "* \"Why\" questions\n", "> Why is the sky blue?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "* Math problems\n", "\n", "
\n", " (from Lu et al., 2021)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Formats\n", "\n", "* Extractive (span selection)\n", "* Cloze\n", "* Boolean\n", "* Multi-choice\n", "* Abstractive" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### QA on collections of documents\n", "\n", "* [TriviaQA](https://nlp.cs.washington.edu/triviaqa/)\n", "* [SearchQA](https://github.com/nyu-dl/dl4ir-searchqA)\n", "* [MS MARCO](https://microsoft.github.io/msmarco/)\n", "* [AmazonQA](https://github.com/amazonqa/amazonqa)\n", "* [TrecQA](https://trec.nist.gov/data/qa.html)\n", "* [WebQA](https://webqna.github.io/)\n", "\n", "> Which politician won the Nobel Peace Prize in 2009?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Multi-hop QA\n", "\n", "* [HotPotQA](https://hotpotqa.github.io/)\n", "* [QAngaroo](https://qangaroo.cs.ucl.ac.uk/)\n", "* [ComplexWebQuestions](https://www.tau-nlp.sites.tau.ac.il/compwebq)\n", "* [HybridQA](https://hybridqa.github.io/)\n", "\n", "> What is the middle name of the player with the second most National Football League career rushing yards?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### QA on structured knowledge\n", "\n", "* [FreebaseQA](https://github.com/kelvin-jiang/FreebaseQA)\n", "* [Event-QA](https://github.com/tarcisiosouza/Event-QA)\n", "* [WikiTableQuestions](https://github.com/ppasupat/WikiTableQuestions)\n", "* [SimpleQuestions](https://github.com/davidgolub/SimpleQA)\n", "* [WikiSQL](https://github.com/salesforce/WikiSQL)\n", "* [RuBQ](https://github.com/vladislavneon/RuBQ)\n", "\n", "> The 1999 film '10 Things I Hate About You' is based on which Shakespeare play?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Extractive QA\n", "\n", "* [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)\n", "* [Natural Questions](https://ai.google.com/research/NaturalQuestions/)\n", "* [NewsQA](https://www.microsoft.com/en-us/research/project/newsqa-dataset/)\n", "* [TyDI-QA](https://github.com/google-research-datasets/tydiqa)\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Rajpurkar et al., 2016)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Information retrieval (IR)-based QA\n", "\n", "General approach:\n", "1. Retrieve relevant **passage**(s)\n", "2. Machine reading comprehension: extract the **answer**, which can be\n", " * A text span from the passage\n", " * Yes/no\n", " * `NULL` (unanswerable)\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Information retrieval\n", "\n", "* Obtain relevant documents/passages given a query\n", "* Example: web search\n", "* Learn more in [NDAK20002U Neural Information Retrieval (NIR)](https://kurser.ku.dk/course/ndak20002u/)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Machine reading comprehension (MRC)\n", "\n", "* Input: (Passage, Question)\n", "* Output: Answer span\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Rajpurkar et al., 2016)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### [demo](https://demo.allennlp.org/reading-comprehension/transformer-qa)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### MRC modeling\n", "\n", "How to model span selection?\n", "\n", "* As sequence labeling (for each token, is it part of the answer?)\n", " * With IOB encoding\n", " \n", "
\n", " \n", "
\n", "
\n", " (from Yoon et al., 2022)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Or as span selection (find start and end of the answer span)\n", "\n", "||||start|end||||\n", "|---|---|---|---|---|---|---|---|\n", "|...|termed|the|interferon|signature|.|Here|...|" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "What are the advantages and disadvantages of each?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### MRC evaluation\n", "\n", "Metrics for binary (yes/no) QA:\n", "* **Accuracy**\n", "* **F-score**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "Metrics for ranking:\n", "* **MRR** (mean reciprocal rank)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Test questions have $k$ gold answers by different human annotators (e.g., $k=3$ for SQuAD and TyDI-QA, $k=5$ for NQ).\n", "Metrics for span selection:\n", "* **Exact match** (EM): text is exactly any of the $k$\n", "* **Word-based f-score** (like in some [sequence labeling](sequence_labeling_slides.ipynb) tasks) averaged over the $k$ gold answers\n", " * Often ignoring punctuation and articles, i.e., `a, an, the`\n", " * As bag-of-words, not exact positions (because the same answer may appear multiple times)\n", " * Macro-averaged: calculate f-score for each question and average the f-scores" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### SQuAD\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from SQuAD leaderboard)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### MRC Models\n", "\n", "![model](https://d3i71xaburhd42.cloudfront.net/1b78ce27180c324f3831f5395a2fdf738e143e74/2-Figure1-1.png)\n", "\n", "
\n", " (from Li et al., 2020)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### MRC with BERT\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Devlin et al., 2019)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### SpanBERT\n", "\n", "\n", "\n", "(from [Joshi et al., 2020](https://aclanthology.org/2020.tacl-1.5))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### MRC skills\n", "\n", "\n", "\n", "(from [Rogers et al., 2022](https://dl.acm.org/doi/10.1145/3560260))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Knowledge-based (KB) question answering\n", "\n", "Information is already organized in tables, databases and knowledge bases!\n", "\n", "1. (Executable) **semantic parsing**: translate natural language question to SQL/SPARQL/logical form **program** (query).\n", "2. **Execute** the program on a database/knowledge-base and return the answer." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "![wikidata](https://upload.wikimedia.org/wikipedia/commons/thumb/6/66/Wikidata-logo-en.svg/500px-Wikidata-logo-en.svg.png)\n", "\n", "[Which university did Turing go to?](https://query.wikidata.org/#select%20distinct%20%3Fitem%20%3FitemLabel%20where%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP31%20wd%3AQ15936437.%0A%20%20%20%20wd%3AQ7251%20wdt%3AP69%20%3Fitem.%0A%20%20%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D%0AORDER%20BY%20DESC%28%3Fsitelinks%29)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "\n", "\n", "(from [Cui et al., 2022](https://aclanthology.org/2022.tacl-1.55/))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Semantic parsing as machine translation\n", "\n", "Can be modeled as sequence-to-sequence (seq2seq), see [machine translation](nmt_slides_active.ipynb).\n", "\n", "\n", "\n", "(from [AllenNLP](https://guide.allennlp.org/semantic-parsing-seq2seq))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Executable semantic parsing to SPARQL\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Keysers et al., 2020)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Executable semantic parsing to SQL\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Oren et al., 2020)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Executable semantic parsing to SQL\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Finegan-Dollak et al., 2018)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Executable semantic parsing to logical form\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Berant et al., 2013)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Relation extraction via reading comprehension\n", "\n", " \n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", " (from Levy et al., 2017; slides)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Summary\n", "\n", "* Relation extraction can be cast as question answering (and vice versa)\n", "* Information retrieval-based question answering require reading comprehension\n", "* Knowledge-based question answering requires semantic parsing" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Background Material\n", "\n", "* Jurafky, Dan and Martin, James H. (2016). [Speech and Language Processing, Chapter 20 (Question Answering)](https://web.stanford.edu/~jurafsky/slp3/20.pdf)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Further Reading\n", "\n", "* [Question Answering. Blog post by Vered Shwartz](http://veredshwartz.blogspot.com/2016/11/question-answering.html)\n", "* Rogers et al., 2022. [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://dl.acm.org/doi/10.1145/3560260)\n", "* Levy et al., 2017. [Zero-Shot Relation Extraction via Reading Comprehension](https://www.aclweb.org/anthology/K17-1034)\n", "* Abdou et al., 2019. [X-WikiRE: A large, multilingual resource for relation extraction as machine comprehension](https://aclanthology.org/D19-6130/)\n", "* [AllenNLP Guide: Semantic Parsing: Intro and Seq2Seq Model](https://guide.allennlp.org/semantic-parsing-seq2seq)" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 1 }