{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", " Text_Extensions_for_Pandas_Overview.ipynb:\n", "

Overview of the basic functionality and usage of Text Extensions for Pandas.

\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Text Extensions for Pandas\n", "\n", "[Text Extensions for Pandas](https://github.com/CODAIT/text-extensions-for-pandas) is a library that provides natural language processing support for Pandas DataFrames. It includes [Pandas](https://pandas.pydata.org) extension arrays that help with natural language processing, and integrates with other popular NLP libraries to provide a workflow centered around the easy to use and powerful Pandas [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).\n", "\n", "This notebook gives an overview of the basic functionality of Text Extensions for Pandas, and serves as a jumping off point to more in-depth examples of specific functionality. See the following notebooks that use Text Extensions for Pandas for data analysis, NLP, and model training:\n", "\n", "- [Analyze_Model_Outputs](./Analyze_Model_Outputs.ipynb) - analyze the outputs of a NLP model on a target corpus\n", "- [Analyze_Text](./Analyze_Text.ipynb) - usage with the IBM Watson cloud API\n", "- [Integrate_NLP_Libraries](./Integrate_NLP_Libraries.ipynb) - integration with SpaCy and IBM Watson\n", "- [Model_Training_with_BERT](./Model_Training_with_BERT.ipynb) - model training for NER with BERT tokenization and embeddings\n", "- [Understand_Tables](./Understand_Tables.ipynb) - integration with IBM Watson Discovery for understanding of tables in PDFs and documents\n", "\n", "API reference can be found at https://text-extensions-for-pandas.readthedocs.io/en/latest/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Environment Setup\n", "\n", "This notebook requires a Python 3.6 or later environment with NumPy, and Pandas. \n", "\n", "The notebook also requires the `text_extensions_for_pandas` library. You can satisfy this dependency in two ways:\n", "\n", "* Run `pip install text_extensions_for_pandas` before running this notebook. This command adds the library to your Python environment.\n", "* Run this notebook out of your local copy of the Text Extensions for Pandas project's [source tree](https://github.com/CODAIT/text-extensions-for-pandas). In this case, the notebook will use the version of Text Extensions for Pandas in your local source tree **if the package is not installed in your Python environment**." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import regex\n", "import sys\n", "import numpy as np\n", "import pandas as pd\n", "\n", "# And of course we need the text_extensions_for_pandas library itself.\n", "try:\n", " import text_extensions_for_pandas as tp\n", "except ModuleNotFoundError as e:\n", " # If we're running from within the project source tree and the parent Python\n", " # environment doesn't have the text_extensions_for_pandas package, use the\n", " # version in the local source tree.\n", " if not os.getcwd().endswith(\"notebooks\"):\n", " raise e\n", " if \"..\" not in sys.path:\n", " sys.path.insert(0, \"..\")\n", " import text_extensions_for_pandas as tp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pandas Extension Arrays\n", "\n", "Text Extensions for Pandas provides several Pandas extension arrays on which much of the functionality is built on top of. This section will introduce and show basic usage of these extension arrays." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### SpanArray\n", "\n", "A `SpanArray` represents a column of character-based spans over a single target text. It is backed by 2 child arrays of integers that are the begin and end offsets of each span item from the target text. Spans can use any offset within the target text and can also overlap with each other. A `SpanArray` can efficiently represent the tokenized result of text because each token is not copied, only offsets are stored. Equality of spans is determined by the text and offset values, so each token will be unique within the text.\n", "\n", "The `SpanArray` is a Pandas extension type, so it can be wrapped as a series and included in a DataFrame to make use of standard Pandas functionality. The values of a `SpanArray` are also designed to render nicely as HTML, for easy display of the span offsets, text and highlighted target text.\n", "\n", "We will show some basic operations of the `SpanArray` by tokenizing a small example piece of text." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Sample text input.\n", "text = \"\"\"\\\n", "In AD 932, King Arthur and his squire, Patsy, travel throughout Britain \\\n", "searching for men to join the Knights of the Round Table. Along the way, \\\n", "he recruits Sir Bedevere the Wise, Sir Lancelot the Brave, Sir Galahad \\\n", "the Pure, Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot, and Sir \\\n", "Not-Appearing-in-this-Film, along with their squires and Robin's troubadours.\\\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Define a crude tokenizer to split by words, for example use only.\n", "def tokenize_with_offsets(text):\n", " \"\"\"Return offsets of tokens from given `text`\"\"\"\n", " splits = text.split(\" \")\n", " begins = np.cumsum([0] + [len(s) + 1 for s in splits[:-1]])\n", " ends = begins + [len(s.strip(\",.\")) for s in splits]\n", " return begins, ends" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", "
beginendcontext
002In
135AD
269932
31115King
41622Arthur
52326and
62730his
73137squire
83944Patsy
94652travel
105363throughout
116471Britain
127281searching
138285for
148689men
159092to
169397join
1798101the
18102109Knights
19110112of
20113116the
21117122Round
22123128Table
23130135Along
24136139the
25140143way
26145147he
27148156recruits
28157160Sir
29161169Bedevere
30170173the
31174178Wise
32180183Sir
33184192Lancelot
34193196the
35197202Brave
36204207Sir
37208215Galahad
38216219the
39220224Pure
40226229Sir
41230235Robin
42236239the
43240274Not-Quite-So-Brave-as-Sir-Lancelot
44276279and
45280283Sir
46284310Not-Appearing-in-this-Film
47312317along
48318322with
49323328their
50329336squires
51337340and
52341348Robin's
53349360troubadours
\n", "

\n", "\n", "\n", "\n", " In\n", "\n", "\n", "\n", " AD\n", "\n", "\n", "\n", " 932\n", "\n", " , \n", "\n", " King\n", "\n", "\n", "\n", " Arthur\n", "\n", "\n", "\n", " and\n", "\n", "\n", "\n", " his\n", "\n", "\n", "\n", " squire\n", "\n", " , \n", "\n", " Patsy\n", "\n", " , \n", "\n", " travel\n", "\n", "\n", "\n", " throughout\n", "\n", "\n", "\n", " Britain\n", "\n", "\n", "\n", " searching\n", "\n", "\n", "\n", " for\n", "\n", "\n", "\n", " men\n", "\n", "\n", "\n", " to\n", "\n", "\n", "\n", " join\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Knights\n", "\n", "\n", "\n", " of\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Round\n", "\n", "\n", "\n", " Table\n", "\n", " . \n", "\n", " Along\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " way\n", "\n", " , \n", "\n", " he\n", "\n", "\n", "\n", " recruits\n", "\n", "\n", "\n", " Sir\n", "\n", "\n", "\n", " Bedevere\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Wise\n", "\n", " , \n", "\n", " Sir\n", "\n", "\n", "\n", " Lancelot\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Brave\n", "\n", " , \n", "\n", " Sir\n", "\n", "\n", "\n", " Galahad\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Pure\n", "\n", " , \n", "\n", " Sir\n", "\n", "\n", "\n", " Robin\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Not-Quite-So-Brave-as-Sir-Lancelot\n", "\n", " , \n", "\n", " and\n", "\n", "\n", "\n", " Sir\n", "\n", "\n", "\n", " Not-Appearing-in-this-Film\n", "\n", " , \n", "\n", " along\n", "\n", "\n", "\n", " with\n", "\n", "\n", "\n", " their\n", "\n", "\n", "\n", " squires\n", "\n", "\n", "\n", " and\n", "\n", "\n", "\n", " Robin's\n", "\n", "\n", "\n", " troubadours\n", " .\n", "

\n", "
\n", "\n", " Your notebook viewer does not support Javascript execution. The above rendering will not be interactive.\n", "
\n", "\n", "\n" ], "text/plain": [ "\n", "[ [0, 2): 'In',\n", " [3, 5): 'AD',\n", " [6, 9): '932',\n", " [11, 15): 'King',\n", " [16, 22): 'Arthur',\n", " [23, 26): 'and',\n", " [27, 30): 'his',\n", " [31, 37): 'squire',\n", " [39, 44): 'Patsy',\n", " [46, 52): 'travel',\n", " [53, 63): 'throughout',\n", " [64, 71): 'Britain',\n", " [72, 81): 'searching',\n", " [82, 85): 'for',\n", " [86, 89): 'men',\n", " [90, 92): 'to',\n", " [93, 97): 'join',\n", " [98, 101): 'the',\n", " [102, 109): 'Knights',\n", " [110, 112): 'of',\n", " [113, 116): 'the',\n", " [117, 122): 'Round',\n", " [123, 128): 'Table',\n", " [130, 135): 'Along',\n", " [136, 139): 'the',\n", " [140, 143): 'way',\n", " [145, 147): 'he',\n", " [148, 156): 'recruits',\n", " [157, 160): 'Sir',\n", " [161, 169): 'Bedevere',\n", " [170, 173): 'the',\n", " [174, 178): 'Wise',\n", " [180, 183): 'Sir',\n", " [184, 192): 'Lancelot',\n", " [193, 196): 'the',\n", " [197, 202): 'Brave',\n", " [204, 207): 'Sir',\n", " [208, 215): 'Galahad',\n", " [216, 219): 'the',\n", " [220, 224): 'Pure',\n", " [226, 229): 'Sir',\n", " [230, 235): 'Robin',\n", " [236, 239): 'the',\n", " [240, 274): 'Not-Quite-So-Brave-as-Sir-Lancelot',\n", " [276, 279): 'and',\n", " [280, 283): 'Sir',\n", " [284, 310): 'Not-Appearing-in-this-Film',\n", " [312, 317): 'along',\n", " [318, 322): 'with',\n", " [323, 328): 'their',\n", " [329, 336): 'squires',\n", " [337, 340): 'and',\n", " [341, 348): 'Robin's',\n", " [349, 360): 'troubadours']\n", "Length: 54, dtype: SpanDtype" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Tokenize the text to get begin, end offsets and construct a `SpanArray`.\n", "begins, ends = tokenize_with_offsets(text)\n", "tokens = tp.SpanArray(text, begins, ends)\n", "\n", "# The array nicely renders in HTML to show offsets, text of the span,\n", "# and highlighted target text.\n", "tokens" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[240, 274): 'Not-Quite-So-Brave-as-Sir-Lancelot'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Indexing the array with an integer will produce a `Span`, which is a single\n", "# element in the array.\n", "tok = tokens[43]\n", "tok" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", "
beginendcontext
0226229Sir
1230235Robin
2236239the
3240274Not-Quite-So-Brave-as-Sir-Lancelot
\n", "

\n", "\n", " In AD 932, King Arthur and his squire, Patsy, travel throughout Britain searching for men to join the Knights of the Round Table. Along the way, he recruits Sir Bedevere the Wise, Sir Lancelot the Brave, Sir Galahad the Pure, \n", "\n", " Sir\n", "\n", "\n", "\n", " Robin\n", "\n", "\n", "\n", " the\n", "\n", "\n", "\n", " Not-Quite-So-Brave-as-Sir-Lancelot\n", " , and Sir Not-Appearing-in-this-Film, along with their squires and Robin's troubadours.\n", "

\n", "
\n", "\n", " Your notebook viewer does not support Javascript execution. The above rendering will not be interactive.\n", "
\n", "\n", "\n" ], "text/plain": [ "\n", "[ [226, 229): 'Sir',\n", " [230, 235): 'Robin',\n", " [236, 239): 'the',\n", " [240, 274): 'Not-Quite-So-Brave-as-Sir-Lancelot']\n", "Length: 4, dtype: SpanDtype" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# It can also be indexed with a slice, producing another `SpanArray`.\n", "toks = tokens[40:44]\n", "toks" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[[226, 229): 'Sir',\n", " [230, 235): 'Robin',\n", " [236, 239): 'the',\n", " [240, 274): 'Not-Quite-So-Brave-as-Sir-Lancelot']" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Iterate over the array to get each `Span`.\n", "toks = [span for span in tokens[40:44]]\n", "toks" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[226, 274): 'Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Addition of `Span`s or `SpanArray`s are supported.\n", "# The result is the minimum `Span` that covers both `Span`s.\n", "result = toks[0] + toks[-1]\n", "result" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# You can check if one `Span` contains another.\n", "result.contains(toks[1])" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Also if two `Span`s overlap.\n", "a = toks[0] + toks[2]\n", "b = toks[2] + toks[3]\n", "a.overlaps(b)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "([204, 207): 'Sir', [226, 229): 'Sir')" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get 2 `Span`s to test equality.\n", "sir = tokens[36]\n", "other_sir = tokens[40]\n", "sir, other_sir" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(False, True)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Equality is determined by text and offset values, not just text.\n", "sir == other_sir, \\\n", "sir.covered_text == other_sir.covered_text" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Only a `Span` from the same target text with matching offsets is equal.\n", "sir == tp.Span(text, 204, 207)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TokenSpanArray\n", "\n", "A `TokenSpanArray` builds on a `SpanArray` with the ability to span text as indices of a `SpanArray` instead of character based offsets. This makes it convenient to use when doing analysis on the token level. Similar to `SpanArray`, a single item in a `TokenSpanArray` is a `TokenSpan`. For an example, let's define a single `TokenSpan` using the target text from above." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[11, 22): 'King Arthur'" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Single `TokenSpan` to cover \"King Arthur\" - notice we begin with the third\n", "# token and end at the fifth.\n", "tp.TokenSpan(tokens, 3, 5)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", "
beginendbegin tokenend tokencontext
0112235King Arthur
1394489Patsy
21571782832Sir Bedevere the Wise
31802023236Sir Lancelot the Brave
42042243640Sir Galahad the Pure
52262744044Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot
62803104547Sir Not-Appearing-in-this-Film
73413485253Robin's
\n", "

\n", "\n", " In AD 932, \n", "\n", " King Arthur\n", "\n", " and his squire, \n", "\n", " Patsy\n", "\n", " , travel throughout Britain searching for men to join the Knights of the Round Table. Along the way, he recruits \n", "\n", " Sir Bedevere the Wise\n", "\n", " , \n", "\n", " Sir Lancelot the Brave\n", "\n", " , \n", "\n", " Sir Galahad the Pure\n", "\n", " , \n", "\n", " Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot\n", "\n", " , and \n", "\n", " Sir Not-Appearing-in-this-Film\n", "\n", " , along with their squires and \n", "\n", " Robin's\n", " troubadours.\n", "

\n", "
\n", "\n", " Your notebook viewer does not support Javascript execution. The above rendering will not be interactive.\n", "
\n", "\n", "\n" ], "text/plain": [ "\n", "[ [11, 22): 'King Arthur',\n", " [39, 44): 'Patsy',\n", " [157, 178): 'Sir Bedevere the Wise',\n", " [180, 202): 'Sir Lancelot the Brave',\n", " [204, 224): 'Sir Galahad the Pure',\n", " [226, 274): 'Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot',\n", " [280, 310): 'Sir Not-Appearing-in-this-Film',\n", " [341, 348): 'Robin's']\n", "Length: 8, dtype: TokenSpanDtype" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# We can also make a `TokenSpanArray` with a list of begin and end offsets of\n", "# measured in tokens. Here we make spans of the names within the target text.\n", "begin_tokens = [3, 8, 28, 32, 36, 40, 45, 52]\n", "end_tokens = [5, 9, 32, 36, 40, 44, 47, 53]\n", "token_spans = tp.TokenSpanArray(tokens, begin_tokens, end_tokens)\n", "token_spans" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", "
beginendcontext
002In
135AD
269932
31115King
41622Arthur
\n", "

\n", "\n", "\n", "\n", " In\n", "\n", "\n", "\n", " AD\n", "\n", "\n", "\n", " 932\n", "\n", " , \n", "\n", " King\n", "\n", "\n", "\n", " Arthur\n", " and his squire, Patsy, travel throughout Britain searching for men to join the Knights of the Round Table. Along the way, he recruits Sir Bedevere the Wise, Sir Lancelot the Brave, Sir Galahad the Pure, Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot, and Sir Not-Appearing-in-this-Film, along with their squires and Robin's troubadours.\n", "

\n", "
\n", "\n", " Your notebook viewer does not support Javascript execution. The above rendering will not be interactive.\n", "
\n", "\n", "\n" ], "text/plain": [ "\n", "[[0, 2): 'In', [3, 5): 'AD', [6, 9): '932', [11, 15): 'King',\n", " [16, 22): 'Arthur']\n", "Length: 5, dtype: SpanDtype" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# When all the spans in a `TokenSpanArray` come from the same document, you can access\n", "# the tokens of that document via the `document_tokens` property:\n", "token_spans.document_tokens[:5]" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", "
beginendbegin tokenend tokencontext
0112235King Arthur
\n", "

\n", "\n", " In AD 932, \n", "\n", " King Arthur\n", " and his squire, Patsy, travel throughout Britain searching for men to join the Knights of the Round Table. Along the way, he recruits Sir Bedevere the Wise, Sir Lancelot the Brave, Sir Galahad the Pure, Sir Robin the Not-Quite-So-Brave-as-Sir-Lancelot, and Sir Not-Appearing-in-this-Film, along with their squires and Robin's troubadours.\n", "

\n", "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", "
beginendbegin tokenend tokencontext
001502Second document
\n", "

\n", "\n", "\n", "\n", " Second document\n", "\n", "

\n", "
\n", "\n", " Your notebook viewer does not support Javascript execution. The above rendering will not be interactive.\n", "
\n", "\n", "\n" ], "text/plain": [ "\n", "[[11, 22): 'King Arthur', [0, 15): 'Second document']\n", "Length: 2, dtype: TokenSpanDtype" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Both SpanArrays and TokenSpanArrays can contain spans from multiple documents.\n", "tokens_2 = tp.SpanArray(\"Second document\", [0, 7], [6, 15])\n", "token_spans_2 = tp.TokenSpanArray(tokens_2, [0], [2])\n", "\n", "two_doc_series = pd.concat([pd.Series(token_spans[0:1]), pd.Series(token_spans_2)])\n", "two_doc_series.array" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the HTML representation now contains the annotated text of two documents. We can use the `tokens` property to view view the two sets of tokens backing the two spans in this array:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([\n", " [ [0, 2): 'In',\n", " [3, 5): 'AD',\n", " [6, 9): '932',\n", " [11, 15): 'King',\n", " [16, 22): 'Arthur',\n", " [23, 26): 'and',\n", " [27, 30): 'his',\n", " [31, 37): 'squire',\n", " [39, 44): 'Patsy',\n", " [46, 52): 'travel',\n", " [53, 63): 'throughout',\n", " [64, 71): 'Britain',\n", " [72, 81): 'searching',\n", " [82, 85): 'for',\n", " [86, 89): 'men',\n", " [90, 92): 'to',\n", " [93, 97): 'join',\n", " [98, 101): 'the',\n", " [102, 109): 'Knights',\n", " [110, 112): 'of',\n", " [113, 116): 'the',\n", " [117, 122): 'Round',\n", " [123, 128): 'Table',\n", " [130, 135): 'Along',\n", " [136, 139): 'the',\n", " [140, 143): 'way',\n", " [145, 147): 'he',\n", " [148, 156): 'recruits',\n", " [157, 160): 'Sir',\n", " [161, 169): 'Bedevere',\n", " [170, 173): 'the',\n", " [174, 178): 'Wise',\n", " [180, 183): 'Sir',\n", " [184, 192): 'Lancelot',\n", " [193, 196): 'the',\n", " [197, 202): 'Brave',\n", " [204, 207): 'Sir',\n", " [208, 215): 'Galahad',\n", " [216, 219): 'the',\n", " [220, 224): 'Pure',\n", " [226, 229): 'Sir',\n", " [230, 235): 'Robin',\n", " [236, 239): 'the',\n", " [240, 274): 'Not-Quite-So-Brave-as-Sir-Lancelot',\n", " [276, 279): 'and',\n", " [280, 283): 'Sir',\n", " [284, 310): 'Not-Appearing-in-this-Film',\n", " [312, 317): 'along',\n", " [318, 322): 'with',\n", " [323, 328): 'their',\n", " [329, 336): 'squires',\n", " [337, 340): 'and',\n", " [341, 348): 'Robin's',\n", " [349, 360): 'troubadours']\n", " Length: 54, dtype: SpanDtype ,\n", " \n", " [[0, 6): 'Second', [7, 15): 'document']\n", " Length: 2, dtype: SpanDtype ], dtype=object)" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "two_doc_series.array.tokens" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Spanner\n", "\n", "The `spanner` module of Text Extensions for Pandas provides span-specific operations\n", "for Pandas DataFrames, based on the Document Spanners formalism, also known as\n", "spanner algebra.\n", "\n", "Spanner algebra is an extension of relational algebra with additional operations\n", "to cover NLP applications. See the paper [\"Document Spanners: A Formal Approach to\n", "Information Extraction\"](\n", "https://researcher.watson.ibm.com/researcher/files/us-fagin/jacm15.pdf) by Fagin et al.\n", "for more information.\n", "\n", "The available operations in `spanner` include: `consolidate()` to eliminate overlap in a span column, extract matching tokens with `extract_dict()` for dictionary matching or `extract_regex_tok()` for regular expression matching, joining series of spans with `adjacent_join()`, `contain_join()`, or `overlap_join()`, and projection on spans with `lemmatize()`.\n", "\n", "Here we will show how to extract tokens matching regular expressions and then join the results to a DataFrame." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
match
0[157, 169): 'Sir Bedevere'
1[180, 192): 'Sir Lancelot'
2[204, 215): 'Sir Galahad'
3[226, 235): 'Sir Robin'
4[280, 310): 'Sir Not-Appearing-in-this-Film'
\n", "
" ], "text/plain": [ " match\n", "0 [157, 169): 'Sir Bedevere'\n", "1 [180, 192): 'Sir Lancelot'\n", "2 [204, 215): 'Sir Galahad'\n", "3 [226, 235): 'Sir Robin'\n", "4 [280, 310): 'Sir Not-Appearing-in-this-Film'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Extract tokens using a regular expression, here we find all the knights.\n", "knights = tp.spanner.extract_regex_tok(tokens, regex.compile(r\"Sir.\\S+\"), max_len=2)\n", "knights" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
match
0[323, 328): 'their'
0[98, 109): 'the Knights'
1[113, 122): 'the Round'
2[136, 143): 'the way'
3[170, 178): 'the Wise'
4[193, 202): 'the Brave'
5[216, 224): 'the Pure'
6[236, 274): 'the Not-Quite-So-Brave-as-Sir-Lan...
\n", "
" ], "text/plain": [ " match\n", "0 [323, 328): 'their'\n", "0 [98, 109): 'the Knights'\n", "1 [113, 122): 'the Round'\n", "2 [136, 143): 'the way'\n", "3 [170, 178): 'the Wise'\n", "4 [193, 202): 'the Brave'\n", "5 [216, 224): 'the Pure'\n", "6 [236, 274): 'the Not-Quite-So-Brave-as-Sir-Lan..." ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Try to find all knight's virtues, not as easy and end up with other spans. \n", "virtues = tp.spanner.extract_regex_tok(tokens, regex.compile(r\"the.\\S+\"), max_len=2)\n", "virtues" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
knightvirtue
0[157, 169): 'Sir Bedevere'[170, 178): 'the Wise'
1[180, 192): 'Sir Lancelot'[193, 202): 'the Brave'
2[204, 215): 'Sir Galahad'[216, 224): 'the Pure'
3[226, 235): 'Sir Robin'[236, 274): 'the Not-Quite-So-Brave-as-Sir-Lan...
\n", "
" ], "text/plain": [ " knight \\\n", "0 [157, 169): 'Sir Bedevere' \n", "1 [180, 192): 'Sir Lancelot' \n", "2 [204, 215): 'Sir Galahad' \n", "3 [226, 235): 'Sir Robin' \n", "\n", " virtue \n", "0 [170, 178): 'the Wise' \n", "1 [193, 202): 'the Brave' \n", "2 [216, 224): 'the Pure' \n", "3 [236, 274): 'the Not-Quite-So-Brave-as-Sir-Lan... " ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calling `tp.spanner.adjacent_join()` will join two span columns, where a pair\n", "# of spans match if they are adjacent in the text.\n", "\n", "# Now, easily join the 2 results and match each knight to their virtue.\n", "tp.spanner.adjacent_join(knights[\"match\"], virtues[\"match\"], first_name=\"knight\", second_name=\"virtue\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TensorArray\n", "\n", "A `TensorArray` represents an array of [tensors](https://en.wikipedia.org/wiki/Tensor#As_multidimensional_arrays) where each element is an N-dimensional tensor of the same shape. If there are M tensor elements in the array, then the entire `TensorArray` will have a shape of M x N, where the outer dimension is the number of elements. Backing the `TensorArray` is a [numpy.ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) with shape M x N. Tensors, or numpy.ndarrays, are often used as feature vectors for machine learning model training and inference results. In Text Extensions for Pandas, they are used to store BERT embeddings from `io.bert.add_embeddings()` that can then be used to train a NLU model.\n", "\n", "`TensorArray`s can be constructed with zero copy from a single `numpy.ndarray` or with a sequence of elements of similar shape. Conversion of a `TensorArray` to a `numpy.ndarray` can be done with zero copy by calling `TensorArray.to_numpy()` or using the provided numpy array interface, e.g. `numpy.asarray(TensorArray(...))`. The `TensorArray` is a Pandas extension type of type `TensorDtype` and can be wrapped in a `pandas.Series` or used as a column in a `pandas.DataFrame` and used in standard Pandas operations. A `NULL` or missing value in the `TensorArray` is represented as a N-dimensional `numpy.ndarray` where all items are `numpy.nan`. Standard arithmetic and comparison operations are supported and delegated to the backing `numpy.ndarray`. Taking a slice or multiple item selection will produce another `TensorArray`, while a single element selection will produce a `TensorElement` that also wraps a view of the `numpy.ndarray`, with similar operator support." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(array([[0, 1],\n", " [2, 3],\n", " [4, 5],\n", " [6, 7],\n", " [8, 9]]),\n", " )" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Construct from a numpy.ndarray.\n", "arr = tp.TensorArray(np.arange(10).reshape(5, 2))\n", "arr, arr.dtype" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 [0, 1]\n", "1 [2, 3]\n", "2 [4, 5]\n", "3 [6, 7]\n", "4 [8, 9]\n", "dtype: TensorDtype" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Wrap in a Pandas Series.\n", "s = pd.Series(arr)\n", "s" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(array([[0, 1],\n", " [2, 3],\n", " [4, 5],\n", " [6, 7],\n", " [8, 9]]),\n", " dtype('int64'))" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Convert back to numpy using the provided array interface.\n", "np_arr = np.asarray(s)\n", "np_arr, np_arr.dtype" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 [ False, False]\n", "1 [ False, False]\n", "2 [ False, True]\n", "3 [ True, True]\n", "4 [ True, True]\n", "dtype: TensorDtype" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Apply operations on the Series, result is another Series of type TensorDtype.\n", "thresh = s > 4\n", "thresh" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(array([False, False, False, True, True]),\n", " text_extensions_for_pandas.array.tensor.TensorArray)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a boolean selection mask. Use `.array` to get the Series as\n", "# a `TensorArray` which can be used directly on numpy operations and\n", "# returns another `TensorArray`\n", "mask = np.all(thresh.array, axis=1)\n", "mask, type(mask)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3 [6, 7]\n", "4 [8, 9]\n", "dtype: TensorDtype" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Apply Pandas selection on the Series of TensorDtype by converting\n", "# the mask to a numpy boolean array.\n", "s[mask.to_numpy()]" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timefeatures
02018-01-01 00:00:00[0, 1]
12018-01-01 01:00:00[2, 3]
22018-01-01 02:00:00[4, 5]
32018-01-01 03:00:00[6, 7]
42018-01-01 04:00:00[8, 9]
\n", "
" ], "text/plain": [ " time features\n", "0 2018-01-01 00:00:00 [0, 1]\n", "1 2018-01-01 01:00:00 [2, 3]\n", "2 2018-01-01 02:00:00 [4, 5]\n", "3 2018-01-01 03:00:00 [6, 7]\n", "4 2018-01-01 04:00:00 [8, 9]" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# TensorArray can also be added to a Pandas DataFrame.\n", "df = pd.DataFrame({\"time\": pd.date_range('2018-01-01', periods=5, freq='H'), \"features\": arr})\n", "df" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timefeatures
42018-01-01 04:00:00[8, 9]
32018-01-01 03:00:00[6, 7]
22018-01-01 02:00:00[4, 5]
12018-01-01 01:00:00[2, 3]
02018-01-01 00:00:00[0, 1]
\n", "
" ], "text/plain": [ " time features\n", "4 2018-01-01 04:00:00 [8, 9]\n", "3 2018-01-01 03:00:00 [6, 7]\n", "2 2018-01-01 02:00:00 [4, 5]\n", "1 2018-01-01 01:00:00 [2, 3]\n", "0 2018-01-01 00:00:00 [0, 1]" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# TensorArray supports many of the standard DataFrame operations.\n", "df.sort_values(by=\"time\", ascending=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Saving Pandas Extension Arrays to Disk\n", "\n", "Pandas supports several built-in I/O formats, but currently the only supported format for saving DataFrames with Text Extensions for Pandas arrays to disk is with [Feather](https://arrow.apache.org/docs/python/feather.html) files. Text Extensions for Pandas arrays can also be converted to Apache Arrow format, see https://arrow.apache.org/docs/python/pandas.html#dataframes for more information." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "# Dummy function to create some features.\n", "def hasher(span, num_features=4):\n", " arr = np.zeros(num_features, dtype=\"int8\")\n", " arr[hash(span.covered_text) % 4] = 1\n", " return arr" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(54, 4)" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create our feature vector.\n", "features = tp.TensorArray([hasher(span) for span in tokens])\n", "features.to_numpy().shape" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
spanfeatures
0[0, 2): 'In'[0, 0, 0, 1]
1[3, 5): 'AD'[1, 0, 0, 0]
2[6, 9): '932'[0, 0, 1, 0]
3[11, 15): 'King'[0, 1, 0, 0]
4[16, 22): 'Arthur'[1, 0, 0, 0]
\n", "
" ], "text/plain": [ " span features\n", "0 [0, 2): 'In' [0, 0, 0, 1]\n", "1 [3, 5): 'AD' [1, 0, 0, 0]\n", "2 [6, 9): '932' [0, 0, 1, 0]\n", "3 [11, 15): 'King' [0, 1, 0, 0]\n", "4 [16, 22): 'Arthur' [1, 0, 0, 0]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add tokens and features to a DataFrame.\n", "df = pd.DataFrame({\"span\": tokens, \"features\": features})\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "# Save DataFrame to a feather file.\n", "# Feather is a lightweight, fast binary columnar format, with basic\n", "# compression and support built into Pandas.\n", "df.to_feather(\"outputs/tp_overview.feather\")" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
spanfeatures
0[0, 2): 'In'[0, 0, 0, 1]
1[3, 5): 'AD'[1, 0, 0, 0]
2[6, 9): '932'[0, 0, 1, 0]
3[11, 15): 'King'[0, 1, 0, 0]
4[16, 22): 'Arthur'[1, 0, 0, 0]
\n", "
" ], "text/plain": [ " span features\n", "0 [0, 2): 'In' [0, 0, 0, 1]\n", "1 [3, 5): 'AD' [1, 0, 0, 0]\n", "2 [6, 9): '932' [0, 0, 1, 0]\n", "3 [11, 15): 'King' [0, 1, 0, 0]\n", "4 [16, 22): 'Arthur' [1, 0, 0, 0]" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Read the file back into a new DataFrame.\n", "\n", "df_load = pd.read_feather(\"outputs/tp_overview.feather\")\n", "df_load.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## NLP Library Input/Output Integration\n", "\n", "Text Extensions for Pandas also provides integration with other NLP libraries and datasets. It takes care of processing the inputs and outputs using Pandas DataFrame as a standard data structure and automatically producing the above extension arrays where applicable. Below is an overview of what each module provides along with more notebooks with example usage.\n", "\n", "### Watson\n", "\n", "The `io.watson` sub-package provides functions to process and help analyze responses the IBM Waton Cloud service APIs.\n", "\n", "In the module `io.watson.nlu` you can use Watson Natural Language Understanding to analyze text and then process the response into Pandas DataFrames containing `SpanArray`s for tokens, sentences and relations. See [getting started on Watson NLU](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-getting-started) for setting up the Watson NLU Cloud Service, and the notebook [Analyze_Text](./Analyze_Text.ipynb) for in-depth examples of using the `io.watson.nlu` module.\n", "\n", "In the module `io.watson.table` you can use Watson Discovery to extract and analyze tables within documents and web pages, and then process the response into Pandas DataFrames that make it easy to reconstruct and work with the extracted tables. See [Waston Discovery Installation](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-install) and [IBM Cloud Pak for Data](https://www.ibm.com/products/cloud-pak-for-data) for getting started with Watson Discovery, and the notebook [Understand_Tables](./Understand_Tables.ipynb) for an in-depth example of using the `watson.table` module.\n", "\n", "### SpaCy\n", "\n", "The `io.spacy` module contains functions to integrate with the popular NLP library [SpaCy](https://spacy.io/). This allows you to use a [SpaCy tokenizer](https://spacy.io/usage/spacy-101#annotations-token) on text and return the tokens as a `SpanArray` in a Pandas DataFrame with `io.spacy.make_tokens()` or with additional token features with `io.spacy.make_tokens_and_features()`. See the notebook [Integrate_NLP_Libraries](./Integrate_NLP_Libraries.ipynb) for more examples with the `io.spacy` module.\n", "\n", "### BERT\n", "\n", "The BERT model is originally from the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. The model is pre-trained with masked language modeling and next sentence prediction objectives, which make it effective for masked token prediction and NLU.\n", "\n", "Text Extension for Pandas integrates with the [Huggingface Transformers](https://huggingface.co/transformers/index.html) library to process the result of BERT tokenization into a Pandas DataFrame with tokens as a`SpanArray` column and compute BERT embbeddings that can also be added to a DataFrame as a `TensorArray`. The embeddings can be used for model training in your NLP application. See the notebook [Model_Training_with_BERT](./Model_Training_with_BERT.ipynb) for an example of tokenizing text with BERT and computing embeddings for model training/scoring.\n", "\n", "### CoNLL\n", "\n", "[CoNLL](https://www.conll.org/), the SIGNLL Conference on Computational Natural Language Learning, is an annual academic conference for natural language processing researchers. Each year's conference features a competition involving a challenging NLP task. The task for the 2003 competition involved identifying mentions of [named entities](https://en.wikipedia.org/wiki/Named-entity_recognition) in English and German news articles from the late 1990's. The corpus for this 2003 competition is one of the most widely-used benchmarks for the performance of named entity recognition models.\n", "\n", "Text Extensions for Pandas contains the module `io.conll` that can help work with an analyze the CoNLL-2003 corpus. The provided functions can help convert between the [IOB2 format](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)) used in the corpus, and `SpanArray` with entity type for easier analysis. See the notebooks [Analyze_Model_Outputs](./Analyze_Model_Outputs.ipynb) for an in-depth analysis of the corpus and the 2003 competition results, and [Model_Training_with_BERT](./Model_Training_with_BERT.ipynb) for using the corpus to train a named entity recognition (NER) model." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.17" } }, "nbformat": 4, "nbformat_minor": 4 }