{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "---\n", "Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Use the Banks example corpus" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load TF\n", "\n", "We (down)load the corpus." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from tf.app import use" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "**Locating corpus resources ...**" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "app: ~/text-fabric-data/github/annotation/banks/app" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "data: ~/text-fabric-data/github/annotation/banks/tf/0.2" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " TF: TF API 12.5.4, annotation/banks/app v3, Search Reference
\n", " Data: annotation - banks 0.2, Character table, Feature docs
\n", "
Node types\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "
Name# of nodes# slots / node% coverage
book199.00100
chapter249.50100
sentence333.00100
line127.6793
word991.00100
\n", " Sets: no custom sets
\n", " Features:
\n", "
Two quotes from Consider Phlebas by Iain M. Banks\n", "
\n", "\n", "
\n", "
\n", "author\n", "
\n", "
str
\n", "\n", " the author of a book\n", "\n", "
\n", "\n", "
\n", "
\n", "gap\n", "
\n", "
int
\n", "\n", " 1 for words that occur between [ ], which are inserted by the editor\n", "\n", "
\n", "\n", "
\n", "
\n", "letters\n", "
\n", "
str
\n", "\n", " the letters of a word\n", "\n", "
\n", "\n", "
\n", "
\n", "number\n", "
\n", "
int
\n", "\n", " number of chapter, or sentence in chapter, or line in sentence\n", "\n", "
\n", "\n", "
\n", "
\n", "otype\n", "
\n", "
str
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "punc\n", "
\n", "
str
\n", "\n", " the punctuation after a word\n", "\n", "
\n", "\n", "
\n", "
\n", "terminator\n", "
\n", "
str
\n", "\n", " the last character of a line\n", "\n", "
\n", "\n", "
\n", "
\n", "title\n", "
\n", "
str
\n", "\n", " the title of a book\n", "\n", "
\n", "\n", "
\n", "
\n", "oslots\n", "
\n", "
none
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "\n", " Settings:
specified
  1. apiVersion: 3
  2. appName: annotation/banks
  3. appPath: /Users/me/text-fabric-data/github/annotation/banks/app
  4. commit: g5e28b54aaf679d5fbbbf7e879a6316f323b9d330
  5. css: ''
  6. dataDisplay:
  7. \n", " textFormats:\n", "
  8. \n", " layout-orig-full:\n", "
    • method: layoutRich
    • style: normal
    \n", "
    9. \n", "
  9. docs:
    • docBase: {docRoot}/{org}/{repo}/blob/master/programs
    • docExt: .ipynb
    • docPage: convert
    • docRoot: {urlNb}
    • featureBase: {docBase}
    • featurePage: convert
  10. interfaceDefaults: {}
  11. isCompatible: True
  12. local: local
  13. localDir: /Users/me/text-fabric-data/github/annotation/banks/_temp
  14. provenanceSpec:
    • corpus: Two quotes from Consider Phlebas by Iain M. Banks
    • doi: 10.5281/zenodo.2630416
    • org: annotation
    • relative: /tf
    • repo: banks
    • version: 0.2
  15. release: v3.1
  16. typeDisplay:
    • book: {featuresBare: author}
    • line:
      • features: terminator
      • label: {number}
      • template: {number}
      • verselike: True
    • word: {features: gap}
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
TF API: names N F E L T S C TF Fs Fall Es Eall Cs Call directly usable
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "A = use(\"annotation/banks\", hoist=globals())" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Exploration\n", "\n", "Let's explore this corpus by means of Text-Fabric.\n", "\n", "### Frequency list\n", "\n", "We can get ordered frequency lists for the values of all features.\n", "\n", "First the words:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(('the', 8),\n", " ('of', 5),\n", " ('and', 4),\n", " ('in', 3),\n", " ('we', 3),\n", " ('everything', 2),\n", " ('know', 2),\n", " ('most', 2),\n", " ('ones', 2),\n", " ('patterns', 2),\n", " ('us', 2),\n", " ('Besides', 1),\n", " ('Culture', 1),\n", " ('Everything', 1),\n", " ('So', 1),\n", " ('a', 1),\n", " ('about', 1),\n", " ('aid', 1),\n", " ('any', 1),\n", " ('around', 1),\n", " ('as', 1),\n", " ('barbarian', 1),\n", " ('bottom', 1),\n", " ('can', 1),\n", " ('care', 1),\n", " ('climbing', 1),\n", " ('composed', 1),\n", " ('control', 1),\n", " ('dead', 1),\n", " ('elegant', 1),\n", " ('enjoyable', 1),\n", " ('final', 1),\n", " ('find', 1),\n", " ('free', 1),\n", " ('games', 1),\n", " ('good', 1),\n", " ('harness', 1),\n", " ('have', 1),\n", " ('high', 1),\n", " ('humans', 1),\n", " ('impossible', 1),\n", " ('is', 1),\n", " ('it', 1),\n", " ('languages', 1),\n", " ('left', 1),\n", " ('life', 1),\n", " ('line', 1),\n", " ('make', 1),\n", " ('mattered', 1),\n", " ('mountains', 1),\n", " ('not', 1),\n", " ('nothing', 1),\n", " ('our', 1),\n", " ('over', 1),\n", " ('own', 1),\n", " ('problems', 1),\n", " ('really', 1),\n", " ('romance', 1),\n", " ('safety', 1),\n", " ('societies', 1),\n", " ('sports', 1),\n", " ('studying', 1),\n", " ('such', 1),\n", " ('take', 1),\n", " ('terms', 1),\n", " ('that', 1),\n", " ('that’s', 1),\n", " ('things', 1),\n", " ('those', 1),\n", " ('to', 1),\n", " ('truth', 1),\n", " ('ultimately', 1),\n", " ('where', 1),\n", " ('why', 1),\n", " ('without', 1))" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "F.letters.freqList()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the node types we can get info by calling this:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "lines_to_next_cell": 2 }, "outputs": [ { "data": { "text/plain": [ "(('book', 99.0, 100, 100),\n", " ('chapter', 49.5, 101, 102),\n", " ('sentence', 33.0, 115, 117),\n", " ('line', 7.666666666666667, 103, 114),\n", " ('word', 1, 1, 99))" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "C.levels.data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It means that chapters are 49.5 words long on average, and that the chapter nodes are 101 and 102.\n", "\n", "And you see that we have 99 words." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Add to the banks corpus\n", "\n", "We are going to make a relationship between each pair of words, and we annotate each related pair with how similar they are.\n", "\n", "We measure the similarity by looking at the distinct letters in each word (lowercase), and computing the percentage of\n", "how many letters they have in common with respect to how many letters they jointly have.\n", "\n", "This will become a symmetric edge feature. Symmetric means, that if a and b are similar, then b and a as well, with the\n", "same similarity.\n", "\n", "We only store one copy of each symmetric pair of edges.\n", "\n", "We can then use\n", "[`E.sim.b(node)`](https://annotation.github.io/text-fabric/tf/core/edgefeature.html)\n", "to find all nodes that are parallel to node.\n", "\n", "If words do not have letters in common, their similarity is 0, and we do not make an edge." ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 2 }, "source": [ "# Preparation\n", "\n", "We pre-compute all letter sets for all words." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def makeSet(w):\n", " return set(F.letters.v(w).lower())" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "lines_to_end_of_cell_marker": 2, "lines_to_next_cell": 2 }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "99 words\n" ] } ], "source": [ "words = {}\n", "\n", "for w in F.otype.s(\"word\"):\n", " words[w] = makeSet(w)\n", "\n", "nWords = len(words)\n", "print(f\"{nWords} words\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "lines_to_next_cell": 2 }, "outputs": [], "source": [ "def sim(wSet, vSet):\n", " return int(round(100 * len(wSet & vSet) / len(wSet | vSet)))" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 2 }, "source": [ "# Compute all similarities\n", "\n", "We are going to perform all comparisons.\n", "\n", "Since there are 99 words, this will amount to only 5000 comparisons.\n", "\n", "For a big corpus, this amount will quickly grow with the number of items to be compared.\n", "\n", "See for example the similarities in the\n", "[Quran](https://nbviewer.jupyter.org/github/q-ran/quran/blob/master/programs/parallels.ipynb)." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "def computeSim():\n", " similarity = {}\n", "\n", " wordNodes = sorted(words.keys())\n", " nWords = len(wordNodes)\n", "\n", " nComparisons = nWords * (nWords - 1) // 2\n", "\n", " print(f\"{nComparisons} comparisons to make\")\n", "\n", " TF.indent(reset=True)\n", "\n", " co = 0\n", " si = 0\n", " stop = False\n", " for i in range(nWords):\n", " nodeI = wordNodes[i]\n", " wordI = words[nodeI]\n", " for j in range(i + 1, nWords):\n", " nodeJ = wordNodes[j]\n", " wordJ = words[nodeJ]\n", " s = sim(wordI, wordJ)\n", " co += 1\n", " if s:\n", " similarity[(nodeI, nodeJ)] = sim(wordI, wordJ)\n", " si += 1\n", " if stop:\n", " break\n", "\n", " TF.info(f\"{co:>4} comparisons and {si:>4} similarities\")\n", " return similarity" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4851 comparisons to make\n", " 0.01s 4851 comparisons and 3332 similarities\n" ] } ], "source": [ "similarity = computeSim()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "7\n", "100\n" ] } ], "source": [ "print(min(similarity.values()))\n", "print(max(similarity.values()))" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "eq = [x for x in similarity.items() if x[1] >= 100]\n", "neq = [x for x in similarity.items() if x[1] <= 50]" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "((1, 4), 100)\n", "((1, 2), 8)\n" ] } ], "source": [ "print(eq[0])\n", "print(neq[0])" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "58\n", "3247\n" ] } ], "source": [ "print(len(eq))\n", "print(len(neq))" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 Everything\n", "4 everything\n" ] } ], "source": [ "print(eq[0][0][0], F.letters.v(eq[0][0][0]))\n", "print(eq[0][0][1], F.letters.v(eq[0][0][1]))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 Everything\n", "2 about\n" ] } ], "source": [ "print(neq[0][0][0], F.letters.v(neq[0][0][0]))\n", "print(neq[0][0][1], F.letters.v(neq[0][0][1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Add parallels to the TF dataset\n", "\n", "We now add this information to the Banks dataset as an *edge feature*." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "import os" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "GH_BASE = os.path.expanduser(\"~/github\")\n", "\n", "path = f\"{A.context.org}/{A.context.repo}/sim/tf\"\n", "location = f\"{GH_BASE}/{path}\"\n", "module = A.context.version" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "metaData = {\n", " \"\": {\n", " \"name\": \"Banks (similar words)\",\n", " \"converters\": \"Dirk Roorda\",\n", " \"sourceUrl\": \"https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/text-fabric/use.ipynb\",\n", " \"version\": \"0.2\",\n", " },\n", " \"sim\": {\n", " \"valueType\": \"int\",\n", " \"edgeValues\": True,\n", " \"description\": \"similarity between words, as a percentage of the common material wrt the combined material\",\n", " },\n", "}" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "simData = {}\n", "for ((f, t), d) in similarity.items():\n", " simData.setdefault(f, {})[t] = d" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s Exporting 0 node and 1 edge and 0 configuration features to ~/github/annotation/banks/sim/tf/0.2:\n", " | 0.00s T sim to ~/github/annotation/banks/sim/tf/0.2\n", " 0.00s Exported 0 node features and 1 edge features and 0 config features to ~/github/annotation/banks/sim/tf/0.2\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "A.api.TF.save(\n", " edgeFeatures=dict(sim=simData), metaData=metaData, location=location, module=module\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "All chapters:\n", "\n", "* *use*\n", "* [share](share.ipynb)\n", "* [app](app.ipynb)\n", "* [repo](repo.ipynb)\n", "* [compose](compose.ipynb)\n", "\n", "---\n", "\n", "CC-BY Dirk Roorda" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.4" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }