{ "cells": [ { "cell_type": "markdown", "id": "7f891ec4-ea4a-4e21-b87a-4c163946080d", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "# [Doc4TF](https://github.com/tonyjurg/Doc4TF)\n", "#### *Automatic creation of feature documentation for existing Text-Fabric datasets*\n", "\n", "Version and release date:" ] }, { "cell_type": "code", "execution_count": 1, "id": "a7b8b60e-16e3-4dcf-bc09-a592c3701aee", "metadata": {}, "outputs": [], "source": [ "scriptVersion=\"0.5.1\"\n", "scriptDate=\"June 16, 2024\"" ] }, { "cell_type": "markdown", "id": "da55f600-c935-4b2e-835f-e8eeaad7d38c", "metadata": { "jp-MarkdownHeadingCollapsed": true, "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Table of content \n", "* 1 - Introduction\n", "* 2 - Setting up the environment\n", "* 3 - Load Text-Fabric data\n", "* 4 - Creation of the dataset\n", " * 4.1 - Setting up some global variables\n", " * 4.2 - Store all relevant data into a dictionary\n", "* 5 - Create the documentation pages\n", " * 5.1 - Create the set of feature pages\n", " * 5.2 - Create the index pages\n", "* 6 - Changelog\n", "* 7 - Licence" ] }, { "cell_type": "markdown", "id": "411d018e-e242-48de-9117-4e2f95f055fc", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "# 1 - Introduction \n", "##### [Back to TOC](#TOC)\n", "\n", "Ideally, a comprehensive documentation set should be created as part of developing a Text-Fabric dataset. However, in practice, this is not always completed during the initial phase or after changes to features. This Jupyter Notebook contains Python code to automatically generate (and thus ensure consistency) a documentation set for any [Text-Fabric](https://github.com/annotation/text-fabric) dataset. It serves as a robust starting point for the development of a brand new documentation set or as validation for an existing one. One major advantage is that the resulting documentation set is fully hyperlinked, a task that can be laborious if done manually.\n", "\n", "The main steps in producing the documentation set are:\n", "* Load a Text-Fabric database\n", "* Execute the code pressent in the subsequent cells. The code will:\n", " * Construct the python dictionary stroring relevant data from the TF datase \n", " * Create separate files for each feature\n", " * Create a set of overview pages sorting the nodes accordingly \n", " \n", "The output format can be either Markdown, the standard for feature documentation stored on GitHub using its on-site processor, or HTML, which facilitates local storage and browsing with any web browser." ] }, { "cell_type": "markdown", "id": "0901649a-bf63-485f-9499-8311210b6ef7", "metadata": {}, "source": [ "# 2. Setting up the environment\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "markdown", "id": "fcac99b8-f133-4fdc-ad10-97427954dee8", "metadata": {}, "source": [ "Your environment should (for obvious reasons) include the Python package `Text-Fabric`. Text-Fabric requires at least Python version 3.7.0. If not installed yet, it can be installed using `pip`. More details on installing the Text-Fabric package can be found in [tf.about.install](https://annotation.github.io/text-fabric/tf/about/install.html).\n", "\n", "Further it is required to be able to invoke the Text-Fabric data set (either from an online resource, or from a localy stored copy). There are no further requirements as the scripts basicly operate 'stand alone'. " ] }, { "cell_type": "markdown", "id": "ab0e6ff5-4f6b-4c2f-b6cb-db2a039d042a", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 3 - Load Text-Fabric data \n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "markdown", "id": "ceb50c9d-31c1-42d9-8e89-1c56b88d4e5f", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "At this step, the Text-Fabric dataset is loaded, which embedded data will be used to create a documentation set. \n", "\n", "Which dataset will be loaded is specified in the parameters as detailed below:\n", "```\n", "A = use (\"{GitHub user name}/{repository name}\", version=\"{version}\", hoist=globals())\n", "```\n", "For various options regarding other possible storage locations, and other load options, see the documentation for function [`use`](https://annotation.github.io/text-fabric/tf/app.html#tf.app.use)." ] }, { "cell_type": "code", "execution_count": 2, "id": "177f91d0-0baf-45eb-8450-dcce0e6b4b86", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 3, "id": "20a63ffb-0b5d-4586-bce3-bcb39a84273d", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "outputs": [], "source": [ "# Loading the Text-Fabric code\n", "# Note: it is assumed Text-Fabric is installed in your environment\n", "from tf.fabric import Fabric\n", "from tf.app import use" ] }, { "cell_type": "code", "execution_count": 4, "id": "1f2d822d-b47a-4766-8226-8157799740c0", "metadata": { "scrolled": true, "slideshow": { "slide_type": "slide" }, "tags": [] }, "outputs": [ { "data": { "text/markdown": [ "**Locating corpus resources ...**" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "app: ~/text-fabric-data/github/saulocantanhede/tfgreek2/app" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "data: ~/text-fabric-data/github/saulocantanhede/tfgreek2/tf/0.5.7" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " TF: TF API 12.4.4, saulocantanhede/tfgreek2/app v3, Search Reference
\n", " Data: saulocantanhede - tfgreek2 0.5.7, Character table, Feature docs
\n", "
Node types\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "
Name# of nodes# slots / node% coverage
book275102.93100
chapter260529.92100
verse794417.34100
sentence1970313.82198
group89457.0146
clause308147.17160
wg1068686.88533
phrase690071.9095
subphrase1161781.60135
word1377791.00100
\n", " Sets: no custom sets
\n", " Features:
\n", "
Nestle 1904 Greek New Testament\n", "
\n", "\n", "
\n", "
\n", "after\n", "
\n", "
str
\n", "\n", " material after the end of the word\n", "\n", "
\n", "\n", "
\n", "
\n", "appositioncontainer\n", "
\n", "
int
\n", "\n", " 1 if it is an apposition container\n", "\n", "
\n", "\n", "
\n", "
\n", "articular\n", "
\n", "
int
\n", "\n", " 1 if the sentence, group, clause, phrase or wg has an article\n", "\n", "
\n", "\n", "
\n", "
\n", "before\n", "
\n", "
str
\n", "\n", " this is XML attribute before\n", "\n", "
\n", "\n", "
\n", "
\n", "book\n", "
\n", "
str
\n", "\n", " book name (full name)\n", "\n", "
\n", "\n", "
\n", "
\n", "bookshort\n", "
\n", "
str
\n", "\n", " book name (abbreviated) from ref attribute in xml\n", "\n", "
\n", "\n", "
\n", "
\n", "case\n", "
\n", "
str
\n", "\n", " grammatical case\n", "\n", "
\n", "\n", "
\n", "
\n", "chapter\n", "
\n", "
int
\n", "\n", " chapter number, from ref attribute in xml\n", "\n", "
\n", "\n", "
\n", "
\n", "clausetype\n", "
\n", "
str
\n", "\n", " clause type\n", "\n", "
\n", "\n", "
\n", "
\n", "cls\n", "
\n", "
str
\n", "\n", " this is XML attribute cls\n", "\n", "
\n", "\n", "
\n", "
\n", "cltype\n", "
\n", "
str
\n", "\n", " clause type\n", "\n", "
\n", "\n", "
\n", "
\n", "criticalsign\n", "
\n", "
str
\n", "\n", " this is XML attribute criticalsign\n", "\n", "
\n", "\n", "
\n", "
\n", "crule\n", "
\n", "
str
\n", "\n", " clause rule (from xml attribute Rule)\n", "\n", "
\n", "\n", "
\n", "
\n", "degree\n", "
\n", "
str
\n", "\n", " grammatical degree\n", "\n", "
\n", "\n", "
\n", "
\n", "discontinuous\n", "
\n", "
int
\n", "\n", " 1 if the word is out of sequence in the xml\n", "\n", "
\n", "\n", "
\n", "
\n", "domain\n", "
\n", "
str
\n", "\n", " domain\n", "\n", "
\n", "\n", "
\n", "
\n", "framespec\n", "
\n", "
str
\n", "\n", " this is XML attribute framespec\n", "\n", "
\n", "\n", "
\n", "
\n", "function\n", "
\n", "
str
\n", "\n", " this is XML attribute function\n", "\n", "
\n", "\n", "
\n", "
\n", "gender\n", "
\n", "
str
\n", "\n", " grammatical gender\n", "\n", "
\n", "\n", "
\n", "
\n", "gloss\n", "
\n", "
str
\n", "\n", " short translation\n", "\n", "
\n", "\n", "
\n", "
\n", "id\n", "
\n", "
str
\n", "\n", " xml id\n", "\n", "
\n", "\n", "
\n", "
\n", "junction\n", "
\n", "
str
\n", "\n", " type of junction\n", "\n", "
\n", "\n", "
\n", "
\n", "lang\n", "
\n", "
str
\n", "\n", " language the text is in\n", "\n", "
\n", "\n", "
\n", "
\n", "lemma\n", "
\n", "
str
\n", "\n", " lexical lemma\n", "\n", "
\n", "\n", "
\n", "
\n", "lemmatranslit\n", "
\n", "
str
\n", "\n", " transliteration of the word lemma\n", "\n", "
\n", "\n", "
\n", "
\n", "ln\n", "
\n", "
str
\n", "\n", " ln\n", "\n", "
\n", "\n", "
\n", "
\n", "mood\n", "
\n", "
str
\n", "\n", " verbal mood\n", "\n", "
\n", "\n", "
\n", "
\n", "morph\n", "
\n", "
str
\n", "\n", " morphological code\n", "\n", "
\n", "\n", "
\n", "
\n", "nodeid\n", "
\n", "
int
\n", "\n", " node id (as in the XML source data\n", "\n", "
\n", "\n", "
\n", "
\n", "normalized\n", "
\n", "
str
\n", "\n", " lemma normalized\n", "\n", "
\n", "\n", "
\n", "
\n", "note\n", "
\n", "
str
\n", "\n", " annotation of linguistic nature\n", "\n", "
\n", "\n", "
\n", "
\n", "num\n", "
\n", "
int
\n", "\n", " generated number (not in xml): book: (Matthew=1, Mark=2, ..., Revelation=27); sentence: numbered per chapter; word: numbered per verse.\n", "\n", "
\n", "\n", "
\n", "
\n", "number\n", "
\n", "
str
\n", "\n", " grammatical number\n", "\n", "
\n", "\n", "
\n", "
\n", "otype\n", "
\n", "
str
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "person\n", "
\n", "
str
\n", "\n", " grammatical person\n", "\n", "
\n", "\n", "
\n", "
\n", "punctuation\n", "
\n", "
str
\n", "\n", " this is XML attribute punctuation\n", "\n", "
\n", "\n", "
\n", "
\n", "ref\n", "
\n", "
str
\n", "\n", " biblical reference with word counting\n", "\n", "
\n", "\n", "
\n", "
\n", "referent\n", "
\n", "
str
\n", "\n", " number of referent\n", "\n", "
\n", "\n", "
\n", "
\n", "rela\n", "
\n", "
str
\n", "\n", " this is XML attribute rela\n", "\n", "
\n", "\n", "
\n", "
\n", "role\n", "
\n", "
str
\n", "\n", " role\n", "\n", "
\n", "\n", "
\n", "
\n", "rule\n", "
\n", "
str
\n", "\n", " syntactical rule\n", "\n", "
\n", "\n", "
\n", "
\n", "sp\n", "
\n", "
str
\n", "\n", " part-of-speach\n", "\n", "
\n", "\n", "
\n", "
\n", "strong\n", "
\n", "
int
\n", "\n", " strong number\n", "\n", "
\n", "\n", "
\n", "
\n", "subjrefspec\n", "
\n", "
str
\n", "\n", " this is XML attribute subjrefspec\n", "\n", "
\n", "\n", "
\n", "
\n", "tense\n", "
\n", "
str
\n", "\n", " verbal tense\n", "\n", "
\n", "\n", "
\n", "
\n", "text\n", "
\n", "
str
\n", "\n", " the text of a word\n", "\n", "
\n", "\n", "
\n", "
\n", "trans\n", "
\n", "
str
\n", "\n", " translation of the word surface text according to the Berean Interlinear Bible\n", "\n", "
\n", "\n", "
\n", "
\n", "translit\n", "
\n", "
str
\n", "\n", " transliteration of the word surface text\n", "\n", "
\n", "\n", "
\n", "
\n", "typ\n", "
\n", "
str
\n", "\n", " this is XML attribute typ\n", "\n", "
\n", "\n", "
\n", "
\n", "type\n", "
\n", "
str
\n", "\n", " morphological type (on word), syntactical type (on sentence, group, clause, phrase or wg)\n", "\n", "
\n", "\n", "
\n", "
\n", "unaccent\n", "
\n", "
str
\n", "\n", " word in unicode characters without accents and diacritical markers\n", "\n", "
\n", "\n", "
\n", "
\n", "unicode\n", "
\n", "
str
\n", "\n", " word in unicode characters plus material after it\n", "\n", "
\n", "\n", "
\n", "
\n", "variant\n", "
\n", "
str
\n", "\n", " this is XML attribute variant\n", "\n", "
\n", "\n", "
\n", "
\n", "verse\n", "
\n", "
int
\n", "\n", " verse number, from ref attribute in xml\n", "\n", "
\n", "\n", "
\n", "
\n", "voice\n", "
\n", "
str
\n", "\n", " verbal voice\n", "\n", "
\n", "\n", "
\n", "
\n", "frame\n", "
\n", "
str
\n", "\n", " frame\n", "\n", "
\n", "\n", "
\n", "
\n", "oslots\n", "
\n", "
none
\n", "\n", " \n", "\n", "
\n", "\n", "
\n", "
\n", "parent\n", "
\n", "
none
\n", "\n", " parent relationship between words\n", "\n", "
\n", "\n", "
\n", "
\n", "sibling\n", "
\n", "
int
\n", "\n", " this is XML attribute sibling\n", "\n", "
\n", "\n", "
\n", "
\n", "subjref\n", "
\n", "
none
\n", "\n", " number of subject referent\n", "\n", "
\n", "\n", "
\n", "
\n", "\n", " Settings:
specified
  1. apiVersion: 3
  2. appName: saulocantanhede/tfgreek2
  3. appPath:C:/Users/tonyj/text-fabric-data/github/saulocantanhede/tfgreek2/app
  4. commit: 352af50c8ce86edd8a0e2d58519453a8f53ee084
  5. css: ''
  6. dataDisplay:
    • excludedFeatures: []
    • noneValues:
      • none
      • unknown
      • no value
      • NA
    • sectionSep1:
    • sectionSep2: :
    • textFormat: text-orig-full
  7. docs:
    • docBase: https://github.com/saulocantanhede/tfgreek2/tree/main/docs
    • docPage: about
    • docRoot: https://github.com/saulocantanhede/tfgreek2
    • featureBase:https://github.com/saulocantanhede/tfgreek2/tree/main/docs/features/<feature>.md
    • featurePage: README
  8. interfaceDefaults: {fmt: text-orig-full}
  9. isCompatible: True
  10. local: local
  11. localDir:C:/Users/tonyj/text-fabric-data/github/saulocantanhede/tfgreek2/_temp
  12. provenanceSpec:
    • branch: main
    • corpus: Nestle 1904 Greek New Testament
    • doi: 10.5281/zenodo.notyet
    • moduleSpecs: []
    • org: saulocantanhede
    • relative: /tf
    • repo: tfgreek2
    • repro: tfgreek2
    • version: 0.5.7
    • webBase: https://learner.bible/text/show_text/nestle1904/
    • webHint: Show this on the website
    • webLang: en
    • webUrl:https://learner.bible/text/show_text/nestle1904/<1>/<2>/<3>
    • webUrlLex: {webBase}/word?version={version}&id=<lid>
  13. release: 0.5.7
  14. typeDisplay:
    • clause:
      • condense: True
      • label: {typ} {function} {rela} \\\\ {cls} {role} {junction}
      • style: ''
    • group:
      • label: {typ} {function} {rela} \\\\ {type} {role} {rule}
      • style: ''
    • phrase:
      • condense: True
      • label: {typ} {function} {rela} \\\\ {type} {role} {rule}
      • style: ''
    • sentence:
      • label: {typ} {function} {rela} \\\\ {role} {rule}
      • style: ''
    • subphrase:
      • label: {typ} {function} {rela} \\\\ {type} {role} {rule}
      • style: ''
    • verse:
      • condense: True
      • label: {book} {chapter}:{verse}
      • style: ''
    • wg:
      • condense: True
      • label: {type} {role} {rule} {junction}
      • style: ''
    • word:
      • features:
        • lemma
        • sp
      • featuresBare: [gloss]
  15. writing: grc
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
TF API: names N F E L T S C TF Fs Fall Es Eall Cs Call directly usable
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "Display is setup for viewtype [syntax-view](https://github.com/saulocantanhede/tfgreek2/blob/main/docs/syntax-view.md#start)" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "See [here](https://github.com/saulocantanhede/tfgreek2/blob/main/docs/viewtypes.md#start) for more information on viewtypes" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# load the app and data\n", "A = use (\"saulocantanhede/tfgreek2\", version=\"0.5.7\", hoist=globals())" ] }, { "cell_type": "markdown", "id": "141768d0-5893-4d44-8667-22dd129a1159", "metadata": {}, "source": [ "# 4 - Creation of the dataset" ] }, { "cell_type": "markdown", "id": "0feac423-fe52-4d79-bd4a-36621c9f795e", "metadata": { "tags": [] }, "source": [ "## 4.1 - Setting up some global variables\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "code", "execution_count": 5, "id": "00f1d619-7481-4a79-b044-15addc2ad979", "metadata": { "tags": [] }, "outputs": [], "source": [ "# If the following variable is set, it will be used as title for all pages. It is intended to the describe the dataset in one line\n", "customPageTitleMD=\"N1904 Greek New Testament [saulocantanhede/tfgreek2 - 0.5.7](https://github.com/saulocantanhede/tfgreek2)\"\n", "customPageTitleHTML=\"N1904 Greek New Testament saulocantanhede/tfgreek2 - 0.5.7\"\n", "\n", "# Specify the location to store the resulting files, relative to the location of this notebook (without a trailing slash).\n", "resultLocation = \"results\"\n", "\n", "# Type of output format ('html' for HTML, 'md' for Mark Down, or 'both' for both HTML and Mark Down)\n", "typeOutput='md'\n", "\n", "# HTML table style definition (only relevant for HTML output format)\n", "htmlStyle=''\n", "\n", "# Limit the number of entries in the frequency tables per node type on each feature description page to this number\n", "tableLimit=10\n", "\n", "# This switch can be set to 'True' if you want additional information, such as dictionary entries and file details, to be printed. For basic output, set this switch to 'False'.\n", "verbose=False\n", "\n", "\n", "\n", "\n", "# Create the footers for MD and HTML, include today's date\n", "from datetime import datetime\n", "today = datetime.today()\n", "formatted_date = today.strftime(\"%b. %d, %Y\")\n", "footerMD=f'\\n\\nCreated on {formatted_date} using [Doc4TF version {scriptVersion} ({scriptDate})](https://github.com/tonyjurg/Doc4TF/blob/main/CreateFeatureDoc.ipynb)'\n", "footerHTML=f'\\n

Created on {formatted_date} using Doc4TF - version {scriptVersion} ({scriptDate})

'" ] }, { "cell_type": "markdown", "id": "9c1870d9-1821-4c14-a57c-f379e92af3ed", "metadata": {}, "source": [ "## 4.2 - Store all relevant data into a dictionary\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "markdown", "id": "5f85a8fc-05c8-4b64-b990-f5d7113b7cf9", "metadata": {}, "source": [ "The following will create a dictionary containing all relevant information for the loaded node and edge features." ] }, { "cell_type": "code", "execution_count": 6, "id": "e2fd353b-f615-4954-917e-65e61ff8a3d8", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Gathering generic details\n", "Analyzing Node Features: .......................................................\n", "Analyzing Edge Features: .....\n", "Finished in 16.41 seconds.\n" ] } ], "source": [ "# Initialize an empty dictionary to store feature data\n", "featureDict = {}\n", "import time\n", "overallTime = time.time()\n", "\n", "def getFeatureDescription(metaData):\n", " \"\"\"\n", " This function looks for the 'description' key in the metadata dictionary. If the key is found,\n", " it returns the corresponding description. If the key is not present, it returns a default \n", " message indicating that no description is available.\n", "\n", " Parameters:\n", " metaData (dict): A dictionary containing metadata about a feature.\n", "\n", " Returns:\n", " str: The description of the feature if available, otherwise a default message.\n", " \"\"\"\n", " return metaData.get('description', \"No feature description\")\n", "\n", "def setDataType(metaData):\n", " \"\"\"\n", " This function checks for the 'valueType' key in the metadata. If the key is present, it\n", " returns 'String' if the value is 'str', and 'Integer' for other types. If the 'valueType' key\n", " is not present, it returns 'Unknown'.\n", "\n", " Parameters:\n", " metaData (dict): A dictionary containing metadata, including the 'valueType' of a feature.\n", "\n", " Returns:\n", " str: A string indicating the determined data type ('String', 'Integer', or 'Unknown').\n", " \"\"\"\n", " if 'valueType' in metaData:\n", " return \"String\" if metaData[\"valueType\"] == 'str' else \"Integer\"\n", " return \"Unknown\"\n", "\n", "def processFeature(feature, featureType, featureMethod):\n", " \"\"\"\n", " Processes a given feature by extracting metadata, description, and data type, and then\n", " compiles frequency data for different node types in a feature dictionary. Certain features\n", " are skipped based on their type. The processed data is added to a global feature dictionary.\n", "\n", " Parameters:\n", " feature (str): The name of the feature to be processed.\n", " featureType (str): The type of the feature ('Node' or 'Edge').\n", " featureMethod (function): A function to obtain feature data.\n", "\n", " Returns:\n", " None: The function updates a global dictionary with processed feature data and does not return anything.\n", " \"\"\"\n", " \n", " # Obtain the meta data\n", " featureMetaData = featureMethod(feature).meta\n", " featureDescription = getFeatureDescription(featureMetaData)\n", " dataType = setDataType(featureMetaData)\n", "\n", " # Initialize dictionary to store feature frequency data\n", " featureFrequencyDict = {}\n", "\n", " # Skip for specific features based on type\n", " if not (featureType == 'Node' and feature == 'otype') and not (featureType == 'Edge' and feature == 'oslots'):\n", " for nodeType in F.otype.all:\n", " frequencyLists = featureMethod(feature).freqList(nodeType)\n", " if not isinstance(frequencyLists, int):\n", " if len(frequencyLists)!=0:\n", " featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': frequencyLists[:tableLimit]}\n", " elif isinstance(frequencyLists, int):\n", " if frequencyLists != 0:\n", " featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': [(\"Link\", frequencyLists)]}\n", "\n", " # Add processed feature data to the main dictionary\n", " featureDict[feature] = {'name': feature, 'descr': featureDescription, 'type': featureType, 'datatype': dataType, 'freqlist': featureFrequencyDict}\n", " \n", "########################################################\n", "# MAIN FUNCTION #\n", "########################################################\n", "\n", "########################################################\n", "# Gather general information #\n", "########################################################\n", "\n", "print('Gathering generic details')\n", "\n", "# Initialize default values\n", "corpusName = A.appName\n", "liveName = ''\n", "versionName = A.version\n", "\n", "# Trying to locate corpus information\n", "if A.provenance:\n", " for parts in A.provenance[0]: \n", " if isinstance(parts, tuple):\n", " key, value = parts[0], parts[1]\n", " if verbose: print (f'General info: {key}={value}')\n", " if key == 'corpus': corpusName = value\n", " if key == 'version': versionName = value\n", " # value for live is a tuple\n", " if key == 'live': liveName=value[1]\n", "if liveName is not None and len(liveName)>1:\n", " # an URL was found\n", " pageTitleMD = f'Doc4TF pages for [{corpusName}]({liveName}) (version {versionName})'\n", " pageTitleHTML = f'

Doc4TF pages for {corpusName} (version {versionName})

'\n", "else:\n", " # No URL found\n", " pageTitleMD = f'Doc4TF pages for {corpusName} (version {versionName})'\n", " pageTitleHTML = f'

Doc4TF pages for {corpusName} (version {versionName})

'\n", "\n", "# Overwrite in case user provided a title\n", "if 'customPageTitleMD_' in globals():\n", " pageTitleMD = customPageTitleMD\n", "if 'customPageTitleHTML' in globals():\n", " pageTitleHTML = customPageTitleHTML\n", "\n", " \n", "########################################################\n", "# Processing node features #\n", "########################################################\n", "\n", "print('Analyzing Node Features: ', end='')\n", "for nodeFeature in Fall():\n", " if not verbose: print('.', end='') # Progress indicator\n", " processFeature(nodeFeature, 'Node', Fs)\n", " if verbose: print(f'\\nFeature {nodeFeature} = {featureDict[nodeFeature]}\\n') # Print feature data if verbose\n", "\n", "########################################################\n", "# Processing edge features #\n", "########################################################\n", "\n", "print('\\nAnalyzing Edge Features: ', end='')\n", "for edgeFeature in Eall():\n", " if not verbose: print('.', end='') # Progress indicator\n", " processFeature(edgeFeature, 'Edge', Es)\n", " if verbose: print(f'\\nFeature {edgeFeature} = {featureDict[edgeFeature]}\\n') # Print feature data if verbose\n", "\n", "print(f'\\nFinished in {time.time() - overallTime:.2f} seconds.')" ] }, { "cell_type": "markdown", "id": "cabf4dd5-b817-42cf-8644-6431970085a7", "metadata": {}, "source": [ "## 5 - Create the documentation pages\n", "\n", "Two types of pages will be created:\n", " * Feature description pages (one per feature)\n", " * Set of index pages (linking to the feature pages)" ] }, { "cell_type": "markdown", "id": "3b9abcc1-1362-42a0-85db-8234766a41f5", "metadata": {}, "source": [ "## 5.1 - Create the set of feature pages\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "code", "execution_count": 7, "id": "c4b5e148-c336-4d95-a238-5fe79fa9525c", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished in 0.06 seconds (written 60 md files to directory C:\\Users\\tonyj\\OneDrive\\Documents\\GitHub\\Doc4TF\\results)\n" ] } ], "source": [ "import os\n", "import time\n", "overallTime = time.time()\n", "\n", "# Initialize a counter for the number of files created\n", "filesCreated = 0\n", "# Get the current working directory and append a backslash for path building\n", "pathFull = os.getcwd() + '\\\\'\n", "\n", "# Iterating over each feature in the feature dictionary\n", "for featureName, featureData in featureDict.items():\n", " # Extracting various properties of each feature\n", " featureDescription = featureData.get('descr')\n", " featureType = featureData.get('type')\n", " featureDataType = featureData.get('datatype')\n", " \n", " # Initializing strings to accumulate HTML and Markdown content\n", " nodeListHTML = nodeListMD = ''\n", " tableListHTML = tableListMD = ''\n", " frequencyData = featureData.get('freqlist')\n", "\n", " # Processing frequency data for each node\n", " for node in frequencyData:\n", " # Building HTML and Markdown links for each node\n", " nodeListHTML += f' {node}'\n", " nodeListMD += f' [`{node}`](featuresbynodetype.md#{node}) '\n", "\n", " # Starting HTML and Markdown tables for frequency data\n", " tableListHTML += f'

Frequency for nodetype {node}

'\n", " tableListMD += f'### Frequency for nodetype [{node}](featuresbynodetype.md#{node})\\nValue|Occurences\\n---|---\\n'\n", "\n", " # Populating tables with frequency data\n", " itemData = frequencyData.get(node).get('freq')\n", " for item in itemData:\n", " handleSpace = item[0] if item[0] != ' ' else 'space' # prevent garbling of tables where the value itself is a space\n", " tableListHTML += f''\n", " tableListMD += f'{handleSpace}|{item[1]}\\n'\n", " tableListHTML += f'
ValueOccurences
{handleSpace}{item[1]}
\\n'\n", "\n", " # Creating info blocks for HTML and Markdown\n", " infoBlockHTML = f'
Data typeFeature typeAvailable for nodes
{featureDataType}{featureType}{nodeListHTML}
'\n", " infoBlockMD = f'Data type|Feature type|Available for nodes\\n---|---|---\\n[`{featureDataType}`](featuresbydatatype.md#{featureDataType.lower()})|[`{featureType}`](featuresbytype.md#{featureType.lower()})|{nodeListMD}'\n", "\n", " # Outputting in Markdown format\n", " if typeOutput in ('md','both'):\n", " pageMD = f'{pageTitleMD}\\n# Feature: {featureName}\\n{infoBlockMD}\\n## Description\\n{featureDescription}\\n## Feature Values\\n{tableListMD} {footerMD} '\n", " fileNameMD = os.path.join(resultLocation, f\"{featureName}.md\")\n", " try:\n", " with open(fileNameMD, \"w\", encoding=\"utf-8\") as file:\n", " file.write(pageMD)\n", " filesCreated += 1\n", " # Log if verbose mode is on\n", " if verbose: print(f\"Markdown content written to {pathFull + fileNameMD}\")\n", " except Exception as e:\n", " print(f\"Exception: {e}\")\n", " break # Stops execution on encountering an exception\n", "\n", " # Outputting in HTML format\n", " if typeOutput in ('html','both'):\n", " pageHTML = f'{htmlStyle}

{pageTitleHTML}

\\n

Feature: {featureName}

\\n{infoBlockHTML}\\n

Description

\\n

{featureDescription}

\\n

Feature Values

\\n{tableListHTML} {footerHTML}'\n", " fileNameHTML = os.path.join(resultLocation, f\"{featureName}.htm\")\n", " try:\n", " with open(fileNameHTML, \"w\", encoding=\"utf-8\") as file:\n", " file.write(pageHTML)\n", " filesCreated += 1\n", " # Log if verbose mode is on\n", " if verbose: print(f\"HTML content written to {pathFull + fileNameHTML}\")\n", " except Exception as e:\n", " print(f\"Exception: {e}\")\n", " break # Stops execution on encountering an exception\n", "\n", "# Reporting the number of files created\n", "if filesCreated != 0:\n", " print(f'Finished in {time.time() - overallTime:.2f} seconds (written {filesCreated} {\"html and md\" if typeOutput == \"both\" else typeOutput} files to directory {pathFull + resultLocation})')\n", "else:\n", " print('No files written')" ] }, { "cell_type": "markdown", "id": "cb9ec070-ae31-4ab5-b356-1b07666e5026", "metadata": {}, "source": [ "## 5.2 - Create the index pages\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "code", "execution_count": 8, "id": "bb568470-a2ed-446b-86c9-e298fb4bc6b3", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished in 0.01 seconds (written 3 md files to directory C:\\Users\\tonyj\\OneDrive\\Documents\\GitHub\\Doc4TF\\results)\n" ] } ], "source": [ "import os\n", "import time\n", "overallTime = time.time()\n", "\n", "# Initialize a counter for the number of files created\n", "filesCreated = 0\n", "\n", "def exampleData(feature):\n", " \"\"\"\n", " This function checks if the specified feature exists in the global `featureDict` and if it \n", " has a non-empty frequency list. If so, it extracts the first few values from this frequency \n", " list to create a list of examples.\n", "\n", " Parameters:\n", " feature (str): The name of the feature for which examples are to be created.\n", "\n", " Returns:\n", " str: A string containing the examples concatenated together. Returns \"No values\" if the \n", " feature does not exist in `featureDict` or if it has an empty frequency list.\n", " \"\"\"\n", " # Check if the feature exists in featureDict and has non-empty freqlist.\n", " if feature in featureDict and featureDict[feature]['freqlist']:\n", " # Get the first value from the freqlist\n", " freq_list = next(iter(featureDict[feature]['freqlist'].values()))['freq']\n", " # Use list comprehension to create the example list. \n", " example_list = ' '.join(f'`{item[0]}`' for item in freq_list[:4])\n", " return example_list\n", " else:\n", " return \"No values\"\n", "\n", " \n", "def writeToFile(fileName, content, fileType, verbose):\n", " \"\"\"\n", " Writes provided content to a specified file. If verbose is True, prints a confirmation message.\n", " This function attempts to write the given content to a file with the specified name. It handles \n", " any exceptions during writing and can optionally print a message upon successful writing. The function \n", " also increments a global counter `filesCreated` for each successful write operation.\n", "\n", " Parameters:\n", " fileName (str): The name of the file to write to.\n", " content (str): The content to be written to the file.\n", " fileType (str): The type of file (used for informational messages; e.g., 'md' for Markdown, 'html' for HTML).\n", " verbose (bool): If True, prints a message upon successful writing.\n", "\n", " Returns:\n", " None: The function does not return a value but writes content to a file and may print messages.\n", " \"\"\"\n", " global filesCreated\n", " try:\n", " with open(fileName, \"w\", encoding=\"utf-8\") as file:\n", " file.write(content)\n", " filesCreated+=1\n", " if verbose: \n", " print(f\"{fileType.upper()} content written to {fileName}\")\n", " except Exception as e:\n", " print(f\"Exception while writing {fileType.upper()} file: {e}\")\n", "\n", "# Set up some lists\n", "nodeFeatureList = []\n", "typeFeatureList = []\n", "dataTypeFeatureList = []\n", "\n", "for featureName, featureData in featureDict.items():\n", " typeFeatureList.append((featureName,featureData.get('type')))\n", " dataTypeFeatureList.append((featureName,featureData.get('datatype')))\n", " for node in featureData.get('freqlist'):\n", " nodeFeatureList.append((node, featureName))\n", " \n", "########################################################### \n", "# Create the page with overview per node type (e.g. word) #\n", "###########################################################\n", " \n", "pageMD=f'{pageTitleMD}\\n# Overview features per nodetype\\n'\n", "pageHTML=f'{htmlStyle}

{pageTitleHTML}

\\n

Overview features per nodetype

'\n", "\n", "# Sort the list alphabetically based on the second item of each tuple (featureName)\n", "nodeFeatureList = sorted(nodeFeatureList, key=lambda x: x[1])\n", "# Iterate over node types\n", "for NodeType in F.otype.all:\n", " NodeItemTextMD=f'## {NodeType}\\n\\nFeature|Featuretype|Datatype|Description|Examples\\n---|---|---|---|---\\n' \n", " NodeItemTextHTML=f'

{NodeType}

\\n\\n' \n", " for node, feature in nodeFeatureList:\n", " if node == NodeType: \n", " featureData=featureDict[feature]\n", " featureDescription=featureData.get('descr') \n", " featureType=featureData.get('type') \n", " featureDataType=featureData.get('datatype')\n", " NodeItemTextMD+=f\"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featuresbytype.md#{featureType})|[`{featureDataType}`](featuresbydatatype.md#{featureDataType})|{featureDescription}|{exampleData(feature)}\\n\"\n", " NodeItemTextHTML+=f\"\\n\"\n", " NodeItemTextHTML+=f\"
FeatureFeaturetypeDatatypeDescriptionExamples
{feature}{featureType}{featureDataType}{featureDescription}{exampleData(feature)}
\\n\"\n", " pageHTML+=NodeItemTextHTML\n", " pageMD+=NodeItemTextMD\n", " \n", "pageHTML+=f'{footerHTML}'\n", "pageMD+=f'{footerMD}'\n", " \n", "# Write to file by calling common function\n", "if typeOutput in ('md','both'):\n", " fileNameMD = os.path.join(resultLocation, \"featuresbynodetype.md\")\n", " writeToFile(fileNameMD, pageMD, 'md', verbose)\n", "\n", "if typeOutput in ('html','both'):\n", " fileNameHTML = os.path.join(resultLocation, \"featuresbynodetype.htm\")\n", " writeToFile(fileNameHTML, pageHTML, 'html', verbose)\n", "\n", "####################################################################\n", "# Create the page with overview per data type (string or integer) #\n", "####################################################################\n", "\n", "pageMD=f'{pageTitleMD}\\n# Overview features per datatype\\n'\n", "pageHTML=f'{htmlStyle}

{pageTitleHTML}

\\n

Overview features per datatype'\n", "\n", "# Sort the list alphabetically based on the second item of each tuple (featureName)\n", "dataTypeFeatureList = sorted(dataTypeFeatureList, key=lambda x: x[1])\n", "\n", "DataItemTextMD=DataItemTextHTML=''\n", "for DataType in ('Integer','String'):\n", " DataItemTextMD=f'## {DataType}\\n\\nFeature|Featuretype|Available on nodes|Description|Examples\\n---|---|---|---|---\\n' \n", " DataItemTextHTML=f'

{DataType}

\\n\\n' \n", " for feature, featureDataType in dataTypeFeatureList: \n", " if featureDataType == DataType: \n", " featureDescription=featureDict[feature].get('descr') \n", " featureType=featureDict[feature].get('type') \n", " nodeListMD=nodeListHTML=''\n", " for thisNode in featureDict[feature]['freqlist']:\n", " nodeListMD+=f'[`{thisNode}`](featuresbynodetype.md#{thisNode}) '\n", " nodeListHTML+=f'{thisNode} '\n", " DataItemTextMD+=f\"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featuresbytype.md#{featureType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(feature)}\\n\"\n", " DataItemTextHTML+=f\"\\n\"\n", " DataItemTextHTML+=f\"
FeatureFeaturetypeAvailable on nodesDescriptionExamples
{feature}{featureType}{nodeListHTML}{featureDescription}{exampleData(feature)}
\\n\"\n", " pageMD+=DataItemTextMD\n", " pageHTML+=DataItemTextHTML\n", "\n", "pageHTML+=f'{footerHTML}'\n", "pageMD+=f'{footerMD}'\n", " \n", " \n", "# Write to file by calling common function\n", "if typeOutput in ('md','both'):\n", " fileNameMD = os.path.join(resultLocation, \"featuresbydatatype.md\")\n", " writeToFile(fileNameMD, pageMD, 'md', verbose)\n", "\n", "if typeOutput in ('html','both'):\n", " fileNameHTML = os.path.join(resultLocation, \"featuresbydatatype.htm\")\n", " writeToFile(fileNameHTML, pageHTML, 'html', verbose)\n", " \n", "##################################################################\n", "# Create the page with overview per feature type (edge or node) #\n", "##################################################################\n", "\n", "pageMD=f'{pageTitleMD}\\n# Overview features per type\\n'\n", "pageHTML=f'{htmlStyle}

{pageTitleHTML}

\\n

Overview features per type'\n", "\n", "# Sort the list alphabetically based on the second item of each tuple (nodetype)\n", "typeFeatureList = sorted(typeFeatureList, key=lambda x: x[1])\n", "for featureType in ('Node','Edge'):\n", " ItemTextMD=f'## {featureType}\\n\\nFeature|Datatype|Available on nodes|Description|Examples\\n---|---|---|---|---\\n' \n", " ItemTextHTML=f'

{featureType}

\\n\\n' \n", " for thisFeature, thisFeatureType in typeFeatureList: \n", " if featureType == thisFeatureType:\n", " featureDescription=featureDict[thisFeature].get('descr')\n", " featureDataType=featureDict[thisFeature].get('datatype')\n", " nodeListMD=nodeListHTML=''\n", " for thisNode in featureDict[thisFeature]['freqlist']:\n", " nodeListMD+=f'[`{thisNode}`](featuresbynodetype.md#{thisNode}) '\n", " nodeListHTML+=f'{thisNode} '\n", " ItemTextMD+=f\"[`{thisFeature}`]({thisFeature}.md#readme)|[`{featureDataType}`](featuresbydatatype.md#{featureDataType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(thisFeature)}\\n\"\n", " ItemTextHTML+=f\"\\n\"\n", " ItemTextHTML+=f\"
FeatureDatatypeAvailable on nodesDescriptionExamples
{thisFeature}{featureDataType}{nodeListHTML}{featureDescription}{exampleData(thisFeature)}
\\n\"\n", " pageMD+=ItemTextMD\n", " pageHTML+=ItemTextHTML\n", "\n", "pageHTML+=f'{footerHTML}'\n", "pageMD+=f'{footerMD}'\n", "\n", "# Write to file by calling common function\n", "if typeOutput in ('md','both'):\n", " fileNameMD = os.path.join(resultLocation, \"featuresbytype.md\")\n", " writeToFile(fileNameMD, pageMD, 'md', verbose)\n", "\n", "if typeOutput in ('html','both'):\n", " fileNameHTML = os.path.join(resultLocation, \"featuresbytype.htm\")\n", " writeToFile(fileNameHTML, pageHTML, 'html', verbose)\n", " \n", "\n", "# Reporting the number of files created\n", "if filesCreated != 0:\n", " print(f'Finished in {time.time() - overallTime:.2f} seconds (written {filesCreated} {\"html and md\" if typeOutput == \"both\" else typeOutput} files to directory {pathFull + resultLocation})')\n", "else:\n", " print('No files written')" ] }, { "cell_type": "markdown", "id": "8cad157a-f173-4e6e-8d53-42b6a4437537", "metadata": {}, "source": [ "# 6 - Change log\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "markdown", "id": "23b5ae25-4032-4b1a-8a77-c8eb35f1afbc", "metadata": {}, "source": [ "Minor changes with version [0.5](https://github.com/tonyjurg/Doc4TF/blob/main/previous_versions/CreateFeatureDoc(0_5).ipynb)):\n", "\n", " - Additional notes on install requirements added May 26, 2024.\n", " - Fixed bug [#18: incorrectly use of HTML page title for MD title](https://github.com/tonyjurg/Doc4TF/issues/18)\n", " \n", "Changes to previous major version ([0.4](https://github.com/tonyjurg/Doc4TF/blob/main/previous_versions/CreateFeatureDoc(0_4).ipynb)):\n", "\n", " * Cleaning up some spelling errors.\n", " * Enhancement [#13](https://github.com/tonyjurg/Doc4TF/issues/13) (update names of index pages)\n", " * Enhancement [#16](https://github.com/tonyjurg/Doc4TF/issues/16) (version number and date at start of Notebook)" ] }, { "cell_type": "markdown", "id": "a40a6eca-dc08-4b51-bece-bbda6439b645", "metadata": {}, "source": [ "# 7 - License\n", "##### [Back to TOC](#TOC)" ] }, { "cell_type": "markdown", "id": "1096c784-46e4-4158-aaf6-5c15310cee67", "metadata": {}, "source": [ "Licenced under [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://github.com/tonyjurg/Doc4TF/blob/main/LICENCE.md)" ] } ], "metadata": { "celltoolbar": "Geen", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }