{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b01be3ab",
   "metadata": {},
   "source": [
    "# PyEvoc Step-by-Step Pipeline Tutorial\n",
    "\n",
    "This notebook guides a new user through the current PyEvoc package structure.\n",
    "\n",
    "It follows the actual module architecture:\n",
    "\n",
    "```text\n",
    "pyevoc.data\n",
    "pyevoc.preprocessing\n",
    "pyevoc.features\n",
    "pyevoc.analysis\n",
    "pyevoc.visualisation\n",
    "```\n",
    "\n",
    "The workflow is intentionally explicit. Each step can be run, inspected, and modified."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0422b130",
   "metadata": {},
   "source": [
    "## 0. How to use PyEvoc locally\n",
    "\n",
    "If PyEvoc is not installed from PyPI, the recommended local usage is:\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/text-lab/pyevoc.git\n",
    "cd pyevoc\n",
    "pip install -e .\n",
    "```\n",
    "\n",
    "The `-e` option installs the package in editable mode, so changes to the local package files are immediately visible to the notebook.\n",
    "\n",
    "Alternatively, install directly from GitHub:\n",
    "\n",
    "```bash\n",
    "pip install git+https://github.com/text-lab/pyevoc.git\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bb9be19",
   "metadata": {},
   "source": [
    "## 1. Import core libraries and inspect PyEvoc\n",
    "\n",
    "This cell verifies that Python can import the local PyEvoc package.\n",
    "\n",
    "To simplify the procedure, ensure the dataset, the anchor list, and this notebook file are in the same folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bed67e7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "import pyevoc\n",
    "\n",
    "print(\"PyEvoc imported from:\")\n",
    "print(pyevoc.__file__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b030a3af",
   "metadata": {},
   "source": [
    "## 2. Load and standardise the dataset\n",
    "\n",
    "PyEvoc expects a standard corpus schema with four required columns:\n",
    "\n",
    "```text\n",
    "user_id | doc_id | time | text\n",
    "```\n",
    "\n",
    "Your original dataset can have any column names. You only need to map your original column names to the PyEvoc standard names.\n",
    "\n",
    "For example, if your file contains:\n",
    "\n",
    "```text\n",
    "account_ID | post_id | publication_time | content\n",
    "```\n",
    "\n",
    "you map them like this:\n",
    "\n",
    "```python\n",
    "column_map={\n",
    "    \"account_id\":       \"user_id\",\n",
    "    \"post_id\":          \"doc_id\",\n",
    "    \"publication_time\": \"time\",\n",
    "    \"content\":          \"text\",\n",
    "}\n",
    "```\n",
    "\n",
    "Any additional columns present in the source file are preserved automatically after the four required ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b88ba25",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.data.dataset import DatasetConfig, load_dataset, standardise_dataset, corpus_summary\n",
    "\n",
    "# -----------------------------------------------------------------------\n",
    "# Change this path to your input file.\n",
    "# Supported formats: .csv, .tsv, .xlsx, .parquet, .json\n",
    "# -----------------------------------------------------------------------\n",
    "INPUT_FILE = \"AGI.xlsx\"\n",
    "\n",
    "# Adapt the left-hand side to match your dataset's column names.\n",
    "# The right-hand side must use PyEvoc standard names.\n",
    "config = DatasetConfig(\n",
    "    column_map={\n",
    "        \"account_ID\":   \"user_id\",\n",
    "        \"tweet_ID\":    \"doc_id\",\n",
    "        \"tweet_pub_time\":   \"time\",\n",
    "        \"tweet_text\":   \"text\",\n",
    "    },\n",
    "    start_date=\"2022-10-01\",\n",
    "    end_date=\"2025-05-01\",\n",
    "    timezone=\"UTC\",\n",
    "    drop_missing_text=True,\n",
    ")\n",
    "\n",
    "corpus = load_dataset(INPUT_FILE, config=config)\n",
    "\n",
    "print(corpus.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f71a39d2",
   "metadata": {},
   "source": [
    "## 3. Language Filtering\n",
    "\n",
    "PyEvoc implements a two-stage language identification workflow based on fastText and Lingua. \n",
    "\n",
    "It works with English, Italian, French, German, Spanish, Portuguese.\n",
    "\n",
    "The primary detection stage relies on the official fastText language-identification model (`lid.176.bin` or `lid.176.ftz`), which provides efficient and accurate language classification for large textual corpora.\n",
    "\n",
    "If the fastText model is not already available locally, it can be downloaded automatically using:\n",
    "\n",
    "```python\n",
    "from pyevoc.preprocessing.language_filtering import download_fasttext_lid_model\n",
    "\n",
    "model_path = download_fasttext_lid_model(compact=False)\n",
    "```\n",
    "\n",
    "For documents with uncertain fastText classifications, PyEvoc can optionally apply a secondary validation step based on the Lingua language detector. To enable this functionality, install the additional dependency:\n",
    "\n",
    "```python\n",
    "pip install lingua-language-detector\n",
    "```\n",
    "\n",
    "For large corpora, language filtering may take time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a268d0f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.preprocessing.language_filtering import (\n",
    "    LanguageFilterConfig,\n",
    "    download_fasttext_lid_model,\n",
    "    filter_by_language,\n",
    ")\n",
    "\n",
    "model_path = download_fasttext_lid_model(compact=False)\n",
    "\n",
    "lang_config = LanguageFilterConfig(\n",
    "    text_col=\"text\",\n",
    "    target_language=\"en\",\n",
    "    min_probability=0.90,\n",
    "    ambiguous_min_probability=0.50,\n",
    "    ambiguous_max_probability=0.90,\n",
    "    use_lingua_fallback=True,\n",
    "    lingua_min_confidence=0.80,\n",
    "    keep_probability=True,\n",
    "    keep_diagnostics=False,\n",
    "    verbose=False,\n",
    ")\n",
    "\n",
    "subcorpus_lin = filter_by_language(\n",
    "    corpus,\n",
    "    config=lang_config,\n",
    "    model_path=model_path,\n",
    ")\n",
    "\n",
    "print(subcorpus_lin.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f1529e7",
   "metadata": {},
   "source": [
    "## 4. Thematic filtering with an external anchor list\n",
    "\n",
    "The thematic filter extracts a domain-specific subcorpus.\n",
    "\n",
    "The anchor file should be a plain text file with one anchor term or phrase per line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "782c1246",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from pyevoc.preprocessing.thematic_filtering import (\n",
    "    ThematicFilterConfig,\n",
    "    build_thematic_subset,\n",
    "    load_anchor_terms\n",
    ")\n",
    "\n",
    "ANCHOR_FILE = \"anchor.txt\"  # change this path\n",
    "\n",
    "theme_config = ThematicFilterConfig(\n",
    "    text_col=\"text\",\n",
    "    min_anchor_hits=1,\n",
    "    similarity_threshold=0.10,\n",
    "    min_expansion_hits=2,\n",
    "    max_features=50000,\n",
    "    lowercase=True,\n",
    "    ngram_range=(1, 2)\n",
    ")\n",
    "\n",
    "subcorpus_the, theme_metadata = build_thematic_subset(\n",
    "    subcorpus_lin,\n",
    "    anchor_file=ANCHOR_FILE,\n",
    "    config=theme_config\n",
    ")\n",
    "\n",
    "print(subcorpus_the.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "889e86d2",
   "metadata": {},
   "source": [
    "## 5. Subcorpus statistics\n",
    "\n",
    "This step gives basic descriptive information about the selected corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcc355af",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.preprocessing.corpus_statistics import corpus_statistics\n",
    "\n",
    "stats = corpus_statistics(\n",
    "    subcorpus_lin,     #if only language filtering is applied\n",
    "#   subcorpus_the,     #if language and thematic filtering is applied\n",
    "    text_col=\"text\",\n",
    ")\n",
    "\n",
    "stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2e437ea",
   "metadata": {},
   "source": [
    "## 6. Basic text cleaning\n",
    "\n",
    "The cleaned text is stored in a new column called `clean_text`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0da6e01",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.preprocessing.cleaning import CleaningConfig, clean_corpus\n",
    "\n",
    "clean_config = CleaningConfig(\n",
    "    lowercase=True,\n",
    "    remove_urls=True,\n",
    "    url_placeholder=\"URL\",\n",
    "    expand_contractions=True,\n",
    "    space_emojis=True,\n",
    "    reduce_elongated_vowels=True,\n",
    "    space_punctuation=True,\n",
    "    show_progress=True,\n",
    ")\n",
    "\n",
    "corpus_clean = clean_corpus(\n",
    "    subcorpus_lin,     #if only language filtering is applied\n",
    "#   subcorpus_the,     #if language and thematic filtering is applied\n",
    "    text_col=\"text\",\n",
    "    output_col=\"clean_text\",\n",
    "    config=clean_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48b69dcc",
   "metadata": {},
   "source": [
    "## 7. Linguistic annotation with Stanza\n",
    "\n",
    "This step performs tokenisation, lemmatisation, POS tagging, and named-entity recognition.\n",
    "\n",
    "The output is a token-level dataframe.\n",
    "\n",
    "If Stanza models are not installed, you may need to run:\n",
    "\n",
    "```python\n",
    "import stanza\n",
    "stanza.download(\"en\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "461291c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.preprocessing import (\n",
    "    AnnotationConfig,\n",
    "    annotate_with_stanza,\n",
    "    annotation_diagnostics,\n",
    "    dependency_diagnostics,\n",
    ")\n",
    "\n",
    "ann_config = AnnotationConfig(\n",
    "    text_col=\"clean_text\",\n",
    "    doc_id_col=\"doc_id\",\n",
    "    user_col=\"user_id\",\n",
    "    time_col=\"time\",\n",
    "    language=\"en\",\n",
    "    processors=\"tokenize,pos,lemma,depparse\",\n",
    "    use_gpu=True,\n",
    "    batch_size=128,\n",
    "    save_parquet=True,\n",
    "    output_dir=\"annotated_outputs\",\n",
    "    output_name=\"corpus_stanza_anno\",\n",
    ")\n",
    "\n",
    "tokens_raw = annotate_with_stanza(\n",
    "    corpus_clean,\n",
    "    config=ann_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09b54781-e775-49ab-b9d6-c329be453676",
   "metadata": {},
   "outputs": [],
   "source": [
    "#RECOVERY FROM ANNOTATION BACKUP\n",
    "\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "OUTPUT_DIR = \"annotated_outputs\"\n",
    "os.makedirs(OUTPUT_DIR, exist_ok=True)\n",
    "\n",
    "tokens_raw = pd.read_parquet(\n",
    "    os.path.join(OUTPUT_DIR, \"corpus_stanza_anno.parquet\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9055dd76",
   "metadata": {},
   "source": [
    "## 8. Emoji assignment\n",
    "\n",
    "Emoji tokens are reassigned to the `EMOJI` UPOS category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1b0eab9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.features.emoji_assignment import assign_emoji_upos\n",
    "\n",
    "tokens, emoji_diag, emoji_top = assign_emoji_upos(\n",
    "    tokens_raw,\n",
    "    token_col=\"token\",\n",
    "    upos_col=\"upos\",\n",
    "    lemma_col=\"lemma\",\n",
    "    return_diagnostics=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae51d630",
   "metadata": {},
   "source": [
    "## 9. Structural foregrounding and positional salience\n",
    "\n",
    "This step computes the indicators needed to reconstruct the AOE-like salience component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5e4993d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.features.foregrounding import (\n",
    "    ForegroundingConfig,\n",
    "    ForegroundingWeights,\n",
    "    add_salience_indicators,\n",
    "    foregrounding_diagnostics,\n",
    "    metadata_coverage,\n",
    ")\n",
    "\n",
    "metadata_coverage(tokens)\n",
    "\n",
    "fg_config = ForegroundingConfig(\n",
    "    doc_col=\"doc_id\",\n",
    "    user_col=\"user_id\",\n",
    "    time_col=\"time\",\n",
    "    token_col=\"token\",\n",
    "    sentence_col=\"sentence_id\",\n",
    "    position_col=\"position\",\n",
    "    weights=ForegroundingWeights(\n",
    "        opening_sentence=0.35,\n",
    "        emphasis=0.25,\n",
    "        list_or_quote=0.20,\n",
    "        intensification=0.20,\n",
    "    ),\n",
    ")\n",
    "\n",
    "tokens = add_salience_indicators(tokens, config=fg_config)\n",
    "\n",
    "foregrounding_diagnostics(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22fc86bd",
   "metadata": {},
   "source": [
    "## 10. Unigram cleaning and selection\n",
    "\n",
    "This step defines the lexical units retained for representational analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaf54ac2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.features.unigram_selection import select_unigrams\n",
    "\n",
    "tokens_selected, unigram_diag, unigram_params = select_unigrams(\n",
    "    tokens,\n",
    "    keep_upos={\"NOUN\", \"ADJ\", \"EMOJI\"},\n",
    "    token_col=\"token\",\n",
    "    lemma_col=\"lemma\",\n",
    "    upos_col=\"upos\",\n",
    "    doc_col=\"doc_id\",\n",
    "    user_col=\"user_id\",\n",
    "    min_docs_per_term=3,\n",
    "    min_users_per_term=3,\n",
    "    return_diagnostics=True,\n",
    ")\n",
    "\n",
    "unigram_diag"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939e259b",
   "metadata": {},
   "source": [
    "## 11. Compute AFE/AOE term-level indices\n",
    "\n",
    "This step computes diffusion, salience, and rank-like AOE indicators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c70ce0c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.features.term_indices import (\n",
    "    SalienceConfig,\n",
    "    compute_term_statistics,\n",
    ")\n",
    "\n",
    "term_stats, pos_thresholds, quadrant_summary = compute_term_statistics(\n",
    "    tokens_selected,\n",
    "    term_col=\"term\",\n",
    "    upos_col=\"upos\",\n",
    "    user_col=\"user_id\",\n",
    "    doc_col=\"doc_id\",\n",
    "    time_col=\"time\",\n",
    "    r_pos_col=\"r_pos\",\n",
    "    r_str_col=\"r_str\",\n",
    "    config=SalienceConfig(\n",
    "        alpha=0.50,\n",
    "        max_rank_value=5.0,\n",
    "        use_users_for_frequency=True,\n",
    "        focal_upos={\"NOUN\", \"ADJ\", \"EMOJI\"},\n",
    "    ),\n",
    ")\n",
    "\n",
    "pos_thresholds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e299fa",
   "metadata": {},
   "source": [
    "## 12. Concreteness labelling\n",
    "\n",
    "This step adds concreteness labels when a lexicon is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f15ed4bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.features.concreteness import (\n",
    "    load_concreteness_lexicon,\n",
    "    add_concreteness_labels,\n",
    ")\n",
    "\n",
    "lexicon = load_concreteness_lexicon()\n",
    "\n",
    "term_stats, concreteness_coverage = add_concreteness_labels(\n",
    "    term_stats,\n",
    "    lexicon=lexicon,\n",
    ")\n",
    "\n",
    "concreteness_coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "717936fe",
   "metadata": {},
   "source": [
    "## 13. Emoji labelling\n",
    "\n",
    "This step adds descriptions for emoji terms and falls back to Unicode names when needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7cb9012",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.features.emoji_labelling import (\n",
    "    load_emoji_lookup,\n",
    "    add_emoji_descriptions,\n",
    ")\n",
    "\n",
    "emoji_lookup = load_emoji_lookup()\n",
    "\n",
    "term_stats, emoji_coverage, emoji_methods, missing_emoji = add_emoji_descriptions(\n",
    "    term_stats,\n",
    "    lookup=emoji_lookup,\n",
    "    min_similarity=0.80,\n",
    ")\n",
    "\n",
    "emoji_coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae3b863a",
   "metadata": {},
   "source": [
    "## 14. EVOC thresholds and quadrants\n",
    "\n",
    "Thresholds are computed separately by POS category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed773ba9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.analysis.quadrants import assign_evoc_quadrants\n",
    "\n",
    "evoc_quadrants, quadrant_counts, pos_thresholds_round = assign_evoc_quadrants(\n",
    "    term_stats,\n",
    "    generate_html=True,\n",
    "    html_output_dir=\"evoc_outputs\",\n",
    "    html_top_n=5,\n",
    ")\n",
    "\n",
    "evoc_quadrants.attrs[\"html_outputs\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8046646c",
   "metadata": {},
   "source": [
    "## 15. Collocations and Named Entities\n",
    "\n",
    "This extracts collocational structures (and optionally named entities) from the selected token table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1dd5212",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.analysis.collocations_entities import extract_collocations_and_entities\n",
    "\n",
    "colloc_results = extract_collocations_and_entities(\n",
    "    tokens_df=tokens_raw,\n",
    "    include_collocations=True,\n",
    "    include_entities=False,\n",
    "    min_freq=2,\n",
    "    min_docs=2,\n",
    "    min_users=2,\n",
    "    write_html=True,\n",
    ")\n",
    "\n",
    "collocations_df = colloc_results[\"collocations_df\"]\n",
    "\n",
    "collocations_df.shape\n",
    "\n",
    "#results = extract_collocations_and_entities(\n",
    "#    tokens_df=tokens,\n",
    "#    include_collocations=True,\n",
    "#    include_entities=True,\n",
    "#    entity_n=2,\n",
    "#    resolve_overlap=True,\n",
    "#    prefer_overlap=\"evidence\",\n",
    "#    write_html=True,\n",
    "#)\n",
    "#\n",
    "#collocations_df = results[\"collocations_df\"]\n",
    "#entities_df = results[\"entities_df\"]\n",
    "#overlap_log_df = results[\"overlap_log_df\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e4fe52d",
   "metadata": {},
   "source": [
    "## 16. Temporal stability\n",
    "\n",
    "Temporal stability analysis is performed by partitioning the corpus into time periods and recomputing the EVOC framework within each period. Depending on the analytical objective, periods may be defined using custom temporal boundaries, equal-duration intervals, or approximately equal-interval intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2191fc64",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.analysis.temporal_stability import run_temporal_stability_analysis\n",
    "\n",
    "temporal_results = run_temporal_stability_analysis(\n",
    "    df=tokens_selected,\n",
    "    evoc_quadrants_df=evoc_quadrants,\n",
    "    pos_thresholds_round_df=pos_thresholds_round,\n",
    "    n_periods=4,\n",
    "    time_period_mode=\"equal_days\",\n",
    "    show_tables=False,\n",
    ")\n",
    "\n",
    "#temporal_results = run_temporal_stability_analysis(\n",
    "#    df=tokens_selected,\n",
    "#    evoc_quadrants_df=evoc_quadrants,\n",
    "#    pos_thresholds_round_df=pos_thresholds_round,\n",
    "#    time_period_mode=\"custom\",\n",
    "#    custom_time_cuts=[\"2022-01-01\", \"2024-01-01\"],\n",
    "#    period_labels=[\"2018-2021\", \"2022-2023\", \"2024-2026\"],\n",
    "#    show_table=False,\n",
    "#)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cdf75b1",
   "metadata": {},
   "source": [
    "## 17. Visualisations\n",
    "\n",
    "The package currently provides plotting functions for EVOC maps, emoji maps, semantic-tree edges, and Sankey diagrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9be7f68c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyevoc.visualisation import (\n",
    "    build_evoc_target_plot,\n",
    "    build_evoc_collocation_tree_for_upos,\n",
    "    build_emoji_evoc_plot,\n",
    "    build_temporal_sankey_ordered,\n",
    ")\n",
    "\n",
    "OUTPUT_DIR = \"evoc_outputs\"\n",
    "\n",
    "# 1. EVOC target maps\n",
    "plot_nouns = build_evoc_target_plot(\n",
    "    evoc_quadrants,\n",
    "    \"NOUN\",\n",
    "    pos_thresholds_round,\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    node_size_mode=\"fixed\",\n",
    ")\n",
    "\n",
    "plot_adjectives = build_evoc_target_plot(\n",
    "    evoc_quadrants,\n",
    "    \"ADJ\",\n",
    "    pos_thresholds_round,\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    node_size_mode=\"fixed\",\n",
    ")\n",
    "\n",
    "# 2. EVOC semantic trees\n",
    "tree_nouns = build_evoc_collocation_tree_for_upos(\n",
    "    evoc_quadrants_df=evoc_quadrants,\n",
    "    collocations_df=collocations_df,\n",
    "    tokens_df=tokens_raw,\n",
    "    upos=\"NOUN\",\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    top_roots_per_quadrant=5,\n",
    "    max_leaves_per_root=3,\n",
    "    min_leaf_freq=5,\n",
    ")\n",
    "\n",
    "tree_adjectives = build_evoc_collocation_tree_for_upos(\n",
    "    evoc_quadrants_df=evoc_quadrants,\n",
    "    collocations_df=collocations_df,\n",
    "    tokens_df=tokens_raw,\n",
    "    upos=\"ADJ\",\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    top_roots_per_quadrant=5,\n",
    "    max_leaves_per_root=3,\n",
    "    min_leaf_freq=5,\n",
    ")\n",
    "\n",
    "# 3. EVOC emoji map\n",
    "fig_emoji, emoji_evoc_df, emoji_evoc_html = build_emoji_evoc_plot(\n",
    "    html_file=f\"{OUTPUT_DIR}/evoc_quadrants_emoji_compact.html\",\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    output_png=None,\n",
    ")\n",
    "\n",
    "# 4. EVOC temporal sankey\n",
    "sankey_results = build_temporal_sankey_ordered(\n",
    "    period_quadrants=temporal_results[\"period_quadrants\"],\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    output_html=\"evoc_temporal_sankey.html\",\n",
    "    output_png=None,\n",
    "    arrangement=\"snap\",\n",
    "    show_zero_nodes=False,\n",
    "    width=1000,\n",
    "    height=500,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a0cfd91",
   "metadata": {},
   "source": [
    "# End of tutorial\n",
    "\n",
    "At this stage you have produced:\n",
    "\n",
    "- a standardised corpus;\n",
    "- a language-filtered corpus;\n",
    "- a thematic subcorpus;\n",
    "- a cleaned and annotated token table;\n",
    "- term-level AFE/AOE indicators;\n",
    "- EVOC quadrants;\n",
    "- collocations;\n",
    "- HTML reports;\n",
    "- visual outputs.\n",
    "\n",
    "For detailed theoretical explanations, consult:\n",
    "\n",
    "```text\n",
    "docs/methodology.md\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}