{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "99b33566",
   "metadata": {},
   "source": [
    "# Cluster morphology using networkx\n",
    "\n",
    "This is a short example demonstrating network analysis with DICES data. I'm using NetworkX to build the network models and Pyplot to visualize them.\n",
    "\n",
    "I'm by no means an expert in network tools. If you have more complex case studies you'd like to share, please get in touch!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b16e7d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import statements\n",
    "import pickle\n",
    "import pandas as pd\n",
    "import os\n",
    "from dicesapi import DicesAPI\n",
    "from dicesapi.jupyter import NotebookPBar\n",
    "from collections import Counter\n",
    "from matplotlib import pyplot as plt\n",
    "import networkx as nx\n",
    "\n",
    "# initialize connection to database\n",
    "api = DicesAPI(\n",
    "    progress_class = NotebookPBar,\n",
    "    logfile = 'dices.log',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e14e25",
   "metadata": {},
   "source": [
    "### Define what we want to study\n",
    "\n",
    "In this case, I'd like to organize every conversation in the corpus according to which parties talk to which other parties.\n",
    "\n",
    "Nodes in my network will be character instances, and edges will be speaker-addressee relationships. I'm not going to consider how many times they speak throughout the conversation, simple whether person A ever speaks to person B.\n",
    "\n",
    "I'll assign numbers to the participants in the order in which they appear.\n",
    "\n",
    "The function below produces a dictionary with three components:\n",
    "- `key`: a shorthand representation of who speaks to whom\n",
    "- `turns`: a table of all the speeches in the cluster\n",
    "- `graph`: a networkx graph representing speaker-addressee relationships"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b022429c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def convo_graph(cluster):\n",
    "    persons = dict()\n",
    "    \n",
    "    def get_id(inst):\n",
    "        name = inst.name if inst is not None else 'N/A'\n",
    "        \n",
    "        return persons.setdefault(name, len(persons) + 1)\n",
    "\n",
    "    turns = pd.DataFrame(dict(\n",
    "        id = cl.id,\n",
    "        source = [get_id(inst) for inst in (s.spkr or [None])],\n",
    "        target = [get_id(inst) for inst in (s.addr or [None])],\n",
    "    ) for s in cluster.getSpeeches())\n",
    "    \n",
    "    all_edges = turns.explode('source').explode('target')\n",
    "    \n",
    "    flat_with_weights = all_edges.groupby(['source','target']\n",
    "                                ).size(\n",
    "                                ).reset_index(name='weight'\n",
    "                                ).sort_values(['source', 'target'])\n",
    "    \n",
    "    graph = nx.from_pandas_edgelist(flat_with_weights, create_using=nx.DiGraph,\n",
    "                            source='source', target='target')\n",
    "    \n",
    "    key = tuple((e.source, e.target) for i, e in flat_with_weights.iterrows())\n",
    "    \n",
    "    return dict(key=key, graph=graph, turns=turns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6033b11",
   "metadata": {},
   "source": [
    "### Download all the speech clusters in the *Iliad*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85533055",
   "metadata": {},
   "outputs": [],
   "source": [
    "clusters = api.getClusters(work_title='Iliad')\n",
    "print(len(clusters), 'clusters')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325ef2b7",
   "metadata": {},
   "source": [
    "### Test out our model\n",
    "\n",
    "Let's try building a couple of graphs to see what they're like. I'm starting with item 0, the first cluster. Try picking other numbers to compare the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "484d45b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "cl = clusters[10]\n",
    "print(cl)\n",
    "\n",
    "pd.DataFrame(dict(\n",
    "    cluster = cl.id,\n",
    "    speech = s.id,\n",
    "    work = f'{s.author.name} {s.work.title}',\n",
    "    first = s.l_fi,\n",
    "    last = s.l_la,\n",
    "    spkr = s.getSpkrString(),\n",
    "    addr = s.getAddrString(),\n",
    ") for s in cl.getSpeeches())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0d52cbb",
   "metadata": {},
   "source": [
    "Run our custom function to produce key, turns, and graph as a dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7230eae",
   "metadata": {},
   "outputs": [],
   "source": [
    "bundle = convo_graph(cl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cdb07c1",
   "metadata": {},
   "source": [
    "Let's start with the turns, since that's the easiest for us to interpret. The speeches are still in order, but the names have been replaced by numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68d9b398",
   "metadata": {},
   "outputs": [],
   "source": [
    "bundle['turns']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaae49d9",
   "metadata": {},
   "source": [
    "The key is a flattened form of this, combining turns that are identical in spkr-addressee relation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12794a81",
   "metadata": {},
   "outputs": [],
   "source": [
    "bundle['key']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55384700",
   "metadata": {},
   "source": [
    "### Build graphs for each cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "979dcb30",
   "metadata": {},
   "outputs": [],
   "source": [
    "pbar = NotebookPBar(max=len(clusters))\n",
    "graphs = []\n",
    "\n",
    "for i, cl in enumerate(clusters):\n",
    "    graphs.append(convo_graph(cl))\n",
    "    pbar.update(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a844bac6",
   "metadata": {},
   "source": [
    "### Organize the clusters graphs according to key.\n",
    "\n",
    "Here we create two dictionaries. One stores all the graphs based on key, the flat representation of the map. The other stores all the turn-taking tables in the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78a4d76a",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_index = {}\n",
    "turns_index = {}\n",
    "\n",
    "for graph in graphs:\n",
    "    k = graph['key']\n",
    "    g = graph['graph']\n",
    "    m = graph['turns']\n",
    "    \n",
    "    if k not in graph_index:\n",
    "        graph_index[k] = []\n",
    "    graph_index[k].append(g)        \n",
    "        \n",
    "    if k not in turns_index:\n",
    "        turns_index[k] = []\n",
    "    turns_index[k].append(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8dc7e6e",
   "metadata": {},
   "source": [
    "### Count conversations according to key\n",
    "\n",
    "Make a quick counter of how many graphs are organized under each key, so we can see which morphologies are most common."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83d8810f",
   "metadata": {},
   "outputs": [],
   "source": [
    "key_count = Counter([g['key'] for g in graphs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b6678f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "key_count.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a9ea63",
   "metadata": {},
   "source": [
    "### Plot the most common morphologies\n",
    "\n",
    "We use the counter to take each successive map in order, from most common down. Then we check the `graph_index` for an example of the graph representing that morphology and plot it. The final line below also saves a copy of the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccff0beb",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(3, 4, figsize=(22,12))\n",
    "plt.subplots_adjust(wspace=1, hspace=.5)\n",
    "\n",
    "for i, rec in enumerate(key_count.most_common(12)):\n",
    "    key, count = rec\n",
    "    row = i % 4\n",
    "    col = i // 4\n",
    "    \n",
    "    plt.sca(ax[col, row])\n",
    "    g = graph_index[key][0]\n",
    "    nx.draw_spring(g, node_color='pink', width=4, with_labels=True)\n",
    "    ax[col,row].set_title(f'n={count}', fontsize=18)\n",
    "\n",
    "plt.savefig('foo.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52b12f6c",
   "metadata": {},
   "source": [
    "### Search for speeches by morphology\n",
    "\n",
    "We can also go the other direction: specify a key and look for examples of it in the corpus by using the indices we built."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e433adc7",
   "metadata": {},
   "source": [
    "#### Define the relationship we're looking for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2b10803",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = (((1), (2)), ((3), (1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56704c9a",
   "metadata": {},
   "source": [
    "#### Visualize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62916618",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# look at first graph\n",
    "graph = graph_index[key][0]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,6))\n",
    "nx.draw(graph, node_color='pink', width=2, with_labels=True)\n",
    "ax.set_title(f'n={len(g)}')\n",
    "fig.savefig('chain.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ab6eac8",
   "metadata": {},
   "source": [
    "#### List all matching conversations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e42a2883",
   "metadata": {},
   "outputs": [],
   "source": [
    "cl_ids = [turns.loc[0,'id'] for turns in turns_index[key]]\n",
    "\n",
    "for cl in clusters.filterIDs(cl_ids):\n",
    "    display(\n",
    "        pd.DataFrame(dict(\n",
    "            author = s.author.name,\n",
    "            work = s.work.title,\n",
    "            lines = s.l_range,\n",
    "            speaker = ', '.join([i.name for i in s.spkr]),\n",
    "            addressee = ', '.join([i.name for i in s.addr]),\n",
    "        ) for s in cl.getSpeeches())\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}