"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduction\n",
"According to the model of complex socio-cultural systems sketched in the introduction, transactions resemble the building blocks of these systems. From all the transactions in a field, a patterned macrobehavior emerges that provides expectations or meaning for future microbehavior. We refer to such structures as meaning structures (Mohr, xxxx). To get a grip on this concept, we mapped the systems model to a unified model for digital behavioral data. The fundamental idea is that agents make transactions and transactions select facts. The latter notion expresses the duality that a selection is both a top-down and a bottom-up hypothesis. In downward causation, selection expresses how agents are influenced by emergent facts -- in Padgett & Powell's (2012) words, \"relations make actors\"; as part of emergence, selection expresses how agents are free to chose facts -- \"actors make relations\".\n",
"\n",
"... Social and cultural meaning structures ... agency\n",
"\n",
"White's market profiles\n",
"\n",
"cores... Fuchs\n",
"\n",
"Flack's power distributions\n",
"\n",
"Zipf's Law\n",
"\n",
"communities and blocks\n",
"\n",
"catalysis\n",
"\n",
"attention economics\n",
"\n",
"Breiger\n",
"\n",
"domains... types of tie"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"||\n",
"|:--|\n",
"|**Figure 1**: Mapping of socio-cultural systems model to mathematical framework (fig. 2 in Lietz, 2020)|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**In this notebook**, we demonstrate how we can use the unified data model to study the meaning encoded in networks. We will lay out a core part of the ``compsoc`` mathematical framework (figure 1). We will see how digital behavioral data can be naturally represented by bipartite matrices, how meaning structures can be obtained by projecting these matrices, how matrix normalization can be used to account for potential behavioral differences in subfields, and how matrix algebra and community detection go hand in hand to uncover multiple layers or domains of social life from data. Two basic functions will be developed. They are largely based on the mathematical framework developed by Batagelj and Cerinšek (2013). Lietz (2020, particularly the [technical appendix](https://link.springer.com/article/10.1007/s11192-020-03527-0#appendices)) can be read to gain traction on the formalism used in this notebook. To be able to process large data collections, this notebook uses sparse matrix methods."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dependencies and settings"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import compsoc as cs\n",
"import networkx as nx\n",
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"#import warnings"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"#warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Understanding the basic formalism\n",
"A simple toy model shall help us to keep things simple and understandable. It contains only the most central aspects of transactions selecting facts and transactions belonging to domains. The data model consists of yellow tables for entities and green tables of relationships among two types of entities:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"||\n",
"|:--|\n",
"|**Figure 2**: Subset of unified model for digital behavioral data|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data, depicted in figure 1, resembles a field that consists of $m=5$ distinct transactions. Our computational framework requires that identifiers are integers $i=0,...,m-1$. Like all entity and relationship tables, transactions are stored in a pandas dataframe. The `transaction_id` column contains the integers $i$ and the `transaction` column contains transaction names:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"fact_id 0 1 2 3 4 5 6 7 8\n",
"transaction_id \n",
"0 0.2 0.2 0.20 0.20 0.200000 0.000000 0.000000 0.0 0.0\n",
"1 0.0 0.0 0.25 0.25 0.250000 0.250000 0.000000 0.0 0.0\n",
"2 0.0 0.0 0.00 0.00 0.333333 0.333333 0.333333 0.0 0.0\n",
"3 0.0 0.0 0.00 0.00 0.000000 0.000000 0.500000 0.5 0.0\n",
"4 0.0 0.0 0.00 0.00 0.000000 0.000000 0.000000 0.0 1.0"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_selections = pd.DataFrame(GN.toarray())\n",
"df_selections.index.name = 'transaction_id'\n",
"df_selections.columns.name = 'fact_id'\n",
"df_selections"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Matrix projection\n",
"The two bold arrows in figure 1 show the two projections that can be obtained via [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication). Projection generally involves multiplications with the transposed selection matrix. The transpose of the selection matrix $G$ is called $G^\\mathrm{T}$ and has dimensionality $n\\times m$, i.e., the rows (columns) of $G$ are its columns (rows). Similarly, the transpose of the normalized selection matrix $G^\\mathrm{N}$ is $(G^\\mathrm{N})^\\mathrm{T}$.\n",
"#### Projection to the transaction mode\n",
"The first kind of projection results in transaction similarity matrices:\n",
"\n",
"- $H=G\\times G^\\mathrm{T}$ is a transaction similarity matrix where weights $x_{ik}\\in \\mathbb{N}$ are the numbers of facts co-selected by transactions $i$ and $k$;\n",
"- $H^\\mathrm{N}=G^\\mathrm{N}\\times(G^\\mathrm{N})^\\mathrm{T}$ is a normalized transaction similarity matrix where weights $x_{ik}^\\mathrm{N}\\in \\mathbb{R}_{[0,1]}$ are the products of the normalized selections made in transactions $i$ and $k$, summed over all facts $j$.\n",
"\n",
"$H^\\mathrm{N}$ is the complementary transformation of the one described by Batagelj and Cerinšek (2013, section 3.4). Transaction matrices are directed but symmetric. Weights $x_{ik}$ and $x_{ik}^\\mathrm{N}$ can be interpreted as transaction similarities.\n",
"\n",
"Transpose the two matrices $G$ and $G^\\mathrm{N}$:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"GT = csr_matrix.transpose(G)\n",
"GNT = csr_matrix.transpose(GN)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that transposing a matrix in [Compressed Sparse Row](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) format creates a matrix in [Compressed Sparse Column](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) format:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"scipy.sparse.csc.csc_matrix"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(GT)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Multiply the matrices to obtain the projections:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"H = G*GT\n",
"HN = GN*GNT"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The format of a projected matrix is determined by the format of the first factor:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"scipy.sparse.csr.csr_matrix"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(HN)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since weights in transaction matrices can be interpreted as transaction similarities, there is no use in carrying along both triangular portions of the matrix. To remove a portion, transform the matrix into [COOrdinate](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.coo_matrix.html) format, ..."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"HN = HN.tocoo()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"... remove the upper portion (you could just as well use ``tril()`` to remove the lower portion) of the matrix, ..."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"HN = triu(HN)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"... and transform the matrix back into [Compressed Sparse Row](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) format:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"HN = HN.tocsr()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To draw $H^\\mathrm{N}$ as a graph, create an edge list (by concatenating three pandas series), ..."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"cs.draw_graph(\n",
" hn, \n",
" node_pos=nx.circular_layout(hn), \n",
" node_size_factor=.25, \n",
" node_color=cs.uniform_vertex_property(hn, 'black'), \n",
" node_border_width=2, \n",
" edge_width_factor=50, \n",
" labels='text', \n",
" font_size_factor=2, \n",
" font_color='white', \n",
" figsize='small', \n",
" margins=.2\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Projection to the fact mode\n",
"The second kind of projection results in fact co-selection matrices:\n",
"\n",
"- $I=G^\\mathrm{T}\\times G$ is a fact co-selection matrix where weights $y_{jl}\\in \\mathbb{N}$ are the numbers of transactions that co-select facts $j$ and $l$ (Batagelj and Cerinšek 2013, section 3.2);\n",
"- $I^\\mathrm{N}=G^\\mathrm{T}\\times G^\\mathrm{N}$ is a normalized fact co-selection matrix where weights $y_{jl}^\\mathrm{N}\\in \\mathbb{R}_{\\geq0}$ are the normalized numbers of transactions that co-select facts $j$ and $l$ (Batagelj and Cerinšek 2013, section 3.3).\n",
"\n",
"Fact matrices are also directed and symmetric. It is these matrices that resemble **meaning structures**. According to our general systems model, they emerge from the collective transactions in a field, and they harbor the patterns that agents observe and which influence future transactions via downward causation. Co-selection weights $y_{ik}$ and $y_{ik}^\\mathrm{N}$ of meaning structures can be interpreted as **catalyses**. Padgett & Powell (2012, chapter 4) have adapted this concept from chemistry to mean that the selection of a product in a social network entails – or makes more likely – the selection of another product (which results in a self-sustaining production cycle). Co-selection is mutual catalysis because the selection of a fact makes the selection of a co-selected fact more likely."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"I = GT*G\n",
"IN = GT*GN"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To correctly access indices and weights, co-selection matrices must be transformed from [Compressed Sparse Column](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) to [Compressed Sparse Row](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) format:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"I = I.tocsr()\n",
"IN = IN.tocsr()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Transform $I^\\mathrm{N}$ to an edge list:"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"cs.draw_graph(\n",
" _in, \n",
" node_pos=nx.circular_layout(_in), \n",
" node_size_factor=.25, \n",
" node_border_width=2, \n",
" edge_width_factor=25, \n",
" edge_transparency=.5, \n",
" curved_edges=True, \n",
" labels='text', \n",
" font_size_factor=2, \n",
" figsize='medium'\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data enrichment\n",
"Normalized fact co-selection matrices $I^\\mathrm{N}$ are additive, i.e., fact co-selections from additional transactions can simply be added to existing co-selection weights. Due to this nature, these matrices have a number of unique properties.\n",
"\n",
"The sum of all weights equals the sum of all weights in $G$:"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"15.0"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"IN.sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Fact attributes\n",
"The summed **weight** of rows or columns in $I^\\mathrm{N}$, i.e., the weighted in- or outdegree $w$ in the corresponding graph, equals the weighted number of selections per fact in the selection matrix $G$. Fact 5 (``fact_id=4``) has been selected most oftern:"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 1.0\n",
"1 1.0\n",
"2 2.0\n",
"3 2.0\n",
"4 3.0\n",
"5 2.0\n",
"6 2.0\n",
"7 1.0\n",
"8 1.0\n",
"dtype: float64"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"w = IN.sum(axis=1)\n",
"w = np.squeeze(np.array(w))\n",
"w = pd.Series(w).round(4)\n",
"w"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Elements $a$ in the diagonal are a measure of how strongly, in absolute terms, the respective facts are **autocatalytic** (catalyze themselves). In our toy example, e.g., transaction 1 selects each of its five facts with a normalized weight of 0.2. Projection of just these five selections to the fact mode would result in a fully connected graph in which facts mutually catalyze each other (including themselves) with a weight of 0.2. Since fact 1 (``fact_id=0``) is not selected in any other transaction, 0.2 is also its final value of autocatalysis, stored in the matrix diagonal. While fact 5 has been selected most often, fact 9 (``fact_id=8``) ist most autocatalytic because it is not co-selected with any other fact:"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 0.2000\n",
"1 0.2000\n",
"2 0.4500\n",
"3 0.4500\n",
"4 0.7833\n",
"5 0.5833\n",
"6 0.8333\n",
"7 0.5000\n",
"8 1.0000\n",
"dtype: float64"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a = np.array(IN.diagonal())\n",
"a = pd.Series(a).round(4)\n",
"a"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The fraction of autocatalyses $a$ among all weighted selections $w$ can be turned into a measure of **embeddedness**, $e=1-a/w$. Here, embeddedness basically tells to what extent facts are co-selected. Facts 1 and 2 are most embedded because they are each co-selected among four other facts. Fact 5 is not embedded because it is not co-selected:"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 0.8000\n",
"1 0.8000\n",
"2 0.7750\n",
"3 0.7750\n",
"4 0.7389\n",
"5 0.7084\n",
"6 0.5834\n",
"7 0.5000\n",
"8 0.0000\n",
"dtype: float64"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"e = (1-a/w).round(4)\n",
"e"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To compute facts' distinct co-selections or **degrees**, autocatalyses (loops) must be removed. This change of the sparsity structure is fastest using the [LInked List](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.lil_matrix.html) matrix format:"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'_shape': (9, 9),\n",
" 'maxprint': 50,\n",
" 'dtype': dtype('float64'),\n",
" 'rows': array([list([1, 2, 3, 4]), list([0, 2, 3, 4]), list([0, 1, 3, 4, 5]),\n",
" list([0, 1, 2, 4, 5]), list([0, 1, 2, 3, 5, 6]),\n",
" list([2, 3, 4, 6]), list([4, 5, 7]), list([6]), list([])],\n",
" dtype=object),\n",
" 'data': array([list([0.2, 0.2, 0.2, 0.2]), list([0.2, 0.2, 0.2, 0.2]),\n",
" list([0.2, 0.2, 0.45, 0.45, 0.25]),\n",
" list([0.2, 0.2, 0.45, 0.45, 0.25]),\n",
" list([0.2, 0.2, 0.45, 0.45, 0.5833333333333333, 0.3333333333333333]),\n",
" list([0.25, 0.25, 0.5833333333333333, 0.3333333333333333]),\n",
" list([0.3333333333333333, 0.3333333333333333, 0.5]), list([0.5]),\n",
" list([])], dtype=object)}"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"IN_nodiag = IN.tolil()\n",
"IN_nodiag.setdiag(values=0)\n",
"IN_nodiag.__dict__"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With a degree of 6, fact 5 is most strongly connected to other facts:"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 4\n",
"1 4\n",
"2 5\n",
"3 5\n",
"4 6\n",
"5 4\n",
"6 3\n",
"7 1\n",
"8 0\n",
"dtype: int64"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"k = [len(i) for i in IN_nodiag.data.tolist()]\n",
"k = pd.Series(k)\n",
"k"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A final score is most intuitive when facts are persons but also applies to cultural entities. A person that only goes to few parties but talks to many persons during those transactions may be called sociable. **Sociability** is the extent to which activity (measured as the number of selections $w$) is turned into connections $k$. Facts 1 and 2 are most sociable because one selection each results in four co-selections. Fact 5 is only moderately sociable because it took three selections to obtain six co-selections."
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 4.0\n",
"1 4.0\n",
"2 2.5\n",
"3 2.5\n",
"4 2.0\n",
"5 2.0\n",
"6 1.5\n",
"7 1.0\n",
"8 0.0\n",
"dtype: float64"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s = (k/w).round(4)\n",
"s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With these fact attributes, we can now enrich the ``facts`` dataframe:"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"Caution\n",
"\n",
"It is important to understand the effects of matrix normalization as we have discussed it so far. For example, a strong co-selection of two facts can result from few transactions in which few facts are co-selected or from many transactions in which many facts are co-selected. Matrix normalization complicates interpretability when the number of selections per transaction exhibits large variance. In small-data settings, it may therefore be advised to not use matrix normalization. In big-data settings, it may be possible to remove outlier transactions with exceptionally many selections (e.g., publications with either very short or very long reference lists).\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Cumulative co-selection fractions\n",
"Sometimes it is can be helpful to filter graphs, e.g., to uncover a pattern that may otherwise be hidden in relational \"noise\". An observer of a meaning structure may neglect weak ties to identify the core facts of the field. Hence, removing weakly selected nodes or edges is not only a step we can do as network analysts, but an analytical step towards understanding what an observer in a field may have observed as he or she was trying to reduce uncertainty.\n",
"\n",
"As an alternative to simply filtering edges with small weights, one can keep those co-selections that collectively account for a specified amount of attention in the field. For this purpose we enrich edges by **cumulative fractions** of total matrix weights.\n",
"\n",
"First, distinct edge weights and the weight (attention) they collectively account for are identified. In the toy example, a single co-selection tie with a ``weight == 1`` also accounts for a total weight of 1. But 16 co-selection ties with ``weight == 0.2`` account for a total weight of 3.2:"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"weight\n",
"1.000000 1.000000\n",
"0.833333 0.833333\n",
"0.783333 0.783333\n",
"0.583333 1.750000\n",
"0.500000 1.500000\n",
"0.450000 3.600000\n",
"0.333333 1.333333\n",
"0.250000 1.000000\n",
"0.200000 3.200000\n",
"Name: weight, dtype: float64"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"co_selections_norm_cumfrac = co_selections_norm.copy()\n",
"co_selections_norm_cumfrac.index = co_selections_norm_cumfrac.weight\n",
"co_selections_norm_cumfrac = co_selections_norm_cumfrac['weight'].groupby(co_selections_norm_cumfrac.index).sum()\n",
"co_selections_norm_cumfrac = co_selections_norm_cumfrac.sort_index(ascending=False)\n",
"co_selections_norm_cumfrac"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Second, the summed weights are turned into cumulative co-selection fractions ``cumfrac`` such that all weights at least as strong as the smallest one account for 100 percent of the attention in the whole field. In the toy example, if we wanted to keep those co-selections that account for no more than 75 percent of the attention (``cumfrac <= .75``), we would have to remove all edges with ``weight < 0.3333``:"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"weight\n",
"1.000000 0.0667\n",
"0.833333 0.1222\n",
"0.783333 0.1744\n",
"0.583333 0.2911\n",
"0.500000 0.3911\n",
"0.450000 0.6311\n",
"0.333333 0.7200\n",
"0.250000 0.7867\n",
"0.200000 1.0000\n",
"Name: cumfrac, dtype: float64"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"co_selections_norm_cumfrac = co_selections_norm_cumfrac.cumsum()/sum(co_selections_norm_cumfrac)\n",
"co_selections_norm_cumfrac = co_selections_norm_cumfrac.round(4)\n",
"co_selections_norm_cumfrac.rename('cumfrac', inplace=True)\n",
"co_selections_norm_cumfrac"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The cumulative fractions can then be merged back into the co-selection dataframe:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"co_selections_norm = pd.merge(left=co_selections_norm, right=co_selections_norm_cumfrac, left_on='weight', right_on=co_selections_norm_cumfrac.index)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Apply the edge filter:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"cs.draw_graph(\n",
" in_filter, \n",
" node_pos=nx.circular_layout(in_filter), \n",
" node_size_factor=.25, \n",
" node_border_width=2, \n",
" edge_width_factor=25, \n",
" edge_transparency=.5, \n",
" curved_edges=True, \n",
" labels='text', \n",
" font_size_factor=2, \n",
" figsize='medium'\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Function\n",
"The ``project_selection_matrix()`` function performs all these steps with just a few parameter inputs. It has two additional features: First, to handle cases where transaction and fact identifiers are not integers from $0$ to $m$ or $n$, that normal form is created (using a function defined in lines 38-40). Second, for transaction matrices the degrees $k$ and selection weights $w$ of transactions are provided."
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"def project_selection_matrix(\n",
" selections, \n",
" how, \n",
" transaction_id, \n",
" fact_id, \n",
" norm=True, \n",
" remove_loops=True, \n",
" symmetrize=True\n",
"):\n",
" '''\n",
" Description: This function uses the terminology of the compsoc unified data model \n",
" according to which \"transactions select facts\"; these selections are stored in a \n",
" selection matrics; the function projects a selection matrix to a transaction \n",
" similarity matrix or a fact co-selection matrix; computes fact attributes; \n",
" computes cumulative co-selection fractions for matrix filtering.\n",
" \n",
" Inputs:\n",
" selections: Dataframe containing the selection matrix indices and data; must \n",
" contain a 'weight' column that contains the cell weights.\n",
" how: String that specifies which projection is to be made; must be either \n",
" 'transactions' or 'facts'; if 'transactions', then the matrix of transactions \n",
" coupled by facts will be created; if 'facts', then the matrix of facts \n",
" coupled by transactions will be created.\n",
" transaction_id: Name of the column of the dataframe ``selections`` that holds the \n",
" identifiers of the transactions selecting facts.\n",
" fact_id: Name of the column of the dataframe ``selections`` that holds the \n",
" identifiers of the facts getting selected in transactions.\n",
" norm: Boolean parameter specifying if matrix normalization should be performed.\n",
" remove_loops: Boolean parameter specifying if the matrix diagonal should be \n",
" removed; if False, loops will be included in computing cumulative \n",
" co-selection fractions.\n",
" symmetrize: Boolean parameter specifying if the lower portion of the matrix \n",
" should be removed.\n",
" \n",
" Output: A dataframe containing the projected matrices (enriched by cumulative \n",
" fractions in the case of a normalized projection to the fact mode); a dataframe \n",
" containing matrix-based attributes of transactions or facts (depending on the \n",
" type of projection)\n",
" '''\n",
" \n",
" # function\n",
" def get_unique(s):\n",
" l = s.unique().tolist()\n",
" return {identifier: index for index, identifier in enumerate(l)}\n",
" \n",
" # map identifiers of transactions and facts to unique integers\n",
" import pandas as pd\n",
" d_transactions_indices = get_unique(selections[transaction_id])\n",
" d_facts_indices = get_unique(selections[fact_id])\n",
" \n",
" # construct selection matrix\n",
" rows = [d_transactions_indices[transaction_id] for transaction_id in selections[transaction_id].values]\n",
" columns = [d_facts_indices[fact_id] for fact_id in selections[fact_id].values]\n",
" cells = selections['weight'].tolist()\n",
" from scipy.sparse import csr_matrix, coo_matrix, triu\n",
" G = coo_matrix((cells, (rows, columns))).tocsr()\n",
" GT = csr_matrix.transpose(G)\n",
" from sklearn.preprocessing import normalize\n",
" GN = normalize(G, norm='l1', axis=1)\n",
" \n",
" # project selection matrix ...\n",
" import numpy as np\n",
" \n",
" # ... to transaction similarity matrix\n",
" if how == 'transactions':\n",
" if norm == True:\n",
" GNT = csr_matrix.transpose(GN)\n",
" H = GN*GNT\n",
" else:\n",
" H = G*GT\n",
" \n",
" # derive transaction attributes dataframe\n",
" H_nodiag = H.tolil()\n",
" H_nodiag.setdiag(values=0)\n",
" \n",
" k = pd.Series([len(i) for i in H_nodiag.data.tolist()])\n",
" w = pd.Series(np.array(H.diagonal()))\n",
" if norm == True:\n",
" w = (1/w).round(4)\n",
" else:\n",
" w = w.round(4)\n",
" \n",
" d_indices_transactions = {index: identifier for identifier, index in d_transactions_indices.items()}\n",
" \n",
" transaction_attributes = pd.concat([pd.Series(d_indices_transactions), k, w], axis=1)\n",
" transaction_attributes.columns = [transaction_id, 'degree', 'weight']\n",
" \n",
" # construct similarities dataframe\n",
" if remove_loops == True:\n",
" H = H.tolil()\n",
" H.setdiag(0)\n",
"\n",
" if symmetrize == True:\n",
" H = triu(H.tocoo()).tocsr()\n",
" else:\n",
" H = H.tocsr()\n",
" \n",
" transaction_id_from = [d_indices_transactions[index] for index in H.nonzero()[0].tolist()]\n",
" transaction_id_to = [d_indices_transactions[index] for index in H.nonzero()[1].tolist()]\n",
" weight = H.data.tolist()\n",
" \n",
" similarities = pd.concat([pd.Series(transaction_id_from), pd.Series(transaction_id_to), pd.Series(weight)], axis=1)\n",
" similarities.columns = [transaction_id+'_from', transaction_id+'_to', 'similarity']\n",
" \n",
" return similarities, transaction_attributes\n",
" \n",
" # ... to fact co-selection matrix\n",
" if how == 'facts':\n",
" if norm == True:\n",
" I = GT*GN\n",
" else:\n",
" I = GT*G\n",
" \n",
" # derive fact attributes dataframe\n",
" I_nodiag = I.tolil()\n",
" I_nodiag.setdiag(values=0)\n",
" \n",
" k = pd.Series([len(i) for i in I_nodiag.data.tolist()])\n",
" \n",
" d_indices_facts = {index: identifier for identifier, index in d_facts_indices.items()}\n",
" \n",
" if norm == True:\n",
" w = pd.Series(np.squeeze(np.array(I.sum(axis=1)))).round(4)\n",
" a = pd.Series(np.array(I.diagonal())).round(4)\n",
" e = (1-a/w).round(4)\n",
" s = (k/w).round(4)\n",
" \n",
" fact_attributes = pd.concat([pd.Series(d_indices_facts), k, w, a, e, s], axis=1)\n",
" fact_attributes.columns = [fact_id, 'degree', 'weight', 'autocatalysis', 'embeddedness', 'sociability']\n",
" \n",
" else:\n",
" fact_attributes = pd.concat([pd.Series(d_indices_facts), k], axis=1)\n",
" fact_attributes.columns = [fact_id, 'degree']\n",
" \n",
" # construct co-selections dataframe with cumulative co-selection fractions\n",
" if remove_loops == True:\n",
" I = I.tolil()\n",
" I.setdiag(0)\n",
" \n",
" if symmetrize == True:\n",
" I = triu(I.tocoo()).tocsr()\n",
" else:\n",
" I = I.tocsr()\n",
" \n",
" fact_id_from = [d_indices_facts[index] for index in I.nonzero()[0].tolist()]\n",
" fact_id_to = [d_indices_facts[index] for index in I.nonzero()[1].tolist()]\n",
" weight = I.data.tolist()\n",
" \n",
" co_selections = pd.concat([pd.Series(fact_id_from), pd.Series(fact_id_to), pd.Series(weight)], axis=1)\n",
" co_selections.columns = [fact_id+'_from', fact_id+'_to', 'weight']\n",
" \n",
" co_selections_cumfrac = co_selections.copy()\n",
" co_selections_cumfrac.index = co_selections_cumfrac.weight\n",
" co_selections_cumfrac = co_selections_cumfrac['weight'].groupby(co_selections_cumfrac.index).sum()\n",
" co_selections_cumfrac = co_selections_cumfrac.sort_index(ascending=False)\n",
" co_selections_cumfrac = co_selections_cumfrac.cumsum()/sum(co_selections_cumfrac)\n",
" co_selections_cumfrac = co_selections_cumfrac.round(4)\n",
" co_selections_cumfrac.rename('cumfrac', inplace=True)\n",
" \n",
" co_selections = pd.merge(left=co_selections, right=co_selections_cumfrac, left_on='weight', right_on=co_selections_cumfrac.index)\n",
" \n",
" return co_selections, fact_attributes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As part of the ``compsoc`` library, we can call this function, e.g., to compare the fact co-selection matrices with and without normalization:"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"with_normalization, _ = cs.project_selection_matrix(\n",
" selections=selections, \n",
" how='facts', \n",
" transaction_id='transaction_id', \n",
" fact_id='fact_id', \n",
" norm=True, \n",
" remove_loops=False, \n",
" symmetrize=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
fact_id_to
\n",
"
0
\n",
"
1
\n",
"
2
\n",
"
3
\n",
"
4
\n",
"
5
\n",
"
6
\n",
"
7
\n",
"
8
\n",
"
\n",
"
\n",
"
fact_id_from
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
0.2
\n",
"
0.2
\n",
"
0.20
\n",
"
0.20
\n",
"
0.2000
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
1
\n",
"
0.2
\n",
"
0.2
\n",
"
0.20
\n",
"
0.20
\n",
"
0.2000
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
2
\n",
"
0.2
\n",
"
0.2
\n",
"
0.45
\n",
"
0.45
\n",
"
0.4500
\n",
"
0.2500
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
3
\n",
"
0.2
\n",
"
0.2
\n",
"
0.45
\n",
"
0.45
\n",
"
0.4500
\n",
"
0.2500
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
4
\n",
"
0.2
\n",
"
0.2
\n",
"
0.45
\n",
"
0.45
\n",
"
0.7833
\n",
"
0.5833
\n",
"
0.3333
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
5
\n",
"
NaN
\n",
"
NaN
\n",
"
0.25
\n",
"
0.25
\n",
"
0.5833
\n",
"
0.5833
\n",
"
0.3333
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
6
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
0.3333
\n",
"
0.3333
\n",
"
0.8333
\n",
"
0.5
\n",
"
NaN
\n",
"
\n",
"
\n",
"
7
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
0.5000
\n",
"
0.5
\n",
"
NaN
\n",
"
\n",
"
\n",
"
8
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
1.0
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
"fact_id_to 0 1 2 3 4 5 6 7 8\n",
"fact_id_from \n",
"0 0.2 0.2 0.20 0.20 0.2000 NaN NaN NaN NaN\n",
"1 0.2 0.2 0.20 0.20 0.2000 NaN NaN NaN NaN\n",
"2 0.2 0.2 0.45 0.45 0.4500 0.2500 NaN NaN NaN\n",
"3 0.2 0.2 0.45 0.45 0.4500 0.2500 NaN NaN NaN\n",
"4 0.2 0.2 0.45 0.45 0.7833 0.5833 0.3333 NaN NaN\n",
"5 NaN NaN 0.25 0.25 0.5833 0.5833 0.3333 NaN NaN\n",
"6 NaN NaN NaN NaN 0.3333 0.3333 0.8333 0.5 NaN\n",
"7 NaN NaN NaN NaN NaN NaN 0.5000 0.5 NaN\n",
"8 NaN NaN NaN NaN NaN NaN NaN NaN 1.0"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.pivot_table(\n",
" data=with_normalization, \n",
" values='weight', \n",
" index='fact_id_from', \n",
" columns='fact_id_to'\n",
").round(4)"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"without_normalization, _ = cs.project_selection_matrix(\n",
" selections=selections, \n",
" how='facts', \n",
" transaction_id='transaction_id', \n",
" fact_id='fact_id', \n",
" norm=False, \n",
" remove_loops=False, \n",
" symmetrize=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
fact_id_to
\n",
"
0
\n",
"
1
\n",
"
2
\n",
"
3
\n",
"
4
\n",
"
5
\n",
"
6
\n",
"
7
\n",
"
8
\n",
"
\n",
"
\n",
"
fact_id_from
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
1.0
\n",
"
1.0
\n",
"
1.0
\n",
"
1.0
\n",
"
1.0
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
1
\n",
"
1.0
\n",
"
1.0
\n",
"
1.0
\n",
"
1.0
\n",
"
1.0
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
2
\n",
"
1.0
\n",
"
1.0
\n",
"
2.0
\n",
"
2.0
\n",
"
2.0
\n",
"
1.0
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
3
\n",
"
1.0
\n",
"
1.0
\n",
"
2.0
\n",
"
2.0
\n",
"
2.0
\n",
"
1.0
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
4
\n",
"
1.0
\n",
"
1.0
\n",
"
2.0
\n",
"
2.0
\n",
"
3.0
\n",
"
2.0
\n",
"
1.0
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
5
\n",
"
NaN
\n",
"
NaN
\n",
"
1.0
\n",
"
1.0
\n",
"
2.0
\n",
"
2.0
\n",
"
1.0
\n",
"
NaN
\n",
"
NaN
\n",
"
\n",
"
\n",
"
6
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
1.0
\n",
"
1.0
\n",
"
2.0
\n",
"
1.0
\n",
"
NaN
\n",
"
\n",
"
\n",
"
7
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
1.0
\n",
"
1.0
\n",
"
NaN
\n",
"
\n",
"
\n",
"
8
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
NaN
\n",
"
1.0
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
"fact_id_to 0 1 2 3 4 5 6 7 8\n",
"fact_id_from \n",
"0 1.0 1.0 1.0 1.0 1.0 NaN NaN NaN NaN\n",
"1 1.0 1.0 1.0 1.0 1.0 NaN NaN NaN NaN\n",
"2 1.0 1.0 2.0 2.0 2.0 1.0 NaN NaN NaN\n",
"3 1.0 1.0 2.0 2.0 2.0 1.0 NaN NaN NaN\n",
"4 1.0 1.0 2.0 2.0 3.0 2.0 1.0 NaN NaN\n",
"5 NaN NaN 1.0 1.0 2.0 2.0 1.0 NaN NaN\n",
"6 NaN NaN NaN NaN 1.0 1.0 2.0 1.0 NaN\n",
"7 NaN NaN NaN NaN NaN NaN 1.0 1.0 NaN\n",
"8 NaN NaN NaN NaN NaN NaN NaN NaN 1.0"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.pivot_table(\n",
" data=without_normalization, \n",
" values='weight', \n",
" index='fact_id_from', \n",
" columns='fact_id_to'\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Multiplex data\n",
"As we have just seen, the weight of edges in meaning structures is intimately related to the number of transactions and the number of selections per transaction in a field. If we simply filter co-selections in a field (using weights or cumulative fractions), we may unwillingly punish a subfield that is not as voluminous or exhibits a different style of behavior as another subfield. For example, an academic field may harbor subfields of different size (different number of publications) and with different publication practices (different conventions of how many references to cite in a publication). In such cases, it is advised to use multiplex data, i.e., to use one network layer per subfield. These layers are the different **domains** of social life. Each domain has its own **type of tie**, but domains can – and typically do – overlap in terms of facts. It is a beauty of the matrix formalism that a mapping of transactions to domains translates to a separate type of tie per domain in the meaning structure (see figure 1).\n",
"\n",
"### Function\n",
"\n",
"The ``meaning_structures()`` function is a software wrapper around, i.e., it calls and does things in addition to, the ``project_selection_matrix()`` function that is tailored to creating multi-layer meaning structures (it is restricted to projecting selection matrices to the fact mode). The ``multiplex`` parameter decides if multi-layer fact matrices are created. If ``multiplex=True``, a ``transactions`` dataframe must be provided that maps a ``transaction_id`` to a ``domain_id``. Consult the description below for details, e.g., about outputs:"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [],
"source": [
"def meaning_structures(\n",
" selections, \n",
" transaction_id, \n",
" fact_id, \n",
" multiplex=False, \n",
" transactions=None, \n",
" domain_id=None, \n",
" facts=None, \n",
" norm=True, \n",
" remove_loops=True, \n",
" symmetrize=True\n",
"):\n",
" '''\n",
" Description: Projects a selection matrix to (multiplex) co-selection matrix.\n",
" \n",
" Inputs:\n",
" selections: Dataframe containing the selection matrix indices and data; must \n",
" contain a 'weight' column that contains the cell weights.\n",
" transaction_id: Name of the column of the dataframe ``selections`` that holds the \n",
" identifiers of the transactions selecting facts.\n",
" fact_id: Name of the column of the dataframe ``selections`` that holds the \n",
" identifiers of the facts getting selected in transactions.\n",
" multiplex: Boolean parameter specifying if selections occurr in multiple domains; \n",
" set to False by default.\n",
" transactions: Dataframe containing the ``transaction_id`` identifiers of the \n",
" ``selections`` dataframe; must be specified if ``multiplex=True``; set to None \n",
" by default.\n",
" domain_id: Name of the column of the dataframe ``transactions`` that holds the \n",
" identifiers of the domains the transactions belong to; must be an integer from \n",
" 0 to d where d is the number of domains; must be specified if \n",
" ``multiplex=True``; set to None by default.\n",
" facts: Dataframe containing the ``fact_id`` identifiers of the ``selections`` \n",
" dataframe; if specified, it will be enriched by fact attributes; set to None \n",
" by default.\n",
" norm: Boolean parameter specifying if matrix normalization should be performed.\n",
" remove_loops: Boolean parameter specifying if the matrix diagonal should be \n",
" removed; if False, loops will be included in computing cumulative \n",
" co-selection fractions.\n",
" symmetrize: Boolean parameter specifying if the lower portion of the matrix \n",
" should be removed.\n",
" \n",
" Output: At least two dataframes will be returned: first, a dataframe containing the \n",
" co-selection matrix independent of domain; second, a dataframe containing fact \n",
" attributes (if no ``facts`` dataframe is provided), or an enriched ``facts`` \n",
" dataframe (if one is provided), independent of domain. When ``multiplex=True`` \n",
" two additional dataframes will be returned: third, a dataframe containing the \n",
" co-selection matrix for domains; fourth, a list of dataframes containing fact \n",
" attributes (if no ``facts`` dataframe is provided), or a list of enriched \n",
" ``facts`` dataframes (if a ``facts`` dataframe is provided), for domains.\n",
" '''\n",
" \n",
" if multiplex == True:\n",
" if transactions is None:\n",
" print('A transactions dataframe must be specified.')\n",
" else:\n",
" if domain_id is None:\n",
" print('The domain identifier for the transactions dataframe must be specified.')\n",
" else:\n",
" if domain_id not in transactions.columns:\n",
" print('The specified domain identifier is not a column in the transactions dataframe.')\n",
" else:\n",
" domain_ids = set(transactions[domain_id])\n",
" if (len(domain_ids) > 1) & (min(domain_ids) == 0) & (max(domain_ids) == len(domain_ids)-1):\n",
" \n",
" # co-selections and fact attributes dataframes independent of domain\n",
" co_selections, fact_attributes = project_selection_matrix(selections=selections, how='facts', transaction_id=transaction_id, fact_id=fact_id, norm=norm, remove_loops=remove_loops, symmetrize=symmetrize)\n",
" \n",
" # co-selections and fact attributes dataframes for domains\n",
" co_selections_domain = pd.DataFrame(columns=[fact_id+'_from', fact_id+'_to', 'weight', 'cumfrac', domain_id])\n",
" fact_attributes_domain = []\n",
" facts_enriched_domain = []\n",
" for identifier in set(transactions[domain_id]):\n",
" df = selections[selections[transaction_id].isin(transactions[transactions[domain_id] == identifier][transaction_id])]\n",
" df_co_selections, df_fact_attributes = project_selection_matrix(selections=df, how='facts', transaction_id=transaction_id, fact_id=fact_id, norm=norm, remove_loops=remove_loops, symmetrize=symmetrize)\n",
" df_co_selections[domain_id] = identifier\n",
" co_selections_domain = pd.concat([co_selections_domain, df_co_selections])\n",
" if facts is None:\n",
" fact_attributes_domain.append(df_fact_attributes)\n",
" else:\n",
" df_facts_enriched = pd.merge(left=facts, right=df_fact_attributes, on=fact_id, how='left')\n",
" facts_enriched_domain.append(df_facts_enriched)\n",
" co_selections_domain.reset_index(drop=True, inplace=True)\n",
" if facts is None:\n",
" return co_selections, fact_attributes, co_selections_domain, fact_attributes_domain\n",
" else:\n",
" facts_enriched = pd.merge(left=facts, right=fact_attributes, on=fact_id, how='left')\n",
" \n",
" return co_selections, facts_enriched, co_selections_domain, facts_enriched_domain\n",
" else:\n",
" print('The specified domain identifier does not contain multiple domains or domains are not coded as integers starting with zero.')\n",
" else:\n",
" \n",
" # co-selections and fact attributes dataframes independent of domain\n",
" co_selections, fact_attributes = project_selection_matrix(selections=selections, how='facts', transaction_id=transaction_id, fact_id=fact_id, norm=norm, remove_loops=remove_loops, symmetrize=symmetrize)\n",
" \n",
" if facts is None:\n",
" return co_selections, fact_attributes\n",
" else:\n",
" facts_enriched = pd.merge(left=facts, right=fact_attributes, on=fact_id, how='left')\n",
" \n",
" return co_selections, facts_enriched"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see how node communities in transaction similarity graphs are mirrored in link communities (types of tie) in fact co-selection graphs. The `transactions` dataframe of the toy example contains the necessary mapping of `transaction_id` to `domain_id`. Since each transaction only belongs to a single domain, we spare the green \"Belong\" table in figure 2 and story the information in the `transactions` table. To use these domain identifiers for coloring nodes in transaction graphs using the `draw_graph()` function, we must create a dictionary that maps a `domain_id` to a color. [colorbrewer2.org](https://colorbrewer2.org/) is a great source of color advice for cartography. In our dictionary, we want the first domain to be colored in red and the second in blue:"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [],
"source": [
"domain_color = {0: '#e41a1c', 1: '#377eb8'}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To draw the transaction graph, this dictionary is the color source for the nodes. Create a dictionary that maps a node to a color by using the `partition_to_vertex_property()` helper described [here](draw_graph.ipynb). The needed partition is the `domain_id` column of the `transactions` table:"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: '#e41a1c', 1: '#e41a1c', 2: '#377eb8', 3: '#377eb8', 4: '#377eb8'}"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"node_color = cs.partition_to_vertex_property(transactions['domain_id'], {0: '#e41a1c', 1: '#377eb8'})\n",
"node_color"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we simply assign this dictionary to the `node_color` parameter of the `draw_graph()` function:"
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADnCAYAAADy1tHpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd3hbx5mv3wHYe5VIqneRVLMlW5Il9yLLshwnjp2yqTdxiTe+u1nvbrIts5Ns4rQt2eQ6Tq7cbhInm+LsWpYs9yLLVrEsW4XqVCcpsYhVbADm/nEAgiokAOIckADnfR4+D0CeM/NR4g/znZmvCK01BoMhfnANtwEGgyEyjGgNhjjDiNZgiDOMaA2GOMOI1mCIM4xoDYY4w4jWYIgzjGgNhjjDiNZgiDOMaA2GOMOI1mCIM4xoDYY4w4jWYIgzjGgNhjjDiNZgiDOMaA2GOCNpuA0wjByUUtnAlcBC/9cCIBdIBXqBDmAv8B6wHdgipawbHmtHL8JUrjAopS4HHkTrTyNEetg3aq0RYj3wKPCilNLrlI2GIEa0oxil1DXAD4DFge8VnzlDUX0DxfX1FDY0kt7ZicvnQwtBT2oKjQWFNBQX0VBURF1pCT63O3DrUeBfgCeklOaPykGMaEchSqlM4BHgIYCU7m5mHjjA7Kp95LW0hD1OZ1oaB2bNZG95Oe052YFvvwTcK6U8brfdBgsj2lGGUupK4DfAVOHzsWDHB8z7cCfJHs+Qx/QJweFp09h81RK609JA6zaE+HMp5S9tM9zQhxHtKEIpdTNa/zdCZBQ0NnLNG29R1Nho2/jn0tPZtHwZx6ZMDnzrG1LK79s2gQEwoh01KKVuROv1CJEyY/8Blm98G7fPZ/s8GthbUc47y64CIcAI13aMaEcBSqn5aL0JITLL91Rx1aZ3EA7PeXD6dN68/tqAcL8gpXza4SlHDUa0CY5SKgXYBsybdvAQ173+huOCDVBVUc47y5eB1u0IMVdKeTRGUyc0JiIq8fkHYF5OSwvLN74dM8ECVFTtZUp1NQiRBaxRSsVy+oTFREQlMEqpBWj99wjB1W9uDHuH2F1WRtptK0ldvozkigrcRUXo3l68x4/T9drrtD/+BL4zZ8Ia66q336G2tJSu9PQbgfuAnw/9NzKAcY8TGqXU74C7K/bs4apN74Z1j7uslLFbNiNcQSfM19qKyMhAJFmf8b6zzTTedx8974Q35uGpU3n9phsATgBTpZRDP18yGPc4UVFKlaH1R4XPx/wdH4Z/o8uKcOp65RUa77ufmoo51JZXUjN9Jg2f+SyeY8dw5edR+PgaXMXFYQ05tbqa3OZmgAnAqoh/GcN5GNEmLl9GiKRJR4+Ree5c2Df5Wlo4s+JWGj//RbrWrUcHIqR6e+l+/Q0aP/t5dGcXrpwcMj/zZ2GNKYDyqr2Btw9G9FsYLsKINgHxb/h8CaCiqiqie3VbG56gwC7Cc/gwPTveByB53rywx51x4CBu65n6FqXUxIiMMpyHEW1iUgZMTOnuprSm1vbBfWfPAiDc4f/5pPb0UHaqJvB2qe1GjSKMaBOThQBFDQ32H/G43aQsWgRA7779Ed1a1NAQeLnQXqNGF0a0iYkl2vqGUNdFTOYXPo977Fi018u5P/wxonuNaO3BiDYxmQNQaGMyAEBS+WxyvvF1ADqefArPgQMR3V/Y0GdPpa2GjTKMaBOTTICU7h7bBnSNGWMd82Rk0PPhTlq++0jEY6T09NmTZZthoxAj2sQkBcBlUxaPyMuj6JlfkzRpEp7qaho/93no7o54nH72pNhi2CjFiDYx6QbwRbC7OxAiO5uiX/+K5PLZeE6epOETn8LXMLRnZV8wyqorasNGMSb2OEFQSqUC1wKrgWsAOtPCr9F2KUR6OoW/fJqUBfPxnj5Nwyc/hbemJvSNA9CZ3mdPa1SGjXKMaOMYpVQxcBuWUFdwwbNiY1EhHDw4tMHT0ih8+klSr7gCb1MTDZ/8NN4jR6Oyt7GoMPAygrhKw4UY0cYR/kinCiyRrsYKUhjwKLY+zNjgi0hOpvD//oLUZcvwNTfT+Kk/i3in+FI0FBUFXm6PerBRjBHtCMefxH4NQaFOCffepsICfELgiiSTy+Wi4P/8hLQbrsfX1kbDZz9H7+7dkZp9SRqKjWjtwIh2BKKUKgRWYon0ViBnKON4kpM5NnkSUyJwa1OuuIL0VVYijkhKpvDxNQNe662ppX7V7WGN25GZQW1pKWjtQYjwcvoMl8SIdgTgd3tnEVxNl2HTzv7eivKIRIsr6G2L9DTc6WkDXqojOPbZN3s22to9flZKGV4GveGSGNEOE0qpZGA5QaFOt30SrakZN46zeXnkW/msIel5dzOnxk2w1QyfEOwrnx14+6itg49CRpRo/StOGTAZSMfaZOkCaoDqeG83oZTK53y3N8/RCa1KiGy/YiE3vvxqTOtD9aeqsoLOjAyAKuCtYTIjYRj2cjNKqUrgHmARViD52AEubQbex9rEeA7YFA8iVkrNJLiaLgfcg99hC1XAWmArWj+NEFnXv/Ia06qrYzD1+bTk5PDsxz+G1ypV8xEp5XMxNyLBGBbR+ndE78SqYnBt/5/1ajcdpOPFhQbc+EinmzTRe+Ewu7BcrV9LKdtiYXc4KKWSgKuwRHoHMDMG03qAN7GEulZK2adOpdT9wGOpXV18/Hd/IL0rdsFIPiFYt3oVp0tKAH4lpfxszCZPYGIuWqXUTcAaYBKAR7uooYhGnUsrmZwjlUsdPabSQy4d5ItWxtFAql/EWtMsBH8FPDVcK69SKhfL3V2NFeyQH4Npm4D1WEJ9UUp5yc5Z/keOl4CbSmpqufWFDSR5ne9IqYEtSxaze95c0LoOISqllE2OTzwKiJlolVI5wA+xymjSrtM5psdyimK8EXqMAh8lNDFRnKZA9C2y64H7pJSn7LR7IJRS0wi6vdcQm/2BffhXU+DdcKsaKqUmoPVmhCibcPw4N778qqPC1cD7Cy9nx8LLQWsfQqyQUr7i2ISjjJiIVilVAawDJvu04KAezxHK0FFvjWjKaKBCHCVZeNGaFiH4uBN/IEopN1YEUkCo5XbPcQm8wEasZ/i1UspDQx1IKVWB1m8hRGFJTS03vvKqI66y1+Viy5LFVM2pBK0Dm2EPSSl/avtkoxTHRauUulxrXhKCwhadyU49jXYybJ0jlR7miGrGiGa0pkcIPiGl/O9ox/V7BysIur2Fg99hC83AC1ir6QYp5Vm7BlZKVaL1SwhRltbZybK3N0V2hhuChsJC3rruGpoKC/sLFqxn7uuklJtsm2wU46holVIVWvOWEBSe0Xns0DPxOZYNqCkXx5gs6tCaXiFYLaV8MdJRlFJTCK6m1wLJdlt6CQ4SdHs3SSkv2nWzC38lxKeA6wGmHK7miq3byGkb+l5eV2oqu+fO4cMF8wMBFJeiFrhcSlk35IkMgIOiVUplAzuByWd0Hu/rmWjH03c1s8UxpljCPScE80O5lH6390qCQp3jsJEAPuBtgru9kVVIixKllAt4AK1/gBCZAOOPn6C8ai8TTpwIO1a5vqiIvZXlHJ42LXCkE4qNwI1OfiiNBpwU7WPA/S06k826Al9MjicBNAvEQUpFE1gH+ddLKc8r4aCUygJuwRLpKmCI6TAR0UrQ7X1hJOyk+r2Kb6L1JxEiDSClu5uihgYKGxopqm8gvbMTt9eLz+2iJyWFxsJCGoqKaCgq4lxWZv/hXsf6wAv1b/kfUsqvOfMbjQ4cEa3/WOdlnxZs0nNtf4YNRTK9XC12Bo6F/kJK+Z9+tzCwml5PbEqeHCbo9m4cqSuMP0HhC8ADRBZO2Qg8CfxcSnlIKbUE64My1CPFp6WUvxmKrQYHROsPnNgPTN7vm0A148K6b3ZZDtfMHkN5WS7jCzLIz0wmJclN87ke9tW08vyOU7y1L/w48zE0sdB1AK3xCMEBrDxUp/EB7xIU6t54iNoK0C+MdKH/az5WqGUa0AN0YEVbbfd/HbyEF/MA8LMQU50Dlkgpd9n6C4wSnBDtPcB/tet03tbzwj7W+dvbK/jYFcFA9Y5uD26XIC056Fa/tqeOf/rDTry+8Gy+XOxnrLBt83Ug2oAXsUS6Xkppf7HhOMIv/CeBz4e49BCwaKCgEMPAOCHaN4Br9/gmc5ySsO+7bX4Z2enJfHCsieON5+jssQ7/x+Skcc/iiXxmuZX7/fPXDvLkm+HF0BbQwmLXwH1pouAowdX0TSmlfbVKEwClVDqwCbgsxKXPAR+9cLU2DI6tovUH/+/2aBev6YURRzoNhvzYXFbOL+Nk0zk+/uONYd6lWS52ki06o51eA5sJCnVPPLm9w4F/k2s7oUM6/1FK+Z0YmJQw2B16dw9ADUW2ChZg76kWVs4voyg7NYK7BCf0WCrE0aFM2cH5bq9J3I4AKeURpdSnscJLB3tG+rZS6r2hnKmPVuw+OF0E0KhzbR4W5k6wUk9rz0a2ajZGVqnlBFbm0EqgSEp5l5TyKSPYoSGl3ADIEJcJ4Bml1GTHDUoQbFtp/RsQCwFayQxxdXikp7gZl5/OnYsmcPPcUgD+sPV4RGN0kI5Hu0gSAz42bSXo9u40bq/tfAe4AuuobSAKgD8qpZZLKaN+lkl0bHumVUqNA072ajev6EUM7hENTHFOKmsfvu6i73f1enl6Y3XYm1D9WSJ2ky/aA2/PAS9jiXSdCatzHqVUHrCN0GfATwJfMh+cg2PnM+1ksFa2oQoWwOfTNLZZBcOy05NJSXLh8fr4fxuP8Ictka2yAdpJJ592gJ8AXzef5rFFStmslPoYsAWrjNBAfNF/zc9jYlicYudKexPwcqPOYau2J45BCBhfkMFnl0/hjsvHU9vcyV/9ajtH6jsiGqdcHGGyOA3wl1LKH9tinCFi/BtTvw5xWS9wtZRySwxMikvs3IgSYJ2N2IXWcKLxHN/9nz08885RSvPS+ee75vXL+IrINDANx4YVKeUzwH+GuCwZ+INSakwMTIpL7Pwj7gKrppMT/H7LMQBmleYwsySy2t2uoE3GLR5+/horw2kwxgO/9dfbMlyAnaKtAUgn8r6l4XCmNTju+ILIusFlBDsr1tpnkWEo+JMm7gFCbQBeD3zXeYviDztFWw20pIleUrE/qq8sPyjUcz2R1DfS5NL3DGx6yIwApJS1wN1YFS0G42+UUnfFwKS4wjbR+rfp3wf6iyQ8I8J4Rv3MMiv22OP1sftEeNXyATLoJll4wfoDeUAptdifBG4YRqSUbwMPh3HpU0qpWNTjihvs/uN9DyBfRNYzeGxuGk/dv4TbLxtHcU4wTFEImFGSjbprLh9ZOB6A3285TltXWEUIAcinr4xKEvAPWDHENUqpNUqpjyil7IkEMQyFnwDPhLgmC3jWXwnFgP0JA8uBjd06mdf1ZWGXlynNS+NPXwvWLO/q9dLZ4yUjxU1qv9S853ec4pHn9oSdmgewWOzpX2b1UnQDr2IFWzwvpTwZ9uCGqPF/aL4LzA1x6R+Bu03ghf2iFVhdvufu8E2njqJQtwCQ5BZcO3sMi6YWUjEul6KsVHIzkun2+Djd0sXuE808v+MUOyNwiwGy6WC5axda0yUELwA3AaE+sXcQDGt836SNOY9SajqWlxYqaP1vpZQ/jIFJIxon8mkfAH7WpLPZoiuIJjoqWipFNRPFGYCfSikfUkqlYlVYDJSdmRRiiFrgeSwBvyqlPOekvaMZpdTtWP/Og+EDbpZSvhYDk0YsTog2W2uOC0Heh75p1MSkZtrF5NLOUrEbrJYyc6SU52XD+72COQQFvJjBP2E6gVcIutHm+MhmlFLfAv4pxGUNWKVYT8TApBGJU4XdvgA82avdbNTz6Y5JDbUgLnwsE7vIspLffySl/JtQ9/gjcFZhCfgWCJmq9B5BN/oD86wVPf5yts9j9UUajK3ANVJKZ4ICRjhOiVZg/TGvOqPz2K5nEUs3eZY4xlRRC1aBucsiTRBQSqUB1xFchUN1WT5J0I1+TUoZu9Z0CYZSqgDrPH1yiEt/LqV8wHmLRh5O1j0epzV7hCD3qC5hr55ELIQ7gdPMcR1Ba3xCsExKuTma8fwfQPMJCviKELeY1L8oUUpdBryDVQVyMP6XlPLJGJg0onC6LchNWrNOCFKO6BL2OSzcCZymUhwJJBQ8KKUMVcozYpRSpQTd6JsZPNUMgkn2zwG7jBsdHkqpz2O1LxmMbmCZlHJURbrFogHXnVrzOyFIrtUF7NFT6LW5PY4LHzPEiYBLDDE6GvBXHbwBS8C3Q8giz8cJPge/MVqfycJFKfUo8JUQlx0DFkopG2Ng0oggVq0ub9GaZ4Ugs1sns1tP4QwFtoydSzvzxGGyRGfAJf6qEytsKPxu9GUE3eiFIW5px2r2HHCj6521MP7wH9G9ibWzPxgvAbdJKZ3vlj0CiGVT6enAE8DVAKd1Pkd1CU3kMBSXOZsOJorTTOBMwB3eD3xRSvmufVYPHX/5nYAbfRODP5+ZEq0DoJQaj7UxFSq/9jtSyn+MgUnDTsxEC33d2r6qNd8TwnoWbNPpnNBjaSSH9kFL1Wgy6CafNsaLM32hif7V9d+Ab47UMjJKqQws4Qbc6FBV3I8Q3I0e9cXQlVLXYZ2Rh6rLe6eU8n+ct2h4ialoA/hXofv8X31/wB7too0M2knHiwsQuPCRQRe5dASydQDQmjYheBp49MLAiZGM/4NrIUE3ekGIWwJtR57Dqr88ap7d+qOUehj4UYjLWrFajRyMgUnDxrCINoBSKhm4EyspeimhN3IasQp/rQV+JaVsD3H9iEcpNQFr9V2Ntak1WDV2H9ZRSMCN3jda3Gj/nsF/YeXhDsZurOZekeWHxhHDKtr+BFqKhLhsjpRyTyzsGQ78fXP7u9GhnuPiopWmXfjT87YAofJrf4vVTnNk/HHbjBHtCMXvRgeKfN9B6NS1FmADI6hptRMopWZjnX2Hytb6mpTyP2JgUswxoo0TlFKTOL8p9mCH3V6s4mlrgbVSygPOWxg7/DWU/xjiMi9wvZQy3G5tcYMRbRzidxNvwRLwKgiZuHyAoBu9SUoZfumPEYpS6nvA10NcdhorI6gmBibFDCPaOMefGbOY4CpcGeKWs8ALWALeIKWMrLLACMFfXnUDcGOIS9/BWnET5tjMiDbBUEpNJbgbfS2Du9EeYCNBN/qQ8xbah1KqGCvwIlQW1k+llA/FwKSYYESbwCilcrHc6DuA2yBk7Og+gm70u/HgRiulrsB6fg+VtP1ZKeWvYmCS4xjRjhL87uRSgm707BC3NGI1hF4LvCiljKzEZgxRSt0L/CLEZZ3AUinlh4OMk4F1nJSD9SHQgxXcsncknfsa0Y5S/LHgAQFfw+Ahgr1YgfsBN/qI8xZGhlJqDfClEJdVY0VMnfXfkw18AlgOLETrCoS4uISo1j6E2Iflir8N/HY4P8SMaA2B/rG3Ygn4NiAvxC17CLrRW0ZCdo2/2sjbhM6uWo+16/wAWn8OIfrOe4XPR15zM6ld3bi9XrxuN91pqTTn5aFd/bSsdQdC/BIrhHaX/b/N4BjRGs7D70Yvw3oOXg3MCHFLPUE3+iUp5aBFpp3Ef5a9HSgM956S2lqmVB+huL6egsYmkrwXf/543G6aCgpoKC7iyNQp1JaV9f/x74GvSinPRGt/uBjRGgZFKTWLoBu9jMHd6B7gdYIVK485b+H5KKVuxjoKupSbC0Lg9niYuf8A5VV7KTh7NuI5zubnsbe8nAOzZuJJTgatGxDiQSnl76P/DUJjRGsIG3/RtZVYAr6V0MXFdxJ0o7fFqvC7UurvGKDjXklNLde8+RY5bdE7BG1ZWWy89mpqxvXluTwF3Od0DLgRrWFI+DO0ria4Ck8LcctpYB2WgF92cjfWnxH0LFYGGQBuj4crt2ylYk+VrVXKNLCvfDZbliy2Vl0rhfIeJ0sJGdEaosYvknKCAl7K4M3duoHXCO5G294/SSmVj1U/Kjulu5sVG15k7GnnHjvPFBfz4soVdKelgRUX/UmnzrlNp21D1PhT4Kr8X99XShVh7UKvBlZwcUZOKpabvRJ4VCn1AcGKlXb1T/oLIDu5p4eV616guKHBhiEHZkx9PbetW8+621fRk5p6F/At4O+dmMustAZHUUqlcH7/pMkhbom6f5JS6jK03ooQSbeue4Hxp05FOsSQqS0tYd3tqwB8CLFUSrnV7jmMaA0xw+9GVxIU8BIGr+rXxfn9k0Jm6/g/JLYB8yp27+Gqd2Jf52/L4ivZNX8eWJ7HQrs7Thj32BAz/G70bv/XI/7+Sf3d6Av7J6VhJT/cDqCU2k5wN3rHAJUpvgbMy25t5Yqt28K2TWRmknrVVSQvmE/KvHkkL5iPu8AK1T59zXV4Dh8Oe6yF723n+KSJtOTlVQB/i+Uq24ZZaQ0jgiH0TzqF5UY/h79/kn9H+yhQtmL9BiacDH9/K23FCgqfWHPJn0UqWoCa0lLWr14FWtcjxAQ7d5PNSmsYEfhdyA3ABqXUV4F5BAV85SVuGQfc7/86p5R6GUvIZblnmxkfgWADeOvr6d25k54PPsRbV0f+D38wxN8GSmtrKWhspKmwsBi4C3hmyINdgBGtYcThd3s/9H/9i1KqhPP7J2VccEsG8JHAm4qqyM9iu15+mboXX+x77x4/fgiWBxFAedVeNl29HOBBbBTtYGdpBsOIQEpZJ6V8XEp5J1ZpnVXAY1gr63m4PR5mHBhC2WOf/cFa0w8eIrmnB2CZUqos1PXhYlZaQ1zh7yKxHlivlHoQq9j7auAzwIyihgZSekdGJdlkj4fi+vpAmONCwJZaVUa0hrjF70bvAHYopTzAd4rqnQ2iiJTChsaAaBdh7XpHjXGPDYnCQoAihyOfIqXfh0ioPN+wMaI1JAqTAXKbW4bZjPPJa+4rdjnZrjGNaA2JQjpAsmdkPM8GSPL05QwM1uo0IoxoDYmCdcozMmKF+hDB4CXbtGZEa0gUOgG8SaFa2MYWT1LfXq9tvZONaA2JQi1Aa3bOcNtxHm05fVmJtXaNaURrSBS2AzQUh2prFFsaivrs2W7XmEa0hkTBEm3RCBNtsf2iNcEVhkRhO0D9mGI8bvclS6GGwpWfH3ydG6xZJ3JzzvuZr7nZquwYAq/LxZkxfX3BjWgNhv5IKU8qpXb0pqRcdmTKFGYciryXWOnunZf8/pi1z533vm7xUrxhZBEdmzwpUDOqCqu7gS0Y99iQSDwKsLeifLjtAKCqoiLw8tEBEvaHhFlpDYnEb9D6R2dKxuY2FBVS1NAY0c2nxoXKuw+fpvx86spKQet2fwsR2zArrSFhkFJ2IMQagE3Ll+ETdlY4Dh+fEGy6epn1Roin7W7WZURrSDS+jdYn68eMYde8ucNiQFVlBadLSkDrOuCbdo9vRGtIKKSULQjxZYDtixbS1G/XNxY05+ay7corrDdC3CelbLJ7DiNaQ8IhpXwReNzndvPiyhW0Z15Y5NEZOjIy2LDyVrxW6OKvpJS25M9eiBGtIVH5C+Cdjqws1q1eRWv2hU0O7KU9M5N1t6+i3Qpb3IpVF8oRjGgNCYm/wdcq4L22nBzWfmQ1J4Pd7WylprSU5z5yB615uQAfACud7NNrRGtIWKSUzcCNwBudGRlsWLWSjVcvp8fqbhc1vUlJbFp2FetXr+JcViZYneivd+I5tj9GtIaExn/ccjPw92jds798Nn+8+y52V1bSnZIypDF7kpOpqijnj3ffxd7KCtC6F2uX+Ab/B4WjmOAKQ8Ljbzn5iFJqLfBkR1bWos3LlvLelYuYdugwUw9XU9TQQKpV7vSSdKek0FhUSPXUqRyaMT3QixZgB0J8QUp56RhIBzCiNYwapJS7lVJLgTuABz3JyTfuL5/N/vLZAGS3tlri7erG7fXidbvpTk2lsaiQ1tyLmt6/gRU2+d9Od36/ECNaw6jCv+o+CzyrlJoNfBlYjtYL2nJyUttyBkii17obIXZiPbeukVJWxcrmCzGiNYxapJT7gL8G8DfvqgDmYjXBTsXqWN8O7EKIPbFeUQfCiNZgAPyCDPQPGtGY3WODIc4wojUY4gwjWoMhzjCiNRjiDCNagyHOMKI1GOIMocMoBekkSql5wD3AdcCyEJdvAzYCzwFv2Vksy2CIF4ZFtEqpVOBjWDmHy/v/rEcn0UEaXqyeLC58ZNBNmrgoLrQKK4zsl3bX4DEYRjIxF61SagXwC2AigEe7OUURjTqHVjLpJJVAA7T+pNBDLh3kizbGUU+asIJTtKZFCB4GnjArr2E0EDPRKqVygR9hxXrSptM5pkuooahvVQ0XgY+xnGWSqKNA9OUabwDulVKGriJtMMQxMRGtUqoSWA9M9GnBQT2eI5ShL7GiRoamjEbKxRFShBetaRWCu6WUL9lgtsEwInFctEqphVrzkhAUNOtMdulptJNh6xwp9DBHHGGsOIvW9ArBJ6SUf7J1EoNhhOCoaJVSlVrzlhAUnNb5fKBn4HPslEkzWxxjiqgLCPcOKeUGhyYzGIYNx0SrlMoBdgKTzug83tcz0Y4fC2tmieNMFbVozTkhWCClPOjwpAZDTHFSRT8CJrXoTHbERLAAgv16IjW6ECHIAJ5QSpkAEkNC4cgftFLqFuBenxbs1NMcdIkvhaBKT6FLJ4N1BvxQDCc3GBzHdvdYKZUCHAAm7fdNoJqh15pNT3Hz268uY2xuOgDf/tMu1n1QE9a9Y2hioesAWtMpBDPNUZAhUXBiCfwoMKlNp3OEsqgGuv+GGX2CjZQzFFCn8xGCdOC+qAwxGEYQToj2QYBjuiSqc9hZpdl8/MoJ7D4x9DKyR3Vp4OW9fg/AYIh7bBWtUmoOcI1Hu6mhaMjjCAFfX10JwA+eH3rRu7Nk06bTAUqAO4c8kMEwgrB7pb0boIbCiEMTzxtk8UQqxuXyp/dOcKAumpYoghN6bODNJ6IYyGAYMdgt2kUAjfqiws5hU5ydyv3Xz6CxrZvHXj0UtUGN9NWxXRj1YAbDCMA20SqlBH5htDD0fqAP31ZOZloSP3lpPx3dnqjtaicdj3YBTFJKFUY9oMEwzNi50pYBY3u0259eFznLZxZzXcVYth9pYsPOWpvMErQGP0TMamuIe+wU7WSAc6RzqSMndr4AAAo8SURBVHzYUKQlu3l4VTm9Hh8/XGdvx4UO0gIvJ9s6sMEwDNgp2nQAzxCHvO+G6ZTmpfPbzcc4Wt9ho1n03xQb2qGvwTCCsFO0Qz6UnVGSzT2LJ1LX3Mnjbxy20aSLiDaB12AYduzs5dMJ4MYX8Y1fWzmbJLeLx149iBBW+OKlSE5ykZ7ixqc13b3hz+MK2tQZsXEGwwjDTtHWAGTQDWgiWdRK/aGK/3zXvEGv+8bqSr6xupLas5189D/eCnv8DLoCL+3a3TIYhg073eMjQHOq6CWVEdER0I8ml75n5O3DaYnBYAe2ZvkopV4Fbtjum8kZCmwbd7NaAUSW5RMggy6udX0AcBooNRUbDfGO3RFR2wHyRTShh/aST19J5O1GsIZEwG7R/g/AOOr7b/4MKxPEmcDL54bTDoPBLuwW7TvAzlThYSxNNg8dOdl0kC/a0ZpW4NfDbY/BYAdOVK64H3jsrM5is65kOI9G54jqwEr7Eynl/x42QwwGG7HzyCfAr7XmkXzRnj9ON3CKYgemCE0ebYznDFrjE4JHh8WIOMRfCG86Vpz2fCAPSAV6gXZgL9bexW4p5UUNlgzO40gJVaXU54Cne7WbjXo+3cS2aIQLH8vETrJEF8APpJRfj6kBcYZSyg3cCtwPXAvBfMaB0JoeIXgPeAL4jZTynLNWGgI4JVqBtfFz+xmdx3Y9i1i6ybPEMaaKWoB9wGVSyq4Qt4xKlFLZWOWBHqBfMkWXTqaFLFrJoEcn48OFQJOMh2xxjhw6Ah+IQF8TtKeAf5dSHovxrzHqcLJYeZnW7BGCvKN6LHv1ZGIh3InUUek6GnCLr5JSbnF80jhEKXUj8DgwCeCcTuW4HksNRWF5Rkl4GMNZJorT5It2ALSmQwj+Bvi5lHJkHB8kIE63BblBa14QgpSjuoS9ehJOCncidVSIowhrivullL9wbLI4RSmVBfwQa3WlRWdyQI+ngTyG+n+TQwfTxClKRN+JwevAl6SUR2ww2XABsWjAdYfW/F4IUup0Abv1FHpJtnUOFz5mihNMEX2hxX8tpfxXWydJAJRSY4AXgQU+LTikx1FNmW3dH0popFIcIUV40JomIbhVSrnNlsENfcSq1eXNWvOsEGR16yT26KmctinMMY825orDZImugEv8FbPCXoxSaizwJjCrQ6exQ8+gLYqyQAORQi/zxCGKRQta0y4EN0spN9s+0Sgmlk2lp2LtNF4LcFrnc1SX0EQOQ3HLsulgkqhjPPUBd3gv8AUp5Vb7rE4M/A293wTmt+oMtunZ9Di4oy/wMU8cpkw0ojXNQnC1lHK3YxOOMmLWZEdKWQ3cADykNefGirMsdu3lavEhk6glmw4Eg32AaDLoZBxnWCJ2s9y1iwmiHsAHfB+43Ah2QP4TmN+u09imyx0VLIDGxU49PdDhIQ/4nVIqLeSNhrCI2UrbH6VUKXAv1rlgX+8Qrxa0kUk7aX0lYlz4yKCbHDpIFt6+MfodM/xMSrk/pr9AHKGUuh1Y69WCTXoeHTGsuHPBefn3pJR/F7PJE5hhEW0ApVQysBqrkPgiYGqIW2qxonGeA56RUtpbTCrBUErl+4/dSvf6JnGU0tA32UwebSwRewB8QrDUeEPR40QYY9hIKXuBZ/1fKKXygcuxDvrTsdz3TvxilVKayhOR8XUhKD2rszhKSdg3rVpQxj99dO6g15zr9nDDd18NOVYz2RyhlKmi1gX8G1b7UUMUDOtKa3AOpVSa1pwUgsJ3fZU0kx32vQHR9np8tHZeugpJZ6+Xj/94Y1jjufFyvXg/8HizQEr5YdjGGC5iWFdag6PcLQSFLTqDZrKGNMCuE808+FT0x6xe3JyimMnUAXwFf2CHYWjEskW7IbZ8BeC4LmEkVI497m+EpjWfUUqFTEgwDIwRbQKilMrWmiU+LahlZLQv6iCdZp2FEGQCS4fbnnjGiDYxuUwIRBsZUbUctZt+brrpqRQF5pk2MYm6eyHAlDFZPPPnyyjLT8fr09Q1d7K1upHfbT5ObXPkdd9bdGbAUzeijQKz0iYmlwO06uhEm5+ZwuSiTLp7vaQmuZg2NptPLZ3Mb/58GbfMjfzMt1/3wsuiMmyUY1baxKQAoGuI4Yr1bd384rVDvF5Vx4mmc3i8mmS3YNHUQh66ZRZTx2TxzY/O4UxrFx8cOxv2uP3sGRkP2nGKWWkTkzQA3xD/e7cebuSJNw9zpL4Dj9c6x+/1at492MC9a7ZworGDJLeLB2+aEdG4/ewx3QujwIg2MfEAIRIwhkZHt4en3qoGYM74PPIyws+N7mfPSOobE3cY0SYmnQBJeENdNyT2nGoBwOUSlOaFv2j2s8d0L4wCI9rE5ABAlnCmQGL/UI1I1vJs+uwxWVlRYESbmLwH9O8WaCsV43L7XtdFcPSTY7oX2oIRbWKyHQKitfe5NiPVzeeutjIo95xspvlc+I+nucKI1g6MaBOTaqAxVfSSQ2QucmleGmu+vJjVl49jbG6w2ESSW7BkehG/+NJiJhVl4vVpHn3lYNjjuvBREOxgaHJqo8Ck5iUoSql/B/7yhB7Dbh2qtkCQ0rw0/vS1a/ved/V66erxkpmaRHKS9Rnf2ePh+2ur2LAz/PTmcdQzz3UYYJuU8sqwbzRchAmuSFweA/6yjAb2MRFPmP/VTe09/Ov6vcyfmMf0sdnkZ6aQlZZEZ4+Xg6fbeK+6kWe3naCuJbKmDRNFXeCl6asUJWalTWCUUi8DNx3wjecw44fNjgJaWeyqQmvOCsE4KaU58okC80yb2HwfYJo4ReYwHY268DJHWMEYQvBjI9joMaJNYKSUrwCPu4VmnjjkSIRUKGaJE2Ra1Rh3AY/E3IAExIg28XlYa07miQ6mi5MxnbiIs0yiDq3xAl80/WztwYg2wZFStgjBl7TGN12cYhKxKWiZTyuXi4MIAULwbSmlOZu1CSPaUYCU8iUhuBegwnWMqZzC7qCL/hTRzCKxD7fwgdVO81uOTTYKMbvHowil1ANa86gQiDpdwB492dYWIS58TBcnmUINLitA+Sngy1JKZzIXRilGtKMMpdQ9WrNGCLJ7dBJVerK/+Ft0FRtzaWeuOEy26ERrtBB8D/hH01zafoxoRyFKqYnAGuBmgFad0dcFPrJCcLqvG3yxaAl88wDWptM79lptCGBEO0pRSgngy1rzHSEoBvBoN6fJp0Vn0UImrWTg6ydigSaTTnJpJ1d0MIazpAtrQ1hruoTgp8A3zVmssxjRjnKUUqnAXcCDwLL+P9MaPLjx4ULgIwkfLnHR38sh4GfAU1LKppgYPcoxojX0oZSag9X0eyGwUGsqhbjIX67GSq3bDrwLvG2eW2OLEa1hQJRSKUAGVqG4HqDTuL7DjxGtwRBnmOAKgyHOMKI1GOIMI1qDIc4wojUY4gwjWoMhzjCiNRjiDCNagyHOMKI1GOIMI1qDIc4wojUY4gwjWoMhzjCiNRjiDCNagyHOMKI1GOIMI1qDIc4wojUY4oz/D1EUfUy17KeUAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"cs.draw_graph(\n",
" hn, \n",
" node_pos=nx.circular_layout(hn), \n",
" node_size_factor=.25, \n",
" node_color=node_color, \n",
" node_border_width=2, \n",
" edge_width_factor=50, \n",
" labels='text', \n",
" font_size_factor=2, \n",
" font_color='white', \n",
" figsize='small', \n",
" margins=.2\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To draw the fact graph, the `domain_color` dictionary is the color source for the edges. We now call the ``meaning_structures()`` function to create a multiplex fact matrix. By setting `multiplex=True`, we specify that selections occurr in multiple domains and that one matrix is created per domain. With `domain_id='domain_id'` we set which column of the `transactions` dataframe contains the information which domain a transaction belongs to. Optionally, we specify the name of the `facts` entity table to enrich it. Note that, since we do not set the `norm` and `remove_loops` parameters, default values will be used, i.e., co-selections will be normalized and self-loops will be removed:"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [],
"source": [
"co_selections, facts_enriched, co_selections_domain, facts_enriched_domain = cs.meaning_structures(\n",
" selections=selections, \n",
" transaction_id='transaction_id', \n",
" fact_id='fact_id', \n",
" multiplex=True, \n",
" transactions=transactions, \n",
" domain_id='domain_id', \n",
" facts=facts, \n",
" symmetrize=False\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first two outputs are the edge and node lists when the domain information is not used, i.e., the first output is the singleplex edge list ..."
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``compsoc`` – *Notebooks for Computational Sociology* (alpha)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Meaning structures: Bipartite matrix projection\n",
"Authors: [Haiko Lietz](https://www.gesis.org/person/haiko.lietz)\n",
"\n",
"Version: 0.91 (14.09.2020)\n",
"\n",
"Please cite as: Lietz, Haiko (2020). Meaning structures: Bipartite matrix projection. Version 0.91 (14.09.2020). *compsoc – Notebooks for Computational Sociology*. GESIS. url:[github.com/gesiscss/compsoc](https://github.com/gesiscss/compsoc)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
|\n",
"|:--|\n",
"|**Figure 1**: Mapping of socio-cultural systems model to mathematical framework (fig. 2 in Lietz, 2020)|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**In this notebook**, we demonstrate how we can use the unified data model to study the meaning encoded in networks. We will lay out a core part of the ``compsoc`` mathematical framework (figure 1). We will see how digital behavioral data can be naturally represented by bipartite matrices, how meaning structures can be obtained by projecting these matrices, how matrix normalization can be used to account for potential behavioral differences in subfields, and how matrix algebra and community detection go hand in hand to uncover multiple layers or domains of social life from data. Two basic functions will be developed. They are largely based on the mathematical framework developed by Batagelj and Cerinšek (2013). Lietz (2020, particularly the [technical appendix](https://link.springer.com/article/10.1007/s11192-020-03527-0#appendices)) can be read to gain traction on the formalism used in this notebook. To be able to process large data collections, this notebook uses sparse matrix methods."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dependencies and settings"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import compsoc as cs\n",
"import networkx as nx\n",
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"#import warnings"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"#warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Understanding the basic formalism\n",
"A simple toy model shall help us to keep things simple and understandable. It contains only the most central aspects of transactions selecting facts and transactions belonging to domains. The data model consists of yellow tables for entities and green tables of relationships among two types of entities:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"|
|\n",
"|:--|\n",
"|**Figure 2**: Subset of unified model for digital behavioral data|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data, depicted in figure 1, resembles a field that consists of $m=5$ distinct transactions. Our computational framework requires that identifiers are integers $i=0,...,m-1$. Like all entity and relationship tables, transactions are stored in a pandas dataframe. The `transaction_id` column contains the integers $i$ and the `transaction` column contains transaction names:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"