{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "TLDR; I show how you can visualize the knowledge distribution of your source code by mining version control systems.\n", "\n", "# Introduction\n", "In software development, it's all about knowledge – both technical and the business domain. But we software developers transfer only a small part of this knowledge into code. But code alone isn't enough to get a glimpse of the greater picture and the interrelations of all the different concepts. There will be always developers that know more about some concept as laid down in source code. It's important to make sure that this knowledge is distributed over more than one head. More developers mean more different perspectives on the problem domain leading to a more robust and understandable code bases.\n", "\n", "How can we get insights about knowledge in code?\n", "\n", "It's possible to estimate the knowledge distribution by analyzing the version control system. We can use active changes in the code as proxy for \"someone knew what he did\" because otherwise, he wouldn't be able to contribute code at all. To find spots where the knowledge about the code could be improved, we can identify areas in the code that are possibly known by only one developer. This gives you a hint where you should start some pair programming or invest in redocumentation.\n", "\n", "In this blog post, we approximate the knowledge distribution by counting the number of additions per file that each developer contributed to a software system. I'll show you step by step how you can do this by using Python and [Pandas](http://pandas.pydata.org/).\n", "\n", "\n", "Attribution: The work is heavily inspired by [Adam Tornhill](http://adamtornhill.com/)'s book [\"Your Code as a Crime Scene\"](https://pragprog.com/book/atcrime/your-code-as-a-crime-scene), who did a similar analysis called \"knowledge map\". I use the similar visualization style of a \"bubble chart\" based on his work as well." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Import history\n", "For this analysis, you need a log from your Git repository. In this example, we analyze a [fork of the Spring PetClinic project](https://github.com/buschmais/spring-petclinic). \n", "\n", "To avoid some noise, we add the parameters --no-merges and --no-renames, too.\n", "\n", "```bash\n", "git log --no-merges --no-renames --numstat --pretty=format:\"%x09%x09%x09%aN\"\n", "```\n", "\n", "We read the log output into a Pandas' DataFrame by using the method described in this [blog post](https://www.feststelltaste.de/reading-a-git-repos-commit-history-with-pandas-efficiently/), but slightly modified (because we need less data):" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
additionsdeletionspathauthor
111docs/README.mdMarkus Harrer
3760docs/README.mdMarkus Harrer
42900docs/assets/css/style.scssMarkus Harrer
5--docs/documentation/images/class-diagram.pngMarkus Harrer
612240docs/documentation/index.htmlMarkus Harrer
\n", "
" ], "text/plain": [ " additions deletions path \\\n", "1 1 1 docs/README.md \n", "3 76 0 docs/README.md \n", "4 290 0 docs/assets/css/style.scss \n", "5 - - docs/documentation/images/class-diagram.png \n", "6 1224 0 docs/documentation/index.html \n", "\n", " author \n", "1 Markus Harrer \n", "3 Markus Harrer \n", "4 Markus Harrer \n", "5 Markus Harrer \n", "6 Markus Harrer " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import git\n", "from io import StringIO\n", "import pandas as pd\n", "\n", "# connect to repo\n", "git_bin = git.Repo(\"../../buschmais-spring-petclinic/\").git\n", "\n", "# execute log command\n", "git_log = git_bin.execute('git log --no-merges --no-renames --numstat --pretty=format:\"%x09%x09%x09%aN\"')\n", "\n", "# read in the log\n", "git_log = pd.read_csv(StringIO(git_log), sep=\"\\x09\", header=None, names=['additions', 'deletions', 'path','author'])\n", "\n", "# convert to DataFrame\n", "commit_data = git_log[['additions', 'deletions', 'path']].join(git_log[['author']].fillna(method='ffill')).dropna()\n", "commit_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Getting data that matters\n", "\n", "In this example, we are only interested in Java source code files that still exist in the software project.\n", "\n", "We can retrieve the existing Java source code files by using Git's ls-files combined with a filter for the Java source code file extension. The command will return a plain text string that we split by the line endings to get a list of files. Because we want to combine this information with the other above, we put it into a DataFrame with the column name path." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
path
0src/main/java/org/springframework/samples/petc...
1src/main/java/org/springframework/samples/petc...
2src/main/java/org/springframework/samples/petc...
3src/main/java/org/springframework/samples/petc...
4src/main/java/org/springframework/samples/petc...
\n", "
" ], "text/plain": [ " path\n", "0 src/main/java/org/springframework/samples/petc...\n", "1 src/main/java/org/springframework/samples/petc...\n", "2 src/main/java/org/springframework/samples/petc...\n", "3 src/main/java/org/springframework/samples/petc...\n", "4 src/main/java/org/springframework/samples/petc..." ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "existing_files = pd.DataFrame(git_bin.execute('git ls-files -- *.java').split(\"\\n\"), columns=['path'])\n", "existing_files.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next step is to combine the commit_data with the existing_files information by using Pandas' merge function. By default, merge will \n", "- combine the data by the columns with the same name in each DataFrame \n", "- only leave those entries that have the same value (using an \"inner join\"). \n", "\n", "In plain English, merge will only leave the still existing Java source code files in the DataFrame. This is exactly what we need." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
additionsdeletionspathauthor
045src/test/java/org/springframework/samples/petc...Antoine Rey
1530src/test/java/org/springframework/samples/petc...Colin But
2257src/test/java/org/springframework/samples/petc...Antoine Rey
31670src/test/java/org/springframework/samples/petc...Colin But
4219src/test/java/org/springframework/samples/petc...Antoine Rey
\n", "
" ], "text/plain": [ " additions deletions path \\\n", "0 4 5 src/test/java/org/springframework/samples/petc... \n", "1 53 0 src/test/java/org/springframework/samples/petc... \n", "2 25 7 src/test/java/org/springframework/samples/petc... \n", "3 167 0 src/test/java/org/springframework/samples/petc... \n", "4 21 9 src/test/java/org/springframework/samples/petc... \n", "\n", " author \n", "0 Antoine Rey \n", "1 Colin But \n", "2 Antoine Rey \n", "3 Colin But \n", "4 Antoine Rey " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "contributions = pd.merge(commit_data, existing_files)\n", "contributions.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now convert some columns to their correct data types. The columns additions and deletions columns are representing the added or deleted lines of code as numbers. We have to convert those accordingly." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
additionsdeletionspathauthor
045src/test/java/org/springframework/samples/petc...Antoine Rey
1530src/test/java/org/springframework/samples/petc...Colin But
2257src/test/java/org/springframework/samples/petc...Antoine Rey
31670src/test/java/org/springframework/samples/petc...Colin But
4219src/test/java/org/springframework/samples/petc...Antoine Rey
\n", "
" ], "text/plain": [ " additions deletions path \\\n", "0 4 5 src/test/java/org/springframework/samples/petc... \n", "1 53 0 src/test/java/org/springframework/samples/petc... \n", "2 25 7 src/test/java/org/springframework/samples/petc... \n", "3 167 0 src/test/java/org/springframework/samples/petc... \n", "4 21 9 src/test/java/org/springframework/samples/petc... \n", "\n", " author \n", "0 Antoine Rey \n", "1 Colin But \n", "2 Antoine Rey \n", "3 Colin But \n", "4 Antoine Rey " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "contributions['additions'] = pd.to_numeric(contributions['additions'])\n", "contributions['deletions'] = pd.to_numeric(contributions['deletions'])\n", "contributions.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Calculating the knowledge about code\n", "We want to estimate the knowledge about code as the proportion of additions to the whole source code file. This means we need to calculate the relative amount of added lines for each developer. To be able to do this, we have to know the sum of all additions for a file.\n", "\n", "Additionally, we calculate it for deletions as well to easily get the number of lines of code later on.\n", "\n", "We use an additional DataFrame to do these calculations." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pathadditionsdeletions
0src/main/java/org/springframework/samples/petc...1110
1src/main/java/org/springframework/samples/petc...7023
2src/main/java/org/springframework/samples/petc...6719
3src/main/java/org/springframework/samples/petc...290137
4src/main/java/org/springframework/samples/petc...7923
\n", "
" ], "text/plain": [ " path additions deletions\n", "0 src/main/java/org/springframework/samples/petc... 111 0\n", "1 src/main/java/org/springframework/samples/petc... 70 23\n", "2 src/main/java/org/springframework/samples/petc... 67 19\n", "3 src/main/java/org/springframework/samples/petc... 290 137\n", "4 src/main/java/org/springframework/samples/petc... 79 23" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "contributions_sum = contributions.groupby('path').sum()[['additions', 'deletions']].reset_index()\n", "contributions_sum.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also want to have an indicator about the quantity of the knowledge. This can be achieved if we calculate the lines of code for each file, which is a simple subtraction of the deletions from the additions (be warned: this does only work for simple use cases where there are no heavy renames of files as in our case)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pathadditionsdeletionslines
0src/main/java/org/springframework/samples/petc...1110111
1src/main/java/org/springframework/samples/petc...702347
2src/main/java/org/springframework/samples/petc...671948
3src/main/java/org/springframework/samples/petc...290137153
4src/main/java/org/springframework/samples/petc...792356
\n", "
" ], "text/plain": [ " path additions deletions \\\n", "0 src/main/java/org/springframework/samples/petc... 111 0 \n", "1 src/main/java/org/springframework/samples/petc... 70 23 \n", "2 src/main/java/org/springframework/samples/petc... 67 19 \n", "3 src/main/java/org/springframework/samples/petc... 290 137 \n", "4 src/main/java/org/springframework/samples/petc... 79 23 \n", "\n", " lines \n", "0 111 \n", "1 47 \n", "2 48 \n", "3 153 \n", "4 56 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "contributions_sum['lines'] = contributions_sum['additions'] - contributions_sum['deletions']\n", "contributions_sum.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We combine both DataFrames with a merge analog as above." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
additionsdeletionspathauthoradditions_sumdeletions_sumlines
045src/test/java/org/springframework/samples/petc...Antoine Rey57552
1530src/test/java/org/springframework/samples/petc...Colin But57552
2257src/test/java/org/springframework/samples/petc...Antoine Rey1927185
31670src/test/java/org/springframework/samples/petc...Colin But1927185
4219src/test/java/org/springframework/samples/petc...Antoine Rey1349125
\n", "
" ], "text/plain": [ " additions deletions path \\\n", "0 4 5 src/test/java/org/springframework/samples/petc... \n", "1 53 0 src/test/java/org/springframework/samples/petc... \n", "2 25 7 src/test/java/org/springframework/samples/petc... \n", "3 167 0 src/test/java/org/springframework/samples/petc... \n", "4 21 9 src/test/java/org/springframework/samples/petc... \n", "\n", " author additions_sum deletions_sum lines \n", "0 Antoine Rey 57 5 52 \n", "1 Colin But 57 5 52 \n", "2 Antoine Rey 192 7 185 \n", "3 Colin But 192 7 185 \n", "4 Antoine Rey 134 9 125 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "contributions_all = pd.merge(\n", " contributions, \n", " contributions_sum, \n", " left_on='path', \n", " right_on='path', \n", " suffixes=['', '_sum'])\n", "contributions_all.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Identify knowledge hotspots" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OK, here comes the key: We group all additions by the file paths and the authors. This gives us all the additions to a file per author. Additionally, we want to keep the sum of all additions as well as the information about the lines of code. Because those are contained in the DataFrame multiple times, we just get the first entry for each." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
additionsadditions_sumlines
pathauthor
src/main/java/org/springframework/samples/petclinic/PetclinicInitializer.javaAntoine Rey111111111
src/main/java/org/springframework/samples/petclinic/model/BaseEntity.javaAntoine Rey37047
Faisal Hameed17047
Gordon Dickens147047
Michael Isvy517047
boly3817047
src/main/java/org/springframework/samples/petclinic/model/NamedEntity.javaAntoine Rey36748
Gordon Dickens156748
Michael Isvy496748
src/main/java/org/springframework/samples/petclinic/model/Owner.javaAntoine Rey14290153
\n", "
" ], "text/plain": [ " additions \\\n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 3 \n", " Faisal Hameed 1 \n", " Gordon Dickens 14 \n", " Michael Isvy 51 \n", " boly38 1 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 3 \n", " Gordon Dickens 15 \n", " Michael Isvy 49 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 14 \n", "\n", " additions_sum \\\n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 70 \n", " Faisal Hameed 70 \n", " Gordon Dickens 70 \n", " Michael Isvy 70 \n", " boly38 70 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 67 \n", " Gordon Dickens 67 \n", " Michael Isvy 67 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 290 \n", "\n", " lines \n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 47 \n", " Faisal Hameed 47 \n", " Gordon Dickens 47 \n", " Michael Isvy 47 \n", " boly38 47 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 48 \n", " Gordon Dickens 48 \n", " Michael Isvy 48 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 153 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grouped_contributions = contributions_all.groupby(\n", " ['path', 'author']).agg(\n", " {'additions' : 'sum',\n", " 'additions_sum' : 'first',\n", " 'lines' : 'first'})\n", "grouped_contributions.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we are ready to calculate the knowledge \"ownership\". The ownership is the relative amount of additions to all additions of one file per author." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
additionsadditions_sumlinesownership
pathauthor
src/main/java/org/springframework/samples/petclinic/PetclinicInitializer.javaAntoine Rey1111111111.000000
src/main/java/org/springframework/samples/petclinic/model/BaseEntity.javaAntoine Rey370470.042857
Faisal Hameed170470.014286
Gordon Dickens1470470.200000
Michael Isvy5170470.728571
\n", "
" ], "text/plain": [ " additions \\\n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 3 \n", " Faisal Hameed 1 \n", " Gordon Dickens 14 \n", " Michael Isvy 51 \n", "\n", " additions_sum \\\n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 70 \n", " Faisal Hameed 70 \n", " Gordon Dickens 70 \n", " Michael Isvy 70 \n", "\n", " lines \\\n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 47 \n", " Faisal Hameed 47 \n", " Gordon Dickens 47 \n", " Michael Isvy 47 \n", "\n", " ownership \n", "path author \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 1.000000 \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 0.042857 \n", " Faisal Hameed 0.014286 \n", " Gordon Dickens 0.200000 \n", " Michael Isvy 0.728571 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grouped_contributions['ownership'] = grouped_contributions['additions'] / grouped_contributions['additions_sum']\n", "grouped_contributions.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Having this data, we can now extract the author with the highest ownership value for each file. This gives us a list with the knowledge \"holder\" for each file." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
authoradditionsadditions_sumlinesownership
path
src/main/java/org/springframework/samples/petclinic/PetclinicInitializer.javaAntoine Rey1111111111.000000
src/main/java/org/springframework/samples/petclinic/model/BaseEntity.javaboly385170470.728571
src/main/java/org/springframework/samples/petclinic/model/NamedEntity.javaMichael Isvy4967480.731343
src/main/java/org/springframework/samples/petclinic/model/Owner.javaMichael Isvy1642901530.565517
src/main/java/org/springframework/samples/petclinic/model/Person.javaMichael Isvy5979560.746835
\n", "
" ], "text/plain": [ " author additions \\\n", "path \n", "src/main/java/org/springframework/samples/petcl... Antoine Rey 111 \n", "src/main/java/org/springframework/samples/petcl... boly38 51 \n", "src/main/java/org/springframework/samples/petcl... Michael Isvy 49 \n", "src/main/java/org/springframework/samples/petcl... Michael Isvy 164 \n", "src/main/java/org/springframework/samples/petcl... Michael Isvy 59 \n", "\n", " additions_sum lines \\\n", "path \n", "src/main/java/org/springframework/samples/petcl... 111 111 \n", "src/main/java/org/springframework/samples/petcl... 70 47 \n", "src/main/java/org/springframework/samples/petcl... 67 48 \n", "src/main/java/org/springframework/samples/petcl... 290 153 \n", "src/main/java/org/springframework/samples/petcl... 79 56 \n", "\n", " ownership \n", "path \n", "src/main/java/org/springframework/samples/petcl... 1.000000 \n", "src/main/java/org/springframework/samples/petcl... 0.728571 \n", "src/main/java/org/springframework/samples/petcl... 0.731343 \n", "src/main/java/org/springframework/samples/petcl... 0.565517 \n", "src/main/java/org/springframework/samples/petcl... 0.746835 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ownerships = grouped_contributions.reset_index().groupby(['path']).max()\n", "ownerships.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Preparing the visualization\n", "Reading tables is not as much fun as a good visualization. I find Adam Tornhill's suggestion of an enclosure or bubble chart very good:\n", "\n", "\n", "\n", "*Source: Thorsten Brunzendorf ([@thbrunzendorf](https://twitter.com/thbrunzendorf/status/857892329285451776))* \n", " \n", "\n", "The visualization is written in [D3](https://d3js.org/) and just need data in a specific format called \"[flare](http://matthewfieger.com/posts/me/2014/06/27/nesting_data_for_d3.html)\". So let's prepare some data for this!\n", "\n", "First, we calculate the responsible author. We say that an author that contributed more than 70% of the source code is the responsible person that we have to ask if we want to know something about the code. For all the other code parts, we assume that the knowledge is distributed among different heads." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pathauthoradditionsadditions_sumlinesownershipresponsible
0src/main/java/org/springframework/samples/petc...Antoine Rey1111111111.000000Antoine Rey
1src/main/java/org/springframework/samples/petc...boly385170470.728571boly38
2src/main/java/org/springframework/samples/petc...Michael Isvy4967480.731343Michael Isvy
3src/main/java/org/springframework/samples/petc...Michael Isvy1642901530.565517None
4src/main/java/org/springframework/samples/petc...Michael Isvy5979560.746835Michael Isvy
\n", "
" ], "text/plain": [ " path author additions \\\n", "0 src/main/java/org/springframework/samples/petc... Antoine Rey 111 \n", "1 src/main/java/org/springframework/samples/petc... boly38 51 \n", "2 src/main/java/org/springframework/samples/petc... Michael Isvy 49 \n", "3 src/main/java/org/springframework/samples/petc... Michael Isvy 164 \n", "4 src/main/java/org/springframework/samples/petc... Michael Isvy 59 \n", "\n", " additions_sum lines ownership responsible \n", "0 111 111 1.000000 Antoine Rey \n", "1 70 47 0.728571 boly38 \n", "2 67 48 0.731343 Michael Isvy \n", "3 290 153 0.565517 None \n", "4 79 56 0.746835 Michael Isvy " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "plot_data = ownerships.reset_index()\n", "plot_data['responsible'] = plot_data['author']\n", "plot_data.loc[plot_data['ownership'] <= 0.7, 'responsible'] = \"None\"\n", "plot_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we need some colors per author to be able to differ them in our visualization. We use the two classic data analysis libraries for this. We just draw some colors from a color map here for each author." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
authorcolor
0Antoine Rey#006837
1boly38#39a758
2Michael Isvy#9dd569
9Tomas Repel#e3f399
42Tejas Metha#fee999
\n", "
" ], "text/plain": [ " author color\n", "0 Antoine Rey #006837\n", "1 boly38 #39a758\n", "2 Michael Isvy #9dd569\n", "9 Tomas Repel #e3f399\n", "42 Tejas Metha #fee999" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "from matplotlib import cm\n", "from matplotlib.colors import rgb2hex\n", "\n", "authors = plot_data[['author']].drop_duplicates()\n", "rgb_colors = [rgb2hex(x) for x in cm.RdYlGn_r(np.linspace(0,1,len(authors)))]\n", "authors['color'] = rgb_colors\n", "authors.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we combine the colors to the plot data and whiten the minor ownership with all the None responsibilities." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pathauthoradditionsadditions_sumlinesownershipresponsibleauthor_colorcolor
0src/main/java/org/springframework/samples/petc...Antoine Rey1111111111.000000Antoine ReyAntoine Rey#006837
1src/main/java/org/springframework/samples/petc...boly385170470.728571boly38boly38#39a758
2src/main/java/org/springframework/samples/petc...Michael Isvy4967480.731343Michael IsvyMichael Isvy#9dd569
3src/main/java/org/springframework/samples/petc...Michael Isvy1642901530.565517NoneNaNwhite
4src/main/java/org/springframework/samples/petc...Michael Isvy5979560.746835Michael IsvyMichael Isvy#9dd569
\n", "
" ], "text/plain": [ " path author additions \\\n", "0 src/main/java/org/springframework/samples/petc... Antoine Rey 111 \n", "1 src/main/java/org/springframework/samples/petc... boly38 51 \n", "2 src/main/java/org/springframework/samples/petc... Michael Isvy 49 \n", "3 src/main/java/org/springframework/samples/petc... Michael Isvy 164 \n", "4 src/main/java/org/springframework/samples/petc... Michael Isvy 59 \n", "\n", " additions_sum lines ownership responsible author_color color \n", "0 111 111 1.000000 Antoine Rey Antoine Rey #006837 \n", "1 70 47 0.728571 boly38 boly38 #39a758 \n", "2 67 48 0.731343 Michael Isvy Michael Isvy #9dd569 \n", "3 290 153 0.565517 None NaN white \n", "4 79 56 0.746835 Michael Isvy Michael Isvy #9dd569 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "colored_plot_data = pd.merge(\n", " plot_data, authors, \n", " left_on='responsible', \n", " right_on='author', \n", " how='left', \n", " suffixes=['', '_color'])\n", "colored_plot_data.loc[colored_plot_data['responsible'] == 'None', 'color'] = \"white\"\n", "colored_plot_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Visualizing\n", "\n", "The [bubble chart](https://github.com/feststelltaste/software-analytics/blob/master/notebooks/vis/knowledge_islands.html) needs D3's flare format for displaying. We just dump the DataFrame data into this hierarchical format. As for hierarchy, we use the Java source files that are structured via directories." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "import os\n", "import json\n", "\n", " \n", "json_data = {}\n", "json_data['name'] = 'flare'\n", "json_data['children'] = []\n", "\n", "for row in colored_plot_data.iterrows():\n", " series = row[1]\n", " path, filename = os.path.split(series['path'])\n", "\n", " last_children = None\n", " children = json_data['children']\n", "\n", " for path_part in path.split(\"/\"):\n", " entry = None\n", "\n", " for child in children:\n", " if \"name\" in child and child[\"name\"] == path_part:\n", " entry = child\n", " if not entry:\n", " entry = {}\n", " children.append(entry)\n", "\n", " entry['name'] = path_part\n", " if not 'children' in entry: \n", " entry['children'] = []\n", "\n", " children = entry['children']\n", " last_children = children\n", "\n", " last_children.append({\n", " 'name' : filename + \" [\" + series['responsible'] + \", \" + \"{:6.2f}\".format(series['ownership']) + \"]\",\n", " 'size' : series['lines'],\n", " 'color' : series['color']})\n", "\n", "with open ( \"vis/flare.json\", mode='w', encoding='utf-8') as json_file:\n", " json_file.write(json.dumps(json_data, indent=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Results\n", "You can see the complete, interactive visualization [here](https://www.feststelltaste.de/wp-content/uploads/demos/knowledge_islands/spring-petclinic.html). Just tap at one of the bubbles and you will see how it works.\n", "\n", "The source code files are ordered hierarchically into bubbles. The size of the bubbles represents the lines of code and the different colors stand for each developer.\n", "\n", "![](./resources/knowledge_island_1.png)\n", "\n", "On the left side, you can see that there are some red bubbles. Drilling down, we see that one developer did add almost all the code for the tests:\n", "\n", "![](./resources/knowledge_island_2.png)\n", "\n", "On the right side, you see that some knowledge is evenly distributed (white bubbles), but there are also some knowledge islands. Especially the PetClinicInitializer.java class got my attention because it's big and only one developer knows what's going on here:\n", "\n", "![](./resources/knowledge_island_3.png)\n", "\n", "\n", "I also did the analysis for the huge repository of [IntelliJ IDEA Community Edition](https://github.com/JetBrains/intellij-community). It contains over 170000 commits for 55391 Java source code files. The visualization works even here (it's just a little bit slow and confusing), but the flare.json file is almost 30 MB and therefore it's not practical for viewing it online. But here is the overview picture:\n", "\n", "![](./resources/knowledge_island_4.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Summary\n", "We can quickly create an impression about the knowledge distribution of a software system. With the bubble chart visualization, you can get an overview as well as detailed information about the contributors of your source code.\n", "\n", "But I want to point out two points against this method:\n", "\n", "- Renamed or split source files will also get new file names. This will \"reset\" the history for older, renamed files. Thus developers that added code before a rename or split aren't included in the result. But we could argue that they can't remember \"the old code\" anyhow ;-)\n", "- We use additions as proxy for knowledge. Developers could also gain knowledge by doing code reviews or working together while coding. We cannot capture those constellations with such a simply analysis.\n", "\n", "But as you have seen, the analysis can guide you nevertheless and gives you great insights very quickly." ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 1 }