{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Whotracks.me April Update\n",
"\n",
"This month we have a big update to the site. We have restructured the data we publish to make it easier to use, increased the number of entries we publish, and we have laid the groundwork for internationalised versions of WhoTracks.Me - that means you can see how tracking differs between different countries.\n",
"\n",
"Thanks to integration with Ghostery 8 we collected significantly more tracker data this month, covering 360 million page loads. This is spread over countries across the world, with Germany and the USA the most represented."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/vnd.plotly.v1+html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from plotly.offline import init_notebook_mode, iplot, offline\n",
"import plotly.graph_objs as go\n",
"from whotracksme.website.plotting.colors import cliqz_colors, palette\n",
"from whotracksme.website.plotting.utils import (\n",
" CliqzFonts,\n",
" div_output,\n",
" set_margins,\n",
" annotation,\n",
" set_line_style,\n",
" set_category_colors\n",
")\n",
"\n",
"import pandas as pd\n",
"init_notebook_mode()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.plotly.v1+json": {
"data": [
{
"hole": 0.45,
"hoverinfo": "label+percent",
"labels": [
"Germany",
"USA",
"France",
"Other",
"Russia",
"UK",
"Poland",
"Netherlands",
"Canada",
"Ukraine",
"Austria",
"Italy",
"Spain",
"Switzerland",
"Belgium"
],
"name": "Data origin",
"pull": 0.07,
"textfont": {
"color": "#1A1A25",
"family": "sans-serif",
"size": 15
},
"textinfo": "label",
"textposition": "outside",
"type": "pie",
"values": [
87124064,
78216572,
40282874,
32326828,
24384449,
16317893,
10554555,
10291928,
10054367,
6268086,
6261035,
6094486,
5753209,
4732324,
4048089
]
}
],
"layout": {
"annotations": [
{
"align": "center",
"ax": 0,
"ay": 0,
"bgcolor": "#1A1A25",
"bordercolor": "#1A1A25",
"borderpad": 5,
"borderwidth": 1,
"font": {
"color": "white",
"family": "sans-serif",
"size": 15
},
"showarrow": true,
"text": "DATA ORIGIN",
"width": 100,
"x": 0.5,
"xref": "x",
"y": 0.5,
"yref": "y"
}
],
"margin": {
"b": 30,
"l": 60,
"pad": 5,
"r": 60,
"t": 30
},
"paper_bgcolor": "#00000000",
"plot_bgcolor": "#FFFFFF",
"showlegend": false,
"xaxis": {
"showgrid": false,
"showline": false,
"showticklabels": false,
"zeroline": false
},
"yaxis": {
"showgrid": false,
"showline": false,
"showticklabels": false,
"zeroline": false
}
}
},
"text/html": [
""
],
"text/vnd.plotly.v1+html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def doughnut_chart(values, labels, name):\n",
" trace = go.Pie(\n",
" values=values,\n",
" labels=labels,\n",
" name=str(name),\n",
" hoverinfo=\"label+percent\",\n",
" textposition=\"outside\",\n",
" hole=0.45,\n",
" pull=0.07,\n",
" textinfo=\"label\",\n",
" textfont=dict(\n",
" family=CliqzFonts.regular,\n",
" color=cliqz_colors[\"black\"],\n",
" size=15\n",
" ) \n",
" )\n",
" data = [trace]\n",
" layout = dict(\n",
" showlegend=False,\n",
" paper_bgcolor=cliqz_colors[\"transparent\"],\n",
" plot_bgcolor=cliqz_colors[\"white\"],\n",
" xaxis=dict(showgrid=False, showline=False, showticklabels=False, zeroline=False),\n",
" yaxis=dict(showgrid=False, showline=False, showticklabels=False, zeroline=False),\n",
" # autosize=True,\n",
" margin=set_margins(t=30, b=30),\n",
" annotations=[\n",
" annotation(\n",
" text=str(name).upper(),\n",
" x=0.5,\n",
" y=0.5,\n",
" background_color=cliqz_colors[\"black\"],\n",
" shift_x=0,\n",
" text_size=15\n",
" )\n",
" ]\n",
" )\n",
" fig = dict(data=data, layout=layout)\n",
" # NB: saving plot requires a manual step, plotly is does not support it yet\n",
" # source: https://github.com/plotly/plotly.py/issues/880\n",
" offline.plot(fig, image='svg')\n",
"\n",
" return iplot(fig)\n",
"\n",
"countries = ['Germany', 'USA', 'France', 'Other', 'Russia', 'UK', 'Poland', 'Netherlands', 'Canada', 'Ukraine', 'Austria', 'Italy', 'Spain', 'Switzerland', 'Belgium']\n",
"page_loads = [87124064, 78216572, 40282874, 32326828, 24384449, 16317893, 10554555, 10291928, 10054367, 6268086, 6261035, 6094486, 5753209, 4732324, 4048089]\n",
"\n",
"doughnut_chart(values=page_loads, labels=countries, name='Data origin')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This volume of data will also enable us to publish separate rankings for individual countries, something we plan to add later this month."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data restructure\n",
"\n",
"We have updated the struture of data which we publish in our [respository](https://github.com/cliqz-oss/whotracks.me/) to make it both easier to use and more scalable as we add more data. We now publish CSV files each month for each of the following:\n",
"\n",
" * `domains.csv`: Top third-party domains seen tracking.\n",
" * `trackers.csv`: Top trackers - this combines domains known be operated by the same tracker.\n",
" * `companies.csv`: Top companies - aggregates the stats for trackers owned by the same company.\n",
" * `sites.csv`: Stats for number of trackers seen on popular websites.\n",
" * `site_trackers.csv`: Stats for each tracker on each site.\n",
"\n",
"These files can then be loaded with popular data-analysis tools such as [Pandas](https://pandas.pydata.org/). We have also rewritten the code to render the site to take advantage of Pandas. We expose the dataframes via the `DataSource` class which loads data from all CSV files:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from whotracksme.data.loader import DataSource\n",
"data = DataSource()\n",
"len(data.trackers.df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have also updated the criteria by which we include trackers and sites on the main site. We now 'rollover' entries, so once they have been included once, we will keep publishing data (until they completely dissappear from the data). This has the effect of naturally growing the number of trackers and sites we publish. We currently have data on 868 trackers and 748 websites published:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_ts():\n",
" df = pd.DataFrame({\n",
" 'trackers': data.trackers.df.groupby('month').count()['tracker'], \n",
" 'sites': data.sites.df.groupby('month').count()['site']\n",
" })\n",
" sites_trace = go.Scatter(\n",
" x=df.index, \n",
" y=df.sites, \n",
" name='Sites',\n",
" line=dict(width=4, color='#9ebcda'),\n",
" )\n",
" trackers_trace = go.Scatter(\n",
" x=df.index, \n",
" y=df.trackers, \n",
" name='Trackers',\n",
" line=dict(width=4, color='#A069AB'),\n",
" )\n",
" \n",
" layout=dict(\n",
" margin=set_margins(t=0,b=30),\n",
" legend=dict(\n",
" x=0.05, y=1,\n",
" bgcolor='#E2E2E2',\n",
" orientation='h'\n",
" )\n",
" )\n",
" fig = dict(data=[sites_trace, trackers_trace], layout=layout)\n",
" offline.plot(fig, image='svg')\n",
"\n",
" iplot(fig)\n",
"\n",
"plot_ts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The per site trend for average number of trackers continues a slightly downward trend, but the average is still above 9. There are several possible reasons for this, it is not necessarily that sites are using fewer trackers. The proportion of data from Ghostery users continues to increase, and these users will disproportionately block many trackers. This has an effect on the average number of trackers, because it prevents the blocked trackers from loading others. The data shows also that the average indcidence of blocking for trackers increased to 25% in March, up from 20% in February. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"traces = [\n",
" go.Box(\n",
" y=data.sites.df[data.sites.df.month == '2018-01'].trackers, \n",
" name='Jan 2018',\n",
" marker=dict(\n",
" color='#c44e52',\n",
" line=dict(\n",
" color='#c44e52',\n",
" width=3\n",
" ),\n",
" )\n",
" ),\n",
" go.Box(\n",
" y=data.sites.df[data.sites.df.month == '2018-02'].trackers, \n",
" name='Feb 2018',\n",
" marker=dict(\n",
" color='#55a868',\n",
" line=dict(\n",
" color='#55a868',\n",
" width=3\n",
" ),\n",
" )\n",
" ),\n",
" go.Box(\n",
" y=data.sites.df[data.sites.df.month == '2018-03'].trackers, \n",
" name='Mar 2018',\n",
" marker=dict(\n",
" color='#4c72b0',\n",
" line=dict(\n",
" color='#4c72b0',\n",
" width=3\n",
" ),\n",
" )\n",
" )\n",
"]\n",
"fig = dict(data=traces, layout=dict(showlegend=False, margin=set_margins(t=0, b=30)))\n",
"offline.plot(fig, image='svg')\n",
"iplot(fig)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Mean occurrence of Blocking per page\n",
"traces = [\n",
" go.Bar(\n",
" x=['Jan 2018', 'Feb 2018', 'Mar 2018'],\n",
" y=[\n",
" data.trackers.df[data.trackers.df.month == '2018-01'].has_blocking.mean()*100,\n",
" data.trackers.df[data.trackers.df.month == '2018-02'].has_blocking.mean()*100,\n",
" data.trackers.df[data.trackers.df.month == '2018-03'].has_blocking.mean()*100\n",
" ],\n",
" marker=dict(\n",
" color=['#A069AB', '#9564c4', '#6564c4'],\n",
" line=dict(\n",
" color='#222',\n",
" width=2\n",
" ),\n",
" )\n",
" )\n",
"]\n",
"fig = dict(data=traces, layout=dict(margin=set_margins(t=0, b=30)))\n",
"offline.plot(fig, image_height=200, image_width=800, image='svg', output_type='file')\n",
"iplot(fig)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As in previous months, we look at sites' changing the trackers. [fewo-direct.de](../websites/fewo-direkt.de.html), [brigitte.de](../websites/brigitte.de.html) and [gutefrage.net](../websites/gutefrage.net.html) all had 5 fewer trackers on average per page this month. However, each of these still has over 50 trackers with some kind of presence, showing that this is more likely a side-effect of increased blocking than an active effort to reduce tracking on their sites. [klingel.de](../websites/klingel.de.html) and [informationvine.com](../websites/informationvine.com.html) see the largest increase in tracking of the sites we currently monitor."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mar_trackers = data.sites.get_snapshot('2018-03').set_index('site')['trackers']\n",
"feb_trackers = data.sites.get_snapshot('2018-02').set_index('site')['trackers']\n",
"site_diffs = pd.DataFrame({\n",
" 'trackers': mar_trackers,\n",
" 'change': (mar_trackers - feb_trackers)\n",
"})\n",
"site_diffs[(site_diffs.change > 5) | (site_diffs.change < -5.5)].sort_values('change')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A side-effect of the filtering we added in this new data pipeline is that the site reach for top trackers has increased. In the previous analysis a long-tail of very rarely visited sites reduced effective site reach. With this factor reduced, we get a real sense of the coverage of the largest trackers, with Google Analytics reaching 85% of popular sites, and Facebook almost 60%."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = data.trackers.get_snapshot().sort_values(by='site_reach', ascending=False).head(10)\n",
"df['name'] = df.id.apply(func=lambda x: data.app_info[x]['name'])\n",
"\n",
"traces = [\n",
" go.Bar(\n",
" x=df.site_reach[::-1]*100,\n",
" y=df.name[::-1],\n",
" orientation='h',\n",
" marker=dict(\n",
" color=palette('#9ebcda', '#A069AB', 10),\n",
" line=dict(\n",
" color='#333',\n",
" width=2\n",
" ),\n",
" )\n",
" )\n",
"]\n",
"layout=dict(margin=set_margins(l=200))\n",
"fig = dict(data=traces, layout=layout)\n",
"offline.plot(fig, image='svg')\n",
"iplot(fig)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you want to delve deeper into our data, it is available on the [Whotracks.me Github Repository](https://github.com/cliqz-oss/whotracks.me/tree/master/whotracksme/data), and as a [pip package](https://pypi.python.org/pypi/whotracksme/)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}