{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Browser Fingerprint Exercise\n", "\n", "
\n", "Is my network traffic lying to me? Most malware authors don\u2019t seem to spend a lot of effort trying to blend into network traffic. I\u2019m pretty sure the reason for this is \u201cthey don\u2019t need to\u201d. By identifying legitimate HTTP requests based on browser request structure we may be able to, more easily, identify malicious traffic. This notebook will focus on some ways to gather legit browser requests, understand them, and use that data to find non-legitiate requests.\n", "\n", " | header_events_json | \n", "origin | \n", "ts | \n", "useragent | \n", "
---|---|---|---|---|
0 | \n", "[{\"ACCEPT\":\"*\\/*\"},{\"ACCEPT-LANGUAGE\":\"en-US\"}... | \n", "client | \n", "2012-03-30 17:32:57.382264 | \n", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ... | \n", "
1 | \n", "[{\"CACHE-CONTROL\":\"no-cache\"},{\"DATE\":\"Fri, 30... | \n", "server | \n", "2012-03-30 17:32:57.382264 | \n", "NA | \n", "
2 | \n", "[{\"ACCEPT\":\"*\\/*\"},{\"ACCEPT-LANGUAGE\":\"en-US\"}... | \n", "client | \n", "2012-03-30 17:32:57.382264 | \n", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ... | \n", "
3 | \n", "[{\"CACHE-CONTROL\":\"no-cache\"},{\"DATE\":\"Fri, 30... | \n", "server | \n", "2012-03-30 17:32:57.382264 | \n", "NA | \n", "
4 | \n", "[{\"ACCEPT\":\"*\\/*\"},{\"ACCEPT-LANGUAGE\":\"en-US\"}... | \n", "client | \n", "2012-03-30 17:32:57.382264 | \n", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ... | \n", "
5 rows \u00d7 4 columns
\n", "\n", " | \n", " | count | \n", "
---|---|---|
short_agent | \n", "features | \n", "\n", " |
memeo:autobackup:/4.60.0.7923:/platform=1 | \n", "ACCEPT-LANGUAGE:ACCEPT:USER-AGENT:HOST:CONNECTION | \n", "2 | \n", "
microsoft-cryptoapi/6.1 | \n", "CACHE-CONTROL:CONNECTION:ACCEPT:IF-MODIFIED-SINCE:IF-NONE-MATCH:USER-AGENT:HOST | \n", "2 | \n", "
CACHE-CONTROL:CONNECTION:ACCEPT:IF-MODIFIED-SINCE:USER-AGENT:HOST | \n", "3 | \n", "|
CONNECTION:ACCEPT:IF-MODIFIED-SINCE:IF-NONE-MATCH:USER-AGENT:HOST | \n", "2 | \n", "|
CONNECTION:ACCEPT:USER-AGENT:HOST | \n", "3 | \n", "|
mozilla/4.0 | \n", "USER-AGENT:HOST | \n", "3 | \n", "
USER-AGENT:HOST:IF-MODIFIED-SINCE:IF-NONE-MATCH:CONNECTION | \n", "3 | \n", "|
mozilla/4.0:msie:6.0:windows:nt:5.1:sv1:.net:clr:1.1.4322 | \n", "ACCEPT:ACCEPT-LANGUAGE:XXXXXXXXXXXXXXX:USER-AGENT:HOST:CONNECTION | \n", "1 | \n", "
mozilla/4.0:msie:7.0:windows:nt:6.1:wow64:trident/5.0:slcc2:n2.0:n3.5:n3.0 | \n", "ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "1 | \n", "
ACCEPT:ACCEPT-LANGUAGE:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "4 | \n", "|
mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "ACCEPT:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "2 | \n", "
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "15 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "12 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:CONTENT-TYPE:CONTENT-LENGTH:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:CACHE-CONTROL:COOKIE | \n", "1 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:CONTENT-TYPE:X-VERIFY:CONTENT-LENGTH:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:CACHE-CONTROL | \n", "1 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:USER-AGENT:ACCEPT-ENCODING:HOST:CONNECTION | \n", "2 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "5 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "6 | \n", "|
ACCEPT:REFERER:ACCEPT-LANGUAGE:USER-AGENT:ACCEPT-ENCODING:COOKIE:CONNECTION:HOST | \n", "1 | \n", "|
ACCEPT:REFERER:ACCEPT-LANGUAGE:USER-AGENT:ACCEPT-ENCODING:HOST:CONNECTION | \n", "77 | \n", "
20 rows \u00d7 1 columns
\n", "\n", " | \n", " | count | \n", "
---|---|---|
\n", " | short_agent | \n", "\n", " |
other | \n", "memeo:autobackup:/4.60.0.7923:/platform=1 | \n", "1 | \n", "
microsoft-cryptoapi/6.1 | \n", "4 | \n", "|
mozilla/4.0 | \n", "2 | \n", "|
msie | \n", "mozilla/4.0:msie:6.0:windows:nt:5.1:sv1:.net:clr:1.1.4322 | \n", "1 | \n", "
mozilla/4.0:msie:7.0:windows:nt:6.1:wow64:trident/5.0:slcc2:n2.0:n3.5:n3.0 | \n", "2 | \n", "|
mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "12 | \n", "|
mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "17 | \n", "|
mozilla/5.0:msie:9.0:windows:nt:6.1:wow64:trident/5.0 | \n", "4 | \n", "|
mozilla/5.0:msie:9.0:windows:nt:6.1:wow64:trident/5.0:np06 | \n", "3 | \n", "|
other | \n", "mozilla/5.0:windows:nt:6.1:wow64:rv:12.0:gecko/20100101:firefox/12.0 | \n", "2 | \n", "
mozilla/5.0:windows:nt:6.1:wow64:rv:14.0:gecko/20100101:firefox/14.0.1 | \n", "5 | \n", "|
mozilla/5.0:windows:nt:6.1:wow64:rv:16.0:gecko/20100101:firefox/16.0 | \n", "25 | \n", "|
mozilla/5.0:windows:u:windows:nt:6.1:en-us:rv:1.9.2.18:gecko/20110614:firefox/3.6.18 | \n", "4 | \n", "|
nis/19.9.0.9:mid/{grpgbpdjzi9qbdsno/32ukaorrc}:sid/fhkuuaaaaaa | \n", "1 | \n", "|
nis/19.9.0.9:mid/{grpgbpdjzi9qbdsno/32ukaorrc}:sid/fhkuuaaaaaa:lue/1.8.2.10:windows6.1sp1.0x64enu | \n", "1 | \n", "|
shasta | \n", "1 | \n", "|
shockwave:flash | \n", "6 | \n", "
17 rows \u00d7 1 columns
\n", "\n", " | \n", " | count | \n", "
---|---|---|
features | \n", "short_agent | \n", "\n", " |
ACCEPT-LANGUAGE:ACCEPT:USER-AGENT:HOST:CONNECTION | \n", "memeo:autobackup:/4.60.0.7923:/platform=1 | \n", "2 | \n", "
ACCEPT:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "2 | \n", "
mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "5 | \n", "|
mozilla/5.0:msie:9.0:windows:nt:6.1:wow64:trident/5.0 | \n", "1 | \n", "|
mozilla/5.0:msie:9.0:windows:nt:6.1:wow64:trident/5.0:np06 | \n", "3 | \n", "|
ACCEPT:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "9 | \n", "
ACCEPT:ACCEPT-ENCODING:USER-AGENT:IF-MODIFIED-SINCE:HOST:CONNECTION:COOKIE | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "8 | \n", "
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "mozilla/4.0:msie:7.0:windows:nt:6.1:wow64:trident/5.0:slcc2:n2.0:n3.5:n3.0 | \n", "1 | \n", "
mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "15 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "12 | \n", "
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:IF-MODIFIED-SINCE:HOST:CONNECTION | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "1 | \n", "
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:CONTENT-TYPE:CONTENT-LENGTH:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:CACHE-CONTROL:COOKIE | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "1 | \n", "
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-FLASH-VERSION:CONTENT-TYPE:X-VERIFY:CONTENT-LENGTH:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:CACHE-CONTROL | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "1 | \n", "
ACCEPT:ACCEPT-LANGUAGE:REFERER:X-SVN-REV:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "1 | \n", "
ACCEPT:ACCEPT-LANGUAGE:USER-AGENT:ACCEPT-ENCODING:HOST:CONNECTION | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "2 | \n", "
mozilla/5.0:msie:9.0:windows:nt:6.1:wow64:trident/5.0:np06 | \n", "2 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:USER-AGENT:ACCEPT-ENCODING:HOST:CONNECTION:COOKIE | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.4:slcc2:n2.0:n3.5 | \n", "5 | \n", "
mozilla/5.0:msie:9.0:windows:nt:6.1:wow64:trident/5.0 | \n", "2 | \n", "|
ACCEPT:ACCEPT-LANGUAGE:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION | \n", "mozilla/4.0:msie:8.0:windows:nt:6.1:wow64:trident/4.0:gtb7.2:slcc2:n2.0:n3.5 | \n", "5 | \n", "
ACCEPT:ACCEPT-LANGUAGE:X-FLASH-VERSION:ACCEPT-ENCODING:USER-AGENT:HOST:CONNECTION:COOKIE | \n", "mozilla/4.0:msie:7.0:windows:nt:6.1:wow64:trident/5.0:slcc2:n2.0:n3.5:n3.0 | \n", "4 | \n", "
20 rows \u00d7 1 columns
\n", "\n", " | header_events_json | \n", "origin | \n", "ts | \n", "useragent | \n", "
---|---|---|---|---|
0 | \n", "[{\"ACCEPT\":\"application\\/octet-stream\"},{\"CONT... | \n", "client | \n", "2013-08-10 23:26:48.150406 | \n", "Alina v5.3 | \n", "
1 | \n", "[{\"DATE\":\"Sun, 11 Aug 2013 05:25:27 GMT\"},{\"SE... | \n", "server | \n", "2013-08-10 23:26:48.150406 | \n", "NA | \n", "
2 | \n", "[{\"ACCEPT\":\"application\\/octet-stream\"},{\"CONT... | \n", "client | \n", "2013-08-10 23:28:40.198085 | \n", "Alina v5.3 | \n", "
3 | \n", "[{\"DATE\":\"Sun, 11 Aug 2013 05:27:19 GMT\"},{\"SE... | \n", "server | \n", "2013-08-10 23:28:40.198085 | \n", "NA | \n", "
4 | \n", "[{\"ACCEPT\":\"application\\/octet-stream\"},{\"CONT... | \n", "client | \n", "2013-08-10 23:32:41.074339 | \n", "Alina v5.3 | \n", "
5 rows \u00d7 4 columns
\n", "