![](\"https://github.com/Emaasit/long-range-extrapolation/blob/dev/blog/data-emails.png?raw=true\")
\n",
"\n",
"Second, most of the current literature focuses on parametric models that impose strong restrictive assumptions by pre-specifying the functional form and number of parameters. Pre-specifying a functional form for a time-series model could lead to either overly complex model specifications or simplistic models. It is difficult to know *a priori* the most appropriate function to use for modeling sophisticated insider-threat behavior or highway-crash scenarios that involve complex hidden patterns and many other influencing factors.\n",
"\n",
"### Source code\n",
"For the impatient reader, two options are provided below to access the source code used for empirical analyses:\n",
"\n",
"1. The entire project (code, notebooks, data, and results) can be found [here on GitHub](https://github.com/Emaasit/long-range-extrapolation).\n",
"\n",
"2. Click the binder icon below to open the notebooks in a web browser and explore the entire project without downloading and installing any software.\n",
"\n",
" [](https://mybinder.org/v2/gh/Emaasit/long-range-extrapolation/master?urlpath=lab)\n",
"\n",
"## Data Science Questions\n",
"Given the above limitations in the current state-of-art, this study formulated the following three Data Science questions. Given limited and quite noisy time-series data for insider-threat detection and highway-safety planning, is it possible to perform:\n",
"\n",
"1. pattern discovery without hard-coding trends into statistical models during formulation?\n",
"\n",
"2. model estimation that precludes pre-specifying a functional form?\n",
"\n",
"3. model estimation that is robust to overfitting and has well-calibrated uncertainty estimates? \n",
"\n",
"## Hypothesis\n",
"To answer these three Data Science questions and address the above-described limitations, this study formulated the following hypothesis:\n",
"This study hypothesizes that by leveraging current state-of-the-art innovations in nonparametric Bayesian methods, such as Gaussian processes with spectral mixture kernels, it is possible to perform pattern discovery without prespecifying functional forms and hard-coding trends into statistical models.\n", "\n", "## Methodology\n", "To test the above hypothesis, a nonparametric Bayesian approach was proposed to implicitly capture hidden structure from time series having limited data. The proposed model, a Gaussian process with a spectral mixture kernel, precludes the need to pre-specify a functional form and hard code trends, is robust to overfitting and has well-calibrated uncertainty estimates.\n", "\n", "Mathematical details of the proposed model formulation are described in a corresponding paper that can be found on arXiv through the link below:\n", "\n", "* Emaasit, D. and Johnson, M. (2018). [Capturing Structure Implicitly from Noisy Time-Series having Limited Data](https://arxiv.org/abs/1803.05867). arXiv preprint arXiv:1803.05867.\n", "\n", "A Brief description of the fundamental concepts of the proposed methodology is as follows. Consider for each data point, $i$, that $y_i$ represents the attachment size in emails sent by an insider to their home account and $x_i$ is a temporal covariate such as month. The task is to estimate a latent function $f$, which maps input data, $x_i$, to output data $y_i$ for $i$ = 1, 2, $\\ldots{}$, $N$, where $N$ is the total number of data points. Each of the input data $x_i$ is of a single dimension $D = 1$, and $\\textbf{X}$ is a $N$ x $D$ matrix with rows $x_i$.\n", "\n", "
![\"\"](\"http://haystax.com/wp-content/uploads/2018/03/gp-pgm-352x300.png\")
\n",
"\n",
"The observations are assumed to satisfy:\n",
"\\begin{equation}\\label{eqn:additivenoise}\n",
"y_i = f(x_i) + \\varepsilon, \\quad where \\, \\, \\varepsilon \\sim \\mathcal{N}(0, \\sigma_{\\varepsilon}^2)\n",
"\\end{equation}\n",
"\n",
"The noise term, $\\varepsilon$, is assumed to be normally distributed with a zero mean and variance, $\\sigma_{\\varepsilon}^2$. Latent function $f$ represents hidden underlying trends that produced the observed time-series data.\n",
"\n",
"Given that it is difficult to know $\\textit{a priori}$ the most appropriate functional form to use for $f$, a prior distribution, $p(\\textbf{f})$, over an infinite number of possible functions of interest is formulated. A natural prior over an infinite space of functions is a Gaussian process prior (Williams and Rasmussen, 2006). A GP is fully parameterized by a mean function, $\\textbf{m}$, and covariance function, $\\textbf{K}_{N,N}$, denoted as:\n",
"\\begin{equation}\\label{eqn:gpsim}\n",
"\\textbf{f} \\sim \\mathcal{GP}(\\textbf{m}, \\textbf{K}_{N,N}),\n",
"\\end{equation}\n",
"\n",
"The posterior distribution over the unknown function evaluations, $\\textbf{f}$, at all data points, $x_i$, was estimated using Bayes theorem as follows:\n",
"\n",
"\\begin{equation}\\label{eqn:bayesinfty}\n",
"\\begin{aligned}\n",
"p(\\textbf{f} \\mid \\textbf{y},\\textbf{X}) &= \\frac{p(\\textbf{y} \\mid \\textbf{f}, \\textbf{X}) \\, p(\\textbf{f})}{p(\\textbf{y} \\mid \\textbf{X})}, \\\\\n",
"&= \\frac{p(\\textbf{y} \\mid \\textbf{f}, \\textbf{X}) \\, \\mathcal{N}(\\textbf{f} \\mid \\textbf{m}, \\textbf{K}_{N,N})}{p(\\textbf{y} \\mid \\textbf{X})},\n",
"\\end{aligned}\n",
"\\end{equation}\n",
"\n",
"where:\n",
"\n",
"\\begin{aligned}\n",
"p(\\textbf{f}\\mid \\textbf{y},\\textbf{X}) &= \\text{the posterior distribution of functions that best explain the response variable, given the covariates} \\\\\n",
"p(\\textbf{y} \\mid \\textbf{f}, \\textbf{X}) &= \\text{the likelihood of response variable, given the functions and covariates} \\\\ \n",
"p(\\textbf{f}) &= \\text{the prior over all possible functions of the response variable} \\\\\n",
"p(\\textbf{y} \\mid \\textbf{X}) &= \\text{the data (constant)}\n",
"\\end{aligned}\n",
"\n",
"A spectral mixture kernel was proposed for the covariance function, $\\textbf{K}_{N,N}$. The resulting posterior, $p(\\textbf{f}\\mid \\textbf{y},\\textbf{X})$ is a Gaussian process composed of a distribution of possible functions that best explain the time-series pattern."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The setup\n",
"Let's first install some python packages that we shall use for our analysis. Also we shall set up our plotting requirements."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/demaasit/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
" return f(*args, **kwds)\n"
]
}
],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import pandas as pd\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"sns.set_context('notebook', font_scale = 1.1)\n",
"np.random.seed(12345)\n",
"rc = {'xtick.labelsize': 40, 'ytick.labelsize': 40, 'axes.labelsize': 40, 'font.size': 40, 'lines.linewidth': 4.0, \n",
" 'lines.markersize': 40, 'font.family': \"serif\", 'font.serif': \"cm\", 'savefig.dpi': 200,\n",
" 'text.usetex': False, 'legend.fontsize': 40.0, 'axes.titlesize': 40, \"figure.figsize\": [24, 16]}\n",
"sns.set(rc = rc)\n",
"sns.set_style(\"darkgrid\")\n",
"from IPython.core.interactiveshell import InteractiveShell\n",
"InteractiveShell.ast_node_interactivity = \"all\"\n",
"import gpflow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Raw data and sample formation\n",
"The insider-threat data used for empirical analysis in this study was provided by the computer emergency response team (CERT) division of the software engineering institute (SEI) at Carnegie Mellon University. The particular insider threat focused on is the case where a known insider sent information as email attachments from their work email to their home email.\n",
"\n",
"First, let's read in the data using `pandas`, view the first three records and the structure of the resulting `pandas` dataframe."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | id | \n", "date | \n", "user | \n", "pc | \n", "to | \n", "cc | \n", "bcc | \n", "from | \n", "activity | \n", "size | \n", "attachments | \n", "content | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "{D0V4-N9KM15BF-0512LLVP} | \n", "2010-01-04 07:36:48 | \n", "BTR2026 | \n", "PC-9562 | \n", "Thaddeus.Brett.Daniel@dtaa.com | \n", "Zorita.Angela.Wilson@dtaa.com | \n", "NaN | \n", "Beau.Todd.Romero@dtaa.com | \n", "Send | \n", "23179 | \n", "NaN | \n", "On November 25, general Savary was sent to the... | \n", "
| 1 | \n", "{L5E5-J1HB80OY-9539AOEC} | \n", "2010-01-04 07:38:18 | \n", "BTR2026 | \n", "PC-9562 | \n", "Beau.Todd.Romero@dtaa.com | \n", "NaN | \n", "NaN | \n", "Marsh_Travis@raytheon.com | \n", "View | \n", "17047 | \n", "NaN | \n", "Early in the morning of May 27, a boat crossed... | \n", "
| 2 | \n", "{Q4V7-V6BR00TZ-5209UVDX} | \n", "2010-01-04 07:53:35 | \n", "BTR2026 | \n", "PC-9562 | \n", "Bianca-Clark@optonline.net | \n", "NaN | \n", "NaN | \n", "Beau_Romero@aol.com | \n", "Send | \n", "26507 | \n", "NaN | \n", "The Americans never held up their side of the ... | \n", "
| \n", " | ds | \n", "y | \n", "
|---|---|---|
| 0 | \n", "2010-01 | \n", "117.809274 | \n", "
| 1 | \n", "2010-02 | \n", "112.461320 | \n", "
| 2 | \n", "2010-03 | \n", "134.592245 | \n", "
| 3 | \n", "2010-04 | \n", "148.911866 | \n", "
| 4 | \n", "2010-05 | \n", "80.689085 | \n", "
| 5 | \n", "2010-06 | \n", "128.024029 | \n", "
| 6 | \n", "2010-07 | \n", "115.046041 | \n", "
| 7 | \n", "2010-08 | \n", "142.607937 | \n", "
| 8 | \n", "2010-09 | \n", "121.728119 | \n", "
| 9 | \n", "2010-10 | \n", "126.041467 | \n", "
| 10 | \n", "2010-11 | \n", "80.338345 | \n", "
| 11 | \n", "2010-12 | \n", "113.142879 | \n", "
| 12 | \n", "2011-01 | \n", "95.485553 | \n", "
| 13 | \n", "2011-02 | \n", "88.279993 | \n", "
| 14 | \n", "2011-03 | \n", "148.193802 | \n", "
| 15 | \n", "2011-04 | \n", "221.337135 | \n", "
| 16 | \n", "2011-05 | \n", "146.220533 | \n", "