![](\"images/RNN-rolled.png\")
\n",
"\n",
"A typical RNN (Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"images/RNN-unrolled.png\")
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A typical RNN looks like above-where X(t) is input, h(t) is output and A is the neural network which gains information from the previous step in a loop. The output of one unit goes into the next one and the information is passed.\n",
"\n",
"But, sometimes we don’t need our network to learn only from immediate past information. Suppose we want to predict the blank word in the text ‘ David, a 36-year old man lives in San Francisco. He has a female friend Maria. Maria works as a cook in a famous restaurant in New York whom he met recently in a school alumni meet. Maria told him that she always had a passion for _________ . Here, we want our network to learn from dependency ‘cook’ to predict ‘cooking. There is a gap between the information what we want to predict and from where we want it to get predicted . This is called long-term dependency. We can say that anything larger than trigram as a long term dependency. Unfortunately, RNN does not work practically in this situation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Why RNN does not work practically\n",
"During the training of RNN, as the information goes in loop again and again which results in very large updates to neural network model weights. This is due to the accumulation of error gradients during an update and hence, results in an unstable network. At an extreme, the values of weights can become so large as to overflow and result in NaN values.The explosion occurs through exponential growth by repeatedly multiplying gradients through the network layers that have values larger than 1 or vanishing occurs if the values are less than 1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Long Short Term Memory Networks (LSTM)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. They were introduced by [Hochreiter & Schmidhuber (1997)](http://www.bioinf.jku.at/publications/older/2604.pdf), and were refined and popularized by many people in following work.1 They work tremendously well on a large variety of problems, and are now widely used.\n",
"\n",
"LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!\n",
"\n",
"All recurrent neural networks have the form of a chain of repeating modules of neural network. In standard RNNs, this repeating module will have a very simple structure, such as a single tanh layer."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"images/LSTM3-SimpleRNN.png\")
\n",
"\n",
"The repeating module in a standard RNN contains a single layer.\n",
" (Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"LSTMs also have this chain like structure, but the repeating module has a different structure. Instead of having a single neural network layer, there are four, interacting in a very special way."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"images/LSTM3-chain.png\")
\n",
"\n",
"The repeating module in an LSTM contains four interacting layers.\n",
" (Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Detailed process"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"images/1_Niu_c_FhGtLuHjrStkB_4Q.png\")
\n",
"\n",
"The repeating module in an LSTM contains four interacting layers.\n",
" (Source: https://towardsdatascience.com/understanding-lstm-and-its-quick-implementation-in-keras-for-sentiment-analysis-af410fd85b47)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The symbols used here have following meaning:\n",
"\n",
"- $x$ : Scaling of information\n",
"\n",
"- $+$ : Adding information\n",
"\n",
"- $\\sigma$ : Sigmoid layer\n",
"\n",
"- $tanh$: tanh layer\n",
"\n",
"- $h_{t-1}$ : Output of last LSTM unit\n",
"\n",
"- $c_{t-1}$ : Memory from last LSTM unit\n",
"\n",
"- $X_t$ : Current input\n",
"\n",
"- $c_t$ : New updated memory\n",
"\n",
"- $h_t$ : Current output"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Information passes through many such LSTM units.There are three main components of an LSTM unit which are labeled in the diagram:\n",
"\n",
"1) LSTM has a special architecture which enables it to forget the unnecessary information .The sigmoid layer takes the input $X_t$ and $h_{t-1}$ and decides which parts from old output should be removed (by outputting a 0). In our example, when the input is ‘He has a female friend Maria’, the gender of ‘David’ can be forgotten because the subject has changed to ‘Maria’. This gate is called forget gate $f(t)$. The output of this gate is $f(t) \\cdot c_{t-1}$.\n",
"\n",
"2) The next step is to decide and store information from the new input $X_t$ in the cell state. A Sigmoid layer decides which of the new information should be updated or ignored. A tanh layer creates a vector of all the possible values from the new input. These two are multiplied to update the new cell sate. This new memory is then added to old memory $c_{t-1}$ to give $c_{t}$. In our example, for the new input ‘ He has a female friend Maria’, the gender of Maria will be updated. When the input is ‘Maria works as a cook in a famous restaurant in New York whom he met recently in a school alumni meet’, the words like ‘famous’, ‘school alumni meet’ can be ignored and words like ‘cook, ‘restaurant’ and ‘New York’ will be updated.\n",
"\n",
"3) Finally, we need to decide what we’re going to output. A sigmoid layer decides which parts of the cell state we are going to output. Then, we put the cell state through a tanh generating all the possible values and multiply it by the output of the sigmoid gate, so that we only output the parts we decided to. In our example, we want to predict the blank word, our model knows that it is a noun related to ‘cook’ from its memory, it can easily answer it as ‘cooking’. Our model does not learn this answer from the immediate dependency, rather it learnt it from long term dependency.\n",
"\n",
"We just saw that there is a big difference in the architecture of a typical RNN and a LSTM. In LSTM, our model learns what information to store in long term memory and what to get rid of.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Example phishing URL detection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on the paper:\n",
"- A. Correa Bahnsen, E. C. Bohorquez, S. Villegas, J. Vargas, and F. A. Gonzalez, “Classifying phishing urls using recurrent neural networks,” in Electronic Crime Research (eCrime), 2017 APWG Symposium on. IEEE, 2017, pp. 1–8.\n",
"https://albahnsen.com/wp-content/uploads/2018/05/classifying-phishing-urls-using-recurrent-neural-networks_cameraready.pdf"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import zipfile\n",
"with zipfile.ZipFile('../datasets/model_deployment/phishing.csv.zip', 'r') as z:\n",
" f = z.open('phishing.csv')\n",
" data = pd.read_csv(f, index_col=False)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | url | \n", "phishing | \n", "
|---|---|---|
| 0 | \n", "http://www.subalipack.com/contact/images/sampl... | \n", "1 | \n", "
| 1 | \n", "http://fasc.maximecapellot-gypsyjazz-ensemble.... | \n", "1 | \n", "
| 2 | \n", "http://theotheragency.com/confirmer/confirmer-... | \n", "1 | \n", "
| 3 | \n", "http://aaalandscaping.com/components/com_smart... | \n", "1 | \n", "
| 4 | \n", "http://paypal.com.confirm-key-21107316126168.s... | \n", "1 | \n", "
| \n", " | url | \n", "phishing | \n", "
|---|---|---|
| 39995 | \n", "http://www.diaperswappers.com/forum/member.php... | \n", "0 | \n", "
| 39996 | \n", "http://posting.bohemian.com/northbay/Tools/Ema... | \n", "0 | \n", "
| 39997 | \n", "http://www.tripadvisor.jp/Hotel_Review-g303832... | \n", "0 | \n", "
| 39998 | \n", "http://www.baylor.edu/content/services/downloa... | \n", "0 | \n", "
| 39999 | \n", "http://www.phinfever.com/forums/viewtopic.php?... | \n", "0 | \n", "