{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# CSV Analysis Example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a demo notebook showing how to use the `orion.analysis.analyze` function on a CSV signal." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Load the data\n", "\n", "In the first step, we setup the environment and load the CSV that we want to process.\n", "\n", "To do so, we need to import the `orion.data.load_signal` function and call it passing\n", "the path to the CSV file.\n", "\n", "In this case, we will be loading the `S-1.csv` file from inside the `data` folder." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import logging;\n", "\n", "logging.basicConfig(level=logging.ERROR)\n", "logging.getLogger().setLevel(level=logging.ERROR)\n", "\n", "import warnings\n", "warnings.simplefilter(\"ignore\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampvalue
01222819200-0.366359
11222840800-0.394108
212228624000.403625
31222884000-0.362759
41222905600-0.370746
\n", "
" ], "text/plain": [ " timestamp value\n", "0 1222819200 -0.366359\n", "1 1222840800 -0.394108\n", "2 1222862400 0.403625\n", "3 1222884000 -0.362759\n", "4 1222905600 -0.370746" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from orion.data import load_signal\n", "\n", "signal_path = 'S-1'\n", "\n", "data = load_signal(signal_path)\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Detect anomalies using a pipeline\n", "\n", "Once we have the data, let us try to use the LSTM pipeline to analyze it and search for anomalies.\n", "\n", "In order to do so, we will have import the `orion.analysis.analyze` function and pass it\n", "the loaded data and the path to the pipeline JSON that we want to use.\n", "\n", "In this case, we will be using the `lstm_dynamic_threshold.json` pipeline from inside the `orion` folder.\n", "\n", "The output will be a ``pandas.DataFrame`` containing a table with the detected anomalies." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 7919 samples, validate on 1980 samples\n", "Epoch 1/5\n", "7919/7919 [==============================] - 35s 4ms/step - loss: 0.1959 - mse: 0.1959 - val_loss: 0.3439 - val_mse: 0.3439\n", "Epoch 2/5\n", "7919/7919 [==============================] - 34s 4ms/step - loss: 0.1915 - mse: 0.1915 - val_loss: 0.3147 - val_mse: 0.3147\n", "Epoch 3/5\n", "7919/7919 [==============================] - 34s 4ms/step - loss: 0.1909 - mse: 0.1909 - val_loss: 0.3534 - val_mse: 0.3534\n", "Epoch 4/5\n", "7919/7919 [==============================] - 34s 4ms/step - loss: 0.1873 - mse: 0.1873 - val_loss: 0.2893 - val_mse: 0.2893\n", "Epoch 5/5\n", "7919/7919 [==============================] - 34s 4ms/step - loss: 0.1891 - mse: 0.1891 - val_loss: 0.3037 - val_mse: 0.3037\n", "9899/9899 [==============================] - 13s 1ms/step\n", "9899/9899 [==============================] - 12s 1ms/step\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
startendscore
0122821920012294720000.656399
1140058720014041728000.166991
\n", "
" ], "text/plain": [ " start end score\n", "0 1228219200 1229472000 0.656399\n", "1 1400587200 1404172800 0.166991" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from orion.analysis import analyze\n", "\n", "hyperparameters = {\n", " 'keras.Sequential.LSTMTimeSeriesRegressor#1': {\n", " 'epochs': 5,\n", " 'verbose': True\n", " }\n", "}\n", "\n", "pipeline = 'lstm_dynamic_threshold'\n", "\n", "anomalies = analyze(pipeline, data, hyperparams=hyperparameters)\n", "anomalies" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 2 }