![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\" "\\"Open"){kind=icon}
![\"Download\"](\"https://img.shields.io/badge/Github-Download-blue.svg\" "\\"Download"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"# Data Reconciliation"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"This notebook demonstrates use of linear programming using GLPK/Mathprog for reconciliation of process measurements."
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"## Data Reconciliation and Gross Error Detection"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"The following demonstration was motivated by a former student Matt N. who was exploring methods for assessing measurement information in a refinery application. The illustration was adapted from Chapter 1 of \"Reconciliation & Gross Error Detection: An Intelligent Use of Process Data\" by Shankar Narasimhan and Cornelius Jordache (Gulf Publishing, 2000)."
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"### Heat Exchanger with Bypass\n",
"
\n",
" 1/1 2/2 4/3 6/5\n",
" -------- SPLIT ----- HTEXG ----- MIXER --------\n",
" | ^ \n",
" | |\n",
" | 3 5/4 |\n",
" +-------> VALVE --------+\n",
"\n",
"\n",
"The flow sheet is labeled by `s/k` pairs which refer to stream `s` and measurement `k`. Streams are numbered `s` in `1..S` and sensors are numbered `k` in `1..K`. Parameter `s[k]` denotes the stream associated with measurement `k`. Note that not all of the streams are measured, and the redundancy in measurements due to material balances."
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": false,
"pycharm": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting tmp/DataRecon.dat\n"
]
}
],
"source": [
"%%writefile tmp/DataRecon.dat\n",
"\n",
"/* All of the relevant sensor information is given in the data section. The\n",
"relevant information includes the number of sensors, and a table indexed by \n",
"sensor number k with information on which stream is measured (s[k]), the \n",
"measurement (y[k]), and 'no-penalty' range for measurement errors (emax[k]). \n",
"*/\n",
"\n",
"set UNITS := SPLIT HTEXG VALVE MIXER;\n",
"set STREAMS := 1 2 3 4 5 6;\n",
"\n",
"/* Mass Balance Coefficients */\n",
"param A :\n",
" 1 2 3 4 5 6 :=\n",
" SPLIT -1 1 1 . . .\n",
" HTEXG . -1 . 1 . .\n",
" VALVE . . -1 . 1 .\n",
" MIXER . . . 1 1 -1 ;\n",
"\n",
"/* Sensors and Measurement Data*/\n",
"param: SENSORS: s y emax := \n",
" 1 1 101.91 1.75\n",
" 2 2 68.45 1.75\n",
" 3 4 64.20 1.75\n",
" 4 5 36.44 1.75\n",
" 5 6 98.88 1.75 ;\n",
"\n",
"end;"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"Given sensor measurements $y_k$, the task is to infer values for the flowrates $x_{s_k}$ using mass balance equations and a sensor model\n",
"\n",
"$$\\begin{align*}\n",
" |y_k-x_{s_k}| & \\leq e_k + M g_k \\\\\n",
" |e_k| & \\leq e^{max}_k\n",
"\\end{align*}$$\n",
"\n",
"$g_k$ is a binary variable indicating the presence of a gross error in sensor $k$. $M$ is a big number. A gross error occurs if the measured data is not consistent with sensor model and mass balances. The optimization objective is to find an estimate of mass flows minimizing the number of gross sensor errors."
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"collapsed": false,
"pycharm": {}
},
"outputs": [],
"source": [
"%%script glpsol -m /dev/stdin -d tmp/DataRecon.dat -y tmp/DataRecon.txt --out output\n",
"\n",
"# Example: DataReconciliation.mod\n",
"\n",
"/* Sets */\n",
"set STREAMS;\n",
"set SENSORS;\n",
"set UNITS;\n",
"\n",
"/* Mass Balances Coefficients */\n",
"param A{UNITS,STREAMS} default 0;\n",
"\n",
"/* Sensors Parameters */\n",
"param y{SENSORS} >= 0; # Measurement Data\n",
"param s{SENSORS} > 0; # Stream measured by each sensor\n",
"param emax{SENSORS} >= 0; # Bound on error for each sensor\n",
"param bigM := 100; # Big M\n",
"\n",
"/* Stream Variables */\n",
"var x{STREAMS} >= 0;\n",
"\n",
"/* Sensor Variables */\n",
"var epos{SENSORS} >= 0;\n",
"var eneg{SENSORS} >= 0;\n",
"var gerr{SENSORS} binary; # Gross Errors\n",
"\n",
"/* Optimization Criterion */\n",
"minimize L1: sum {k in SENSORS} (epos[k]+eneg[k] + bigM*gerr[k]);\n",
"\n",
"/* Mass Balances */\n",
"s.t. MASSBAL {i in UNITS}: sum{j in STREAMS} A[i,j]*x[j] = 0;\n",
"\n",
"/* Error Model */\n",
"s.t. s_err {k in SENSORS}: y[k] = x[s[k]] + epos[k] - eneg[k];\n",
"s.t. e_pos {k in SENSORS}: epos[k] <= emax[k] + bigM*gerr[k];\n",
"s.t. e_neg {k in SENSORS}: eneg[k] <= emax[k] + bigM*gerr[k];\n",
"\n",
"/* Solve and Display Solution */\n",
"solve;\n",
"\n",
"printf \"Measurement Reconciliation\";\n",
"printf \"\\n\\n Sensor Measured Reconciled Difference GrossError\";\n",
"for {k in SENSORS}{\n",
" printf \"\\n %s %7.2f %7.2f %7.2f\",\n",
" k,y[k],x[s[k]],(epos[k]-eneg[k]),gerr[k];\n",
" printf \" %3s\", if gerr[k]=1 then \"X\" else \" \";\n",
"}\n",
"printf \"\\n\\n\\nStream Estimates\";\n",
"printf \"\\n\\n Stream Estimate\";\n",
"printf {i in STREAMS} \"\\n %s %7.2f\",i,x[i];\n",
"\n",
"table tab1 {k in SENSORS} OUT \"CSV\" \"tmp/MeasurementError.csv\":\n",
" k~Sensor,\n",
" y[k]~Measurement,\n",
" x[s[k]]~Reconciled,\n",
" (epos[k]-eneg[k])~Error,\n",
" gerr[k]~GrossError;\n",
"\n",
"data;\n",
"\n",
"end;\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"collapsed": false,
"pycharm": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Measurement Reconciliation\n",
"\n",
" Sensor Measured Reconciled Difference GrossError\n",
" 1 101.91 100.63 1.28 \n",
" 2 68.45 64.20 4.25 X\n",
" 3 64.20 64.20 0.00 \n",
" 4 36.44 36.43 0.01 \n",
" 5 98.88 100.63 -1.75 \n",
"\n",
"\n",
"Stream Estimates\n",
"\n",
" Stream Estimate\n",
" 1 100.63\n",
" 2 64.20\n",
" 3 36.43\n",
" 4 64.20\n",
" 5 36.43\n",
" 6 100.63\n"
]
}
],
"source": [
"f = open('tmp/DataRecon.txt')\n",
"print(f.read())\n",
"f.close()"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"## Visualization"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"collapsed": false,
"pycharm": {}
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | Measurement | \n", "Reconciled | \n", "Error | \n", "GrossError | \n", "
|---|---|---|---|---|
| Sensor | \n", "\n", " | \n", " | \n", " | \n", " |
| 1 | \n", "101.91 | \n", "100.63 | \n", "1.28 | \n", "0 | \n", "
| 2 | \n", "68.45 | \n", "64.20 | \n", "4.25 | \n", "1 | \n", "
| 3 | \n", "64.20 | \n", "64.20 | \n", "0.00 | \n", "0 | \n", "
| 4 | \n", "36.44 | \n", "36.43 | \n", "0.01 | \n", "0 | \n", "
| 5 | \n", "98.88 | \n", "100.63 | \n", "-1.75 | \n", "0 | \n", "
"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 0
}