{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[中文版](http://nbviewer.jupyter.org/github/w407022008/All-of-Notes/blob/master/System_Science_and_Control_Engineering/贝尔曼规划，离散系统最优控制，增强学习.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bellman optimality principle\n",
    "\n",
    "- **Markov property**：In the case of a decision input $U(k)$,in the process where the state $X(k)$ is transferred to the state $X(k+1)$,the state $X(k+1)$ is only related to the input $U(k)$ and the first state $X(k)$,then this process has Markovity and is written as：$X(k+1)=f(X(k),U(k),k)$  \n",
    "- For an N-stage general decision process, the input set $U^*(0,N-1)$ is necessary and sufficient as an optimal decision,if:  \n",
    "the optimal performance function satisfies:$$J^*(X(0),U^*(0,N-1))=\\min_{U(0,k-1)} [J(X(0),U(0,k-1)+\\min_{U(k,N-1)}J(X(k),U(k,N-1))],\\forall k\\in (0,N-1)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discrete system dynamic programming\n",
    "\n",
    "State equation of discrete systems：$X(k+1)=f(X(k),U(k),k)$  \n",
    "Satisfy a performance indicator function：$$\\min_{U(0,k-1)} J(u)=\\min_{U(0,k-1)} [S(X(k_f),k_f)+\\sum_{k=k_0}^{k_f-1}L(X(k),U(k),k)]$$\n",
    "Then the Bellman's recurrence formula is：$$\\begin{cases}J^*(X(k),U^*(k,k_f-1))=\\min_{U(k)\\in U} [L(X(k),U(k),k)+J^*(X(k+1),U^*(k+1,k_f-1))]\\\\J^*(X(k_f),k_f)=S(X(k_f,k_f)\\end{cases}$$\n",
    "**Matlab code**  \n",
    "- discrete-time algebric Riccati equations\n",
    "``` matlab\n",
    "[P,L,K]=dare(G,H,Q,R);\n",
    "```\n",
    "$K=(H^TPH+R)^{-1}H^TPG$ \n",
    "- linear-quadratic regulater\n",
    "``` matlab\n",
    "[K,P,L]=dlqr(G,H,Q,R);\n",
    "```\n",
    "L represents a pole array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforce learning\n",
    "\n",
    "! to be continued"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Matlab",
   "language": "matlab",
   "name": "matlab"
  },
  "language_info": {
   "codemirror_mode": "octave",
   "file_extension": ".m",
   "help_links": [
    {
     "text": "MetaKernel Magics",
     "url": "https://github.com/calysto/metakernel/blob/master/metakernel/magics/README.md"
    }
   ],
   "mimetype": "text/x-octave",
   "name": "matlab",
   "version": "0.15.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}