The Hamilton-Jacobi-Bellman (HJB) equation is a partial differential equation which is central to optimal control theory. The solution of the HJB equation is the 'value function', which gives the optimal cost-to-go for a given dynamical system with an associated cost function. Classical variational problems, for example, the brachistochrone problem can be solved using this method as well. The equation is a result of the theory of dynamic programming which was pioneered in the 1950s by Richard Bellman and coworkers. The corresponding discrete-time equation is usually referred to as the Bellman equation. In continuous time, the result can be seen as an extension of earlier work in classical physics on the Hamilton-Jacobi equation by William Rowan Hamilton and Carl Gustav Jacob Jacobi. Consider the following problem in deterministic optimal control
- <math> \min \int_0^T C[x(t),u(t)]\,dt + D[x(T)] </math>
subject to
- <math> \dot{x}(t)=F[x(t),u(t)] </math>
where <math>x(t)</math> is the system state, <math>x(0)</math> is assumed given, and <math>u(t)</math> for <math>0\leq t\leq T</math> is the control that we are trying to find. For this simple system, the Hamilton Jacobi Bellman partial differential equation is
- <math>
\frac{\partial}{\partial t} V(x,t) + \min_u \left\{ \left\langle \frac{\partial}{\partial x}V(x,t), F(x, u) \right\rangle + C(x,u) \right\} = 0 </math> subject to the terminal condition
- <math>
V(x,T) = D(x).\, </math> The unknown <math>V(t, x)</math> in the above PDE is the Bellman 'value function', that is the cost incurred from starting in state <math>x</math> at time <math>t</math> and controlling the system optimally from then until time <math>T</math>. The HJB equation needs to be solved backwards in time, starting from <math>t = T</math> and ending at <math>t = 0</math>. (The notation <math>\langle a,b \rangle </math> means the inner product of the vectors a and b). The HJB equation is a sufficient condition for an optimum. If we can solve for <math>V</math> then we can find from it a control <math>u</math> that achieves the minimum cost. The HJB method can be generalized to stochastic systems as well. In general case, the HJB equation does not have a classical (smooth) solution. Several notions of generalized solutions have been developed to cover such situations, including viscosity solution (Pierre-Louis Lions and Michael Crandall), minimax solution (Andrei Izmailovich Subbotin), and others.
References
- R. E. Bellman. Dynamic Programming. Princeton, NJ, 1957.
