Continuous Backpropagation

Define adjoint state

$$a(t) = \frac{\partial{Loss}}{\partial{z(t)}} \tag1$$

From $t$ to $t + \epsilon$ ($\epsilon$ change in time) we have

$$z(t + \epsilon) = \int_{t}^{t + \epsilon}f(z(t),t,\theta)\partial{t}+z(t) = T_\epsilon(z(t),t) \tag2$$

And because of chain rule ( $\frac{\partial{y}}{\partial{x}} = \frac{\partial{y}}{\partial{u}} \frac{\partial{u}}{\partial{x}}$ )

$$a(t) = a(t + \epsilon)\frac{\partial{T_\epsilon}(z(t),t)}{\partial{z(t)}} \tag3$$

Take the definition of derivative:

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t)}{\epsilon} \tag4$$

Substitue (3) in (4)

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t + \epsilon)\frac{\partial{T_\epsilon}(z(t),t)}{\partial{z(t)}}}{\epsilon} \tag5$$

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t + \epsilon)\frac{\partial}{\partial{z(t)}}{T_\epsilon}(z(t),t)}{\epsilon} \tag6$$

Taylor series around $z(t)$ in (6)

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t + \epsilon)\frac{\partial}{\partial{z(t)}}(z(t)+\epsilon{f(z(t),t,\theta)+\mathcal{O}(\epsilon^2)})}{\epsilon} \tag7$$

aka ${T_\epsilon}(z(t),t)$ to $z(t)+\epsilon{f(z(t),t,\theta)+\mathcal{O}(\epsilon^2)}$ when $\lim_{\epsilon\to0}$

aka when $\epsilon$ change in time is small, take range $\epsilon - 0 = \epsilon$ and become $\epsilon{f(z(t),t,\theta)}$, to make up for the loss add $\mathcal{O}(\epsilon^2)$ at the end (notice it is related to $\epsilon$)

Expand (7)

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t + \epsilon)(\frac{\partial}{\partial{z(t)}}z(t)+\frac{\partial}{\partial{z(t)}}\epsilon{f(z(t),t,\theta)+\mathcal{O}(\epsilon^2)})}{\epsilon} \tag8$$

aka $\frac{\partial}{\partial{z(t)}}z(t) = I$

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t + \epsilon)(I+\frac{\partial}{\partial{z(t)}}\epsilon{f(z(t),t,\theta)+\mathcal{O}(\epsilon^2)})}{\epsilon} \tag9$$

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{a(t+\epsilon) - a(t + \epsilon)(I+\epsilon\frac{\partial{f(z(t),t,\theta)}}{\partial{z(t)}}{+\mathcal{O}(\epsilon^2)})}{\epsilon} \tag{10}$$

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}\frac{- a(t + \epsilon)\epsilon\frac{\partial{f(z(t),t,\theta)}}{\partial{z(t)}}{+\mathcal{O}(\epsilon^2)}}{\epsilon} \tag{11}$$

aka $a(t + \epsilon) - a(t + \epsilon)I = 0$

$$\frac{\partial{a(t)}}{\partial{t}} = \lim_{\epsilon\to0}{- a(t + \epsilon)\frac{\partial{f(z(t),t,\theta)}}{\partial{z(t)}}{+\mathcal{O}(\epsilon)}} \tag{12}$$

and because $\lim_{\epsilon\to0}$

$$\frac{\partial{a(t)}}{\partial{t}} = - a(t)\frac{\partial{f(z(t),t,\theta)}}{\partial{z(t)}} \tag{13}$$