Neural Ordinary Differential Equations (ODE)

return X, y #upper case: matrix, lower case: vector

Traditional neural network:

$$x \rightarrow f(x) \rightarrow y$$

$$f(x) = ax + b$$

$$loss = (f(x) - y)^2 = (ax + b -y)^2$$

$$a = a - \frac{\partial loss}{\partial a} \tag{gradient descent}$$

$$a = a - 2(ax + b - y) \cdot x \cdot LR$$

$$b = b - 2(ax + b - y) \cdot Y \cdot LR$$

Neural ODE:

$$\theta = [a,b]$$

$$\frac{\partial z}{\partial t} = f(z, t, \theta)$$

$$x \rightarrow g(x) \rightarrow y = z_0 \rightarrow g(x) \rightarrow z_t$$

$$loss = (z_t - ODE(f(z_0)))^2$$

$$\frac{\partial loss}{\partial z_T} = 2 \times (z_t - ODE(f(z_0)))$$

$$\theta = \theta - 2\frac{\partial loss}{\partial \theta} \cdot LR$$

ODE solver