🌻
Models
  • Step by step intro
  • Bash
  • Git
    • Remove folder
  • Embedding
    • Normalize Input
    • One-hot
  • Hyperparameter tuning
    • Test vs Validation
    • Bias vs Variance
    • Input
      • Normalize input
      • Initialize weight
    • Hidden Layer
      • Hidden layer size
    • Learning Rate
      • Oscillate learning rate
      • Learning rate finder
    • Batch Size
    • Epoch
    • Gradient
      • Vanishing / Exploding Gradients
      • Gradient Checking
    • Cost Function
      • Loss Binary Cross Entropy
    • Regularization
      • L₂ regularization
      • L₁ regularization
      • Dropout regularization
      • Data augmentation
      • Early stopping
  • Fine-tuning
    • Re-train on new data
    • Freeze layer/weight
  • Common Graphing Stats
    • Confidence interval (CI) and error bar
    • Confusion matrix and type I type II error
    • Effect size
  • Models
    • Inverted Pendulum Model
    • Recurrent Neural Networks
      • GRU and LSTM
      • Neural Turing Machines
    • Hopfield
    • Attention
      • Re-attention
      • Enformer
    • Differential Equations
      • Ordinary Differential Equations
        • Language Ordinary Differential Equations (ODE)
        • Neural Ordinary Differential Equations (ODE)
          • Adjoint Sensitive Method
          • Continuous Backpropagation
          • Adjoint ODE
      • Partial Differential Equations
      • Stochastic Differential Equations
    • Knowledge Tracing Models
      • Bayesian Knowledge Tracing
    • trRosetta
    • Curve up grades
  • deeplearning.ai
    • Neural Networks and Deep Learning
      • Wk2 - Python Basics with Numpy
      • Wk2 - Logistic Regression with a Neural Network mindset
      • Wk3 - Planar data classification with a hidden layer
      • Wk4 - Building your Deep Neural Network: Step by Step
      • Wk4 - Deep Neural Network - Application
    • Hyperparameter Tuning, Regularization and Optimization
      • Wk1 - Initialization
      • Wk1 - Regularization
      • Wk1 - Gradient Checking
    • Structuring Machine Learning Projects
    • Convolutional Neural Networks
    • Sequence Models
  • Neuroscience Paper
    • Rotation and Head Direction
    • Computational Models of Memory Search
    • Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating
    • Sensory uncertainty and spatial decisions
    • A Neural Implementation of the Kalman Filter
    • Place cells, spatial maps and the population code for memory (Hopfield)
    • Spatial Cognitive Map
    • Event Perception and Memory
    • Interplay of Hippocampus and Prefrontal Cortex in Memory
    • The Molecular and Systems Biology of Memory
    • Reconsidering the Evidence for Learning in Single Cells
    • Single Cortical Neurons as Deep Artificial Neural Networks
    • Magnetic resonance-based eye tracking using deep neural networks
Powered by GitBook
On this page

Was this helpful?

  1. Models
  2. Differential Equations
  3. Ordinary Differential Equations
  4. Neural Ordinary Differential Equations (ODE)

Adjoint Sensitive Method

Apply adjoint sensitive method to loss:

From loss=(zt−ODE(z0))2loss = (z_t - ODE(z_0))^2loss=(zt​−ODE(z0​))2 to ∂at∂t=−atT∂f(zt,t,θ)∂z\frac{\partial a_t}{\partial t} = -a_t^T \frac{\partial f(z_t, t, \theta)}{\partial z}∂t∂at​​=−atT​∂z∂f(zt​,t,θ)​ where at=∂loss∂zta_t = \frac{\partial loss}{\partial z_t}at​=∂zt​∂loss​

When there is a unknown vector u and two known matrix A and B, and we want to compute a product

uTB such that AB=Cu^T B \text{ such that } AB = CuTB such that AB=C

If there is a vector v and compute

vTC such that ATv=uv^T C \text{ such that } A^Tv = uvTC such that ATv=u

Thus

vTC=vTAB=(ATv)TB=uTBv^T C = v^T AB = (A^T v)^T B = u^TBvTC=vTAB=(ATv)TB=uTB

Turn unknown B and unknown u to one known C and one unknown v

PreviousNeural Ordinary Differential Equations (ODE)NextContinuous Backpropagation

Last updated 3 years ago

Was this helpful?