| --- |
| title: Forward and Backward for Inference and Learning |
| --- |
| # Forward and Backward |
|
|
| The forward and backward passes are the essential computations of a [Net](net_layer_blob.html). |
|
|
| <img src="fig/forward_backward.png" alt="Forward and Backward" width="480"> |
|
|
| Let's consider a simple logistic regression classifier. |
|
|
| The **forward** pass computes the output given the input for inference. |
| In forward Caffe composes the computation of each layer to compute the "function" represented by the model. |
| This pass goes from bottom to top. |
|
|
| <img src="fig/forward.jpg" alt="Forward pass" width="320"> |
|
|
| The data $$x$$ is passed through an inner product layer for $$g(x)$$ then through a softmax for $$h(g(x))$$ and softmax loss to give $$f_W(x)$$. |
| |
| The **backward** pass computes the gradient given the loss for learning. |
| In backward Caffe reverse-composes the gradient of each layer to compute the gradient of the whole model by automatic differentiation. |
| This is back-propagation. |
| This pass goes from top to bottom. |
| |
| <img src="fig/backward.jpg" alt="Backward pass" width="320"> |
| |
| The backward pass begins with the loss and computes the gradient with respect to the output $$\frac{\partial f_W}{\partial h}$$. The gradient with respect to the rest of the model is computed layer-by-layer through the chain rule. Layers with parameters, like the `INNER_PRODUCT` layer, compute the gradient with respect to their parameters $$\frac{\partial f_W}{\partial W_{\text{ip}}}$$ during the backward step. |
|
|
| These computations follow immediately from defining the model: Caffe plans and carries out the forward and backward passes for you. |
|
|
| - The `Net::Forward()` and `Net::Backward()` methods carry out the respective passes while `Layer::Forward()` and `Layer::Backward()` compute each step. |
| - Every layer type has `forward_{cpu,gpu}()` and `backward_{cpu,gpu}()` methods to compute its steps according to the mode of computation. A layer may only implement CPU or GPU mode due to constraints or convenience. |
|
|
| The [Solver](solver.html) optimizes a model by first calling forward to yield the output and loss, then calling backward to generate the gradient of the model, and then incorporating the gradient into a weight update that attempts to minimize the loss. Division of labor between the Solver, Net, and Layer keep Caffe modular and open to development. |
|
|
| For the details of the forward and backward steps of Caffe's layer types, refer to the [layer catalogue](layers.html). |
|
|
|
|
