# Pedagogical Reference & User Guide

Welcome to the **Autodiff Visualizer**. This guide explains how to read the visual diagram and step-by-step walkthrough, the mathematical concepts of reverse-mode automatic differentiation, notation conventions, and supported mathematical syntax.

---

## 1. Reading the Visualization

The visual diagram represents numerical computations as a directed graph organized into three structural regions:
* **Inputs (bottom)**: Independent input variables and constant values.
* **Operations (middle)**: Intermediate calculations, local backward transformations, and gradient accumulations.
* **Output (top)**: The final scalar computation and seed adjoint introduction.

### Nodes store; arrows carry
A core principle of the visual grammar is: **Nodes store computational state; arrows carry information in transit.**

* **Quantity nodes (two compartments)**:
  Every differentiable quantity (inputs, intermediates, and final output) displays:
  - **Upper compartment (mint)**: Stored primal value computed during the forward pass ($v$).
  - **Lower compartment (pale violet)**: Accumulated adjoint ($\bar{v}$) representing total sensitivity of the final output with respect to $v$.
  - In *Forward Only* mode, the lower compartment and divider are omitted.
* **Constants (gray single-compartment)**:
  Literal constants (e.g. `2`, `0.5`) participate only in forward computation and do not have an adjoint compartment.
* **Arrows in transit**:
  - **Solid green arrows**: Carry forward primal values upward toward consuming operations.
  - **Dotted violet arrows**: Carry established adjoints downward or introduce accumulated adjoints to a quantity.
  - **Dotted plum arrows**: Carry individual gradient contributions toward accumulation ($\Sigma$) nodes.

### Local backward steps & accumulation
When backpropagating from a consumer to an operand:
1. **Derivative trapezoid (rose)**: Consumes an incoming adjoint and applies the local derivative rule to produce one numerical contribution.
2. **Summation node ($\Sigma$)**: When an operand is used multiple times (such as $x \cdot x$) or across multiple operations, its incoming contributions gather at an explicit $\Sigma$ node before introducing the final accumulated adjoint. Unary accumulation (a single contribution) is visually elided directly into the quantity.

---

## 2. Reading the Step-by-Step Walkthrough

The **Step-by-Step Walkthrough** linearizes the execution of the computation into sequential explanatory cards.

### The two box types

1. **Primal box (Forward pass)**:
   $$\boxed{n_1 = x \cdot x = 3 \cdot 3 = 9}$$
   Read from left to right:
   $$\text{quantity} = \text{computation} = \text{values used} = \text{result}$$
   Each forward step shows the operation formula, substitutes the evaluated operand values, and establishes the resulting primal quantity.

2. **Adjoint box (Backward pass)**:
   $$\bar{n}_1 = \frac{\partial \text{out}}{\partial n_1} = 4$$
   $$\bar{n}_1\text{ tells us how sensitive the final output is to }n_1\text{, including all downstream paths.}$$
   Each backward card begins with the quantity's established adjoint.

3. **Derivative and contribution pairs**:
   For each operand contributing to that adjoint, the walkthrough details:
   $$\text{Local derivative rule:}\quad \frac{\partial (u \cdot v)}{\partial u} = v = 4$$
   $$\text{Contribution:}\quad c_{n \to u} = \bar{n} \cdot v = 1 \cdot 4 = 4$$
   The first equation gives the local symbolic derivative with substituted values. The second line shows the chain-rule product (incoming adjoint $\times$ local derivative) that yields the contribution.

### Headlines vs. Everything
* **Headlines (Overview)**: Summarizes the key inputs, the final output, and the overall input sensitivities ($\bar{x} = \partial\text{out}/\partial x$).
* **Everything (Full trace)**: Expands all intermediate computational steps, local derivative rules, individual branch contributions, and step-by-step additions.

---

## 3. Core Concepts of Reverse-Mode AD

Automatic differentiation decomposes complex mathematical expressions into primitive operations whose local derivatives are known.

1. **Primal evaluation (Forward pass)**:
   $$v = f(u_1, u_2, \dots)$$
   Evaluates each operation from inputs to the scalar root output ($\text{out}$).

2. **Seed adjoint**:
   The backward pass always begins at the scalar output with:
   $$\bar{\text{out}} = \frac{\partial \text{out}}{\partial \text{out}} = 1$$

3. **Gradient contribution**:
   When an operation $v = f(u)$ receives an adjoint $\bar{v}$ from its consumers, it transmits a contribution to operand $u$ by the chain rule:
   $$c_{v \to u} = \bar{v} \cdot \frac{\partial v}{\partial u}$$

4. **Accumulation of adjoints**:
   By the multivariate chain rule, if a quantity $u$ contributes to multiple downstream computations (or appears multiple times as an operand), its total adjoint is the sum of all incoming contributions:
   $$\bar{u} = \sum_{v \in \text{Consumers}(u)} c_{v \to u}$$
   Reverse-mode AD computes exact derivatives for all inputs in a single backward pass, regardless of how many input variables exist.

---

## 4. Special Cases

### Single vs. multiple contributions
* **Single contribution**: When a variable is used only once, its single incoming contribution directly becomes its adjoint.
* **Multiple contributions**: When a variable is reused across different operations (e.g. $x \cdot y + x$) or repeated in the same operation (e.g. $x \cdot x$), each use produces a separate branch and contribution. The total sensitivity $\bar{x}$ is their sum.

### Repeated operands ($x \cdot x$)
In $n_1 = x \cdot x$, use independent operand slots $a = x$ and $b = x$:
$$\frac{\partial(a \cdot b)}{\partial a} = b = x = 3, \qquad \frac{\partial(a \cdot b)}{\partial b} = a = x = 3.$$
With incoming adjoint $\bar n_1 = 4$, the two contributions are
$c_{n_1\to x,1}=12$ and $c_{n_1\to x,2}=12$. Their sum establishes $\bar x=24$.
The slot letters explain which occurrence is differentiated; they are not new graph quantities.

### Constants
Literal numbers like `2` or `-0.5` have zero sensitivity with respect to parameters. They participate in forward arithmetic but emit no backward branches or adjoint compartments.

### Stop-gradient boundaries (`nograd`)
Wrapping an expression in `nograd(u)` computes its value normally in the forward pass ($v = u$), but its mathematical derivative is structurally zero:
$$\frac{\partial\, \text{nograd}(u)}{\partial u} \equiv 0$$
`nograd` acts as a complete boundary for backpropagation: no derivative trapezoid or contribution arrow is constructed toward $u$.

---

## 5. Vectors, Matrices, and Vector-Jacobian Products (VJPs)

The diagram and walkthrough show the same two lines: the symbolic contribution
rule above the divider and its numerical application below. Scalar steps instead
show the operation's local derivative above the divider, followed by the named
chain-rule contribution below.

For vector- and matrix-valued operations, forming full Jacobian matrices would be memory-prohibitive. Reverse-mode AD computes **Vector-Jacobian Products (VJPs)** directly.

* **Scalar rule $\to$ Tensor rule**:
  Instead of $\bar{v} \cdot \frac{\partial v}{\partial u}$, the contribution is:
  $$c_{\mathbf{v} \to \mathbf{u}} = \mathbf{J}^T \bar{\mathbf{v}}$$
  where $\mathbf{J}$ is the Jacobian of the operation.
* **Matrix multiplication**:
  For $\mathbf{a} = \mathbf{W}\mathbf{x}$ with incoming adjoint $\bar{\mathbf{a}}$:
  $$c_{\mathbf{a} \to \mathbf{x}} = \mathbf{W}^T \bar{\mathbf{a}}, \qquad c_{\mathbf{a} \to \mathbf{W}} = \bar{\mathbf{a}} \mathbf{x}^T$$
* **Element-wise activations**:
  For $\mathbf{a} = \sigma(\mathbf{z})$:
  $$c_{\mathbf{a} \to \mathbf{z}} = \bar{\mathbf{a}} \odot \sigma'(\mathbf{z})$$
* **Softmax and selections**:
  Selecting a class probability $p_k = \text{softmax}(\mathbf{z})_k$ sends a gradient back through the softmax function, creating nonzero sensitivity across all class logits $\mathbf{z}$.

---

## 6. Notation and Visual Reference

### Summary of quantities and symbols

| Concept | Mathbook notation | Code notation | Role & Meaning |
| :--- | :--- | :--- | :--- |
| **Primal value** | $n_1 = 9$ | `n1 = 9` | Value computed in the forward pass |
| **Accumulated adjoint** | $\bar{x} = 24$ | `x_bar = 24` | Total derivative $\partial\text{out}/\partial x$ |
| **Local derivative** | $\frac{\partial v}{\partial u} = x$ | `diff(v, u) = x` | Derivative of immediate operation w.r.t operand |
| **Contribution** | $c_{v \to u} = 12$ | `c_v_to_u = 12` | Chain-rule product $\bar{v} \cdot \frac{\partial v}{\partial u}$ |
| **Accumulation** | $\Sigma$ | `sum` | Sums multiple incoming contributions |
| **Seed adjoint** | $\bar{\text{out}} = 1$ | `out_bar = 1` | Base case $\partial\text{out}/\partial\text{out} = 1$ |

### Visual roles in cards and diagrams

* **Mint / Green**: Forward computations, primal values, and forward data-flow arrows.
* **Violet**: Established adjoints stored in node lower compartments and introduction arrows.
* **Plum / Magenta**: Individual branch gradient contributions and contribution expressions.
* **Rose**: Local derivative transformation rules.
* **Gray**: Literal constants.

---

## 7. Supported Expressions & Input Syntax

The visualizer includes both a **scalar calculus engine** and a **vector/matrix machine learning engine**.

### Operators and functions

| Engine | Operators / Functions | Notes & Syntax |
| :--- | :--- | :--- |
| **Both** | `+`, `-`, `*`, `/`, unary `-` | Standard arithmetic; array operations are element-wise. |
| **Both** | `**` | Powers ($x^y$ for scalars, integer/float powers like `v**2` for vectors). |
| **Both** | `exp`, `log`, `tanh` | Natural exponential, natural logarithm, hyperbolic tangent. |
| **Both** | `nograd(x)` | Stops backward gradient propagation through $x$. |
| **Scalar only** | `sin`, `cos`, `tan` | Trigonometric functions (radians). |
| **Scalar only** | `asin`, `acos`, `atan` | Inverse trigonometric functions. |
| **Scalar only** | `sinh`, `cosh` | Hyperbolic sine and cosine. |
| **Scalar only** | `sqrt`, `ln`, `abs` | Square root, natural log, absolute value. |
| **Tensor only** | `@` | Matrix multiplication, matrix-vector product, vector dot product. |
| **Tensor only** | `dot(u, v)` | Dot product of two vectors of equal length. |
| **Tensor only** | `sum(x)`, `mean(x)`, `max(x)`, `min(x)` | Full reduction over all entries to a scalar. |
| **Tensor only** | `sigmoid(x)`, `relu(x)` | Common neural network activation functions. |
| **Tensor only** | `softmax(v)` | Softmax probability distribution over vector logits. |
| **Tensor only** | `v[i]`, `pick(v, i)` | Index a vector entry (`0` is first, `-1` is last). |

### Input value syntax
Enter comma-separated assignments in the **Input Values** box:
* **Scalars**: `x = 3.0, y = 4.0`
* **Vectors**: `v = [1.0, -2.0, 3.0], b = 0.5`
* **Matrices**: `W = [[1.0, 2.0], [3.0, 4.0]], x = [0.5, -1.0]`

> **Scalar Output Requirement**: The overall formula must evaluate to a single scalar (e.g. via `sum(...)`, `mean(...)`, `dot(...)`, indexing `v[0]`, or scalar operations), because reverse-mode autodiff propagates sensitivities from a scalar root.

---

## 8. Using the Examples

The **Examples** tab provides curated models ranging from introductory toy expressions to real-world machine learning layers:
* **Toy examples**: Inspect scalar product rules, repeated variable accumulations, vector reductions, and tied max pooling.
* **ML examples**: Explore linear regressions, logistic regression neurons, binary cross-entropy, softmax classification losses, and multi-head attention blocks.
* **No auto-execution**: Selecting an example loads its formula and numerical assignments into the editor and displays an overview of what the example demonstrates. You can inspect or modify the inputs before clicking **Generate Diagram** or **Generate Walkthrough**.

---

## 9. Exporting Diagrams

You can save and export the computational graph directly from the **Export** panel below the visual diagram:
* **PDF**: Vector graphic ideal for slides (PowerPoint, Keynote, LaTeX Beamer), documents, and printing.
* **PNG**: High-resolution raster image suitable for web pages and digital notes.
* **SVG**: Scalable vector format suitable for web publishing and vector editing.
* **Mermaid (.mmd)**: Raw diagram markup file for offline rendering or archiving.

To inspect or copy the raw diagram markup directly to your clipboard, expand the **Mermaid Source** panel below the export buttons.

---

## 10. Using the Python / Notebook API

The application exposes clean, programmatic API endpoints that allow you to generate diagrams and export files directly inside Jupyter notebooks, Google Colab, or automated scripts using `gradio_client`:

```bash
pip install gradio_client
```

### Displaying a Diagram in a Jupyter Notebook

```python
from gradio_client import Client
from IPython.display import Image, display

# Connect to the Hugging Face Space (or local "http://127.0.0.1:7860")
client = Client("simon-clmtd/autodiff-visualizer")

# Generate high-resolution PNG
png_path = client.predict(
    expr="0.5 * sum((W @ x + b - y)**2)",
    vals="W=[[0.5, -0.2], [-0.1, 0.8]], x=[1, 2], b=[0.1, -0.1], y=[0, 1]",
    mode="Forward & Backward",
    notation="∂(...) / ∂x [Mathbook]",
    api_name="/generate_png"
)

# Display inline
display(Image(png_path))
```

### Available API Endpoints

| Endpoint | Inputs | Return Type | Description |
| :--- | :--- | :--- | :--- |
| `/generate_png` | `expr`, `vals`, `mode`, `notation` | `str` (file path) | Generates and downloads a high-res PNG file. |
| `/generate_svg` | `expr`, `vals`, `mode`, `notation` | `str` (file path) | Generates and downloads a scalable vector SVG file. |
| `/generate_pdf` | `expr`, `vals`, `mode`, `notation` | `str` (file path) | Generates and downloads a vector/printable PDF file. |
| `/generate_mermaid` | `expr`, `vals`, `mode`, `notation` | `str` | Returns the raw Mermaid `.mmd` string. |
| `/generate_walkthrough` | `expr`, `vals`, `mode`, `notation` | `str` (HTML) | Returns the step-by-step trace walkthrough HTML. |