Spaces:
Sleeping
Sleeping
| # Pedagogical Reference & User Guide | |
| Welcome to the **Autodiff Visualizer**. This guide explains how to read the visual diagram and step-by-step walkthrough, the mathematical concepts of reverse-mode automatic differentiation, notation conventions, and supported mathematical syntax. | |
| --- | |
| ## 1. Reading the Visualization | |
| The visual diagram represents numerical computations as a directed graph organized into three structural regions: | |
| * **Inputs (bottom)**: Independent input variables and constant values. | |
| * **Operations (middle)**: Intermediate calculations, local backward transformations, and gradient accumulations. | |
| * **Output (top)**: The final scalar computation and seed adjoint introduction. | |
| ### Nodes store; arrows carry | |
| A core principle of the visual grammar is: **Nodes store computational state; arrows carry information in transit.** | |
| * **Quantity nodes (two compartments)**: | |
| Every differentiable quantity (inputs, intermediates, and final output) displays: | |
| - **Upper compartment (mint)**: Stored primal value computed during the forward pass ($v$). | |
| - **Lower compartment (pale violet)**: Accumulated adjoint ($\bar{v}$) representing total sensitivity of the final output with respect to $v$. | |
| - In *Forward Only* mode, the lower compartment and divider are omitted. | |
| * **Constants (gray single-compartment)**: | |
| Literal constants (e.g. `2`, `0.5`) participate only in forward computation and do not have an adjoint compartment. | |
| * **Arrows in transit**: | |
| - **Solid green arrows**: Carry forward primal values upward toward consuming operations. | |
| - **Dotted violet arrows**: Carry established adjoints downward or introduce accumulated adjoints to a quantity. | |
| - **Dotted plum arrows**: Carry individual gradient contributions toward accumulation ($\Sigma$) nodes. | |
| ### Local backward steps & accumulation | |
| When backpropagating from a consumer to an operand: | |
| 1. **Derivative trapezoid (rose)**: Consumes an incoming adjoint and applies the local derivative rule to produce one numerical contribution. | |
| 2. **Summation node ($\Sigma$)**: When an operand is used multiple times (such as $x \cdot x$) or across multiple operations, its incoming contributions gather at an explicit $\Sigma$ node before introducing the final accumulated adjoint. Unary accumulation (a single contribution) is visually elided directly into the quantity. | |
| --- | |
| ## 2. Reading the Step-by-Step Walkthrough | |
| The **Step-by-Step Walkthrough** linearizes the execution of the computation into sequential explanatory cards. | |
| ### The two box types | |
| 1. **Primal box (Forward pass)**: | |
| $$\boxed{n_1 = x \cdot x = 3 \cdot 3 = 9}$$ | |
| Read from left to right: | |
| $$\text{quantity} = \text{computation} = \text{values used} = \text{result}$$ | |
| Each forward step shows the operation formula, substitutes the evaluated operand values, and establishes the resulting primal quantity. | |
| 2. **Adjoint box (Backward pass)**: | |
| $$\bar{n}_1 = \frac{\partial \text{out}}{\partial n_1} = 4$$ | |
| $$\bar{n}_1\text{ tells us how sensitive the final output is to }n_1\text{, including all downstream paths.}$$ | |
| Each backward card begins with the quantity's established adjoint. | |
| 3. **Derivative and contribution pairs**: | |
| For each operand contributing to that adjoint, the walkthrough details: | |
| $$\text{Local derivative rule:}\quad \frac{\partial (u \cdot v)}{\partial u} = v = 4$$ | |
| $$\text{Contribution:}\quad c_{n \to u} = \bar{n} \cdot v = 1 \cdot 4 = 4$$ | |
| The first equation gives the local symbolic derivative with substituted values. The second line shows the chain-rule product (incoming adjoint $\times$ local derivative) that yields the contribution. | |
| ### Headlines vs. Everything | |
| * **Headlines (Overview)**: Summarizes the key inputs, the final output, and the overall input sensitivities ($\bar{x} = \partial\text{out}/\partial x$). | |
| * **Everything (Full trace)**: Expands all intermediate computational steps, local derivative rules, individual branch contributions, and step-by-step additions. | |
| --- | |
| ## 3. Core Concepts of Reverse-Mode AD | |
| Automatic differentiation decomposes complex mathematical expressions into primitive operations whose local derivatives are known. | |
| 1. **Primal evaluation (Forward pass)**: | |
| $$v = f(u_1, u_2, \dots)$$ | |
| Evaluates each operation from inputs to the scalar root output ($\text{out}$). | |
| 2. **Seed adjoint**: | |
| The backward pass always begins at the scalar output with: | |
| $$\bar{\text{out}} = \frac{\partial \text{out}}{\partial \text{out}} = 1$$ | |
| 3. **Gradient contribution**: | |
| When an operation $v = f(u)$ receives an adjoint $\bar{v}$ from its consumers, it transmits a contribution to operand $u$ by the chain rule: | |
| $$c_{v \to u} = \bar{v} \cdot \frac{\partial v}{\partial u}$$ | |
| 4. **Accumulation of adjoints**: | |
| By the multivariate chain rule, if a quantity $u$ contributes to multiple downstream computations (or appears multiple times as an operand), its total adjoint is the sum of all incoming contributions: | |
| $$\bar{u} = \sum_{v \in \text{Consumers}(u)} c_{v \to u}$$ | |
| Reverse-mode AD computes exact derivatives for all inputs in a single backward pass, regardless of how many input variables exist. | |
| --- | |
| ## 4. Special Cases | |
| ### Single vs. multiple contributions | |
| * **Single contribution**: When a variable is used only once, its single incoming contribution directly becomes its adjoint. | |
| * **Multiple contributions**: When a variable is reused across different operations (e.g. $x \cdot y + x$) or repeated in the same operation (e.g. $x \cdot x$), each use produces a separate branch and contribution. The total sensitivity $\bar{x}$ is their sum. | |
| ### Repeated operands ($x \cdot x$) | |
| In $n_1 = x \cdot x$, use independent operand slots $a = x$ and $b = x$: | |
| $$\frac{\partial(a \cdot b)}{\partial a} = b = x = 3, \qquad \frac{\partial(a \cdot b)}{\partial b} = a = x = 3.$$ | |
| With incoming adjoint $\bar n_1 = 4$, the two contributions are | |
| $c_{n_1\to x,1}=12$ and $c_{n_1\to x,2}=12$. Their sum establishes $\bar x=24$. | |
| The slot letters explain which occurrence is differentiated; they are not new graph quantities. | |
| ### Constants | |
| Literal numbers like `2` or `-0.5` have zero sensitivity with respect to parameters. They participate in forward arithmetic but emit no backward branches or adjoint compartments. | |
| ### Stop-gradient boundaries (`nograd`) | |
| Wrapping an expression in `nograd(u)` computes its value normally in the forward pass ($v = u$), but its mathematical derivative is structurally zero: | |
| $$\frac{\partial\, \text{nograd}(u)}{\partial u} \equiv 0$$ | |
| `nograd` acts as a complete boundary for backpropagation: no derivative trapezoid or contribution arrow is constructed toward $u$. | |
| --- | |
| ## 5. Vectors, Matrices, and Vector-Jacobian Products (VJPs) | |
| The diagram and walkthrough show the same two lines: the symbolic contribution | |
| rule above the divider and its numerical application below. Scalar steps instead | |
| show the operation's local derivative above the divider, followed by the named | |
| chain-rule contribution below. | |
| For vector- and matrix-valued operations, forming full Jacobian matrices would be memory-prohibitive. Reverse-mode AD computes **Vector-Jacobian Products (VJPs)** directly. | |
| * **Scalar rule $\to$ Tensor rule**: | |
| Instead of $\bar{v} \cdot \frac{\partial v}{\partial u}$, the contribution is: | |
| $$c_{\mathbf{v} \to \mathbf{u}} = \mathbf{J}^T \bar{\mathbf{v}}$$ | |
| where $\mathbf{J}$ is the Jacobian of the operation. | |
| * **Matrix multiplication**: | |
| For $\mathbf{a} = \mathbf{W}\mathbf{x}$ with incoming adjoint $\bar{\mathbf{a}}$: | |
| $$c_{\mathbf{a} \to \mathbf{x}} = \mathbf{W}^T \bar{\mathbf{a}}, \qquad c_{\mathbf{a} \to \mathbf{W}} = \bar{\mathbf{a}} \mathbf{x}^T$$ | |
| * **Element-wise activations**: | |
| For $\mathbf{a} = \sigma(\mathbf{z})$: | |
| $$c_{\mathbf{a} \to \mathbf{z}} = \bar{\mathbf{a}} \odot \sigma'(\mathbf{z})$$ | |
| * **Softmax and selections**: | |
| Selecting a class probability $p_k = \text{softmax}(\mathbf{z})_k$ sends a gradient back through the softmax function, creating nonzero sensitivity across all class logits $\mathbf{z}$. | |
| --- | |
| ## 6. Notation and Visual Reference | |
| ### Summary of quantities and symbols | |
| | Concept | Mathbook notation | Code notation | Role & Meaning | | |
| | :--- | :--- | :--- | :--- | | |
| | **Primal value** | $n_1 = 9$ | `n1 = 9` | Value computed in the forward pass | | |
| | **Accumulated adjoint** | $\bar{x} = 24$ | `x_bar = 24` | Total derivative $\partial\text{out}/\partial x$ | | |
| | **Local derivative** | $\frac{\partial v}{\partial u} = x$ | `diff(v, u) = x` | Derivative of immediate operation w.r.t operand | | |
| | **Contribution** | $c_{v \to u} = 12$ | `c_v_to_u = 12` | Chain-rule product $\bar{v} \cdot \frac{\partial v}{\partial u}$ | | |
| | **Accumulation** | $\Sigma$ | `sum` | Sums multiple incoming contributions | | |
| | **Seed adjoint** | $\bar{\text{out}} = 1$ | `out_bar = 1` | Base case $\partial\text{out}/\partial\text{out} = 1$ | | |
| ### Visual roles in cards and diagrams | |
| * **Mint / Green**: Forward computations, primal values, and forward data-flow arrows. | |
| * **Violet**: Established adjoints stored in node lower compartments and introduction arrows. | |
| * **Plum / Magenta**: Individual branch gradient contributions and contribution expressions. | |
| * **Rose**: Local derivative transformation rules. | |
| * **Gray**: Literal constants. | |
| --- | |
| ## 7. Supported Expressions & Input Syntax | |
| The visualizer includes both a **scalar calculus engine** and a **vector/matrix machine learning engine**. | |
| ### Operators and functions | |
| | Engine | Operators / Functions | Notes & Syntax | | |
| | :--- | :--- | :--- | | |
| | **Both** | `+`, `-`, `*`, `/`, unary `-` | Standard arithmetic; array operations are element-wise. | | |
| | **Both** | `**` | Powers ($x^y$ for scalars, integer/float powers like `v**2` for vectors). | | |
| | **Both** | `exp`, `log`, `tanh` | Natural exponential, natural logarithm, hyperbolic tangent. | | |
| | **Both** | `nograd(x)` | Stops backward gradient propagation through $x$. | | |
| | **Scalar only** | `sin`, `cos`, `tan` | Trigonometric functions (radians). | | |
| | **Scalar only** | `asin`, `acos`, `atan` | Inverse trigonometric functions. | | |
| | **Scalar only** | `sinh`, `cosh` | Hyperbolic sine and cosine. | | |
| | **Scalar only** | `sqrt`, `ln`, `abs` | Square root, natural log, absolute value. | | |
| | **Tensor only** | `@` | Matrix multiplication, matrix-vector product, vector dot product. | | |
| | **Tensor only** | `dot(u, v)` | Dot product of two vectors of equal length. | | |
| | **Tensor only** | `sum(x)`, `mean(x)`, `max(x)`, `min(x)` | Full reduction over all entries to a scalar. | | |
| | **Tensor only** | `sigmoid(x)`, `relu(x)` | Common neural network activation functions. | | |
| | **Tensor only** | `softmax(v)` | Softmax probability distribution over vector logits. | | |
| | **Tensor only** | `v[i]`, `pick(v, i)` | Index a vector entry (`0` is first, `-1` is last). | | |
| ### Input value syntax | |
| Enter comma-separated assignments in the **Input Values** box: | |
| * **Scalars**: `x = 3.0, y = 4.0` | |
| * **Vectors**: `v = [1.0, -2.0, 3.0], b = 0.5` | |
| * **Matrices**: `W = [[1.0, 2.0], [3.0, 4.0]], x = [0.5, -1.0]` | |
| > **Scalar Output Requirement**: The overall formula must evaluate to a single scalar (e.g. via `sum(...)`, `mean(...)`, `dot(...)`, indexing `v[0]`, or scalar operations), because reverse-mode autodiff propagates sensitivities from a scalar root. | |
| --- | |
| ## 8. Using the Examples | |
| The **Examples** tab provides curated models ranging from introductory toy expressions to real-world machine learning layers: | |
| * **Toy examples**: Inspect scalar product rules, repeated variable accumulations, vector reductions, and tied max pooling. | |
| * **ML examples**: Explore linear regressions, logistic regression neurons, binary cross-entropy, softmax classification losses, and multi-head attention blocks. | |
| * **No auto-execution**: Selecting an example loads its formula and numerical assignments into the editor and displays an overview of what the example demonstrates. You can inspect or modify the inputs before clicking **Generate Diagram** or **Generate Walkthrough**. | |
| --- | |
| ## 9. Exporting Diagrams | |
| You can save and export the computational graph directly from the **Export** panel below the visual diagram: | |
| * **PDF**: Vector graphic ideal for slides (PowerPoint, Keynote, LaTeX Beamer), documents, and printing. | |
| * **PNG**: High-resolution raster image suitable for web pages and digital notes. | |
| * **SVG**: Scalable vector format suitable for web publishing and vector editing. | |
| * **Mermaid (.mmd)**: Raw diagram markup file for offline rendering or archiving. | |
| To inspect or copy the raw diagram markup directly to your clipboard, expand the **Mermaid Source** panel below the export buttons. | |
| --- | |
| ## 10. Using the Python / Notebook API | |
| The application exposes clean, programmatic API endpoints that allow you to generate diagrams and export files directly inside Jupyter notebooks, Google Colab, or automated scripts using `gradio_client`: | |
| ```bash | |
| pip install gradio_client | |
| ``` | |
| ### Displaying a Diagram in a Jupyter Notebook | |
| ```python | |
| from gradio_client import Client | |
| from IPython.display import Image, display | |
| # Connect to the Hugging Face Space (or local "http://127.0.0.1:7860") | |
| client = Client("simon-clmtd/autodiff-visualizer") | |
| # Generate high-resolution PNG | |
| png_path = client.predict( | |
| expr="0.5 * sum((W @ x + b - y)**2)", | |
| vals="W=[[0.5, -0.2], [-0.1, 0.8]], x=[1, 2], b=[0.1, -0.1], y=[0, 1]", | |
| mode="Forward & Backward", | |
| notation="∂(...) / ∂x [Mathbook]", | |
| api_name="/generate_png" | |
| ) | |
| # Display inline | |
| display(Image(png_path)) | |
| ``` | |
| ### Available API Endpoints | |
| | Endpoint | Inputs | Return Type | Description | | |
| | :--- | :--- | :--- | :--- | | |
| | `/generate_png` | `expr`, `vals`, `mode`, `notation` | `str` (file path) | Generates and downloads a high-res PNG file. | | |
| | `/generate_svg` | `expr`, `vals`, `mode`, `notation` | `str` (file path) | Generates and downloads a scalable vector SVG file. | | |
| | `/generate_pdf` | `expr`, `vals`, `mode`, `notation` | `str` (file path) | Generates and downloads a vector/printable PDF file. | | |
| | `/generate_mermaid` | `expr`, `vals`, `mode`, `notation` | `str` | Returns the raw Mermaid `.mmd` string. | | |
| | `/generate_walkthrough` | `expr`, `vals`, `mode`, `notation` | `str` (HTML) | Returns the step-by-step trace walkthrough HTML. | | |