USER_GUIDE.md

Pedagogical Reference & User Guide

Welcome to the Autodiff Visualizer. This guide explains how to read the visual diagram and step-by-step walkthrough, the mathematical concepts of reverse-mode automatic differentiation, notation conventions, and supported mathematical syntax.

---

1. Reading the Visualization

The visual diagram represents numerical computations as a directed graph organized into three structural regions:

• Inputs (bottom): Independent input variables and constant values.
• Operations (middle): Intermediate calculations, local backward transformations, and gradient accumulations.
• Output (top): The final scalar computation and seed adjoint introduction.

Nodes store; arrows carry

A core principle of the visual grammar is: Nodes store computational state; arrows carry information in transit.

• Quantity nodes (two compartments): Every differentiable quantity (inputs, intermediates, and final output) displays:
  • Upper compartment (mint): Stored primal value computed during the forward pass ($v$).
  • Lower compartment (pale violet): Accumulated adjoint ($\bar{v}$) representing total sensitivity of the final output with respect to $v$.
  • In Forward Only mode, the lower compartment and divider are omitted.
• Constants (gray single-compartment): Literal constants (e.g. 2, 0.5) participate only in forward computation and do not have an adjoint compartment.
• Arrows in transit:
  • Solid green arrows: Carry forward primal values upward toward consuming operations.
  • Dotted violet arrows: Carry established adjoints downward or introduce accumulated adjoints to a quantity.
  • Dotted plum arrows: Carry individual gradient contributions toward accumulation ($\Sigma$) nodes.

Local backward steps & accumulation

When backpropagating from a consumer to an operand:

1. Derivative trapezoid (rose): Consumes an incoming adjoint and applies the local derivative rule to produce one numerical contribution.
2. Summation node ($\Sigma$): When an operand is used multiple times (such as $x \cdot x$) or across multiple operations, its incoming contributions gather at an explicit $\Sigma$ node before introducing the final accumulated adjoint. Unary accumulation (a single contribution) is visually elided directly into the quantity.

---

2. Reading the Step-by-Step Walkthrough

The Step-by-Step Walkthrough linearizes the execution of the computation into sequential explanatory cards.

The two box types

1. Primal box (Forward pass): $$\boxed{n_1 = x \cdot x = 3 \cdot 3 = 9}$$ Read from left to right: $$\text{quantity} = \text{computation} = \text{values used} = \text{result}$$ Each forward step shows the operation formula, substitutes the evaluated operand values, and establishes the resulting primal quantity.

2. Adjoint box (Backward pass): $$\bar{n}_1 = \frac{\partial \text{out}}{\partial n_1} = 4$$ $$\bar{n}_1\text{ tells us how sensitive the final output is to }n_1\text{, including all downstream paths.}$$ Each backward card begins with the quantity's established adjoint.

3. Derivative and contribution pairs: For each operand contributing to that adjoint, the walkthrough details: $$\text{Local derivative rule:}\quad \frac{\partial (u \cdot v)}{\partial u} = v = 4$$ $$\text{Contribution:}\quad c_{n \to u} = \bar{n} \cdot v = 1 \cdot 4 = 4$$ The first equation gives the local symbolic derivative with substituted values. The second line shows the chain-rule product (incoming adjoint $\times$ local derivative) that yields the contribution.

Headlines vs. Everything

• Headlines (Overview): Summarizes the key inputs, the final output, and the overall input sensitivities ($\bar{x} = \partial\text{out}/\partial x$).
• Everything (Full trace): Expands all intermediate computational steps, local derivative rules, individual branch contributions, and step-by-step additions.

---

3. Core Concepts of Reverse-Mode AD

Automatic differentiation decomposes complex mathematical expressions into primitive operations whose local derivatives are known.

1. Primal evaluation (Forward pass): $$v = f(u_1, u_2, \dots)$$ Evaluates each operation from inputs to the scalar root output ($\text{out}$).

2. Seed adjoint: The backward pass always begins at the scalar output with: $$\bar{\text{out}} = \frac{\partial \text{out}}{\partial \text{out}} = 1$$

3. Gradient contribution: When an operation $v = f(u)$ receives an adjoint $\bar{v}$ from its consumers, it transmits a contribution to operand $u$ by the chain rule: $$c_{v \to u} = \bar{v} \cdot \frac{\partial v}{\partial u}$$

4. Accumulation of adjoints: By the multivariate chain rule, if a quantity $u$ contributes to multiple downstream computations (or appears multiple times as an operand), its total adjoint is the sum of all incoming contributions: $$\bar{u} = \sum_{v \in \text{Consumers}(u)} c_{v \to u}$$ Reverse-mode AD computes exact derivatives for all inputs in a single backward pass, regardless of how many input variables exist.

---

4. Special Cases

Single vs. multiple contributions

• Single contribution: When a variable is used only once, its single incoming contribution directly becomes its adjoint.
• Multiple contributions: When a variable is reused across different operations (e.g. $x \cdot y + x$) or repeated in the same operation (e.g. $x \cdot x$), each use produces a separate branch and contribution. The total sensitivity $\bar{x}$ is their sum.

Repeated operands ($x \cdot x$)

In $n_1 = x \cdot x$, use independent operand slots $a = x$ and $b = x$: $$\frac{\mathrm{\partial }(a\cdot b)}{\mathrm{\partial }a}=b=x=3,\frac{\mathrm{\partial }(a\cdot b)}{\mathrm{\partial }b}=a=x=3.\backslash frac\{\backslash partial(a\; \backslash cdot\; b)\}\{\backslash partial\; a\}\; =\; b\; =\; x\; =\; 3,\; \backslash qquad\; \backslash frac\{\backslash partial(a\; \backslash cdot\; b)\}\{\backslash partial\; b\}\; =\; a\; =\; x\; =\; 3.$$ With incoming adjoint $\bar n_1 = 4$, the two contributions are $c_{n_1\to x,1}=12$ and $c_{n_1\to x,2}=12$. Their sum establishes $\bar x=24$. The slot letters explain which occurrence is differentiated; they are not new graph quantities.

Constants

Literal numbers like 2 or -0.5 have zero sensitivity with respect to parameters. They participate in forward arithmetic but emit no backward branches or adjoint compartments.

Stop-gradient boundaries (nograd)

Wrapping an expression in nograd(u) computes its value normally in the forward pass ($v = u$), but its mathematical derivative is structurally zero: $$\frac{\mathrm{\partial }\text{nograd}(u)}{\mathrm{\partial }u}\equiv 0\backslash frac\{\backslash partial\backslash ,\; \backslash text\{nograd\}(u)\}\{\backslash partial\; u\}\; \backslash equiv\; 0$$ nograd acts as a complete boundary for backpropagation: no derivative trapezoid or contribution arrow is constructed toward $u$.

---

5. Vectors, Matrices, and Vector-Jacobian Products (VJPs)

The diagram and walkthrough show the same two lines: the symbolic contribution rule above the divider and its numerical application below. Scalar steps instead show the operation's local derivative above the divider, followed by the named chain-rule contribution below.

For vector- and matrix-valued operations, forming full Jacobian matrices would be memory-prohibitive. Reverse-mode AD computes Vector-Jacobian Products (VJPs) directly.

• Scalar rule $\to$ Tensor rule: Instead of $\bar{v} \cdot \frac{\partial v}{\partial u}$, the contribution is: $$c_{\mathbf{v} \to \mathbf{u}} = \mathbf{J}^T \bar{\mathbf{v}}$$ where $\mathbf{J}$ is the Jacobian of the operation.
• Matrix multiplication: For $\mathbf{a} = \mathbf{W}\mathbf{x}$ with incoming adjoint $\bar{\mathbf{a}}$: $$c_{\mathbf{a} \to \mathbf{x}} = \mathbf{W}^T \bar{\mathbf{a}}, \qquad c_{\mathbf{a} \to \mathbf{W}} = \bar{\mathbf{a}} \mathbf{x}^T$$
• Element-wise activations: For $\mathbf{a} = \sigma(\mathbf{z})$: $$c_{\mathbf{a} \to \mathbf{z}} = \bar{\mathbf{a}} \odot \sigma'(\mathbf{z})$$
• Softmax and selections: Selecting a class probability $p_k = \text{softmax}(\mathbf{z})_k$ sends a gradient back through the softmax function, creating nonzero sensitivity across all class logits $\mathbf{z}$.

---

6. Notation and Visual Reference

Summary of quantities and symbols

Concept	Mathbook notation	Code notation	Role & Meaning
Primal value	$n_1 = 9$	n1 = 9	Value computed in the forward pass
Accumulated adjoint	$\bar{x} = 24$	x_bar = 24	Total derivative $\partial\text{out}/\partial x$
Local derivative	$\frac{\partial v}{\partial u} = x$	diff(v, u) = x	Derivative of immediate operation w.r.t operand
Contribution	$c_{v \to u} = 12$	c_v_to_u = 12	Chain-rule product $\bar{v} \cdot \frac{\partial v}{\partial u}$
Accumulation	$\Sigma$	sum	Sums multiple incoming contributions
Seed adjoint	$\bar{\text{out}} = 1$	out_bar = 1	Base case $\partial\text{out}/\partial\text{out} = 1$

Visual roles in cards and diagrams

• Mint / Green: Forward computations, primal values, and forward data-flow arrows.
• Violet: Established adjoints stored in node lower compartments and introduction arrows.
• Plum / Magenta: Individual branch gradient contributions and contribution expressions.
• Rose: Local derivative transformation rules.
• Gray: Literal constants.

---

7. Supported Expressions & Input Syntax

The visualizer includes both a scalar calculus engine and a vector/matrix machine learning engine.

Operators and functions

Engine	Operators / Functions	Notes & Syntax
Both	+, -, *, /, unary -	Standard arithmetic; array operations are element-wise.
Both	**	Powers ($x^y$ for scalars, integer/float powers like v**2 for vectors).
Both	exp, log, tanh	Natural exponential, natural logarithm, hyperbolic tangent.
Both	nograd(x)	Stops backward gradient propagation through $x$.
Scalar only	sin, cos, tan	Trigonometric functions (radians).
Scalar only	asin, acos, atan	Inverse trigonometric functions.
Scalar only	sinh, cosh	Hyperbolic sine and cosine.
Scalar only	sqrt, ln, abs	Square root, natural log, absolute value.
Tensor only	@	Matrix multiplication, matrix-vector product, vector dot product.
Tensor only	dot(u, v)	Dot product of two vectors of equal length.
Tensor only	sum(x), mean(x), max(x), min(x)	Full reduction over all entries to a scalar.
Tensor only	sigmoid(x), relu(x)	Common neural network activation functions.
Tensor only	softmax(v)	Softmax probability distribution over vector logits.
Tensor only	v[i], pick(v, i)	Index a vector entry (0 is first, -1 is last).

Input value syntax

Enter comma-separated assignments in the Input Values box:

• Scalars: x = 3.0, y = 4.0
• Vectors: v = [1.0, -2.0, 3.0], b = 0.5
• Matrices: W = [[1.0, 2.0], [3.0, 4.0]], x = [0.5, -1.0]

Scalar Output Requirement: The overall formula must evaluate to a single scalar (e.g. via sum(...), mean(...), dot(...), indexing v[0], or scalar operations), because reverse-mode autodiff propagates sensitivities from a scalar root.

---

8. Using the Examples

The Examples tab provides curated models ranging from introductory toy expressions to real-world machine learning layers:

• Toy examples: Inspect scalar product rules, repeated variable accumulations, vector reductions, and tied max pooling.
• ML examples: Explore linear regressions, logistic regression neurons, binary cross-entropy, softmax classification losses, and multi-head attention blocks.
• No auto-execution: Selecting an example loads its formula and numerical assignments into the editor and displays an overview of what the example demonstrates. You can inspect or modify the inputs before clicking Generate Diagram or Generate Walkthrough.

---

9. Exporting Diagrams

You can save and export the computational graph directly from the Export panel below the visual diagram:

• PDF: Vector graphic ideal for slides (PowerPoint, Keynote, LaTeX Beamer), documents, and printing.
• PNG: High-resolution raster image suitable for web pages and digital notes.
• SVG: Scalable vector format suitable for web publishing and vector editing.
• Mermaid (.mmd): Raw diagram markup file for offline rendering or archiving.

To inspect or copy the raw diagram markup directly to your clipboard, expand the Mermaid Source panel below the export buttons.

---

10. Using the Python / Notebook API

The application exposes clean, programmatic API endpoints that allow you to generate diagrams and export files directly inside Jupyter notebooks, Google Colab, or automated scripts using gradio_client:

pip install gradio_client

Displaying a Diagram in a Jupyter Notebook

from gradio_client import Client
from IPython.display import Image, display

# Connect to the Hugging Face Space (or local "http://127.0.0.1:7860")
client = Client("simon-clmtd/autodiff-visualizer")

# Generate high-resolution PNG
png_path = client.predict(
    expr="0.5 * sum((W @ x + b - y)**2)",
    vals="W=[[0.5, -0.2], [-0.1, 0.8]], x=[1, 2], b=[0.1, -0.1], y=[0, 1]",
    mode="Forward & Backward",
    notation="∂(...) / ∂x [Mathbook]",
    api_name="/generate_png"
)

# Display inline
display(Image(png_path))

Available API Endpoints

Endpoint	Inputs	Return Type	Description
/generate_png	expr, vals, mode, notation	str (file path)	Generates and downloads a high-res PNG file.
/generate_svg	expr, vals, mode, notation	str (file path)	Generates and downloads a scalable vector SVG file.
/generate_pdf	expr, vals, mode, notation	str (file path)	Generates and downloads a vector/printable PDF file.
/generate_mermaid	expr, vals, mode, notation	str	Returns the raw Mermaid .mmd string.
/generate_walkthrough	expr, vals, mode, notation	str (HTML)	Returns the step-by-step trace walkthrough HTML.