# Pulpie WebGPU Technical Overview

This document describes the architecture, pipeline stages, and execution flow of the Pulpie WebGPU web application.

## High-Level Architecture

Pulpie WebGPU runs the 210M-parameter EuroBERT token classification model entirely inside the user's browser via WebGPU. It eliminates the need for server-side Python backends, CUDA installations, or remote inference APIs.

```text
+-------------------------------------------------------------------------+
|                              Browser Window                             |
|                                                                         |
|  [Input: URL or HTML]                                                   |
|           |                                                             |
|           v                                                             |
|  [DOMParser & Sanitization] -> Cleans scripts, styles, metadata         |
|           |                                                             |
|           v                                                             |
|  [Block Extractor] ----------> Tags semantic blocks with _item_id       |
|           |                                                             |
|           v                                                             |
|  [EuroBERT Tokenizer] -------> Encodes block text, injects <|sep|>      |
|           |                                                             |
|           v                                                             |
|  [Chunk Packer] -------------> Bundles blocks up to 1,024-2,048 tokens |
|           |                                                             |
|           v                                                             |
|  [WebGPU ONNX Runtime] ------> Executes model.onnx via FP32 shaders      |
|           |                                                             |
|           v                                                             |
|  [Logit Evaluator] ----------> Argmax at <|sep|> indices (1=main, 0=drop) |
|           |                                                             |
|           v                                                             |
|  [DOM Reconstructor] --------> Retains kept blocks, drops boilerplate   |
|           |                                                             |
|           v                                                             |
|  [Turndown Service] ---------> Converts cleaned HTML to Markdown        |
|           |                                                             |
|           v                                                             |
|  [Output Views] -------------> Reader Preview, Markdown, Clean HTML     |
+-------------------------------------------------------------------------+
```

## Core Pipeline Stages

### 1. HTML Simplification and Block Segmentation (`html-simplifier.js`)
* Uses browser-native `DOMParser` to build a clean document object model.
* Strips non-content elements: `script`, `style`, `noscript`, `iframe`, `frame`, `meta`, `link`, and hidden CSS nodes (`display: none`, `visibility: hidden`).
* Rewrites relative `src` and `href` attributes to absolute URLs against the page's base origin so that images continue rendering properly in Markdown.
* Traverses the DOM tree to extract semantic content blocks: headings (`h1` through `h6`), paragraphs (`p`), preformatted code (`pre`), blockquotes, list items (`li`), and tables.
* Assigns an incremental `_item_id` attribute (e.g. `_item_id="0"`) to each extracted block.

### 2. Tokenization and Chunking (`tokenizer.js` & `chunker.js`)
* Uses `@huggingface/transformers` to run the EuroBERT fast tokenizer directly in JavaScript.
* Identifies boundary token IDs:
  * BOS: `<|begin_of_text|>` (128000)
  * EOS: `<|end_of_text|>` (128001)
  * SEP: `<|sep|>` (128256)
* Packs blocks into contiguous chunks conforming to the 8,192 token limit:
  `[BOS] block_0 [<|sep|>] block_1 [<|sep|>] ... block_N [<|sep|>] [EOS]`
* Keeps a parallel array tracking which original block indices correspond to each chunk.

### 3. WebGPU Neural Inference (`webgpu-engine.js`)
* Evaluates `navigator.gpu` and requests an adapter using the `high-performance` power preference.
* Verifies whether the hardware supports the `shader-f16` WGSL extension.
* Prepares two 64-bit integer tensors for ONNX Runtime:
  * `input_ids`: Shape `[1, sequence_length]`, containing the packed token IDs.
  * `attention_mask`: Shape `[1, sequence_length]`, initialized to ones.
* Runs the model forward pass on the GPU:
  `const outputMap = await session.run({ input_ids, attention_mask });`
* Returns a `Float32Array` of logits with shape `[1, sequence_length, 2]`.

### 4. Delimiter Logit Evaluation
* Finds the exact position of every `<|sep|>` token in the sequence.
* For each delimiter position `p`:
  * Evaluates `logitMain = logits[p * 2 + 1]` versus `logitOther = logits[p * 2 + 0]`.
  * If `logitMain > logitOther`, the block is classified as primary content (`main`).
  * Otherwise, the block is classified as extraneous boilerplate (`other`).

### 5. Content Reconstruction and Markdown Conversion (`reconstructor.js`)
* Clones the tagged DOM and identifies all elements tagged with an `_item_id` that received a `main` classification.
* Marks those elements, their descendants, and their ancestors up to the document body to be retained.
* Prunes all other elements from the tree.
* Strips zero-sized tracking pixels and 1x1 spacer images.
* Passes the cleaned HTML to a configured `TurndownService` instance that applies GitHub-flavored Markdown rules (fenced code blocks, tables, and images).

### 6. Persistent Model Caching (`model-cache.js`)
* Interacts with the browser's native `CacheStorage` API (`caches.open('pulpie-onnx-v2')`).
* Intercepts downloads of `model.onnx` (808 MB) and streams chunks while updating the user interface progress bar.
* Stores the resulting `Response` in cache so future sessions load in less than 200 milliseconds without network traffic.

## File Organization

| Path | Purpose | Lines of Code |
|---|---|---:|
| `index.html` | Semantic markup, modal dialogs, and layout structure | ~355 |
| `src/main.js` | Event wiring, state coordination, and execution flow | ~474 |
| `src/style.css` | Vercel-style dark charcoal theme and responsive rules | ~974 |
| `src/lib/webgpu-engine.js` | WebGPU device detection, session lifecycle, and tensor execution | ~197 |
| `src/lib/model-cache.js` | Persistent browser cache storage and download progress reporting | ~160 |
| `src/lib/tokenizer.js` | EuroBERT fast tokenizer wrapper and special token management | ~104 |
| `src/lib/chunker.js` | Sequence packing within 1,024-2,048 token context window | ~103 |
| `src/lib/html-simplifier.js` | DOM cleaning, URL resolution, and block segmentation | ~217 |
| `src/lib/reconstructor.js` | Element pruning and HTML to Markdown conversion | ~173 |
| `src/lib/ui-renderer.js` | Results view, stats computation, and block inspection rendering | ~81 |
| src/worker.js | Cloudflare Worker edge handler with /api/proxy and static asset routing | ~80 |
| src/lib/url-fetcher.js | Webpage fetching with dedicated edge proxy and fallback | ~95 |
| `src/lib/samples.js` | Realistic HTML test documents for offline benchmarking | ~135 |