# System Architecture This document provides a comprehensive overview of the Visual Notebook system architecture, including the high-level design, data flow, core abstractions, and technology decisions. ## High-Level Architecture The Visual Notebook follows a classic client-server architecture with a clear separation between the Python backend and the React frontend. The backend serves both the REST API and the compiled frontend assets in production deployments. ``` +----------------------------------------------------------+ | Browser | | | | +-------------------+ +------------------------------+ | | | React + TS | | Zustand Stores | | | | Components |<-| notebook, session, ui, hist | | | +-------------------+ +------------------------------+ | | | ^ | | | Axios / REST | | | v | | +----------------------------------------------------------+ | | v | +----------------------------------------------------------+ | FastAPI Backend | | | | +------------+ +----------+ +---------------------+ | | | API Routes |->| Deps.py |->| Session (cookie) | | | +------------+ +----------+ +---------------------+ | | | | | | | v v v | | +------------+ +----------+ +---------------------+ | | | Pydantic | | Domain | | DuckDB Connection | | | | Schemas | | Layer | | (per session) | | | +------------+ +----------+ +---------------------+ | | | | | +------------+------------+ | | v v v | | +----------+ +----------+ +-----------+ | | |DataFrame | | History | | Viz | | | |Ref | | Tree | | Comput. | | | +----------+ +----------+ +-----------+ | | | | | | | v v v | | +-------+ +--------+ +--------+ | | |DuckDB | | Polars | | DuckDB | | | |(SQL) | |(Lazy) | | (SQL) | | | +-------+ +--------+ +--------+ | | | | +---------------------------------------------------+ | | | Checkpoint Store (Parquet + JSON on disk) | | | +---------------------------------------------------+ | +----------------------------------------------------------+ ``` ## Data Flow The primary data flow follows a pipeline from raw file upload through to interactive visualization: 1. **CSV Upload**: The user uploads a CSV file via the frontend. The file is sent as multipart form data to `POST /api/upload/csv`. 2. **Disk Persistence**: The backend saves the file to the configured `DATA_DIR` with a unique dataset ID prefix to avoid name collisions. 3. **DuckDB Registration**: The saved file is loaded into DuckDB using `read_csv_auto`, which performs automatic type inference, delimiter detection, and header parsing. The data is stored as an in-memory DuckDB table. 4. **Schema Inference**: The `SUMMARIZE` command is run on the registered table to extract column statistics including min, max, mean, standard deviation, quartiles, null percentage, and approximate unique count. Column types are classified into one of six categories (numeric, categorical, datetime, boolean, text, geo_lat_lon). 5. **DataFrameRef Creation**: A `DataFrameRef` object wrapping the DuckDB table is created and stored in the session. This serves as the unified reference for all subsequent operations. 6. **Operations**: When the user applies a transformation (filter, sort, select, rename, groupby, fill_nulls, clip), the operation produces a new `DataFrameRef`. If the operation uses the Polars backend, the result is materialized back into DuckDB for consistent downstream querying. 7. **Visualization**: Chart computation reads directly from DuckDB tables using SQL queries. The computed ECharts specification is returned as a JSON object that the frontend renders without further data movement. ## Session Model Each user interaction is bound to a session object. Sessions are identified by a UUID stored in an HTTP-only cookie named `session_id` with a 7-day expiration. - **SessionStore**: An in-memory dictionary mapping session IDs to `Session` objects. On each request, the `get_session` dependency reads the cookie, looks up the session, or creates a new one if the cookie is absent or invalid. - **Session Data**: Each session holds a DuckDB connection, the current `DataFrameRef`, a `History` tree, and the current table name. The DuckDB connection is shared across all operations within the session, providing transactional consistency. - **Isolation**: Because each session has its own DuckDB connection and table namespace (tables are prefixed with `ds_` followed by a unique hex identifier), multiple concurrent users cannot interfere with each other's data. ## Backend Double-Engine Strategy The backend uses both DuckDB and Polars as computation engines, each chosen for its strengths in specific scenarios: | Engine | Strength | Use Cases | |--------|----------|-----------| | DuckDB | SQL-heavy analytics, aggregation, JOINs, window functions | Schema inference (SUMMARIZE), chart computation, CSV loading, export | | Polars | CPU-parallel lazy transformations, type-safe expressions | Filter, sort, select, rename, groupby, fill_nulls, clip operations | When an operation is applied using the Polars backend, the resulting `DataFrameRef` is automatically materialized back into a DuckDB table. This ensures that subsequent operations, schema inference, and chart computation always have a DuckDB table to query. The materialization uses Apache Arrow as the interchange format, which avoids serialization overhead between Polars and DuckDB. ## DataFrameRef: The Unified Abstraction `DataFrameRef` is a dataclass that abstracts over the two backend engines. It holds: - `backend`: Either `"duckdb"` or `"polars"`, indicating which engine holds the data. - `name`: The DuckDB table name (when backend is `"duckdb"`). - `lf`: A Polars `LazyFrame` reference (when backend is `"polars"`). - `con`: The DuckDB connection reference. - `source_path`: The original CSV file path. All code paths in the application interact with data through `DataFrameRef` methods (`head()`, `schema()`, `to_polars_df()`, `row_count()`, `materialize_to_duckdb()`, `clone()`), never directly with the underlying engine. This allows operations to switch engines transparently and ensures consistent behavior regardless of which backend produced the data. ## History Tree (Memento Pattern) The history system implements the Memento design pattern with a tree structure: - **State**: Each history entry captures the `DataFrameRef`, the operation that produced it, the operation parameters, and a reference to the parent state. - **Linear Order**: A simple list (`order`) tracks the chronological sequence of states, enabling straightforward undo/redo by moving a pointer backward or forward. - **Branching**: The `branch(state_id)` operation resets the current pointer to a previous state, allowing the user to explore an alternate path. The tree structure (parent-child tracking) preserves the full history. - **Tree Export**: The `to_tree()` method serializes the full history for the frontend, including node IDs, parent-child relationships, and which node is current. ## Frontend State Management The frontend uses four Zustand stores, each responsible for a distinct domain: | Store | Purpose | Key State | |-------|---------|-----------| | `useSessionStore` | Session and dataset metadata | sessionId, tableName, columns, rowCount | | `useNotebookStore` | Notebook cell management | cells, activeCellId, add/remove/move cells | | `useUIStore` | UI panel toggles and errors | showUploadDialog, showManipulationPanel, errors | | `useHistoryStore` | Undo/redo state | nodes, currentId, canUndo, canRedo | Zustand was chosen over Redux or Context API for its minimal boilerplate, built-in selector support, and TypeScript compatibility. Stores are independent and do not reference each other directly, which simplifies testing and reasoning about state changes. ## Technology Stack and Rationale | Technology | Rationale | |------------|-----------| | **FastAPI** | Async-first, automatic OpenAPI docs, Pydantic integration, dependency injection. Ideal for data-intensive APIs. | | **DuckDB** | In-process SQL database optimized for OLAP workloads. Zero-config, runs entirely in memory, handles CSV auto-detection and complex aggregations efficiently. | | **Polars** | Rust-based DataFrame library with lazy evaluation and multi-threaded execution. Provides type-safe expressions that are more maintainable than raw SQL strings for transformations. | | **React + TypeScript** | Industry-standard component model with compile-time type safety. TypeScript catches API contract mismatches at build time. | | **Vite** | Fast HMR during development, optimized Rollup-based production builds with code splitting support. | | **Tailwind CSS** | Utility-first approach eliminates CSS-in-JS overhead and ensures consistent spacing, color, and typography across all components. | | **Zustand** | Lightweight state management with minimal boilerplate. No reducers, actions, or middleware configuration required. | | **ECharts** | Rich chart types, interactive tooltips, responsive rendering, and a declarative JSON spec format that aligns with the backend chart computation model. | | **uv** | Fast Python package installer and resolver. Replaces pip and virtualenv with a single tool, dramatically reducing dependency installation time in CI/CD. |