docs/domain-layer.md

Domain Layer

This document describes the core domain model of the Visual Notebook backend, including the DataFrameRef abstraction, the history tree, the operations registry, the checkpoint system, and validation helpers.

DataFrameRef: Unified Wrapper

DataFrameRef is the central abstraction in the domain layer. It provides a uniform interface over two very different computation engines -- DuckDB and Polars -- allowing the rest of the application to interact with data without knowing which engine holds it.

Dataclass Definition

@dataclass
class DataFrameRef:
    backend: Literal["duckdb", "polars"]
    name: str | None = None           # DuckDB table name
    lf: pl.LazyFrame | None = None    # Polars LazyFrame
    con: duckdb.DuckDBPyConnection | None = None
    source_path: str | None = None    # Original CSV path
    metadata: dict[str, Any] = field(default_factory=dict)

The backend field is a discriminating literal that determines which code path each method follows. The other fields are populated according to the backend: DuckDB-backed refs set name and con, while Polars-backed refs set lf.

Methods

head(n=50, offset=0) -> pl.DataFrame

Returns a Polars DataFrame containing the first n rows starting from offset. For DuckDB, this executes SELECT * FROM "{name}" LIMIT {n} OFFSET {offset} and converts the result rows into a Polars DataFrame. For Polars, it uses lf.slice(offset, n).collect().

The return type is always pl.DataFrame regardless of backend, ensuring a consistent interface for the API layer.

schema() -> list[dict[str, str]]

Returns column names and types. For DuckDB, it queries information_schema.columns. For Polars, it reads lf.schema. The return format is a list of dictionaries with name and dtype keys.

to_polars_df() -> pl.DataFrame

Materializes the entire dataset as a Polars DataFrame. For DuckDB, this executes a full table scan (SELECT *) and converts the result. For Polars, it calls lf.collect(). This method should be used with caution on large datasets as it loads all data into memory.

to_lazy() -> pl.LazyFrame

Returns a Polars LazyFrame representation. For Polars, it returns lf directly. For DuckDB, it materializes the full DataFrame and converts it to a LazyFrame via df.lazy(). This is used by operations that require the Polars expression API.

row_count() -> int

Returns the total number of rows. For DuckDB, this executes SELECT COUNT(*). For Polars, it uses lf.select(pl.len()).collect().item(). Both approaches are optimized for their respective engines.

materialize_to_duckdb(table_name, con) -> DataFrameRef

Converts a Polars LazyFrame into a DuckDB table and returns a new DataFrameRef with backend set to "duckdb". This is the key method for bridging the two engines:

df = self.lf.collect()
arrow_tbl = df.to_arrow()
con.register("pl_df", arrow_tbl)
con.execute(f'CREATE OR REPLACE TABLE "{table_name}" AS SELECT * FROM pl_df')
con.unregister("pl_df")

The transfer uses Apache Arrow as the interchange format. Polars converts its in-memory representation to an Arrow table, which DuckDB registers as a virtual table, then materializes into a native DuckDB table. This path avoids serialization to CSV or JSON, preserving type information and achieving high throughput.

clone() -> DataFrameRef

Creates a shallow copy of the DataFrameRef. The underlying data (DuckDB table or Polars LazyFrame) is not copied, only the reference. This is used when creating history snapshots that need independent metadata.

Why Two Backends

The dual-engine strategy exists because DuckDB and Polars have complementary strengths:

• DuckDB excels at SQL-based analytics. Its SUMMARIZE command, read_csv_auto, COPY for export, and window functions are all SQL-native features that would require significant effort to replicate in Polars. DuckDB also handles large datasets that exceed memory by spilling to disk.

• Polars excels at programmatic transformations. Its expression API (pl.col(), .filter(), .sort(), .group_by()) provides type-safe, composable operations that are easier to maintain and test than dynamically generated SQL strings. Polars' lazy evaluation enables query optimization, and its Rust-based engine provides excellent CPU parallelism.

The architecture keeps DuckDB as the canonical storage and query engine, using Polars only for transformation steps. After each Polars operation, the result is materialized back into DuckDB via materialize_to_duckdb(), ensuring that schema inference and chart computation always operate on DuckDB tables.

History Tree: Memento Pattern

The history system implements the Memento design pattern with a tree structure that supports linear undo/redo and non-linear branching.

State Dataclass

@dataclass
class State:
    state_id: str
    ref: DataFrameRef
    operation: str | None = None
    params: dict[str, Any] = field(default_factory=dict)
    metadata: dict[str, Any] = field(default_factory=dict)
    parent_id: str | None = None

Each State captures:

• state_id: An 8-character hex identifier for quick lookup.
• ref: The DataFrameRef at this point in the history. Because DuckDB tables are immutable once created (operations create new tables), the ref can be safely shared across states.
• operation: The operation key (e.g., "filter", "sort", "upload") that produced this state.
• params: The validated parameters used for the operation.
• parent_id: The state ID of the parent, enabling tree traversal.

History Class

class History:
    states: dict[str, State]          # All states by ID
    current_id: str | None             # Current state pointer
    children: dict[str, list[str]]     # Parent -> child ID mapping
    order: list[str]                   # Linear order for undo/redo

commit(ref, operation, params) -> State

Creates a new State, appends it to the history, and updates the current pointer:

1. Generates a unique state_id.
2. Creates a State with the current ID as parent_id.
3. Stores the state in self.states.
4. Adds the state ID to self.children[parent_id].
5. Appends the state ID to self.order.
6. Sets self.current_id to the new state ID.

undo() -> State | None

Moves the current pointer backward in the linear order:

idx = self.order.index(self.current_id)
if idx > 0:
    self.current_id = self.order[idx - 1]
    return self.current

Returns None if already at the first state. The tree structure is preserved; undo does not remove states.

redo() -> State | None

Moves the current pointer forward in the linear order:

idx = self.order.index(self.current_id)
if idx < len(self.order) - 1:
    self.current_id = self.order[idx + 1]
    return self.current

Returns None if already at the last state.

branch(state_id) -> State | None

Resets the current pointer to an arbitrary state by ID, effectively creating a branch point. The next commit() will create a child of this state. This allows users to explore alternate transformation paths from any historical state.

can_undo() / can_redo() -> bool

Check whether the current state is at the boundary of the linear order. These are used by the frontend to enable or disable undo/redo buttons.

to_tree() -> dict

Serializes the entire history as a JSON-compatible tree structure for the frontend:

{
    "nodes": [
        {"id": "abc123", "operation": "upload", "params": {}, "parent_id": None, "children": ["def456"], "is_current": False},
        {"id": "def456", "operation": "filter", "params": {"column": "age", "operator": "gt", "value": 30}, "parent_id": "abc123", "children": [], "is_current": True}
    ],
    "current_id": "def456",
    "order": ["abc123", "def456"]
}

Tree Structure vs. Linear Order

The children dictionary enables tree visualization in the frontend, showing branching paths. The order list provides a simple linear sequence for undo/redo. In practice, most users interact with history linearly, but the tree structure preserves all exploration paths.

Operations Registry

The operations registry is a dictionary mapping operation keys to OperationDef instances. Each operation is self-contained with its own validation and apply logic.

OperationDef Dataclass

@dataclass
class OperationDef:
    key: str                                    # Unique identifier
    label: str                                  # Human-readable name
    description: str                            # Short description
    param_schema: dict[str, Any]                # JSON Schema for parameters
    validate: Callable[[dict, list[dict]], dict] # (params, schema) -> validated params
    apply: Callable[[DataFrameRef, dict], DataFrameRef]  # (ref, params) -> new ref

The validate function receives the raw parameters and the current table schema, and returns validated parameters or raises ValueError. The apply function receives a DataFrameRef and validated parameters, and returns a new DataFrameRef.

All Seven Operations

Key	Label	Description	Required Params
filter	Filter	Filter rows by condition	column, operator
sort	Sort	Sort by column	column
select	Select Columns	Keep only selected columns	columns
rename	Rename Columns	Rename columns using a mapping	mapping
groupby	Group By	Aggregate data by groups	group_by, aggregations
fill_nulls	Fill Nulls	Fill null values using a strategy	(none required)
clip	Clip Values	Clip values to a range	column

Dual-Engine Implementation

Each operation implements both DuckDB and Polars code paths:

DuckDB path: Generates SQL statements and executes CREATE OR REPLACE TABLE to produce a new table. Each operation creates a uniquely named table (e.g., _filt_a3b2c1 for filter, _sort_d4e5f6 for sort) to avoid collisions with existing tables.

Polars path: Uses Polars' expression API to build lazy transformations. The resulting DataFrameRef has backend="polars" and contains the transformed LazyFrame.

Example for the filter operation:

# DuckDB
if ref.backend == "duckdb":
    new_name = f"_filt_{uuid.uuid4().hex[:6]}"
    sql = f'SELECT * FROM "{ref.name}" WHERE "{column}" = {value!r}'
    ref.con.execute(f'CREATE OR REPLACE TABLE "{new_name}" AS {sql}')
    return DataFrameRef(backend="duckdb", name=new_name, con=ref.con)

# Polars
elif ref.backend == "polars":
    lf = ref.lf.filter(pl.col(column) == value)
    return DataFrameRef(backend="polars", lf=lf, source_path=ref.source_path)

Polars Materialization Back to DuckDB

After each operation, the route handler checks if the result is a Polars ref that needs to be materialized:

if new_ref.backend == "polars" and session.current_ref.backend == "duckdb":
    new_table_name = f"_op_{uuid.uuid4().hex[:6]}"
    new_ref = new_ref.materialize_to_duckdb(new_table_name, ref.con)

This ensures that the system always returns to a DuckDB-backed state after each operation, maintaining consistency for downstream queries.

Filter Operators

The filter operation supports nine comparison operators:

Operator	DuckDB SQL	Polars Expression
eq	= value	col == value
neq	!= value	col != value
gt	> value	col > value
lt	< value	col < value
gte	>= value	col >= value
lte	<= value	col <= value
contains	LIKE '%value%'	col.cast(Utf8).str.contains(value)
is_null	IS NULL	col.is_null()
is_not_null	IS NOT NULL	col.is_not_null()

GroupBy Aggregation Functions

The groupby operation supports five aggregation functions: mean, sum, count, min, and max. Each is expressed both as a SQL aggregate function and as a Polars expression:

# DuckDB
agg_parts.append(f'{func}("{col}") AS "{col}_{func}"')

# Polars
if func == "mean":
    agg_exprs.append(pl.col(col).mean().alias(f"{col}_mean"))

Fill Nulls Strategies

Strategy	DuckDB	Polars
forward	COALESCE(col, LAG(col) OVER (...))	lf.fill_null(strategy="forward")
backward	(same pattern with LEAD)	lf.fill_null(strategy="backward")
mean	COALESCE(col, AVG(col) OVER ())	lf.fill_null(strategy="mean")
zero	COALESCE(col, 0)	lf.fill_null(value=0)
constant	COALESCE(col, 'value')	lf.fill_null(value=fill_value)

Checkpoints

The checkpoint system serializes the session state to disk, allowing users to preserve their work.

Save Checkpoint

The save_checkpoint() function writes two files to the checkpoint directory:

1. JSON metadata (checkpoint_YYYYMMDD_HHMMSS.json): Contains the session ID, timestamp, and the full history tree (nodes, current_id, order).

2. Parquet data (data_YYYYMMDD_HHMMSS.parquet): Contains the current table data exported via DuckDB's COPY ... FORMAT PARQUET. This provides a compact, typed representation of the data.

The Parquet export is non-critical -- if it fails, the checkpoint still saves the metadata. This ensures that a schema-only or history-only checkpoint is always available.

Load Checkpoint

The load_latest_checkpoint() function finds the most recent checkpoint JSON file (sorted by filename in reverse order) and returns the parsed metadata.

List Checkpoints

The list_checkpoints() function returns a list of all checkpoints for a session, sorted from newest to oldest, including the file path, timestamp, and session ID.

Validators

The validators.py module provides helper functions for operation parameter validation:

Function	Purpose
validate_column_exists(column, schema)	Returns True if the column name is found in the schema
validate_columns_exist(columns, schema)	Returns a list of column names that do NOT exist
validate_numeric_column(column, schema)	Returns True if the column is numeric or geo_lat_lon type

These helpers are used by the operation validate functions to ensure that user-provided column names exist and have appropriate types before the operation is applied.