# Domain Layer

This document describes the core domain model of the Visual Notebook backend, including the `DataFrameRef` abstraction, the history tree, the operations registry, the checkpoint system, and validation helpers.

## DataFrameRef: Unified Wrapper

`DataFrameRef` is the central abstraction in the domain layer. It provides a uniform interface over two very different computation engines -- DuckDB and Polars -- allowing the rest of the application to interact with data without knowing which engine holds it.

### Dataclass Definition

```python
@dataclass
class DataFrameRef:
    backend: Literal["duckdb", "polars"]
    name: str | None = None           # DuckDB table name
    lf: pl.LazyFrame | None = None    # Polars LazyFrame
    con: duckdb.DuckDBPyConnection | None = None
    source_path: str | None = None    # Original CSV path
    metadata: dict[str, Any] = field(default_factory=dict)
```

The `backend` field is a discriminating literal that determines which code path each method follows. The other fields are populated according to the backend: DuckDB-backed refs set `name` and `con`, while Polars-backed refs set `lf`.

### Methods

#### `head(n=50, offset=0) -> pl.DataFrame`

Returns a Polars DataFrame containing the first `n` rows starting from `offset`. For DuckDB, this executes `SELECT * FROM "{name}" LIMIT {n} OFFSET {offset}` and converts the result rows into a Polars DataFrame. For Polars, it uses `lf.slice(offset, n).collect()`.

The return type is always `pl.DataFrame` regardless of backend, ensuring a consistent interface for the API layer.

#### `schema() -> list[dict[str, str]]`

Returns column names and types. For DuckDB, it queries `information_schema.columns`. For Polars, it reads `lf.schema`. The return format is a list of dictionaries with `name` and `dtype` keys.

#### `to_polars_df() -> pl.DataFrame`

Materializes the entire dataset as a Polars DataFrame. For DuckDB, this executes a full table scan (`SELECT *`) and converts the result. For Polars, it calls `lf.collect()`. This method should be used with caution on large datasets as it loads all data into memory.

#### `to_lazy() -> pl.LazyFrame`

Returns a Polars LazyFrame representation. For Polars, it returns `lf` directly. For DuckDB, it materializes the full DataFrame and converts it to a LazyFrame via `df.lazy()`. This is used by operations that require the Polars expression API.

#### `row_count() -> int`

Returns the total number of rows. For DuckDB, this executes `SELECT COUNT(*)`. For Polars, it uses `lf.select(pl.len()).collect().item()`. Both approaches are optimized for their respective engines.

#### `materialize_to_duckdb(table_name, con) -> DataFrameRef`

Converts a Polars LazyFrame into a DuckDB table and returns a new `DataFrameRef` with backend set to `"duckdb"`. This is the key method for bridging the two engines:

```python
df = self.lf.collect()
arrow_tbl = df.to_arrow()
con.register("pl_df", arrow_tbl)
con.execute(f'CREATE OR REPLACE TABLE "{table_name}" AS SELECT * FROM pl_df')
con.unregister("pl_df")
```

The transfer uses Apache Arrow as the interchange format. Polars converts its in-memory representation to an Arrow table, which DuckDB registers as a virtual table, then materializes into a native DuckDB table. This path avoids serialization to CSV or JSON, preserving type information and achieving high throughput.

#### `clone() -> DataFrameRef`

Creates a shallow copy of the DataFrameRef. The underlying data (DuckDB table or Polars LazyFrame) is not copied, only the reference. This is used when creating history snapshots that need independent metadata.

### Why Two Backends

The dual-engine strategy exists because DuckDB and Polars have complementary strengths:

- **DuckDB** excels at SQL-based analytics. Its `SUMMARIZE` command, `read_csv_auto`, `COPY` for export, and window functions are all SQL-native features that would require significant effort to replicate in Polars. DuckDB also handles large datasets that exceed memory by spilling to disk.

- **Polars** excels at programmatic transformations. Its expression API (`pl.col()`, `.filter()`, `.sort()`, `.group_by()`) provides type-safe, composable operations that are easier to maintain and test than dynamically generated SQL strings. Polars' lazy evaluation enables query optimization, and its Rust-based engine provides excellent CPU parallelism.

The architecture keeps DuckDB as the canonical storage and query engine, using Polars only for transformation steps. After each Polars operation, the result is materialized back into DuckDB via `materialize_to_duckdb()`, ensuring that schema inference and chart computation always operate on DuckDB tables.

## History Tree: Memento Pattern

The history system implements the Memento design pattern with a tree structure that supports linear undo/redo and non-linear branching.

### State Dataclass

```python
@dataclass
class State:
    state_id: str
    ref: DataFrameRef
    operation: str | None = None
    params: dict[str, Any] = field(default_factory=dict)
    metadata: dict[str, Any] = field(default_factory=dict)
    parent_id: str | None = None
```

Each `State` captures:

- **state_id**: An 8-character hex identifier for quick lookup.
- **ref**: The `DataFrameRef` at this point in the history. Because DuckDB tables are immutable once created (operations create new tables), the ref can be safely shared across states.
- **operation**: The operation key (e.g., `"filter"`, `"sort"`, `"upload"`) that produced this state.
- **params**: The validated parameters used for the operation.
- **parent_id**: The state ID of the parent, enabling tree traversal.

### History Class

```python
class History:
    states: dict[str, State]          # All states by ID
    current_id: str | None             # Current state pointer
    children: dict[str, list[str]]     # Parent -> child ID mapping
    order: list[str]                   # Linear order for undo/redo
```

#### `commit(ref, operation, params) -> State`

Creates a new `State`, appends it to the history, and updates the current pointer:

1. Generates a unique `state_id`.
2. Creates a `State` with the current ID as `parent_id`.
3. Stores the state in `self.states`.
4. Adds the state ID to `self.children[parent_id]`.
5. Appends the state ID to `self.order`.
6. Sets `self.current_id` to the new state ID.

#### `undo() -> State | None`

Moves the current pointer backward in the linear order:

```python
idx = self.order.index(self.current_id)
if idx > 0:
    self.current_id = self.order[idx - 1]
    return self.current
```

Returns `None` if already at the first state. The tree structure is preserved; undo does not remove states.

#### `redo() -> State | None`

Moves the current pointer forward in the linear order:

```python
idx = self.order.index(self.current_id)
if idx < len(self.order) - 1:
    self.current_id = self.order[idx + 1]
    return self.current
```

Returns `None` if already at the last state.

#### `branch(state_id) -> State | None`

Resets the current pointer to an arbitrary state by ID, effectively creating a branch point. The next `commit()` will create a child of this state. This allows users to explore alternate transformation paths from any historical state.

#### `can_undo() / can_redo() -> bool`

Check whether the current state is at the boundary of the linear order. These are used by the frontend to enable or disable undo/redo buttons.

#### `to_tree() -> dict`

Serializes the entire history as a JSON-compatible tree structure for the frontend:

```python
{
    "nodes": [
        {"id": "abc123", "operation": "upload", "params": {}, "parent_id": None, "children": ["def456"], "is_current": False},
        {"id": "def456", "operation": "filter", "params": {"column": "age", "operator": "gt", "value": 30}, "parent_id": "abc123", "children": [], "is_current": True}
    ],
    "current_id": "def456",
    "order": ["abc123", "def456"]
}
```

### Tree Structure vs. Linear Order

The `children` dictionary enables tree visualization in the frontend, showing branching paths. The `order` list provides a simple linear sequence for undo/redo. In practice, most users interact with history linearly, but the tree structure preserves all exploration paths.

## Operations Registry

The operations registry is a dictionary mapping operation keys to `OperationDef` instances. Each operation is self-contained with its own validation and apply logic.

### OperationDef Dataclass

```python
@dataclass
class OperationDef:
    key: str                                    # Unique identifier
    label: str                                  # Human-readable name
    description: str                            # Short description
    param_schema: dict[str, Any]                # JSON Schema for parameters
    validate: Callable[[dict, list[dict]], dict] # (params, schema) -> validated params
    apply: Callable[[DataFrameRef, dict], DataFrameRef]  # (ref, params) -> new ref
```

The `validate` function receives the raw parameters and the current table schema, and returns validated parameters or raises `ValueError`. The `apply` function receives a `DataFrameRef` and validated parameters, and returns a new `DataFrameRef`.

### All Seven Operations

| Key | Label | Description | Required Params |
|-----|-------|-------------|-----------------|
| `filter` | Filter | Filter rows by condition | `column`, `operator` |
| `sort` | Sort | Sort by column | `column` |
| `select` | Select Columns | Keep only selected columns | `columns` |
| `rename` | Rename Columns | Rename columns using a mapping | `mapping` |
| `groupby` | Group By | Aggregate data by groups | `group_by`, `aggregations` |
| `fill_nulls` | Fill Nulls | Fill null values using a strategy | (none required) |
| `clip` | Clip Values | Clip values to a range | `column` |

### Dual-Engine Implementation

Each operation implements both DuckDB and Polars code paths:

**DuckDB path**: Generates SQL statements and executes `CREATE OR REPLACE TABLE` to produce a new table. Each operation creates a uniquely named table (e.g., `_filt_a3b2c1` for filter, `_sort_d4e5f6` for sort) to avoid collisions with existing tables.

**Polars path**: Uses Polars' expression API to build lazy transformations. The resulting `DataFrameRef` has `backend="polars"` and contains the transformed `LazyFrame`.

Example for the filter operation:

```python
# DuckDB
if ref.backend == "duckdb":
    new_name = f"_filt_{uuid.uuid4().hex[:6]}"
    sql = f'SELECT * FROM "{ref.name}" WHERE "{column}" = {value!r}'
    ref.con.execute(f'CREATE OR REPLACE TABLE "{new_name}" AS {sql}')
    return DataFrameRef(backend="duckdb", name=new_name, con=ref.con)

# Polars
elif ref.backend == "polars":
    lf = ref.lf.filter(pl.col(column) == value)
    return DataFrameRef(backend="polars", lf=lf, source_path=ref.source_path)
```

### Polars Materialization Back to DuckDB

After each operation, the route handler checks if the result is a Polars ref that needs to be materialized:

```python
if new_ref.backend == "polars" and session.current_ref.backend == "duckdb":
    new_table_name = f"_op_{uuid.uuid4().hex[:6]}"
    new_ref = new_ref.materialize_to_duckdb(new_table_name, ref.con)
```

This ensures that the system always returns to a DuckDB-backed state after each operation, maintaining consistency for downstream queries.

### Filter Operators

The filter operation supports nine comparison operators:

| Operator | DuckDB SQL | Polars Expression |
|----------|------------|-------------------|
| `eq` | `= value` | `col == value` |
| `neq` | `!= value` | `col != value` |
| `gt` | `> value` | `col > value` |
| `lt` | `< value` | `col < value` |
| `gte` | `>= value` | `col >= value` |
| `lte` | `<= value` | `col <= value` |
| `contains` | `LIKE '%value%'` | `col.cast(Utf8).str.contains(value)` |
| `is_null` | `IS NULL` | `col.is_null()` |
| `is_not_null` | `IS NOT NULL` | `col.is_not_null()` |

### GroupBy Aggregation Functions

The groupby operation supports five aggregation functions: `mean`, `sum`, `count`, `min`, and `max`. Each is expressed both as a SQL aggregate function and as a Polars expression:

```python
# DuckDB
agg_parts.append(f'{func}("{col}") AS "{col}_{func}"')

# Polars
if func == "mean":
    agg_exprs.append(pl.col(col).mean().alias(f"{col}_mean"))
```

### Fill Nulls Strategies

| Strategy | DuckDB | Polars |
|----------|--------|--------|
| `forward` | `COALESCE(col, LAG(col) OVER (...))` | `lf.fill_null(strategy="forward")` |
| `backward` | (same pattern with LEAD) | `lf.fill_null(strategy="backward")` |
| `mean` | `COALESCE(col, AVG(col) OVER ())` | `lf.fill_null(strategy="mean")` |
| `zero` | `COALESCE(col, 0)` | `lf.fill_null(value=0)` |
| `constant` | `COALESCE(col, 'value')` | `lf.fill_null(value=fill_value)` |

## Checkpoints

The checkpoint system serializes the session state to disk, allowing users to preserve their work.

### Save Checkpoint

The `save_checkpoint()` function writes two files to the checkpoint directory:

1. **JSON metadata** (`checkpoint_YYYYMMDD_HHMMSS.json`): Contains the session ID, timestamp, and the full history tree (nodes, current_id, order).

2. **Parquet data** (`data_YYYYMMDD_HHMMSS.parquet`): Contains the current table data exported via DuckDB's `COPY ... FORMAT PARQUET`. This provides a compact, typed representation of the data.

The Parquet export is non-critical -- if it fails, the checkpoint still saves the metadata. This ensures that a schema-only or history-only checkpoint is always available.

### Load Checkpoint

The `load_latest_checkpoint()` function finds the most recent checkpoint JSON file (sorted by filename in reverse order) and returns the parsed metadata.

### List Checkpoints

The `list_checkpoints()` function returns a list of all checkpoints for a session, sorted from newest to oldest, including the file path, timestamp, and session ID.

## Validators

The `validators.py` module provides helper functions for operation parameter validation:

| Function | Purpose |
|----------|---------|
| `validate_column_exists(column, schema)` | Returns True if the column name is found in the schema |
| `validate_columns_exist(columns, schema)` | Returns a list of column names that do NOT exist |
| `validate_numeric_column(column, schema)` | Returns True if the column is numeric or geo_lat_lon type |

These helpers are used by the operation `validate` functions to ensure that user-provided column names exist and have appropriate types before the operation is applied.