Blame for docs/development.md

# Development Guide

This document provides a comprehensive guide for setting up a local development environment, understanding the project structure, and extending the Visual Notebook with new operations, chart types, and frontend components.

## Prerequisites

Before setting up the development environment, ensure you have the following tools installed:

| Tool | Version | Purpose |
|------|---------|---------|
| Python | 3.11+ | Backend runtime |
| Node.js | 20+ | Frontend build and development |
| npm | 10+ | Frontend package management |
| uv | Latest | Python package management (replaces pip/venv) |
| Git | 2.40+ | Version control |

### Installing uv

```bash
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```

## Local Development Setup

### Backend Setup

1. **Navigate to the project root**:

```bash
cd /path/to/va_project
```

2. **Create a virtual environment and install dependencies**:

```bash
uv sync
```

This command reads `pyproject.toml`, resolves dependencies, creates a virtual environment in `.venv/`, and installs all packages including development dependencies (pytest, httpx).

3. **Run the backend server**:

```bash
uv run uvicorn visual_notebook.main:app --reload --host 0.0.0.0 --port 8000
```

The `--reload` flag enables automatic server restart when source files change.

4. **Verify the server is running**:

```bash
curl http://localhost:8000/health
# Expected: {"status":"ok","version":"0.1.0"}
```

5. **Access the API documentation**:

Open `http://localhost:8000/docs` in a browser to view the Swagger UI generated by FastAPI.

### Frontend Setup

1. **Navigate to the frontend directory**:

```bash
cd frontend
```

2. **Install dependencies**:

```bash
npm install
```

3. **Start the development server**:

```bash
npm run dev
```

The Vite dev server starts at `http://localhost:5173` with hot module replacement (HMR). API requests to `/api` and `/health` are automatically proxied to the backend at `http://localhost:8000` as configured in `vite.config.ts`.

4. **Build for production**:

```bash
npm run build
```

The production build outputs to `frontend/dist/` with code-split chunks for ECharts, React, and vendor libraries.

### Running Both Together

For the best development experience, run the backend and frontend in separate terminal sessions:

- Terminal 1: `uv run uvicorn visual_notebook.main:app --reload --port 8000`
- Terminal 2: `cd frontend && npm run dev`

Visit `http://localhost:5173` for the full application with live reloading on both frontend and backend changes.

## Project Structure Overview

```
va_project/
    pyproject.toml                    # Root workspace config
    Dockerfile                        # Combined build (frontend + backend)
    Dockerfile.backend                # Backend-only build
    docker-compose.yml                # Development Docker Compose
    datasets/                         # Uploaded CSV files
    checkpoints/                      # Session checkpoint files
    static/                           # Built frontend (production)
    backend/
        pyproject.toml                # Backend dependencies
        visual_notebook/
            main.py                   # FastAPI application entry
            config.py                 # Settings from environment
            session.py                # Session and SessionStore
            api/
                routes.py             # API endpoint definitions
                schemas.py            # Pydantic request/response models
                deps.py               # Dependency injection
            domain/
                dataframe.py          # DataFrameRef wrapper
                history.py            # History tree (Memento)
                operations.py         # Operation registry
                checkpoints.py        # Checkpoint persistence
                validators.py         # Validation helpers
            data/
                csv_loader.py         # CSV loading and export
                schema.py             # Schema inference
                types.py              # Column type classification
            viz/
                _common.py            # Shared chart utilities
                histogram.py          # Histogram computation
                scatter.py            # Scatter plot computation
                bar_topn.py           # Top-N bar chart
                timeline.py           # Timeline/line chart
                heatmap.py            # Heatmap computation
                map.py                # Map computation
            tests/
                test_history.py       # History tests
                test_operations.py    # Operations tests
    frontend/
        package.json                  # Frontend dependencies
        vite.config.ts                # Vite configuration
        tailwind.config.js            # Tailwind CSS theme
        tsconfig.json                 # TypeScript configuration
        index.html                    # HTML entry point
        src/
            main.tsx                  # React entry point
            App.tsx                   # Root component with ErrorBoundary
            api/
                client.ts             # Axios instance and API functions
                hooks.ts              # React hooks for data fetching
                types.ts              # TypeScript type definitions
            store/
                session.ts            # Session state store
                notebook.ts           # Notebook cell store
                ui.ts                 # UI toggle store
                history.ts            # Undo/redo state store
            components/
                Header.tsx            # Top navigation bar
                NotebookPage.tsx      # Main notebook layout
                UploadPanel.tsx       # CSV upload dialog
                TableCellView.tsx     # Table data display
                ChartCellView.tsx     # ECharts rendering
                SchemaView.tsx        # Column schema display
                ManipulationPanel.tsx  # Operation controls
                VisualizationPanel.tsx # Chart configuration
                TablePreview.tsx      # Table row preview
                MapCanvas.tsx         # Map point visualization
                ErrorToast.tsx        # Error notification
            lib/
                chartTypes.ts         # Chart type definitions
                csv.ts                # CSV utility functions
                format.ts             # Number/formatting helpers
            styles/
                index.css             # Global styles and Tailwind
    docs/                            # Documentation (this directory)
```

## Adding a New Operation

Follow these steps to add a new data operation (e.g., `deduplicate`) to the system:

### Step 1: Define the OperationDef

Add a new entry to the `OPERATIONS` dictionary in `backend/visual_notebook/domain/operations.py`:

```python
"deduplicate": OperationDef(
    key="deduplicate",
    label="Remove Duplicates",
    description="Remove duplicate rows based on specified columns",
    param_schema={
        "type": "object",
        "required": ["columns"],
        "properties": {
            "columns": {"type": "array", "items": {"type": "string"}},
            "keep": {"type": "string", "enum": ["first", "last", "none"], "default": "first"},
        },
    },
    validate=_validate_deduplicate,
    apply=_apply_deduplicate,
),
```

### Step 2: Implement the Validate Function

```python
def _validate_deduplicate(params: dict, schema: list[dict]) -> dict:
    columns = params.get("columns", [])
    if not columns:
        raise ValueError("'columns' list is required")
    params.setdefault("keep", "first")
    return params
```

The validate function should check that all required parameters are present and have valid values. It receives the raw user params and the current table schema.

### Step 3: Implement the Apply Function

Provide both DuckDB and Polars implementations:

```python
def _apply_deduplicate(ref: DataFrameRef, params: dict) -> DataFrameRef:
    columns = params["columns"]
    keep = params.get("keep", "first")

    if ref.backend == "duckdb" and ref.name and ref.con:
        new_name = f"_dedup_{uuid.uuid4().hex[:6]}"
        cols = ", ".join(f'"{c}"' for c in columns)

        if keep == "first":
            sql = f'''
                SELECT DISTINCT ON ({cols}) *
                FROM "{ref.name}"
                ORDER BY {cols}
            '''
        elif keep == "last":
            sql = f'''
                SELECT DISTINCT ON ({cols}) *
                FROM "{ref.name}"
                ORDER BY {cols} DESC
            '''
        else:
            sql = f'''
                SELECT * FROM "{ref.name}"
                GROUP BY {cols}
                HAVING COUNT(*) = 1
            '''

        ref.con.execute(f'CREATE OR REPLACE TABLE "{new_name}" AS {sql}')
        return DataFrameRef(backend="duckdb", name=new_name, con=ref.con, source_path=ref.source_path)

    elif ref.backend == "polars" and ref.lf is not None:
        lf = ref.lf.unique(subset=columns, keep=keep)
        return DataFrameRef(backend="polars", lf=lf, source_path=ref.source_path)

    raise ValueError("Invalid DataFrameRef")
```

### Step 4: Add Tests

Add test cases to `backend/visual_notebook/tests/test_operations.py`:

```python
def test_deduplicate_validation():
    op = OPERATIONS["deduplicate"]
    with pytest.raises(ValueError):
        op.validate({}, [])

    result = op.validate({"columns": ["a", "b"]}, [])
    assert result["keep"] == "first"

def test_deduplicate_in_registry():
    assert "deduplicate" in OPERATIONS
```

### Step 5: Verify

1. Run the tests: `uv run pytest backend/visual_notebook/tests/ -v`
2. Start the server and verify the operation appears in `GET /api/ops`.
3. Test the operation via the Swagger UI at `http://localhost:8000/docs`.

## Adding a New Chart Type

Follow these steps to add a new chart type (e.g., `pie`) to the system:

### Step 1: Create the Computation Module

Create a new file `backend/visual_notebook/viz/pie.py`:

```python
"""Pie chart generation."""
from __future__ import annotations
from typing import Any
import duckdb
from visual_notebook.viz._common import PALETTE, base_chart, title, tooltip_style, palette_colors


def compute_pie(
    con: duckdb.DuckDBPyConnection,
    table_name: str,
    category_col: str,
    value_col: str | None = None,
    n: int = 10,
) -> dict[str, Any]:
    """Compute pie chart data and return ECharts spec."""
    if value_col:
        result = con.execute(f'''
            SELECT "{category_col}" as cat, SUM("{value_col}") as val
            FROM "{table_name}"
            WHERE "{category_col}" IS NOT NULL
            GROUP BY cat
            ORDER BY val DESC
            LIMIT {n}
        ''').fetchall()
    else:
        result = con.execute(f'''
            SELECT "{category_col}" as cat, COUNT(*) as val
            FROM "{table_name}"
            WHERE "{category_col}" IS NOT NULL
            GROUP BY cat
            ORDER BY val DESC
            LIMIT {n}
        ''').fetchall()

    data = [{"name": str(r[0]), "value": float(r[1])} for r in result]
    colors = palette_colors(len(data))

    chart = base_chart()
    chart["title"] = title(f"Distribution of {category_col}")
    chart["tooltip"] = {**tooltip_style(), "trigger": "item", "formatter": "{b}: {c} ({d}%)"}
    chart["legend"] = {"orient": "vertical", "left": "left", "top": "middle"}
    chart["series"] = [{
        "type": "pie",
        "radius": ["40%", "70%"],
        "data": data,
        "emphasis": {"itemStyle": {"shadowBlur": 10, "shadowOffsetX": 0, "shadowColor": "rgba(0,0,0,0.5)"}},
    }]

    return chart
```

### Step 2: Register in the Route Handler

Add the import and handler in `backend/visual_notebook/api/routes.py`:

```python
from visual_notebook.viz.pie import compute_pie

# In the create_chart endpoint:
elif req.chart_type == "pie":
    spec = compute_pie(con, table, params["category_col"], params.get("value_col"), params.get("n", 10))
```

### Step 3: Add Frontend Support

Update `frontend/src/lib/chartTypes.ts` to include the new chart type, and add a configuration form in the `VisualizationPanel` component.

## Adding a New Frontend Component

### Step 1: Create the Component File

Create a new file in `frontend/src/components/`:

```typescript
// StatsPanel.tsx
import React from 'react';
import { useSessionStore } from '../store/session';

export function StatsPanel() {
    const { columns, rowCount } = useSessionStore();

    return (
        <div className="bg-white rounded-lg shadow p-4">
            <h3 className="text-sm font-semibold text-gray-700 mb-3">Dataset Statistics</h3>
            <div className="space-y-2">
                <div className="flex justify-between text-sm">
                    <span className="text-gray-500">Total Rows</span>
                    <span className="font-mono">{rowCount.toLocaleString()}</span>
                </div>
                <div className="flex justify-between text-sm">
                    <span className="text-gray-500">Total Columns</span>
                    <span className="font-mono">{columns.length}</span>
                </div>
            </div>
        </div>
    );
}
```

### Step 2: Use Tailwind Classes

Follow the project's Tailwind conventions:

- Use `brand-*` colors for primary actions and accents.
- Use `gray-*` colors for neutral text, borders, and backgrounds.
- Use `text-sm` for body text, `text-xs` for secondary labels.
- Use `rounded-lg` for card corners, `shadow` for elevation.
- Use `flex` and `grid` for layout, with `gap-*` for spacing.

### Step 3: Integrate into the Application

Import and render the component in the appropriate parent:

```typescript
// In NotebookPage.tsx
import { StatsPanel } from './StatsPanel';

// Render conditionally
{showStatsPanel && <StatsPanel />}
```

### Step 4: Add a UI Toggle

If the component should be togglable, add a flag to `useUIStore`:

```typescript
showStatsPanel: boolean;
toggleStatsPanel: () => void;
```

## Testing

### Backend Tests

The backend uses pytest with test files in `backend/visual_notebook/tests/`.

**Run all tests**:

```bash
uv run pytest backend/visual_notebook/tests/ -v
```

**Run a specific test file**:

```bash
uv run pytest backend/visual_notebook/tests/test_operations.py -v
```

**Run with coverage** (if pytest-cov is installed):

```bash
uv run pytest backend/visual_notebook/tests/ -v --cov=visual_notebook
```

**Test structure**:

- `test_history.py`: Tests for commit, undo, redo, branch, and tree serialization.
- `test_operations.py`: Tests for the operations registry, parameter validation, and default values.

### Frontend Testing

The frontend currently does not have automated tests configured. For future implementation, consider:

- **Vitest**: Fast unit testing framework compatible with Vite.
- **React Testing Library**: Component testing with DOM assertions.
- **Playwright**: End-to-end testing for API integration.

## Code Style and Conventions

### Python (Backend)

- **Line length**: 100 characters maximum.
- **Type hints**: Use type hints on all function signatures. Import `from __future__ import annotations` for modern type syntax.
- **Docstrings**: Use triple-quoted docstrings for all public functions and classes.
- **Naming**: `snake_case` for functions and variables, `PascalCase` for classes.
- **Imports**: Standard library first, then third-party, then local. Use absolute imports within the package.
- **f-strings**: Use f-strings for SQL query construction (the SQL injection risk is mitigated by server-generated table names and column names from the schema).
- **Error handling**: Raise `ValueError` in validators, `HTTPException` in route handlers.

### TypeScript (Frontend)

- **Strict mode**: TypeScript strict mode is enabled. All variables must have explicit or inferred types.
- **Functional components**: Use function declarations with explicit return types.
- **Naming**: `camelCase` for variables and functions, `PascalCase` for components and types.
- **Imports**: Use relative paths for local modules, package names for third-party libraries.
- **Hooks**: Custom hooks in `api/hooks.ts` follow the `use[X]` naming pattern and return `{ action, loading, error }`.
- **CSS**: Use Tailwind utility classes exclusively. Avoid inline styles except for dynamic values (chart dimensions, SVG coordinates).

## Git Workflow

### Branch Naming

- `feature/[description]`: New features
- `fix/[description]`: Bug fixes
- `docs/[description]`: Documentation changes
- `refactor/[description]`: Code refactoring

### Commit Messages

Follow conventional commit format:

```
type(scope): description

[optional body]
```

Types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`

Examples:
- `feat(operations): add deduplicate operation`
- `fix(history): correct undo pointer after branch`
- `docs(api): update endpoint descriptions`

### Pull Request Process

1. Create a feature branch from `main`.
2. Implement changes with tests.
3. Run the full test suite: `uv run pytest`.
4. Ensure the frontend builds: `cd frontend && npm run build`.
5. Open a pull request with a description of the changes and any manual testing performed.
6. Request review from a maintainer.
7. Address review feedback and re-run tests.
8. Merge when approved and all checks pass.