docs/development.md

Development Guide

This document provides a comprehensive guide for setting up a local development environment, understanding the project structure, and extending the Visual Notebook with new operations, chart types, and frontend components.

Prerequisites

Before setting up the development environment, ensure you have the following tools installed:

Tool	Version	Purpose
Python	3.11+	Backend runtime
Node.js	20+	Frontend build and development
npm	10+	Frontend package management
uv	Latest	Python package management (replaces pip/venv)
Git	2.40+	Version control

Installing uv

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Local Development Setup

Backend Setup

1. Navigate to the project root:

cd /path/to/va_project

2. Create a virtual environment and install dependencies:

uv sync

This command reads pyproject.toml, resolves dependencies, creates a virtual environment in .venv/, and installs all packages including development dependencies (pytest, httpx).

3. Run the backend server:

uv run uvicorn visual_notebook.main:app --reload --host 0.0.0.0 --port 8000

The --reload flag enables automatic server restart when source files change.

4. Verify the server is running:

curl http://localhost:8000/health
# Expected: {"status":"ok","version":"0.1.0"}

5. Access the API documentation:

Open http://localhost:8000/docs in a browser to view the Swagger UI generated by FastAPI.

Frontend Setup

1. Navigate to the frontend directory:

cd frontend

2. Install dependencies:

npm install

3. Start the development server:

npm run dev

The Vite dev server starts at http://localhost:5173 with hot module replacement (HMR). API requests to /api and /health are automatically proxied to the backend at http://localhost:8000 as configured in vite.config.ts.

4. Build for production:

npm run build

The production build outputs to frontend/dist/ with code-split chunks for ECharts, React, and vendor libraries.

Running Both Together

For the best development experience, run the backend and frontend in separate terminal sessions:

• Terminal 1: uv run uvicorn visual_notebook.main:app --reload --port 8000
• Terminal 2: cd frontend && npm run dev

Visit http://localhost:5173 for the full application with live reloading on both frontend and backend changes.

Project Structure Overview

va_project/
    pyproject.toml                    # Root workspace config
    Dockerfile                        # Combined build (frontend + backend)
    Dockerfile.backend                # Backend-only build
    docker-compose.yml                # Development Docker Compose
    datasets/                         # Uploaded CSV files
    checkpoints/                      # Session checkpoint files
    static/                           # Built frontend (production)
    backend/
        pyproject.toml                # Backend dependencies
        visual_notebook/
            main.py                   # FastAPI application entry
            config.py                 # Settings from environment
            session.py                # Session and SessionStore
            api/
                routes.py             # API endpoint definitions
                schemas.py            # Pydantic request/response models
                deps.py               # Dependency injection
            domain/
                dataframe.py          # DataFrameRef wrapper
                history.py            # History tree (Memento)
                operations.py         # Operation registry
                checkpoints.py        # Checkpoint persistence
                validators.py         # Validation helpers
            data/
                csv_loader.py         # CSV loading and export
                schema.py             # Schema inference
                types.py              # Column type classification
            viz/
                _common.py            # Shared chart utilities
                histogram.py          # Histogram computation
                scatter.py            # Scatter plot computation
                bar_topn.py           # Top-N bar chart
                timeline.py           # Timeline/line chart
                heatmap.py            # Heatmap computation
                map.py                # Map computation
            tests/
                test_history.py       # History tests
                test_operations.py    # Operations tests
    frontend/
        package.json                  # Frontend dependencies
        vite.config.ts                # Vite configuration
        tailwind.config.js            # Tailwind CSS theme
        tsconfig.json                 # TypeScript configuration
        index.html                    # HTML entry point
        src/
            main.tsx                  # React entry point
            App.tsx                   # Root component with ErrorBoundary
            api/
                client.ts             # Axios instance and API functions
                hooks.ts              # React hooks for data fetching
                types.ts              # TypeScript type definitions
            store/
                session.ts            # Session state store
                notebook.ts           # Notebook cell store
                ui.ts                 # UI toggle store
                history.ts            # Undo/redo state store
            components/
                Header.tsx            # Top navigation bar
                NotebookPage.tsx      # Main notebook layout
                UploadPanel.tsx       # CSV upload dialog
                TableCellView.tsx     # Table data display
                ChartCellView.tsx     # ECharts rendering
                SchemaView.tsx        # Column schema display
                ManipulationPanel.tsx  # Operation controls
                VisualizationPanel.tsx # Chart configuration
                TablePreview.tsx      # Table row preview
                MapCanvas.tsx         # Map point visualization
                ErrorToast.tsx        # Error notification
            lib/
                chartTypes.ts         # Chart type definitions
                csv.ts                # CSV utility functions
                format.ts             # Number/formatting helpers
            styles/
                index.css             # Global styles and Tailwind
    docs/                            # Documentation (this directory)

Adding a New Operation

Follow these steps to add a new data operation (e.g., deduplicate) to the system:

Step 1: Define the OperationDef

Add a new entry to the OPERATIONS dictionary in backend/visual_notebook/domain/operations.py:

"deduplicate": OperationDef(
    key="deduplicate",
    label="Remove Duplicates",
    description="Remove duplicate rows based on specified columns",
    param_schema={
        "type": "object",
        "required": ["columns"],
        "properties": {
            "columns": {"type": "array", "items": {"type": "string"}},
            "keep": {"type": "string", "enum": ["first", "last", "none"], "default": "first"},
        },
    },
    validate=_validate_deduplicate,
    apply=_apply_deduplicate,
),

Step 2: Implement the Validate Function

def _validate_deduplicate(params: dict, schema: list[dict]) -> dict:
    columns = params.get("columns", [])
    if not columns:
        raise ValueError("'columns' list is required")
    params.setdefault("keep", "first")
    return params

The validate function should check that all required parameters are present and have valid values. It receives the raw user params and the current table schema.

Step 3: Implement the Apply Function

Provide both DuckDB and Polars implementations:

def _apply_deduplicate(ref: DataFrameRef, params: dict) -> DataFrameRef:
    columns = params["columns"]
    keep = params.get("keep", "first")

    if ref.backend == "duckdb" and ref.name and ref.con:
        new_name = f"_dedup_{uuid.uuid4().hex[:6]}"
        cols = ", ".join(f'"{c}"' for c in columns)

        if keep == "first":
            sql = f'''
                SELECT DISTINCT ON ({cols}) *
                FROM "{ref.name}"
                ORDER BY {cols}
            '''
        elif keep == "last":
            sql = f'''
                SELECT DISTINCT ON ({cols}) *
                FROM "{ref.name}"
                ORDER BY {cols} DESC
            '''
        else:
            sql = f'''
                SELECT * FROM "{ref.name}"
                GROUP BY {cols}
                HAVING COUNT(*) = 1
            '''

        ref.con.execute(f'CREATE OR REPLACE TABLE "{new_name}" AS {sql}')
        return DataFrameRef(backend="duckdb", name=new_name, con=ref.con, source_path=ref.source_path)

    elif ref.backend == "polars" and ref.lf is not None:
        lf = ref.lf.unique(subset=columns, keep=keep)
        return DataFrameRef(backend="polars", lf=lf, source_path=ref.source_path)

    raise ValueError("Invalid DataFrameRef")

Step 4: Add Tests

Add test cases to backend/visual_notebook/tests/test_operations.py:

def test_deduplicate_validation():
    op = OPERATIONS["deduplicate"]
    with pytest.raises(ValueError):
        op.validate({}, [])

    result = op.validate({"columns": ["a", "b"]}, [])
    assert result["keep"] == "first"

def test_deduplicate_in_registry():
    assert "deduplicate" in OPERATIONS

Step 5: Verify

1. Run the tests: uv run pytest backend/visual_notebook/tests/ -v
2. Start the server and verify the operation appears in GET /api/ops.
3. Test the operation via the Swagger UI at http://localhost:8000/docs.

Adding a New Chart Type

Follow these steps to add a new chart type (e.g., pie) to the system:

Step 1: Create the Computation Module

Create a new file backend/visual_notebook/viz/pie.py:

"""Pie chart generation."""
from __future__ import annotations
from typing import Any
import duckdb
from visual_notebook.viz._common import PALETTE, base_chart, title, tooltip_style, palette_colors


def compute_pie(
    con: duckdb.DuckDBPyConnection,
    table_name: str,
    category_col: str,
    value_col: str | None = None,
    n: int = 10,
) -> dict[str, Any]:
    """Compute pie chart data and return ECharts spec."""
    if value_col:
        result = con.execute(f'''
            SELECT "{category_col}" as cat, SUM("{value_col}") as val
            FROM "{table_name}"
            WHERE "{category_col}" IS NOT NULL
            GROUP BY cat
            ORDER BY val DESC
            LIMIT {n}
        ''').fetchall()
    else:
        result = con.execute(f'''
            SELECT "{category_col}" as cat, COUNT(*) as val
            FROM "{table_name}"
            WHERE "{category_col}" IS NOT NULL
            GROUP BY cat
            ORDER BY val DESC
            LIMIT {n}
        ''').fetchall()

    data = [{"name": str(r[0]), "value": float(r[1])} for r in result]
    colors = palette_colors(len(data))

    chart = base_chart()
    chart["title"] = title(f"Distribution of {category_col}")
    chart["tooltip"] = {**tooltip_style(), "trigger": "item", "formatter": "{b}: {c} ({d}%)"}
    chart["legend"] = {"orient": "vertical", "left": "left", "top": "middle"}
    chart["series"] = [{
        "type": "pie",
        "radius": ["40%", "70%"],
        "data": data,
        "emphasis": {"itemStyle": {"shadowBlur": 10, "shadowOffsetX": 0, "shadowColor": "rgba(0,0,0,0.5)"}},
    }]

    return chart

Step 2: Register in the Route Handler

Add the import and handler in backend/visual_notebook/api/routes.py:

from visual_notebook.viz.pie import compute_pie

# In the create_chart endpoint:
elif req.chart_type == "pie":
    spec = compute_pie(con, table, params["category_col"], params.get("value_col"), params.get("n", 10))

Step 3: Add Frontend Support

Update frontend/src/lib/chartTypes.ts to include the new chart type, and add a configuration form in the VisualizationPanel component.

Adding a New Frontend Component

Step 1: Create the Component File

Create a new file in frontend/src/components/:

// StatsPanel.tsx
import React from 'react';
import { useSessionStore } from '../store/session';

export function StatsPanel() {
    const { columns, rowCount } = useSessionStore();

    return (
        <div className="bg-white rounded-lg shadow p-4">
            <h3 className="text-sm font-semibold text-gray-700 mb-3">Dataset Statistics</h3>
            <div className="space-y-2">
                <div className="flex justify-between text-sm">
                    <span className="text-gray-500">Total Rows</span>
                    <span className="font-mono">{rowCount.toLocaleString()}</span>
                </div>
                <div className="flex justify-between text-sm">
                    <span className="text-gray-500">Total Columns</span>
                    <span className="font-mono">{columns.length}</span>
                </div>
            </div>
        </div>
    );
}

Step 2: Use Tailwind Classes

Follow the project's Tailwind conventions:

• Use brand-* colors for primary actions and accents.
• Use gray-* colors for neutral text, borders, and backgrounds.
• Use text-sm for body text, text-xs for secondary labels.
• Use rounded-lg for card corners, shadow for elevation.
• Use flex and grid for layout, with gap-* for spacing.

Step 3: Integrate into the Application

Import and render the component in the appropriate parent:

// In NotebookPage.tsx
import { StatsPanel } from './StatsPanel';

// Render conditionally
{showStatsPanel && <StatsPanel />}

Step 4: Add a UI Toggle

If the component should be togglable, add a flag to useUIStore:

showStatsPanel: boolean;
toggleStatsPanel: () => void;

Testing

Backend Tests

The backend uses pytest with test files in backend/visual_notebook/tests/.

Run all tests:

uv run pytest backend/visual_notebook/tests/ -v

Run a specific test file:

uv run pytest backend/visual_notebook/tests/test_operations.py -v

Run with coverage (if pytest-cov is installed):

uv run pytest backend/visual_notebook/tests/ -v --cov=visual_notebook

Test structure:

• test_history.py: Tests for commit, undo, redo, branch, and tree serialization.
• test_operations.py: Tests for the operations registry, parameter validation, and default values.

Frontend Testing

The frontend currently does not have automated tests configured. For future implementation, consider:

• Vitest: Fast unit testing framework compatible with Vite.
• React Testing Library: Component testing with DOM assertions.
• Playwright: End-to-end testing for API integration.

Code Style and Conventions

Python (Backend)

• Line length: 100 characters maximum.
• Type hints: Use type hints on all function signatures. Import from __future__ import annotations for modern type syntax.
• Docstrings: Use triple-quoted docstrings for all public functions and classes.
• Naming: snake_case for functions and variables, PascalCase for classes.
• Imports: Standard library first, then third-party, then local. Use absolute imports within the package.
• f-strings: Use f-strings for SQL query construction (the SQL injection risk is mitigated by server-generated table names and column names from the schema).
• Error handling: Raise ValueError in validators, HTTPException in route handlers.

TypeScript (Frontend)

• Strict mode: TypeScript strict mode is enabled. All variables must have explicit or inferred types.
• Functional components: Use function declarations with explicit return types.
• Naming: camelCase for variables and functions, PascalCase for components and types.
• Imports: Use relative paths for local modules, package names for third-party libraries.
• Hooks: Custom hooks in api/hooks.ts follow the use[X] naming pattern and return { action, loading, error }.
• CSS: Use Tailwind utility classes exclusively. Avoid inline styles except for dynamic values (chart dimensions, SVG coordinates).

Git Workflow

Branch Naming

• feature/[description]: New features
• fix/[description]: Bug fixes
• docs/[description]: Documentation changes
• refactor/[description]: Code refactoring

Commit Messages

Follow conventional commit format:

type(scope): description

[optional body]

Types: feat, fix, docs, refactor, test, chore

Examples:

• feat(operations): add deduplicate operation
• fix(history): correct undo pointer after branch
• docs(api): update endpoint descriptions

Pull Request Process

1. Create a feature branch from main.
2. Implement changes with tests.
3. Run the full test suite: uv run pytest.
4. Ensure the frontend builds: cd frontend && npm run build.
5. Open a pull request with a description of the changes and any manual testing performed.
6. Request review from a maintainer.
7. Address review feedback and re-run tests.
8. Merge when approved and all checks pass.