docs/backend.md

Backend Deep Dive

This document covers the FastAPI backend architecture in detail, including application lifecycle, middleware, dependency injection, request validation, session management, and error handling.

Application Structure

The backend code lives under backend/visual_notebook/ and follows a layered architecture:

backend/visual_notebook/
    main.py              # FastAPI app, lifespan, middleware, static serving
    config.py            # Settings dataclass from environment variables
    session.py           # Session and SessionStore classes
    api/
        routes.py        # All API endpoint definitions
        schemas.py       # Pydantic request/response models
        deps.py          # FastAPI dependency injection functions
    domain/
        dataframe.py     # DataFrameRef unified wrapper
        history.py       # History tree (Memento pattern)
        operations.py    # Operation registry and implementations
        checkpoints.py   # Checkpoint persistence
        validators.py    # Column validation helpers
    data/
        csv_loader.py    # CSV loading, sampling, and export
        schema.py        # Schema inference and statistics
        types.py         # ColumnType enum and classification
    viz/
        _common.py       # Shared chart utilities
        histogram.py     # Histogram computation
        scatter.py       # Scatter plot computation
        bar_topn.py      # Top-N bar chart computation
        timeline.py      # Timeline/line chart computation
        heatmap.py       # Heatmap computation
        map.py           # Map/geo computation
    tests/
        test_history.py  # History module tests
        test_operations.py # Operations module tests

FastAPI Application Entry Point (main.py)

Lifespan Handler

The application uses FastAPI's lifespan context manager to manage startup and shutdown resources. On startup, it:

1. Creates the DATA_DIR directory if it does not exist, ensuring uploaded CSV files have a storage location.
2. Creates the CHECKPOINT_DIR directory for session checkpoint persistence.
3. Instantiates a SessionStore and attaches it to app.state.session_store.
4. Creates a DuckDB in-memory connection and attaches it to app.state.duckdb_con. This connection serves as the default for new sessions.

On shutdown, the DuckDB connection is closed. Because the connection is in-memory (using :memory:), closing it releases all associated resources.

@asynccontextmanager
async def lifespan(app: FastAPI):
    Path(settings.data_dir).mkdir(parents=True, exist_ok=True)
    Path(settings.checkpoint_dir).mkdir(parents=True, exist_ok=True)
    app.state.session_store = SessionStore()
    app.state.duckdb_con = duckdb.connect(":memory:", read_only=False)
    yield
    app.state.duckdb_con.close()

CORS Middleware

Cross-Origin Resource Sharing is configured via the CORSMiddleware. The allowed origins are read from the CORS_ORIGINS environment variable (comma-separated). Credentials are enabled so that the session cookie is sent with cross-origin requests. All HTTP methods and headers are permitted.

app.add_middleware(
    CORSMiddleware,
    allow_origins=settings.cors_origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

In production, the CORS_ORIGINS variable should be restricted to the actual frontend domain. The default development values allow http://localhost:5173 and http://localhost:3000.

API Router

All API endpoints are defined in api/routes.py and mounted under the /api prefix:

app.include_router(api_router, prefix="/api")

Health Endpoint

A health check endpoint is defined directly on the application (not under /api) for infrastructure monitoring:

@app.get("/health")
async def health():
    return {"status": "ok", "version": "0.1.0"}

API Routes Layer

All endpoints are organized in a single APIRouter instance within routes.py. The router handles the following endpoint groups:

Group	Endpoints	Purpose
Data loading	POST /upload/csv	Upload and register CSV files
Schema	GET /schema	Retrieve column schema and statistics
Rows	GET /rows	Paginated row retrieval
Operations	GET /ops, POST /ops/{op_key}	List and apply data transformations
History	POST /history/undo, POST /history/redo, POST /history/branch, GET /history	Undo/redo/branch navigation
Visualization	POST /viz/chart	Chart computation
Export	GET /export/csv	Export current data as CSV
Checkpoints	POST /checkpoint, GET /checkpoints	Save and list session checkpoints

Each endpoint uses FastAPI's dependency injection to obtain the current session, DuckDB connection, and DataFrameRef as needed.

Dependency Injection (deps.py)

The deps.py module provides four dependency injection functions that are reused across all API endpoints:

get_session_store

Retrieves the SessionStore from the application state. This is a simple accessor that reads request.app.state.session_store.

get_duckdb_con

Retrieves the shared DuckDB connection from request.app.state.duckdb_con. This connection is used for CSV registration and as the default connection for new sessions.

get_session

The primary session resolution dependency. It reads the session_id cookie from the request and calls SessionStore.get_or_create(). If the cookie is absent or the session ID is not found, a new session is created automatically. This ensures that every request has a valid session.

def get_session(request: Request, store: SessionStore = Depends(get_session_store), con=Depends(get_duckdb_con)) -> Session:
    session_id = request.cookies.get("session_id")
    session = store.get_or_create(session_id, con)
    return session

get_current_ref

Returns the current DataFrameRef from the session. Raises HTTPException(400) if no dataset has been loaded. This dependency is used by endpoints that require an active dataset (schema, rows, operations, visualization).

get_history

Returns the History tree from the current session. A convenience dependency that chains get_session and accesses session.history.

Pydantic Schemas

All request and response payloads are validated through Pydantic models defined in api/schemas.py. Key models include:

Model	Purpose	Key Fields
ColumnInfo	Column metadata and statistics	name, dtype, column_type, null_count, unique_count, min, max, mean, std, q25, q50, q75, count
UploadResponse	CSV upload result	dataset_id, table_name, row_count, columns, session_id
SchemaResponse	Schema query result	table_name, row_count, columns
ApplyOpRequest	Operation application request	op_key, params
ApplyOpResponse	Operation application result	state_id, columns_schema, row_count, sample_rows
ChartRequest	Chart computation request	chart_type, params
ChartResponse	Chart computation result	chart_type, spec
HistoryNode	History tree node	id, operation, params, parent_id, children, is_current
HistoryResponse	Full history tree	nodes, current_id, order
UndoRedoResponse	Undo/redo result	state_id, operation, columns_schema, row_count
OperationInfo	Operation descriptor	key, label, description, param_schema
CheckpointResponse	Checkpoint save result	path, timestamp, state_id

Pydantic v2 is used throughout, providing automatic serialization, validation, and OpenAPI schema generation. The model_dump() method is used to serialize responses to dictionaries.

Session Management

Session Dataclass

Each Session object holds:

• id: A 12-character hex string identifying the session.
• duckdb_con: The DuckDB connection for this session.
• current_table: The name of the currently active DuckDB table.
• current_ref: The DataFrameRef wrapping the current data state.
• history: The History tree tracking all operations.
• metadata: An extensible dictionary for future session-level data.

SessionStore

The SessionStore is an in-memory dictionary that maps session IDs to Session objects. It provides:

• create(duckdb_con): Generates a new session ID, creates a Session with a fresh History, and stores it.
• get(session_id): Returns the session or None.
• get_or_create(session_id, duckdb_con): Returns an existing session or creates a new one.
• remove(session_id): Removes a session from the store.

Cookie-Based Session Binding

After a CSV upload, the response sets an httponly cookie named session_id with a 7-day expiration:

resp.set_cookie("session_id", session.id, httponly=True, max_age=86400 * 7)

The httponly flag prevents JavaScript access, improving security against XSS attacks. Subsequent requests include this cookie automatically, allowing the backend to resolve the session.

CORS Configuration

The CORS policy is configured through the CORS_ORIGINS environment variable. In development, this defaults to http://localhost:5173,http://localhost:3000. In production Docker deployments, it is typically set to * or the specific frontend domain. Credentials are enabled to support cookie-based session management across origins.

Static File Serving for Production

In production, the backend serves the compiled React frontend. The _try_mount_static() function attempts to mount the static directory (defaulting to /app/static):

1. Mounts /assets as a static file directory for JS, CSS, and other build artifacts.
2. Mounts /static as a catch-all for other static resources.
3. Registers a catch-all /{full_path:path} route that serves index.html for client-side routing (SPA fallback).

The function gracefully handles missing directories and permission errors, making it safe to run in development (where the frontend is served by Vite) without the static build present.

Error Handling Patterns

The backend uses a layered error handling approach:

HTTPException in Routes

Endpoints raise HTTPException with appropriate status codes:

Status	Condition
400	No dataset loaded, invalid operation state
404	Operation key not found, history state not found
413	File exceeds MAX_UPLOAD_SIZE
422	Validation error (from Pydantic or custom validators)
500	Operation execution failure, chart computation failure

Validation Errors

Operation validators raise ValueError for invalid parameters. The route handler catches these and converts them to HTTPException(422) with the validation message as the detail.

Graceful Degradation

Non-critical operations such as checkpoint Parquet export are wrapped in try-except blocks that silently ignore failures. This ensures that a checkpoint save still succeeds even if the Parquet export fails.

Frontend Error Display

The response error interceptor in the Axios client extracts the detail field from error responses and surfaces it through the ErrorToast component, providing user-visible feedback for all backend errors.