Spaces:

Sid-WC121
/

visual-notebook

Sleeping

Sid-WC121

bug fix1

018ab56 6 months ago

11.3 kB

# Backend Deep Dive

This document covers the FastAPI backend architecture in detail, including application lifecycle, middleware, dependency injection, request validation, session management, and error handling.

## Application Structure

The backend code lives under `backend/visual_notebook/` and follows a layered architecture:

```
backend/visual_notebook/
    main.py              # FastAPI app, lifespan, middleware, static serving
    config.py            # Settings dataclass from environment variables
    session.py           # Session and SessionStore classes
    api/
        routes.py        # All API endpoint definitions
        schemas.py       # Pydantic request/response models
        deps.py          # FastAPI dependency injection functions
    domain/
        dataframe.py     # DataFrameRef unified wrapper
        history.py       # History tree (Memento pattern)
        operations.py    # Operation registry and implementations
        checkpoints.py   # Checkpoint persistence
        validators.py    # Column validation helpers
    data/
        csv_loader.py    # CSV loading, sampling, and export
        schema.py        # Schema inference and statistics
        types.py         # ColumnType enum and classification
    viz/
        _common.py       # Shared chart utilities
        histogram.py     # Histogram computation
        scatter.py       # Scatter plot computation
        bar_topn.py      # Top-N bar chart computation
        timeline.py      # Timeline/line chart computation
        heatmap.py       # Heatmap computation
        map.py           # Map/geo computation
    tests/
        test_history.py  # History module tests
        test_operations.py # Operations module tests
```

## FastAPI Application Entry Point (main.py)

### Lifespan Handler

The application uses FastAPI's `lifespan` context manager to manage startup and shutdown resources. On startup, it:

1. Creates the `DATA_DIR` directory if it does not exist, ensuring uploaded CSV files have a storage location.
2. Creates the `CHECKPOINT_DIR` directory for session checkpoint persistence.
3. Instantiates a `SessionStore` and attaches it to `app.state.session_store`.
4. Creates a DuckDB in-memory connection and attaches it to `app.state.duckdb_con`. This connection serves as the default for new sessions.

On shutdown, the DuckDB connection is closed. Because the connection is in-memory (using `:memory:`), closing it releases all associated resources.

```python
@asynccontextmanager
async def lifespan(app: FastAPI):
    Path(settings.data_dir).mkdir(parents=True, exist_ok=True)
    Path(settings.checkpoint_dir).mkdir(parents=True, exist_ok=True)
    app.state.session_store = SessionStore()
    app.state.duckdb_con = duckdb.connect(":memory:", read_only=False)
    yield
    app.state.duckdb_con.close()
```

### CORS Middleware

Cross-Origin Resource Sharing is configured via the `CORSMiddleware`. The allowed origins are read from the `CORS_ORIGINS` environment variable (comma-separated). Credentials are enabled so that the session cookie is sent with cross-origin requests. All HTTP methods and headers are permitted.

```python
app.add_middleware(
    CORSMiddleware,
    allow_origins=settings.cors_origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)
```

In production, the `CORS_ORIGINS` variable should be restricted to the actual frontend domain. The default development values allow `http://localhost:5173` and `http://localhost:3000`.

### API Router

All API endpoints are defined in `api/routes.py` and mounted under the `/api` prefix:

```python
app.include_router(api_router, prefix="/api")
```

### Health Endpoint

A health check endpoint is defined directly on the application (not under `/api`) for infrastructure monitoring:

```python
@app.get("/health")
async def health():
    return {"status": "ok", "version": "0.1.0"}
```

## API Routes Layer

All endpoints are organized in a single `APIRouter` instance within `routes.py`. The router handles the following endpoint groups:

| Group | Endpoints | Purpose |
|-------|-----------|---------|
| Data loading | `POST /upload/csv` | Upload and register CSV files |
| Schema | `GET /schema` | Retrieve column schema and statistics |
| Rows | `GET /rows` | Paginated row retrieval |
| Operations | `GET /ops`, `POST /ops/{op_key}` | List and apply data transformations |
| History | `POST /history/undo`, `POST /history/redo`, `POST /history/branch`, `GET /history` | Undo/redo/branch navigation |
| Visualization | `POST /viz/chart` | Chart computation |
| Export | `GET /export/csv` | Export current data as CSV |
| Checkpoints | `POST /checkpoint`, `GET /checkpoints` | Save and list session checkpoints |

Each endpoint uses FastAPI's dependency injection to obtain the current session, DuckDB connection, and DataFrameRef as needed.

## Dependency Injection (deps.py)

The `deps.py` module provides four dependency injection functions that are reused across all API endpoints:

### `get_session_store`

Retrieves the `SessionStore` from the application state. This is a simple accessor that reads `request.app.state.session_store`.

### `get_duckdb_con`

Retrieves the shared DuckDB connection from `request.app.state.duckdb_con`. This connection is used for CSV registration and as the default connection for new sessions.

### `get_session`
The primary session resolution dependency. It reads the `session_id` cookie from the request and calls `SessionStore.get_or_create()`. If the cookie is absent or the session ID is not found, a new session is created automatically. This ensures that every request has a valid session.

```python
def get_session(request: Request, store: SessionStore = Depends(get_session_store), con=Depends(get_duckdb_con)) -> Session:
    session_id = request.cookies.get("session_id")
    session = store.get_or_create(session_id, con)
    return session
```

### `get_current_ref`

Returns the current `DataFrameRef` from the session. Raises `HTTPException(400)` if no dataset has been loaded. This dependency is used by endpoints that require an active dataset (schema, rows, operations, visualization).

### `get_history`
Returns the `History` tree from the current session. A convenience dependency that chains `get_session` and accesses `session.history`.

## Pydantic Schemas

All request and response payloads are validated through Pydantic models defined in `api/schemas.py`. Key models include:

| Model | Purpose | Key Fields |
|-------|---------|------------|
| `ColumnInfo` | Column metadata and statistics | name, dtype, column_type, null_count, unique_count, min, max, mean, std, q25, q50, q75, count |
| `UploadResponse` | CSV upload result | dataset_id, table_name, row_count, columns, session_id |
| `SchemaResponse` | Schema query result | table_name, row_count, columns |
| `ApplyOpRequest` | Operation application request | op_key, params |
| `ApplyOpResponse` | Operation application result | state_id, columns_schema, row_count, sample_rows |
| `ChartRequest` | Chart computation request | chart_type, params |
| `ChartResponse` | Chart computation result | chart_type, spec |
| `HistoryNode` | History tree node | id, operation, params, parent_id, children, is_current |
| `HistoryResponse` | Full history tree | nodes, current_id, order |
| `UndoRedoResponse` | Undo/redo result | state_id, operation, columns_schema, row_count |
| `OperationInfo` | Operation descriptor | key, label, description, param_schema |
| `CheckpointResponse` | Checkpoint save result | path, timestamp, state_id |

Pydantic v2 is used throughout, providing automatic serialization, validation, and OpenAPI schema generation. The `model_dump()` method is used to serialize responses to dictionaries.

## Session Management

### Session Dataclass

Each `Session` object holds:

- `id`: A 12-character hex string identifying the session.
- `duckdb_con`: The DuckDB connection for this session.
- `current_table`: The name of the currently active DuckDB table.
- `current_ref`: The `DataFrameRef` wrapping the current data state.
- `history`: The `History` tree tracking all operations.
- `metadata`: An extensible dictionary for future session-level data.

### SessionStore

The `SessionStore` is an in-memory dictionary that maps session IDs to `Session` objects. It provides:

- `create(duckdb_con)`: Generates a new session ID, creates a `Session` with a fresh `History`, and stores it.
- `get(session_id)`: Returns the session or `None`.
- `get_or_create(session_id, duckdb_con)`: Returns an existing session or creates a new one.
- `remove(session_id)`: Removes a session from the store.

### Cookie-Based Session Binding

After a CSV upload, the response sets an `httponly` cookie named `session_id` with a 7-day expiration:

```python
resp.set_cookie("session_id", session.id, httponly=True, max_age=86400 * 7)
```

The `httponly` flag prevents JavaScript access, improving security against XSS attacks. Subsequent requests include this cookie automatically, allowing the backend to resolve the session.

## CORS Configuration

The CORS policy is configured through the `CORS_ORIGINS` environment variable. In development, this defaults to `http://localhost:5173,http://localhost:3000`. In production Docker deployments, it is typically set to `*` or the specific frontend domain. Credentials are enabled to support cookie-based session management across origins.

## Static File Serving for Production

In production, the backend serves the compiled React frontend. The `_try_mount_static()` function attempts to mount the static directory (defaulting to `/app/static`):

1. Mounts `/assets` as a static file directory for JS, CSS, and other build artifacts.
2. Mounts `/static` as a catch-all for other static resources.
3. Registers a catch-all `/{full_path:path}` route that serves `index.html` for client-side routing (SPA fallback).

The function gracefully handles missing directories and permission errors, making it safe to run in development (where the frontend is served by Vite) without the static build present.

## Error Handling Patterns

The backend uses a layered error handling approach:

### HTTPException in Routes

Endpoints raise `HTTPException` with appropriate status codes:

| Status | Condition |
|--------|-----------|
| 400 | No dataset loaded, invalid operation state |
| 404 | Operation key not found, history state not found |
| 413 | File exceeds `MAX_UPLOAD_SIZE` |
| 422 | Validation error (from Pydantic or custom validators) |
| 500 | Operation execution failure, chart computation failure |

### Validation Errors

Operation validators raise `ValueError` for invalid parameters. The route handler catches these and converts them to `HTTPException(422)` with the validation message as the detail.

### Graceful Degradation

Non-critical operations such as checkpoint Parquet export are wrapped in try-except blocks that silently ignore failures. This ensures that a checkpoint save still succeeds even if the Parquet export fails.

### Frontend Error Display

The response error interceptor in the Axios client extracts the `detail` field from error responses and surfaces it through the `ErrorToast` component, providing user-visible feedback for all backend errors.

	# Backend Deep Dive

	This document covers the FastAPI backend architecture in detail, including application lifecycle, middleware, dependency injection, request validation, session management, and error handling.

	## Application Structure

	The backend code lives under `backend/visual_notebook/` and follows a layered architecture:

	```
	backend/visual_notebook/
	main.py # FastAPI app, lifespan, middleware, static serving
	config.py # Settings dataclass from environment variables
	session.py # Session and SessionStore classes
	api/
	routes.py # All API endpoint definitions
	schemas.py # Pydantic request/response models
	deps.py # FastAPI dependency injection functions
	domain/
	dataframe.py # DataFrameRef unified wrapper
	history.py # History tree (Memento pattern)
	operations.py # Operation registry and implementations
	checkpoints.py # Checkpoint persistence
	validators.py # Column validation helpers
	data/
	csv_loader.py # CSV loading, sampling, and export
	schema.py # Schema inference and statistics
	types.py # ColumnType enum and classification
	viz/
	_common.py # Shared chart utilities
	histogram.py # Histogram computation
	scatter.py # Scatter plot computation
	bar_topn.py # Top-N bar chart computation
	timeline.py # Timeline/line chart computation
	heatmap.py # Heatmap computation
	map.py # Map/geo computation
	tests/
	test_history.py # History module tests
	test_operations.py # Operations module tests
	```

	## FastAPI Application Entry Point (main.py)

	### Lifespan Handler

	The application uses FastAPI's `lifespan` context manager to manage startup and shutdown resources. On startup, it:

	1. Creates the `DATA_DIR` directory if it does not exist, ensuring uploaded CSV files have a storage location.
	2. Creates the `CHECKPOINT_DIR` directory for session checkpoint persistence.
	3. Instantiates a `SessionStore` and attaches it to `app.state.session_store`.
	4. Creates a DuckDB in-memory connection and attaches it to `app.state.duckdb_con`. This connection serves as the default for new sessions.

	On shutdown, the DuckDB connection is closed. Because the connection is in-memory (using `:memory:`), closing it releases all associated resources.

	```python
	@asynccontextmanager
	async def lifespan(app: FastAPI):
	Path(settings.data_dir).mkdir(parents=True, exist_ok=True)
	Path(settings.checkpoint_dir).mkdir(parents=True, exist_ok=True)
	app.state.session_store = SessionStore()
	app.state.duckdb_con = duckdb.connect(":memory:", read_only=False)
	yield
	app.state.duckdb_con.close()
	```

	### CORS Middleware

	Cross-Origin Resource Sharing is configured via the `CORSMiddleware`. The allowed origins are read from the `CORS_ORIGINS` environment variable (comma-separated). Credentials are enabled so that the session cookie is sent with cross-origin requests. All HTTP methods and headers are permitted.

	```python
	app.add_middleware(
	CORSMiddleware,
	allow_origins=settings.cors_origins,
	allow_credentials=True,
	allow_methods=["*"],
	allow_headers=["*"],
	)
	```

	In production, the `CORS_ORIGINS` variable should be restricted to the actual frontend domain. The default development values allow `http://localhost:5173` and `http://localhost:3000`.

	### API Router

	All API endpoints are defined in `api/routes.py` and mounted under the `/api` prefix:

	```python
	app.include_router(api_router, prefix="/api")
	```

	### Health Endpoint

	A health check endpoint is defined directly on the application (not under `/api`) for infrastructure monitoring:

	```python
	@app.get("/health")
	async def health():
	return {"status": "ok", "version": "0.1.0"}
	```

	## API Routes Layer

	All endpoints are organized in a single `APIRouter` instance within `routes.py`. The router handles the following endpoint groups:

	\| Group \| Endpoints \| Purpose \|
	\|-------\|-----------\|---------\|
	\| Data loading \| `POST /upload/csv` \| Upload and register CSV files \|
	\| Schema \| `GET /schema` \| Retrieve column schema and statistics \|
	\| Rows \| `GET /rows` \| Paginated row retrieval \|
	\| Operations \| `GET /ops`, `POST /ops/{op_key}` \| List and apply data transformations \|
	\| History \| `POST /history/undo`, `POST /history/redo`, `POST /history/branch`, `GET /history` \| Undo/redo/branch navigation \|
	\| Visualization \| `POST /viz/chart` \| Chart computation \|
	\| Export \| `GET /export/csv` \| Export current data as CSV \|
	\| Checkpoints \| `POST /checkpoint`, `GET /checkpoints` \| Save and list session checkpoints \|

	Each endpoint uses FastAPI's dependency injection to obtain the current session, DuckDB connection, and DataFrameRef as needed.

	## Dependency Injection (deps.py)

	The `deps.py` module provides four dependency injection functions that are reused across all API endpoints:

	### `get_session_store`

	Retrieves the `SessionStore` from the application state. This is a simple accessor that reads `request.app.state.session_store`.

	### `get_duckdb_con`

	Retrieves the shared DuckDB connection from `request.app.state.duckdb_con`. This connection is used for CSV registration and as the default connection for new sessions.

	### `get_session`

	The primary session resolution dependency. It reads the `session_id` cookie from the request and calls `SessionStore.get_or_create()`. If the cookie is absent or the session ID is not found, a new session is created automatically. This ensures that every request has a valid session.

	```python
	def get_session(request: Request, store: SessionStore = Depends(get_session_store), con=Depends(get_duckdb_con)) -> Session:
	session_id = request.cookies.get("session_id")
	session = store.get_or_create(session_id, con)
	return session
	```

	### `get_current_ref`

	Returns the current `DataFrameRef` from the session. Raises `HTTPException(400)` if no dataset has been loaded. This dependency is used by endpoints that require an active dataset (schema, rows, operations, visualization).

	### `get_history`

	Returns the `History` tree from the current session. A convenience dependency that chains `get_session` and accesses `session.history`.

	## Pydantic Schemas

	All request and response payloads are validated through Pydantic models defined in `api/schemas.py`. Key models include:

	\| Model \| Purpose \| Key Fields \|
	\|-------\|---------\|------------\|
	\| `ColumnInfo` \| Column metadata and statistics \| name, dtype, column_type, null_count, unique_count, min, max, mean, std, q25, q50, q75, count \|
	\| `UploadResponse` \| CSV upload result \| dataset_id, table_name, row_count, columns, session_id \|
	\| `SchemaResponse` \| Schema query result \| table_name, row_count, columns \|
	\| `ApplyOpRequest` \| Operation application request \| op_key, params \|
	\| `ApplyOpResponse` \| Operation application result \| state_id, columns_schema, row_count, sample_rows \|
	\| `ChartRequest` \| Chart computation request \| chart_type, params \|
	\| `ChartResponse` \| Chart computation result \| chart_type, spec \|
	\| `HistoryNode` \| History tree node \| id, operation, params, parent_id, children, is_current \|
	\| `HistoryResponse` \| Full history tree \| nodes, current_id, order \|
	\| `UndoRedoResponse` \| Undo/redo result \| state_id, operation, columns_schema, row_count \|
	\| `OperationInfo` \| Operation descriptor \| key, label, description, param_schema \|
	\| `CheckpointResponse` \| Checkpoint save result \| path, timestamp, state_id \|

	Pydantic v2 is used throughout, providing automatic serialization, validation, and OpenAPI schema generation. The `model_dump()` method is used to serialize responses to dictionaries.

	## Session Management

	### Session Dataclass

	Each `Session` object holds:

	- `id`: A 12-character hex string identifying the session.
	- `duckdb_con`: The DuckDB connection for this session.
	- `current_table`: The name of the currently active DuckDB table.
	- `current_ref`: The `DataFrameRef` wrapping the current data state.
	- `history`: The `History` tree tracking all operations.
	- `metadata`: An extensible dictionary for future session-level data.

	### SessionStore

	The `SessionStore` is an in-memory dictionary that maps session IDs to `Session` objects. It provides:

	- `create(duckdb_con)`: Generates a new session ID, creates a `Session` with a fresh `History`, and stores it.
	- `get(session_id)`: Returns the session or `None`.
	- `get_or_create(session_id, duckdb_con)`: Returns an existing session or creates a new one.
	- `remove(session_id)`: Removes a session from the store.

	### Cookie-Based Session Binding

	After a CSV upload, the response sets an `httponly` cookie named `session_id` with a 7-day expiration:

	```python
	resp.set_cookie("session_id", session.id, httponly=True, max_age=86400 * 7)
	```

	The `httponly` flag prevents JavaScript access, improving security against XSS attacks. Subsequent requests include this cookie automatically, allowing the backend to resolve the session.

	## CORS Configuration

	The CORS policy is configured through the `CORS_ORIGINS` environment variable. In development, this defaults to `http://localhost:5173,http://localhost:3000`. In production Docker deployments, it is typically set to `*` or the specific frontend domain. Credentials are enabled to support cookie-based session management across origins.

	## Static File Serving for Production

	In production, the backend serves the compiled React frontend. The `_try_mount_static()` function attempts to mount the static directory (defaulting to `/app/static`):

	1. Mounts `/assets` as a static file directory for JS, CSS, and other build artifacts.
	2. Mounts `/static` as a catch-all for other static resources.
	3. Registers a catch-all `/{full_path:path}` route that serves `index.html` for client-side routing (SPA fallback).

	The function gracefully handles missing directories and permission errors, making it safe to run in development (where the frontend is served by Vite) without the static build present.

	## Error Handling Patterns

	The backend uses a layered error handling approach:

	### HTTPException in Routes

	Endpoints raise `HTTPException` with appropriate status codes:

	\| Status \| Condition \|
	\|--------\|-----------\|
	\| 400 \| No dataset loaded, invalid operation state \|
	\| 404 \| Operation key not found, history state not found \|
	\| 413 \| File exceeds `MAX_UPLOAD_SIZE` \|
	\| 422 \| Validation error (from Pydantic or custom validators) \|
	\| 500 \| Operation execution failure, chart computation failure \|

	### Validation Errors

	Operation validators raise `ValueError` for invalid parameters. The route handler catches these and converts them to `HTTPException(422)` with the validation message as the detail.

	### Graceful Degradation

	Non-critical operations such as checkpoint Parquet export are wrapped in try-except blocks that silently ignore failures. This ensures that a checkpoint save still succeeds even if the Parquet export fails.

	### Frontend Error Display

	The response error interceptor in the Axios client extracts the `detail` field from error responses and surfaces it through the `ErrorToast` component, providing user-visible feedback for all backend errors.