Blame for docs/deployment.md

# Deployment Guide

This document covers all deployment options for the Visual Notebook, including Docker builds, Docker Compose for development, CI/CD pipelines, Hugging Face Spaces deployment, environment configuration, and production considerations.

## Docker Build Options

The project provides three Dockerfile configurations for different deployment scenarios:

### Combined Build (Dockerfile)

The combined Dockerfile builds both the frontend and backend into a single image. This is the recommended approach for most deployments, including HF Spaces.

**Build stages**:

1. **frontend-build** (Node.js 20): Installs npm dependencies and builds the React application with `npm run build`. The output is written to `frontend/dist/`.

2. **Runtime** (Python 3.11): Installs the backend using `uv pip install --system -e .`, copies the built frontend from the previous stage into `/app/static`, and configures environment variables.

```dockerfile
# Stage 1: Build frontend
FROM node:20-slim AS frontend-build
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json* ./
RUN npm install
COPY frontend/ ./
RUN npm run build

# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app
RUN pip install uv
COPY backend/pyproject.toml backend/
COPY backend/visual_notebook/ backend/visual_notebook/
RUN cd backend && uv pip install --system -e .
COPY datasets/ /app/datasets/
COPY --from=frontend-build /app/frontend/dist /app/static
ENV DATA_DIR=/app/datasets
ENV CHECKPOINT_DIR=/app/checkpoints
ENV CORS_ORIGINS=*
ENV HOST=0.0.0.0
ENV PORT=7860
EXPOSE 7860
CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
```

**Build and run**:

```bash
docker build -t visual-notebook .
docker run -p 7860:7860 -v $(pwd)/datasets:/app/datasets visual-notebook
```

### Standalone Backend (Dockerfile.backend)

The backend-only Dockerfile skips the frontend build entirely. It is intended for deployments where the frontend is served separately (e.g., by a CDN or Nginx).

```dockerfile
FROM python:3.11-slim
WORKDIR /app
RUN pip install uv
COPY backend/pyproject.toml backend/
COPY backend/visual_notebook/ backend/visual_notebook/
COPY datasets/ /app/datasets/
RUN cd backend && uv pip install --system -e .
ENV DATA_DIR=/app/datasets
ENV CHECKPOINT_DIR=/app/checkpoints
ENV CORS_ORIGINS=*
ENV HOST=0.0.0.0
ENV PORT=7860
EXPOSE 7860
CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
```

### Frontend-Only (frontend/Dockerfile)

The frontend-only Dockerfile builds the React application and serves it with Nginx:

```dockerfile
FROM node:20-slim AS build
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
```

The Nginx configuration (`frontend/nginx.conf`) is set up to proxy API requests to the backend and serve static files for all other routes.

## Docker Compose for Development

The `docker-compose.yml` defines two services for local development:

```yaml
services:
  backend:
    build:
      context: .
      dockerfile: Dockerfile.backend
    ports:
      - "8000:7860"
    volumes:
      - ./datasets:/app/datasets
      - ./checkpoints:/app/checkpoints
    environment:
      - DATA_DIR=/app/datasets
      - CHECKPOINT_DIR=/app/checkpoints
      - CORS_ORIGINS=http://localhost:5173,http://localhost:3000

  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    ports:
      - "5173:80"
    environment:
      - VITE_API_URL=http://localhost:8000
    depends_on:
      - backend
```

**Usage**:

```bash
# Start both services
docker compose up --build

# Backend available at http://localhost:8000
# Frontend available at http://localhost:5173
```

The volume mounts ensure that uploaded datasets and checkpoints persist across container restarts.

## GitHub Actions CI/CD

The recommended GitHub Actions workflow follows a test-build-deploy pipeline:

### Workflow Stages

1. **Test**: Run pytest against the backend with `uv run pytest`.
2. **Build**: Build the Docker image using the combined Dockerfile.
3. **Deploy**: Push the image to a container registry and deploy.

### Example Workflow

```yaml
name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
        with:
          version: "latest"
      - name: Install dependencies
        run: uv sync
      - name: Run tests
        run: uv run pytest backend/visual_notebook/tests/ -v

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
```

### UV Caching

The `setup-uv` action caches the uv installation and downloaded packages. For Docker builds, the `cache-from` and `cache-to` options use GitHub Actions cache to persist Docker layers between builds, significantly reducing build times for subsequent pushes.

## GitLab CI/CD Pipeline

For GitLab deployments, the following pipeline structure is recommended:

```yaml
stages:
  - test
  - build
  - deploy

test:
  stage: test
  image: python:3.11-slim
  before_script:
    - pip install uv
    - uv sync
  script:
    - uv run pytest backend/visual_notebook/tests/ -v

build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main

deploy:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - curl -X POST "$DEPLOY_WEBHOOK_URL"
  only:
    - main
```

## Hugging Face Spaces Deployment

The combined Dockerfile is designed to be compatible with HF Spaces requirements:

- **Port 7860**: HF Spaces expects the application to listen on port 7860. The Dockerfile `EXPOSE 7860` and the uvicorn command use this port.
- **Static file serving**: The backend serves the frontend from `/app/static`, which is populated during the Docker build.
- **Environment variables**: HF Spaces allows setting secrets and environment variables through the Space settings UI.

### Deployment Steps

1. Create a new Docker Space on Hugging Face.
2. Clone the Space repository.
3. Copy the project files (or add the Space as a git remote).
4. Push to the Space repository. HF Spaces will build and deploy automatically.

### Space Configuration

Set the following secrets in the Space settings:

| Variable | Value |
|----------|-------|
| `DATA_DIR` | `/app/datasets` |
| `CHECKPOINT_DIR` | `/app/checkpoints` |
| `CORS_ORIGINS` | `*` (or your specific domain) |
| `MAX_UPLOAD_SIZE` | `524288000` (500 MB) |

### Persistent Storage

HF Spaces offers persistent storage for paid tiers. To enable it, mount a persistent volume at `/app/datasets` and `/app/checkpoints`. Without persistent storage, uploaded files are lost when the Space restarts.

## Environment Variable Configuration

All configuration is managed through environment variables, read by the `Settings` dataclass in `config.py`:

| Variable | Default | Description |
|----------|---------|-------------|
| `DATA_DIR` | `datasets` | Directory for uploaded CSV files |
| `CHECKPOINT_DIR` | `checkpoints` | Directory for session checkpoints |
| `MAX_UPLOAD_SIZE` | `524288000` (500 MB) | Maximum CSV file size in bytes |
| `PREVIEW_ROWS` | `50` | Number of rows returned by default in `head()` |
| `SAMPLE_ROWS` | `10000` | Number of rows for Polars CSV sampling |
| `CORS_ORIGINS` | `http://localhost:5173,http://localhost:3000` | Comma-separated allowed origins |
| `HOST` | `0.0.0.0` | Server bind address |
| `PORT` | `8000` | Server bind port (7860 in Docker) |

## Production Considerations

### Session Storage

The current `SessionStore` is in-memory, meaning all session data is lost when the server restarts. For production deployments, consider replacing it with a persistent store:

- **Redis**: Store session metadata in Redis with TTL-based expiration. DuckDB tables would need to be exported to Parquet and reloaded on session restore.
- **Database-backed**: Use a lightweight database (SQLite, PostgreSQL) to store session state and checkpoint references.
- **File-based**: Serialize session metadata to JSON files in the checkpoint directory.

### Static File Serving

In production, the backend serves static files from `/app/static`. For high-traffic deployments, consider:

- Using a CDN in front of the application for static asset caching.
- Serving static files directly from Nginx rather than through FastAPI's `StaticFiles` middleware.
- Enabling gzip compression for JS and CSS assets.

### Health Checks

The `/health` endpoint returns a JSON response with status and version. Configure your load balancer or orchestrator to use this endpoint for health monitoring:

```json
{"status": "ok", "version": "0.1.0"}
```

For more comprehensive health checks, consider extending the endpoint to verify DuckDB connectivity and disk space availability.

### Resource Limits

DuckDB operates entirely in memory. For datasets that exceed available RAM:

- Set `max_memory` in the DuckDB connection to enable disk spilling: `duckdb.connect(":memory:", config={"memory_limit": "2GB"})`.
- Use `PREVIEW_ROWS` and `SAMPLE_ROWS` to limit data sent to the frontend.
- Monitor memory usage and set container memory limits appropriately.

### Scaling Considerations

The current architecture is single-process and single-session-per-container. To scale horizontally:

- **Session affinity**: Use cookie-based sticky sessions to route each user to the same backend instance.
- **Shared storage**: Use a networked filesystem (EFS, S3) for `DATA_DIR` so all instances can access uploaded files.
- **Connection pooling**: If moving to a shared DuckDB instance, implement connection pooling to manage concurrent access.
- **Load balancing**: Place multiple backend instances behind a load balancer with health check routing.

For most use cases, a single instance with sufficient RAM (4-8 GB) handles dozens of concurrent users comfortably, given that CSV analysis is typically an interactive, low-concurrency workload.