docs/deployment.md

Deployment Guide

This document covers all deployment options for the Visual Notebook, including Docker builds, Docker Compose for development, CI/CD pipelines, Hugging Face Spaces deployment, environment configuration, and production considerations.

Docker Build Options

The project provides three Dockerfile configurations for different deployment scenarios:

Combined Build (Dockerfile)

The combined Dockerfile builds both the frontend and backend into a single image. This is the recommended approach for most deployments, including HF Spaces.

Build stages:

1. frontend-build (Node.js 20): Installs npm dependencies and builds the React application with npm run build. The output is written to frontend/dist/.

2. Runtime (Python 3.11): Installs the backend using uv pip install --system -e ., copies the built frontend from the previous stage into /app/static, and configures environment variables.

# Stage 1: Build frontend
FROM node:20-slim AS frontend-build
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json* ./
RUN npm install
COPY frontend/ ./
RUN npm run build

# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app
RUN pip install uv
COPY backend/pyproject.toml backend/
COPY backend/visual_notebook/ backend/visual_notebook/
RUN cd backend && uv pip install --system -e .
COPY datasets/ /app/datasets/
COPY --from=frontend-build /app/frontend/dist /app/static
ENV DATA_DIR=/app/datasets
ENV CHECKPOINT_DIR=/app/checkpoints
ENV CORS_ORIGINS=*
ENV HOST=0.0.0.0
ENV PORT=7860
EXPOSE 7860
CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]

Build and run:

docker build -t visual-notebook .
docker run -p 7860:7860 -v $(pwd)/datasets:/app/datasets visual-notebook

Standalone Backend (Dockerfile.backend)

The backend-only Dockerfile skips the frontend build entirely. It is intended for deployments where the frontend is served separately (e.g., by a CDN or Nginx).

FROM python:3.11-slim
WORKDIR /app
RUN pip install uv
COPY backend/pyproject.toml backend/
COPY backend/visual_notebook/ backend/visual_notebook/
COPY datasets/ /app/datasets/
RUN cd backend && uv pip install --system -e .
ENV DATA_DIR=/app/datasets
ENV CHECKPOINT_DIR=/app/checkpoints
ENV CORS_ORIGINS=*
ENV HOST=0.0.0.0
ENV PORT=7860
EXPOSE 7860
CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]

Frontend-Only (frontend/Dockerfile)

The frontend-only Dockerfile builds the React application and serves it with Nginx:

FROM node:20-slim AS build
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

The Nginx configuration (frontend/nginx.conf) is set up to proxy API requests to the backend and serve static files for all other routes.

Docker Compose for Development

The docker-compose.yml defines two services for local development:

services:
  backend:
    build:
      context: .
      dockerfile: Dockerfile.backend
    ports:
      - "8000:7860"
    volumes:
      - ./datasets:/app/datasets
      - ./checkpoints:/app/checkpoints
    environment:
      - DATA_DIR=/app/datasets
      - CHECKPOINT_DIR=/app/checkpoints
      - CORS_ORIGINS=http://localhost:5173,http://localhost:3000

  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    ports:
      - "5173:80"
    environment:
      - VITE_API_URL=http://localhost:8000
    depends_on:
      - backend

Usage:

# Start both services
docker compose up --build

# Backend available at http://localhost:8000
# Frontend available at http://localhost:5173

The volume mounts ensure that uploaded datasets and checkpoints persist across container restarts.

GitHub Actions CI/CD

The recommended GitHub Actions workflow follows a test-build-deploy pipeline:

Workflow Stages

1. Test: Run pytest against the backend with uv run pytest.
2. Build: Build the Docker image using the combined Dockerfile.
3. Deploy: Push the image to a container registry and deploy.

Example Workflow

name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
        with:
          version: "latest"
      - name: Install dependencies
        run: uv sync
      - name: Run tests
        run: uv run pytest backend/visual_notebook/tests/ -v

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

UV Caching

The setup-uv action caches the uv installation and downloaded packages. For Docker builds, the cache-from and cache-to options use GitHub Actions cache to persist Docker layers between builds, significantly reducing build times for subsequent pushes.

GitLab CI/CD Pipeline

For GitLab deployments, the following pipeline structure is recommended:

stages:
  - test
  - build
  - deploy

test:
  stage: test
  image: python:3.11-slim
  before_script:
    - pip install uv
    - uv sync
  script:
    - uv run pytest backend/visual_notebook/tests/ -v

build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main

deploy:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - curl -X POST "$DEPLOY_WEBHOOK_URL"
  only:
    - main

Hugging Face Spaces Deployment

The combined Dockerfile is designed to be compatible with HF Spaces requirements:

• Port 7860: HF Spaces expects the application to listen on port 7860. The Dockerfile EXPOSE 7860 and the uvicorn command use this port.
• Static file serving: The backend serves the frontend from /app/static, which is populated during the Docker build.
• Environment variables: HF Spaces allows setting secrets and environment variables through the Space settings UI.

Deployment Steps

1. Create a new Docker Space on Hugging Face.
2. Clone the Space repository.
3. Copy the project files (or add the Space as a git remote).
4. Push to the Space repository. HF Spaces will build and deploy automatically.

Space Configuration

Set the following secrets in the Space settings:

Variable	Value
DATA_DIR	/app/datasets
CHECKPOINT_DIR	/app/checkpoints
CORS_ORIGINS	* (or your specific domain)
MAX_UPLOAD_SIZE	524288000 (500 MB)

Persistent Storage

HF Spaces offers persistent storage for paid tiers. To enable it, mount a persistent volume at /app/datasets and /app/checkpoints. Without persistent storage, uploaded files are lost when the Space restarts.

Environment Variable Configuration

All configuration is managed through environment variables, read by the Settings dataclass in config.py:

Variable	Default	Description
DATA_DIR	datasets	Directory for uploaded CSV files
CHECKPOINT_DIR	checkpoints	Directory for session checkpoints
MAX_UPLOAD_SIZE	524288000 (500 MB)	Maximum CSV file size in bytes
PREVIEW_ROWS	50	Number of rows returned by default in head()
SAMPLE_ROWS	10000	Number of rows for Polars CSV sampling
CORS_ORIGINS	http://localhost:5173,http://localhost:3000	Comma-separated allowed origins
HOST	0.0.0.0	Server bind address
PORT	8000	Server bind port (7860 in Docker)

Production Considerations

Session Storage

The current SessionStore is in-memory, meaning all session data is lost when the server restarts. For production deployments, consider replacing it with a persistent store:

• Redis: Store session metadata in Redis with TTL-based expiration. DuckDB tables would need to be exported to Parquet and reloaded on session restore.
• Database-backed: Use a lightweight database (SQLite, PostgreSQL) to store session state and checkpoint references.
• File-based: Serialize session metadata to JSON files in the checkpoint directory.

Static File Serving

In production, the backend serves static files from /app/static. For high-traffic deployments, consider:

• Using a CDN in front of the application for static asset caching.
• Serving static files directly from Nginx rather than through FastAPI's StaticFiles middleware.
• Enabling gzip compression for JS and CSS assets.

Health Checks

The /health endpoint returns a JSON response with status and version. Configure your load balancer or orchestrator to use this endpoint for health monitoring:

{"status": "ok", "version": "0.1.0"}

For more comprehensive health checks, consider extending the endpoint to verify DuckDB connectivity and disk space availability.

Resource Limits

DuckDB operates entirely in memory. For datasets that exceed available RAM:

• Set max_memory in the DuckDB connection to enable disk spilling: duckdb.connect(":memory:", config={"memory_limit": "2GB"}).
• Use PREVIEW_ROWS and SAMPLE_ROWS to limit data sent to the frontend.
• Monitor memory usage and set container memory limits appropriately.

Scaling Considerations

The current architecture is single-process and single-session-per-container. To scale horizontally:

• Session affinity: Use cookie-based sticky sessions to route each user to the same backend instance.
• Shared storage: Use a networked filesystem (EFS, S3) for DATA_DIR so all instances can access uploaded files.
• Connection pooling: If moving to a shared DuckDB instance, implement connection pooling to manage concurrent access.
• Load balancing: Place multiple backend instances behind a load balancer with health check routing.

For most use cases, a single instance with sufficient RAM (4-8 GB) handles dozens of concurrent users comfortably, given that CSV analysis is typically an interactive, low-concurrency workload.