Spaces:

Sid-WC121
/

visual-notebook

Sleeping

Sid-WC121

bug fix1

018ab56 6 months ago

10.9 kB

# Deployment Guide

This document covers all deployment options for the Visual Notebook, including Docker builds, Docker Compose for development, CI/CD pipelines, Hugging Face Spaces deployment, environment configuration, and production considerations.

## Docker Build Options

The project provides three Dockerfile configurations for different deployment scenarios:

### Combined Build (Dockerfile)

The combined Dockerfile builds both the frontend and backend into a single image. This is the recommended approach for most deployments, including HF Spaces.

**Build stages**:

1. **frontend-build** (Node.js 20): Installs npm dependencies and builds the React application with `npm run build`. The output is written to `frontend/dist/`.

2. **Runtime** (Python 3.11): Installs the backend using `uv pip install --system -e .`, copies the built frontend from the previous stage into `/app/static`, and configures environment variables.

```dockerfile
# Stage 1: Build frontend
FROM node:20-slim AS frontend-build
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json* ./
RUN npm install
COPY frontend/ ./
RUN npm run build
# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app
RUN pip install uv
COPY backend/pyproject.toml backend/
COPY backend/visual_notebook/ backend/visual_notebook/
RUN cd backend && uv pip install --system -e .
COPY datasets/ /app/datasets/
COPY --from=frontend-build /app/frontend/dist /app/static
ENV DATA_DIR=/app/datasets
ENV CHECKPOINT_DIR=/app/checkpoints
ENV CORS_ORIGINS=*
ENV HOST=0.0.0.0
ENV PORT=7860
EXPOSE 7860
CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
```

**Build and run**:

```bash
docker build -t visual-notebook .
docker run -p 7860:7860 -v $(pwd)/datasets:/app/datasets visual-notebook
```

### Standalone Backend (Dockerfile.backend)

The backend-only Dockerfile skips the frontend build entirely. It is intended for deployments where the frontend is served separately (e.g., by a CDN or Nginx).

```dockerfile
FROM python:3.11-slim
WORKDIR /app
RUN pip install uv
COPY backend/pyproject.toml backend/
COPY backend/visual_notebook/ backend/visual_notebook/
COPY datasets/ /app/datasets/
RUN cd backend && uv pip install --system -e .
ENV DATA_DIR=/app/datasets
ENV CHECKPOINT_DIR=/app/checkpoints
ENV CORS_ORIGINS=*
ENV HOST=0.0.0.0
ENV PORT=7860
EXPOSE 7860
CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
```

### Frontend-Only (frontend/Dockerfile)

The frontend-only Dockerfile builds the React application and serves it with Nginx:

```dockerfile
FROM node:20-slim AS build
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install
COPY . .
RUN npm run build
FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
```

The Nginx configuration (`frontend/nginx.conf`) is set up to proxy API requests to the backend and serve static files for all other routes.

## Docker Compose for Development

The `docker-compose.yml` defines two services for local development:

```yaml
services:
  backend:
    build:
      context: .
      dockerfile: Dockerfile.backend
    ports:
      - "8000:7860"
    volumes:
      - ./datasets:/app/datasets
      - ./checkpoints:/app/checkpoints
    environment:
      - DATA_DIR=/app/datasets
      - CHECKPOINT_DIR=/app/checkpoints
      - CORS_ORIGINS=http://localhost:5173,http://localhost:3000
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    ports:
      - "5173:80"
    environment:
      - VITE_API_URL=http://localhost:8000
    depends_on:
      - backend
```

**Usage**:

```bash
# Start both services
docker compose up --build
# Backend available at http://localhost:8000
# Frontend available at http://localhost:5173
```

The volume mounts ensure that uploaded datasets and checkpoints persist across container restarts.

## GitHub Actions CI/CD

The recommended GitHub Actions workflow follows a test-build-deploy pipeline:

### Workflow Stages

1. **Test**: Run pytest against the backend with `uv run pytest`.
2. **Build**: Build the Docker image using the combined Dockerfile.
3. **Deploy**: Push the image to a container registry and deploy.

### Example Workflow

```yaml
name: CI/CD
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
        with:
          version: "latest"
      - name: Install dependencies
        run: uv sync
      - name: Run tests
        run: uv run pytest backend/visual_notebook/tests/ -v
  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
```

### UV Caching

The `setup-uv` action caches the uv installation and downloaded packages. For Docker builds, the `cache-from` and `cache-to` options use GitHub Actions cache to persist Docker layers between builds, significantly reducing build times for subsequent pushes.

## GitLab CI/CD Pipeline

For GitLab deployments, the following pipeline structure is recommended:

```yaml
stages:
  - test
  - build
  - deploy
test:
  stage: test
  image: python:3.11-slim
  before_script:
    - pip install uv
    - uv sync
  script:
    - uv run pytest backend/visual_notebook/tests/ -v
build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main
deploy:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - curl -X POST "$DEPLOY_WEBHOOK_URL"
  only:
    - main
```

## Hugging Face Spaces Deployment

The combined Dockerfile is designed to be compatible with HF Spaces requirements:

- **Port 7860**: HF Spaces expects the application to listen on port 7860. The Dockerfile `EXPOSE 7860` and the uvicorn command use this port.
- **Static file serving**: The backend serves the frontend from `/app/static`, which is populated during the Docker build.
- **Environment variables**: HF Spaces allows setting secrets and environment variables through the Space settings UI.

### Deployment Steps

1. Create a new Docker Space on Hugging Face.
2. Clone the Space repository.
3. Copy the project files (or add the Space as a git remote).
4. Push to the Space repository. HF Spaces will build and deploy automatically.

### Space Configuration

Set the following secrets in the Space settings:

| Variable | Value |
|----------|-------|
| `DATA_DIR` | `/app/datasets` |
| `CHECKPOINT_DIR` | `/app/checkpoints` |
| `CORS_ORIGINS` | `*` (or your specific domain) |
| `MAX_UPLOAD_SIZE` | `524288000` (500 MB) |

### Persistent Storage

HF Spaces offers persistent storage for paid tiers. To enable it, mount a persistent volume at `/app/datasets` and `/app/checkpoints`. Without persistent storage, uploaded files are lost when the Space restarts.

## Environment Variable Configuration

All configuration is managed through environment variables, read by the `Settings` dataclass in `config.py`:

| Variable | Default | Description |
|----------|---------|-------------|
| `DATA_DIR` | `datasets` | Directory for uploaded CSV files |
| `CHECKPOINT_DIR` | `checkpoints` | Directory for session checkpoints |
| `MAX_UPLOAD_SIZE` | `524288000` (500 MB) | Maximum CSV file size in bytes |
| `PREVIEW_ROWS` | `50` | Number of rows returned by default in `head()` |
| `SAMPLE_ROWS` | `10000` | Number of rows for Polars CSV sampling |
| `CORS_ORIGINS` | `http://localhost:5173,http://localhost:3000` | Comma-separated allowed origins |
| `HOST` | `0.0.0.0` | Server bind address |
| `PORT` | `8000` | Server bind port (7860 in Docker) |

## Production Considerations

### Session Storage

The current `SessionStore` is in-memory, meaning all session data is lost when the server restarts. For production deployments, consider replacing it with a persistent store:

- **Redis**: Store session metadata in Redis with TTL-based expiration. DuckDB tables would need to be exported to Parquet and reloaded on session restore.
- **Database-backed**: Use a lightweight database (SQLite, PostgreSQL) to store session state and checkpoint references.
- **File-based**: Serialize session metadata to JSON files in the checkpoint directory.

### Static File Serving

In production, the backend serves static files from `/app/static`. For high-traffic deployments, consider:

- Using a CDN in front of the application for static asset caching.
- Serving static files directly from Nginx rather than through FastAPI's `StaticFiles` middleware.
- Enabling gzip compression for JS and CSS assets.

### Health Checks

The `/health` endpoint returns a JSON response with status and version. Configure your load balancer or orchestrator to use this endpoint for health monitoring:

```json
{"status": "ok", "version": "0.1.0"}
```

For more comprehensive health checks, consider extending the endpoint to verify DuckDB connectivity and disk space availability.

### Resource Limits

DuckDB operates entirely in memory. For datasets that exceed available RAM:

- Set `max_memory` in the DuckDB connection to enable disk spilling: `duckdb.connect(":memory:", config={"memory_limit": "2GB"})`.
- Use `PREVIEW_ROWS` and `SAMPLE_ROWS` to limit data sent to the frontend.
- Monitor memory usage and set container memory limits appropriately.

### Scaling Considerations

The current architecture is single-process and single-session-per-container. To scale horizontally:

- **Session affinity**: Use cookie-based sticky sessions to route each user to the same backend instance.
- **Shared storage**: Use a networked filesystem (EFS, S3) for `DATA_DIR` so all instances can access uploaded files.
- **Connection pooling**: If moving to a shared DuckDB instance, implement connection pooling to manage concurrent access.
- **Load balancing**: Place multiple backend instances behind a load balancer with health check routing.

For most use cases, a single instance with sufficient RAM (4-8 GB) handles dozens of concurrent users comfortably, given that CSV analysis is typically an interactive, low-concurrency workload.

	# Deployment Guide

	This document covers all deployment options for the Visual Notebook, including Docker builds, Docker Compose for development, CI/CD pipelines, Hugging Face Spaces deployment, environment configuration, and production considerations.

	## Docker Build Options

	The project provides three Dockerfile configurations for different deployment scenarios:

	### Combined Build (Dockerfile)

	The combined Dockerfile builds both the frontend and backend into a single image. This is the recommended approach for most deployments, including HF Spaces.

	Build stages:

	1. frontend-build (Node.js 20): Installs npm dependencies and builds the React application with `npm run build`. The output is written to `frontend/dist/`.

	2. Runtime (Python 3.11): Installs the backend using `uv pip install --system -e .`, copies the built frontend from the previous stage into `/app/static`, and configures environment variables.

	```dockerfile
	# Stage 1: Build frontend
	FROM node:20-slim AS frontend-build
	WORKDIR /app/frontend
	COPY frontend/package.json frontend/package-lock.json* ./
	RUN npm install
	COPY frontend/ ./
	RUN npm run build

	# Stage 2: Runtime
	FROM python:3.11-slim
	WORKDIR /app
	RUN pip install uv
	COPY backend/pyproject.toml backend/
	COPY backend/visual_notebook/ backend/visual_notebook/
	RUN cd backend && uv pip install --system -e .
	COPY datasets/ /app/datasets/
	COPY --from=frontend-build /app/frontend/dist /app/static
	ENV DATA_DIR=/app/datasets
	ENV CHECKPOINT_DIR=/app/checkpoints
	ENV CORS_ORIGINS=*
	ENV HOST=0.0.0.0
	ENV PORT=7860
	EXPOSE 7860
	CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
	```

	Build and run:

	```bash
	docker build -t visual-notebook .
	docker run -p 7860:7860 -v $(pwd)/datasets:/app/datasets visual-notebook
	```

	### Standalone Backend (Dockerfile.backend)

	The backend-only Dockerfile skips the frontend build entirely. It is intended for deployments where the frontend is served separately (e.g., by a CDN or Nginx).

	```dockerfile
	FROM python:3.11-slim
	WORKDIR /app
	RUN pip install uv
	COPY backend/pyproject.toml backend/
	COPY backend/visual_notebook/ backend/visual_notebook/
	COPY datasets/ /app/datasets/
	RUN cd backend && uv pip install --system -e .
	ENV DATA_DIR=/app/datasets
	ENV CHECKPOINT_DIR=/app/checkpoints
	ENV CORS_ORIGINS=*
	ENV HOST=0.0.0.0
	ENV PORT=7860
	EXPOSE 7860
	CMD ["uvicorn", "visual_notebook.main:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
	```

	### Frontend-Only (frontend/Dockerfile)

	The frontend-only Dockerfile builds the React application and serves it with Nginx:

	```dockerfile
	FROM node:20-slim AS build
	WORKDIR /app
	COPY package.json package-lock.json* ./
	RUN npm install
	COPY . .
	RUN npm run build

	FROM nginx:alpine
	COPY --from=build /app/dist /usr/share/nginx/html
	COPY nginx.conf /etc/nginx/conf.d/default.conf
	EXPOSE 80
	CMD ["nginx", "-g", "daemon off;"]
	```

	The Nginx configuration (`frontend/nginx.conf`) is set up to proxy API requests to the backend and serve static files for all other routes.

	## Docker Compose for Development

	The `docker-compose.yml` defines two services for local development:

	```yaml
	services:
	backend:
	build:
	context: .
	dockerfile: Dockerfile.backend
	ports:
	- "8000:7860"
	volumes:
	- ./datasets:/app/datasets
	- ./checkpoints:/app/checkpoints
	environment:
	- DATA_DIR=/app/datasets
	- CHECKPOINT_DIR=/app/checkpoints
	- CORS_ORIGINS=http://localhost:5173,http://localhost:3000

	frontend:
	build:
	context: ./frontend
	dockerfile: Dockerfile
	ports:
	- "5173:80"
	environment:
	- VITE_API_URL=http://localhost:8000
	depends_on:
	- backend
	```

	Usage:

	```bash
	# Start both services
	docker compose up --build

	# Backend available at http://localhost:8000
	# Frontend available at http://localhost:5173
	```

	The volume mounts ensure that uploaded datasets and checkpoints persist across container restarts.

	## GitHub Actions CI/CD

	The recommended GitHub Actions workflow follows a test-build-deploy pipeline:

	### Workflow Stages

	1. Test: Run pytest against the backend with `uv run pytest`.
	2. Build: Build the Docker image using the combined Dockerfile.
	3. Deploy: Push the image to a container registry and deploy.

	### Example Workflow

	```yaml
	name: CI/CD

	on:
	push:
	branches: [main]
	pull_request:
	branches: [main]

	jobs:
	test:
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4
	- uses: astral-sh/setup-uv@v4
	with:
	version: "latest"
	- name: Install dependencies
	run: uv sync
	- name: Run tests
	run: uv run pytest backend/visual_notebook/tests/ -v

	build:
	needs: test
	runs-on: ubuntu-latest
	if: github.ref == 'refs/heads/main'
	steps:
	- uses: actions/checkout@v4
	- name: Set up Docker Buildx
	uses: docker/setup-buildx-action@v3
	- name: Login to registry
	uses: docker/login-action@v3
	with:
	registry: ghcr.io
	username: ${{ github.actor }}
	password: ${{ secrets.GITHUB_TOKEN }}
	- name: Build and push
	uses: docker/build-push-action@v5
	with:
	context: .
	push: true
	tags: ghcr.io/${{ github.repository }}:latest
	cache-from: type=gha
	cache-to: type=gha,mode=max
	```

	### UV Caching

	The `setup-uv` action caches the uv installation and downloaded packages. For Docker builds, the `cache-from` and `cache-to` options use GitHub Actions cache to persist Docker layers between builds, significantly reducing build times for subsequent pushes.

	## GitLab CI/CD Pipeline

	For GitLab deployments, the following pipeline structure is recommended:

	```yaml
	stages:
	- test
	- build
	- deploy

	test:
	stage: test
	image: python:3.11-slim
	before_script:
	- pip install uv
	- uv sync
	script:
	- uv run pytest backend/visual_notebook/tests/ -v

	build:
	stage: build
	image: docker:24
	services:
	- docker:24-dind
	script:
	- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
	- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
	only:
	- main

	deploy:
	stage: deploy
	image: alpine:latest
	script:
	- apk add --no-cache curl
	- curl -X POST "$DEPLOY_WEBHOOK_URL"
	only:
	- main
	```

	## Hugging Face Spaces Deployment

	The combined Dockerfile is designed to be compatible with HF Spaces requirements:

	- Port 7860: HF Spaces expects the application to listen on port 7860. The Dockerfile `EXPOSE 7860` and the uvicorn command use this port.
	- Static file serving: The backend serves the frontend from `/app/static`, which is populated during the Docker build.
	- Environment variables: HF Spaces allows setting secrets and environment variables through the Space settings UI.

	### Deployment Steps

	1. Create a new Docker Space on Hugging Face.
	2. Clone the Space repository.
	3. Copy the project files (or add the Space as a git remote).
	4. Push to the Space repository. HF Spaces will build and deploy automatically.

	### Space Configuration

	Set the following secrets in the Space settings:

	\| Variable \| Value \|
	\|----------\|-------\|
	\| `DATA_DIR` \| `/app/datasets` \|
	\| `CHECKPOINT_DIR` \| `/app/checkpoints` \|
	\| `CORS_ORIGINS` \| `*` (or your specific domain) \|
	\| `MAX_UPLOAD_SIZE` \| `524288000` (500 MB) \|

	### Persistent Storage

	HF Spaces offers persistent storage for paid tiers. To enable it, mount a persistent volume at `/app/datasets` and `/app/checkpoints`. Without persistent storage, uploaded files are lost when the Space restarts.

	## Environment Variable Configuration

	All configuration is managed through environment variables, read by the `Settings` dataclass in `config.py`:

	\| Variable \| Default \| Description \|
	\|----------\|---------\|-------------\|
	\| `DATA_DIR` \| `datasets` \| Directory for uploaded CSV files \|
	\| `CHECKPOINT_DIR` \| `checkpoints` \| Directory for session checkpoints \|
	\| `MAX_UPLOAD_SIZE` \| `524288000` (500 MB) \| Maximum CSV file size in bytes \|
	\| `PREVIEW_ROWS` \| `50` \| Number of rows returned by default in `head()` \|
	\| `SAMPLE_ROWS` \| `10000` \| Number of rows for Polars CSV sampling \|
	\| `CORS_ORIGINS` \| `http://localhost:5173,http://localhost:3000` \| Comma-separated allowed origins \|
	\| `HOST` \| `0.0.0.0` \| Server bind address \|
	\| `PORT` \| `8000` \| Server bind port (7860 in Docker) \|

	## Production Considerations

	### Session Storage

	The current `SessionStore` is in-memory, meaning all session data is lost when the server restarts. For production deployments, consider replacing it with a persistent store:

	- Redis: Store session metadata in Redis with TTL-based expiration. DuckDB tables would need to be exported to Parquet and reloaded on session restore.
	- Database-backed: Use a lightweight database (SQLite, PostgreSQL) to store session state and checkpoint references.
	- File-based: Serialize session metadata to JSON files in the checkpoint directory.

	### Static File Serving

	In production, the backend serves static files from `/app/static`. For high-traffic deployments, consider:

	- Using a CDN in front of the application for static asset caching.
	- Serving static files directly from Nginx rather than through FastAPI's `StaticFiles` middleware.
	- Enabling gzip compression for JS and CSS assets.

	### Health Checks

	The `/health` endpoint returns a JSON response with status and version. Configure your load balancer or orchestrator to use this endpoint for health monitoring:

	```json
	{"status": "ok", "version": "0.1.0"}
	```

	For more comprehensive health checks, consider extending the endpoint to verify DuckDB connectivity and disk space availability.

	### Resource Limits

	DuckDB operates entirely in memory. For datasets that exceed available RAM:

	- Set `max_memory` in the DuckDB connection to enable disk spilling: `duckdb.connect(":memory:", config={"memory_limit": "2GB"})`.
	- Use `PREVIEW_ROWS` and `SAMPLE_ROWS` to limit data sent to the frontend.
	- Monitor memory usage and set container memory limits appropriately.

	### Scaling Considerations

	The current architecture is single-process and single-session-per-container. To scale horizontally:

	- Session affinity: Use cookie-based sticky sessions to route each user to the same backend instance.
	- Shared storage: Use a networked filesystem (EFS, S3) for `DATA_DIR` so all instances can access uploaded files.
	- Connection pooling: If moving to a shared DuckDB instance, implement connection pooling to manage concurrent access.
	- Load balancing: Place multiple backend instances behind a load balancer with health check routing.

	For most use cases, a single instance with sufficient RAM (4-8 GB) handles dozens of concurrent users comfortably, given that CSV analysis is typically an interactive, low-concurrency workload.