Blame for docs/DOCUMENTATION_INTEGRATION.md

# Documentation Integration - Implementation Summary
## Quick Summary

This document describes how Scribe Security documentation is integrated into the Attestation Store for both production and local development. Documentation is built from source (Docusaurus) during Docker builds and via a local script, ensuring all internal links work at `/docs`. For airgapped environments, docs can be pre-built and transferred. Troubleshooting and maintenance tips are included.

**Key Points:**
- Documentation is served locally at `/docs` (not from GitHub Pages)
- Docker builds and local scripts build docs from source for correct routing
- All internal links and assets work as expected
- See troubleshooting section for common issues


## Overview
This implementation adds Scribe Security documentation from GitHub Pages to the Attestation Store application during build time for production and provides easy local development access.

## What Was Implemented

### 1. **Multi-Stage Dockerfile** (`Dockerfile`)
- **Stage 1**: Downloads documentation from `https://scribe-security.github.io/docs/` using `wget`
- **Stage 2**: Copies downloaded docs to `/app/static/docs` in the application image
- Documentation is embedded in the Docker image automatically during build

### 2. **Local Development Script** (`scripts/download-docs.sh`)
- Bash script to download documentation for local development
- Features:
  - Checks if docs already exist (skips re-download unless `--force` flag is used)
  - Uses same `wget` parameters as Dockerfile for consistency
  - Provides helpful output and error handling
  - Made executable with `chmod +x`

### 3. **Makefile** (`Makefile`)

Convenient targets for common operations:
- `make docs` - Build documentation locally
- `make docs-force` - Force rebuild documentation
- `make clean-docs` - Remove built documentation
- `make dev` - Build docs (if needed) and start dev server
- `make build` - Build Docker image with documentation
- `make help` - Show available targets

### 4. **Flask Routes** (`web/routes.py`)

Added two new routes to serve documentation:
- `GET /docs` or `GET /docs/` - Serves documentation index page
- `GET /docs/<path:filename>` - Serves documentation files and assets

**Features:**
- Friendly error message if docs haven't been built yet
- Automatic fallback to file listing if no `index.html`
- Uses Flask's `send_from_directory` for secure file serving
- Opens in new tab via `target="_blank"` in navigation links

### 5. **Git Configuration** (`.gitignore`)

- Added `static/docs/` to prevent committing built documentation
- Keeps repository clean (docs are ~10MB+ when built)
- Prevents duplication since docs are built during CI/deployment

### 6. **Docker Configuration** (`.dockerignore`)

- Added `static/docs/` to exclude locally built docs from build context
- Ensures Docker build uses docs from the multi-stage build process
- Reduces build context size and prevents conflicts

### 7. **Documentation Updates**

**`README.md`:**
- Added comprehensive documentation section
- Explains local development requirements (Node.js 18+, Git)
- Covers production/Docker build process
- Includes airgapped environment instructions
- Links to this detailed implementation guide

**`templates/base.html`:**
- Updated documentation link from `https://scribe-security.github.io/docs/` to `/docs`
- Retained `target="_blank"` to open in new tab
- Documentation now served locally instead of external link

## How It Works

### Production/Docker Build

```bash
docker build -t attstore:latest .
```

**Build Process:**

1. **Stage 1 - Documentation Builder (`node:18-alpine`)**:
   - Installs system dependencies (git, build tools)
   - Clones `https://github.com/scribe-security/docs.git`
   - Runs `npm ci` to install Node.js dependencies
   - Builds with `DOCUSAURUS_BASE_URL=/docs npm run build`
   - Output: Static site in `/docs-build/build`

2. **Stage 2 - Application (`python:3.11-slim`)**:
   - Sets up Python environment
   - Installs application dependencies
   - Copies application code
   - **Copies built documentation from stage 1** to `/app/static/docs`

3. **Result**: 
   - Documentation embedded in Docker image (~10MB)
   - All internal links work correctly at `/docs`
   - Assets (CSS/JS/images) load properly
   - Client-side routing functions as expected

**Run the container:**
```bash
docker run -p 5003:5003 attstore:latest
```

### Local Development

**Requirements:** 
- Node.js 18+ (for building Docusaurus)
- Git (for cloning the docs repo)
- ~2-3 minutes for initial build
- Build tools may be needed on some systems

**Build Commands:**

```bash
# Option 1: Using Make (recommended)
make dev              # Builds docs if needed, then starts server

# Option 2: Manual build then run
make docs            # Just build documentation
python main.py       # Start server separately

# Option 3: Direct script
bash scripts/download-docs.sh
python main.py

# Force rebuild
make docs-force
```

**Build Process (Local):**
1. Clones docs repository to temporary directory
2. Runs `npm ci` to install dependencies
3. Builds with `DOCUSAURUS_BASE_URL=/docs npm run build`
4. Copies built files to `static/docs/`
5. Cleans up temporary directory

### Accessing Documentation

Once the application is running (locally or in Docker):
- **Application**: `http://localhost:5003`
- **Documentation**: `http://localhost:5003/docs`
- Click "Scribe Documentation" link in footer (opens in new tab)
- All internal navigation works correctly
- Client-side routing functions properly

## Airgapped Environment Support

For environments without internet access during build:

### Option 1: Pre-build Documentation (Recommended)

```bash
# On internet-connected machine:
bash scripts/download-docs.sh

# Transfer entire project to airgapped environment
# The built docs in static/docs/ will be included

# In airgapped environment:
docker build -t attstore:latest .
```

**Note**: The Docker build will still try to clone the repository in stage 1, but will use the locally built docs if the build stage fails.

### Option 2: Modify Dockerfile for Airgap

Comment out the docs-builder stage and copy local docs:

```dockerfile
# Comment out or remove the docs-builder stage:
# FROM node:18-alpine AS docs-builder
# ...entire stage...

# In the application stage:
FROM python:3.11-slim

# ... existing app setup ...

# Copy pre-built docs from local filesystem
COPY static/docs /app/static/docs
```

Then build:
```bash
docker build -t attstore:latest .
```

### Option 3: Save Built Image

Build on internet-connected machine, save, and transfer:

```bash
# On internet-connected machine:
docker build -t attstore:latest .
docker save -o attstore-latest.tar attstore:latest

# Transfer attstore-latest.tar to airgapped environment

# On airgapped machine:
docker load -i attstore-latest.tar
docker run -p 5003:5003 attstore:latest
```

## File Changes Summary

| File | Change Type | Purpose |
|------|-------------|---------|
| `Dockerfile` | Modified | Added multi-stage build: Node.js stage builds docs, Python stage includes them |
| `.gitignore` | Modified | Exclude `static/docs/` (built documentation) |
| `.dockerignore` | Modified | Exclude `static/docs/` from build context |
| `scripts/download-docs.sh` | Created | Build script for local development (clones, builds, copies) |
| `Makefile` | Created | Convenient targets (`docs`, `dev`, `build`, `clean-docs`) |
| `web/routes.py` | Modified | Added `/docs` and `/docs/<path>` routes |
| `templates/base.html` | Modified | Updated doc link from external URL to `/docs` |
| `README.md` | Modified | Added documentation section with build instructions |
| `docs/DOCUMENTATION_INTEGRATION.md` | Created | This comprehensive implementation guide |

## Testing Checklist

### Docker Build & Run
- [ ] Test Docker build: `docker build -t attstore:latest .`
  - Verify stage 1 clones repo successfully
  - Verify `npm ci` completes without errors
  - Verify build completes with correct baseUrl
  - Build time should be ~4-5 minutes
- [ ] Test Docker run: `docker run -p 5003:5003 attstore:latest`
- [ ] Access application at `http://localhost:5003`
- [ ] Access documentation at `http://localhost:5003/docs`
- [ ] Click "Scribe Documentation" link in footer
- [ ] Verify docs open in new tab
- [ ] Test internal navigation within docs

### Local Development
- [ ] Test local build: `make docs`
  - Verify Node.js and Git are detected
  - Verify clone succeeds
  - Verify npm install completes
  - Verify build completes
  - Check `static/docs/` directory exists
- [ ] Test force rebuild: `make docs-force`
- [ ] Test clean: `make clean-docs`
- [ ] Test dev mode: `make dev`
- [ ] Access `/docs` in browser from local development
- [ ] Test internal links and navigation

### Git & Docker Configuration
- [ ] Verify docs excluded from git: `git status` should not show `static/docs/`
- [ ] Verify `.dockerignore` excludes `static/docs/`
- [ ] Build Docker image and verify it doesn't use local `static/docs/`

### Airgapped Environment (if applicable)
- [ ] Build docs locally
- [ ] Transfer project to airgapped system
- [ ] Build Docker image in airgapped environment
- [ ] Verify documentation is included and works

## Maintenance

### Updating Documentation

Documentation is built from the latest `main` branch of the docs repository:

**For Production:**
```bash
docker build -t attstore:latest .
```
This will clone and build the latest documentation automatically.

**For Local Development:**
```bash
make docs-force
# or
bash scripts/download-docs.sh --force
```

**Cleaning Up:**
```bash
make clean-docs  # Remove built documentation
```

### Build Details

**What gets built:**
- Static HTML, CSS, JavaScript files
- Images and assets
- Search index (Lunr.js)
- Total size: ~10MB

**Build time:**
- Docker build (first time): ~4-5 minutes
- Local build (first time): ~2-3 minutes
- Subsequent builds (with cache): ~1-2 minutes

### Internal Links & Routing

**Why build from source instead of downloading pre-built HTML?**

The documentation uses **Docusaurus 2**, a React-based static site generator with:
- Client-side routing (React Router)
- Dynamic base URL configuration
- Asset bundling with webpack

**The Problem:**
- GitHub Pages builds with `baseUrl=/` (root path)
- We need to serve at `/docs` (sub-path)
- Downloading pre-built HTML would have wrong paths

**The Solution:**
- Build from source with `DOCUSAURUS_BASE_URL=/docs`
- All generated URLs use `/docs` as the base
- Internal links: `/docs/guides/...` instead of `/guides/...`
- Asset paths: `/docs/assets/...` instead of `/assets/...`
- Client-side routing configured for `/docs` base

**This ensures:**
- ✅ Internal navigation works correctly
- ✅ Asset paths (CSS/JS/images) resolve properly  
- ✅ Client-side routing functions as expected
- ✅ Search works within the docs
- ✅ No broken links or 404 errors

### Troubleshooting

#### "Documentation Not Available" Error

**Local Development:**
```bash
make docs  # Requires Node.js 18+ and Git
```
If the build succeeded but docs still aren't available:
- Check that `static/docs/` directory exists
- Check that it contains files (especially `index.html`)
- Verify Flask can access the directory (permissions)

**Docker:**
- Check build logs for errors in the `docs-builder` stage
- Look for git clone failures or npm install errors
- Verify the COPY command succeeded in stage 2

#### Node.js Not Found (Local Dev)

**Solution:**
- Install Node.js 18+ from https://nodejs.org/
- Or use `nvm`: `nvm install 18 && nvm use 18`
- Or use Docker build which includes Node.js

**Verify installation:**
```bash
node --version  # Should show v18.x.x or higher
npm --version
```

#### Build Fails During `npm ci`

**Possible causes:**
- No internet connectivity
- Cannot access npmjs.org registry
- Firewall/proxy blocking npm
- Network timeout

**Solutions:**
```bash
# Check connectivity
curl https://registry.npmjs.org/

# Configure npm proxy if needed
npm config set proxy http://proxy.example.com:8080
npm config set https-proxy http://proxy.example.com:8080

# Use longer timeout
npm config set fetch-timeout 60000
```

#### Build Fails with gifsicle/Native Module Errors

**Error example:**
```
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2'
autoreconf: not found
```

**Cause:** Native Node.js modules (like `gifsicle` for image optimization) need to be compiled, but build tools are missing.

**Solution (already implemented):**
The Dockerfile now includes all necessary build dependencies:
- `python3` - Required by node-gyp
- `make`, `g++` - C/C++ build tools
- `autoconf`, `automake`, `libtool` - Build configuration
- `nasm` - Assembler for image optimization

**If you still encounter this on Mac (ARM64):**
- Ensure Docker Desktop is up to date
- Try rebuilding without cache: `docker build --no-cache -t attstore:latest .`
- Check that BuildKit is enabled (it is by default)

#### Docs Not Showing in Container

**Check if docs exist in image:**
```bash
docker run --rm attstore:latest ls -la /app/static/docs
```

Should show files including `index.html`, `assets/`, etc.

**Check build logs:**
```bash
docker build -t attstore:latest . 2>&1 | grep -A 10 "docs-builder"
```

Look for errors in:
- Git clone
- npm ci
- npm run build
- COPY command

#### Internal Links Not Working

**Symptoms:**
- Clicking links returns 404
- Assets don't load (no CSS/JS)
- Pages redirect incorrectly

**Check:**
1. Open browser DevTools console for errors
2. Verify docs were built with correct baseUrl:
   ```bash
   grep -r 'href="/docs' static/docs/index.html
   ```
   Should find links starting with `/docs/`

3. Check Flask route is working:
   ```bash
   curl http://localhost:5003/docs/
   ```
   Should return the HTML content

4. Verify baseUrl in build:
   ```bash
   # In built docs
   cat static/docs/index.html | grep baseUrl
   ```

#### Clone/Git Errors in Docker Build

**Error:** `fatal: unable to access 'https://github.com/...': Could not resolve host`

**Solutions:**
- Check internet connectivity
- Verify DNS resolution works in Docker
- Try using Docker's default DNS: `docker build --network=host ...`
- Check firewall/proxy settings

#### Build Too Slow

**Optimization tips:**
```bash
# Use BuildKit for better caching
DOCKER_BUILDKIT=1 docker build -t attstore:latest .

# Use --cache-from to leverage previous builds
docker pull attstore:latest
docker build --cache-from attstore:latest -t attstore:latest .
```

**Local development:**
- Built docs are cached in `static/docs/`
- Rebuild only when needed with `make docs-force`
- Don't commit `static/docs/` to git (it's ignored)