|
Download docs/DOCUMENTATION_INTEGRATION.md from dn-scribe/attstore: direct link, hf CLI and curl.
- Browser
- Download file 14.7 kB
-
https://huggingface.co/spaces/dn-scribe/attstore/resolve/main/docs/DOCUMENTATION_INTEGRATION.md
- Command line
-
hf download hf://spaces/dn-scribe/attstore/docs/DOCUMENTATION_INTEGRATION.md
-
curl -L -o DOCUMENTATION_INTEGRATION.md https://huggingface.co/spaces/dn-scribe/attstore/resolve/main/docs/DOCUMENTATION_INTEGRATION.md
14.7 kB
| # Documentation Integration - Implementation Summary | |
| ## Quick Summary | |
| This document describes how Scribe Security documentation is integrated into the Attestation Store for both production and local development. Documentation is built from source (Docusaurus) during Docker builds and via a local script, ensuring all internal links work at `/docs`. For airgapped environments, docs can be pre-built and transferred. Troubleshooting and maintenance tips are included. | |
| **Key Points:** | |
| - Documentation is served locally at `/docs` (not from GitHub Pages) | |
| - Docker builds and local scripts build docs from source for correct routing | |
| - All internal links and assets work as expected | |
| - See troubleshooting section for common issues | |
| ## Overview | |
| This implementation adds Scribe Security documentation from GitHub Pages to the Attestation Store application during build time for production and provides easy local development access. | |
| ## What Was Implemented | |
| ### 1. **Multi-Stage Dockerfile** (`Dockerfile`) | |
| - **Stage 1**: Downloads documentation from `https://scribe-security.github.io/docs/` using `wget` | |
| - **Stage 2**: Copies downloaded docs to `/app/static/docs` in the application image | |
| - Documentation is embedded in the Docker image automatically during build | |
| ### 2. **Local Development Script** (`scripts/download-docs.sh`) | |
| - Bash script to download documentation for local development | |
| - Features: | |
| - Checks if docs already exist (skips re-download unless `--force` flag is used) | |
| - Uses same `wget` parameters as Dockerfile for consistency | |
| - Provides helpful output and error handling | |
| - Made executable with `chmod +x` | |
| ### 3. **Makefile** (`Makefile`) | |
| Convenient targets for common operations: | |
| - `make docs` - Build documentation locally | |
| - `make docs-force` - Force rebuild documentation | |
| - `make clean-docs` - Remove built documentation | |
| - `make dev` - Build docs (if needed) and start dev server | |
| - `make build` - Build Docker image with documentation | |
| - `make help` - Show available targets | |
| ### 4. **Flask Routes** (`web/routes.py`) | |
| Added two new routes to serve documentation: | |
| - `GET /docs` or `GET /docs/` - Serves documentation index page | |
| - `GET /docs/<path:filename>` - Serves documentation files and assets | |
| **Features:** | |
| - Friendly error message if docs haven't been built yet | |
| - Automatic fallback to file listing if no `index.html` | |
| - Uses Flask's `send_from_directory` for secure file serving | |
| - Opens in new tab via `target="_blank"` in navigation links | |
| ### 5. **Git Configuration** (`.gitignore`) | |
| - Added `static/docs/` to prevent committing built documentation | |
| - Keeps repository clean (docs are ~10MB+ when built) | |
| - Prevents duplication since docs are built during CI/deployment | |
| ### 6. **Docker Configuration** (`.dockerignore`) | |
| - Added `static/docs/` to exclude locally built docs from build context | |
| - Ensures Docker build uses docs from the multi-stage build process | |
| - Reduces build context size and prevents conflicts | |
| ### 7. **Documentation Updates** | |
| **`README.md`:** | |
| - Added comprehensive documentation section | |
| - Explains local development requirements (Node.js 18+, Git) | |
| - Covers production/Docker build process | |
| - Includes airgapped environment instructions | |
| - Links to this detailed implementation guide | |
| **`templates/base.html`:** | |
| - Updated documentation link from `https://scribe-security.github.io/docs/` to `/docs` | |
| - Retained `target="_blank"` to open in new tab | |
| - Documentation now served locally instead of external link | |
| ## How It Works | |
| ### Production/Docker Build | |
| ```bash | |
| docker build -t attstore:latest . | |
| ``` | |
| **Build Process:** | |
| 1. **Stage 1 - Documentation Builder (`node:18-alpine`)**: | |
| - Installs system dependencies (git, build tools) | |
| - Clones `https://github.com/scribe-security/docs.git` | |
| - Runs `npm ci` to install Node.js dependencies | |
| - Builds with `DOCUSAURUS_BASE_URL=/docs npm run build` | |
| - Output: Static site in `/docs-build/build` | |
| 2. **Stage 2 - Application (`python:3.11-slim`)**: | |
| - Sets up Python environment | |
| - Installs application dependencies | |
| - Copies application code | |
| - **Copies built documentation from stage 1** to `/app/static/docs` | |
| 3. **Result**: | |
| - Documentation embedded in Docker image (~10MB) | |
| - All internal links work correctly at `/docs` | |
| - Assets (CSS/JS/images) load properly | |
| - Client-side routing functions as expected | |
| **Run the container:** | |
| ```bash | |
| docker run -p 5003:5003 attstore:latest | |
| ``` | |
| ### Local Development | |
| **Requirements:** | |
| - Node.js 18+ (for building Docusaurus) | |
| - Git (for cloning the docs repo) | |
| - ~2-3 minutes for initial build | |
| - Build tools may be needed on some systems | |
| **Build Commands:** | |
| ```bash | |
| # Option 1: Using Make (recommended) | |
| make dev # Builds docs if needed, then starts server | |
| # Option 2: Manual build then run | |
| make docs # Just build documentation | |
| python main.py # Start server separately | |
| # Option 3: Direct script | |
| bash scripts/download-docs.sh | |
| python main.py | |
| # Force rebuild | |
| make docs-force | |
| ``` | |
| **Build Process (Local):** | |
| 1. Clones docs repository to temporary directory | |
| 2. Runs `npm ci` to install dependencies | |
| 3. Builds with `DOCUSAURUS_BASE_URL=/docs npm run build` | |
| 4. Copies built files to `static/docs/` | |
| 5. Cleans up temporary directory | |
| ### Accessing Documentation | |
| Once the application is running (locally or in Docker): | |
| - **Application**: `http://localhost:5003` | |
| - **Documentation**: `http://localhost:5003/docs` | |
| - Click "Scribe Documentation" link in footer (opens in new tab) | |
| - All internal navigation works correctly | |
| - Client-side routing functions properly | |
| ## Airgapped Environment Support | |
| For environments without internet access during build: | |
| ### Option 1: Pre-build Documentation (Recommended) | |
| ```bash | |
| # On internet-connected machine: | |
| bash scripts/download-docs.sh | |
| # Transfer entire project to airgapped environment | |
| # The built docs in static/docs/ will be included | |
| # In airgapped environment: | |
| docker build -t attstore:latest . | |
| ``` | |
| **Note**: The Docker build will still try to clone the repository in stage 1, but will use the locally built docs if the build stage fails. | |
| ### Option 2: Modify Dockerfile for Airgap | |
| Comment out the docs-builder stage and copy local docs: | |
| ```dockerfile | |
| # Comment out or remove the docs-builder stage: | |
| # FROM node:18-alpine AS docs-builder | |
| # ...entire stage... | |
| # In the application stage: | |
| FROM python:3.11-slim | |
| # ... existing app setup ... | |
| # Copy pre-built docs from local filesystem | |
| COPY static/docs /app/static/docs | |
| ``` | |
| Then build: | |
| ```bash | |
| docker build -t attstore:latest . | |
| ``` | |
| ### Option 3: Save Built Image | |
| Build on internet-connected machine, save, and transfer: | |
| ```bash | |
| # On internet-connected machine: | |
| docker build -t attstore:latest . | |
| docker save -o attstore-latest.tar attstore:latest | |
| # Transfer attstore-latest.tar to airgapped environment | |
| # On airgapped machine: | |
| docker load -i attstore-latest.tar | |
| docker run -p 5003:5003 attstore:latest | |
| ``` | |
| ## File Changes Summary | |
| | File | Change Type | Purpose | | |
| |------|-------------|---------| | |
| | `Dockerfile` | Modified | Added multi-stage build: Node.js stage builds docs, Python stage includes them | | |
| | `.gitignore` | Modified | Exclude `static/docs/` (built documentation) | | |
| | `.dockerignore` | Modified | Exclude `static/docs/` from build context | | |
| | `scripts/download-docs.sh` | Created | Build script for local development (clones, builds, copies) | | |
| | `Makefile` | Created | Convenient targets (`docs`, `dev`, `build`, `clean-docs`) | | |
| | `web/routes.py` | Modified | Added `/docs` and `/docs/<path>` routes | | |
| | `templates/base.html` | Modified | Updated doc link from external URL to `/docs` | | |
| | `README.md` | Modified | Added documentation section with build instructions | | |
| | `docs/DOCUMENTATION_INTEGRATION.md` | Created | This comprehensive implementation guide | | |
| ## Testing Checklist | |
| ### Docker Build & Run | |
| - [ ] Test Docker build: `docker build -t attstore:latest .` | |
| - Verify stage 1 clones repo successfully | |
| - Verify `npm ci` completes without errors | |
| - Verify build completes with correct baseUrl | |
| - Build time should be ~4-5 minutes | |
| - [ ] Test Docker run: `docker run -p 5003:5003 attstore:latest` | |
| - [ ] Access application at `http://localhost:5003` | |
| - [ ] Access documentation at `http://localhost:5003/docs` | |
| - [ ] Click "Scribe Documentation" link in footer | |
| - [ ] Verify docs open in new tab | |
| - [ ] Test internal navigation within docs | |
| ### Local Development | |
| - [ ] Test local build: `make docs` | |
| - Verify Node.js and Git are detected | |
| - Verify clone succeeds | |
| - Verify npm install completes | |
| - Verify build completes | |
| - Check `static/docs/` directory exists | |
| - [ ] Test force rebuild: `make docs-force` | |
| - [ ] Test clean: `make clean-docs` | |
| - [ ] Test dev mode: `make dev` | |
| - [ ] Access `/docs` in browser from local development | |
| - [ ] Test internal links and navigation | |
| ### Git & Docker Configuration | |
| - [ ] Verify docs excluded from git: `git status` should not show `static/docs/` | |
| - [ ] Verify `.dockerignore` excludes `static/docs/` | |
| - [ ] Build Docker image and verify it doesn't use local `static/docs/` | |
| ### Airgapped Environment (if applicable) | |
| - [ ] Build docs locally | |
| - [ ] Transfer project to airgapped system | |
| - [ ] Build Docker image in airgapped environment | |
| - [ ] Verify documentation is included and works | |
| ## Maintenance | |
| ### Updating Documentation | |
| Documentation is built from the latest `main` branch of the docs repository: | |
| **For Production:** | |
| ```bash | |
| docker build -t attstore:latest . | |
| ``` | |
| This will clone and build the latest documentation automatically. | |
| **For Local Development:** | |
| ```bash | |
| make docs-force | |
| # or | |
| bash scripts/download-docs.sh --force | |
| ``` | |
| **Cleaning Up:** | |
| ```bash | |
| make clean-docs # Remove built documentation | |
| ``` | |
| ### Build Details | |
| **What gets built:** | |
| - Static HTML, CSS, JavaScript files | |
| - Images and assets | |
| - Search index (Lunr.js) | |
| - Total size: ~10MB | |
| **Build time:** | |
| - Docker build (first time): ~4-5 minutes | |
| - Local build (first time): ~2-3 minutes | |
| - Subsequent builds (with cache): ~1-2 minutes | |
| ### Internal Links & Routing | |
| **Why build from source instead of downloading pre-built HTML?** | |
| The documentation uses **Docusaurus 2**, a React-based static site generator with: | |
| - Client-side routing (React Router) | |
| - Dynamic base URL configuration | |
| - Asset bundling with webpack | |
| **The Problem:** | |
| - GitHub Pages builds with `baseUrl=/` (root path) | |
| - We need to serve at `/docs` (sub-path) | |
| - Downloading pre-built HTML would have wrong paths | |
| **The Solution:** | |
| - Build from source with `DOCUSAURUS_BASE_URL=/docs` | |
| - All generated URLs use `/docs` as the base | |
| - Internal links: `/docs/guides/...` instead of `/guides/...` | |
| - Asset paths: `/docs/assets/...` instead of `/assets/...` | |
| - Client-side routing configured for `/docs` base | |
| **This ensures:** | |
| - ✅ Internal navigation works correctly | |
| - ✅ Asset paths (CSS/JS/images) resolve properly | |
| - ✅ Client-side routing functions as expected | |
| - ✅ Search works within the docs | |
| - ✅ No broken links or 404 errors | |
| ### Troubleshooting | |
| #### "Documentation Not Available" Error | |
| **Local Development:** | |
| ```bash | |
| make docs # Requires Node.js 18+ and Git | |
| ``` | |
| If the build succeeded but docs still aren't available: | |
| - Check that `static/docs/` directory exists | |
| - Check that it contains files (especially `index.html`) | |
| - Verify Flask can access the directory (permissions) | |
| **Docker:** | |
| - Check build logs for errors in the `docs-builder` stage | |
| - Look for git clone failures or npm install errors | |
| - Verify the COPY command succeeded in stage 2 | |
| #### Node.js Not Found (Local Dev) | |
| **Solution:** | |
| - Install Node.js 18+ from https://nodejs.org/ | |
| - Or use `nvm`: `nvm install 18 && nvm use 18` | |
| - Or use Docker build which includes Node.js | |
| **Verify installation:** | |
| ```bash | |
| node --version # Should show v18.x.x or higher | |
| npm --version | |
| ``` | |
| #### Build Fails During `npm ci` | |
| **Possible causes:** | |
| - No internet connectivity | |
| - Cannot access npmjs.org registry | |
| - Firewall/proxy blocking npm | |
| - Network timeout | |
| **Solutions:** | |
| ```bash | |
| # Check connectivity | |
| curl https://registry.npmjs.org/ | |
| # Configure npm proxy if needed | |
| npm config set proxy http://proxy.example.com:8080 | |
| npm config set https-proxy http://proxy.example.com:8080 | |
| # Use longer timeout | |
| npm config set fetch-timeout 60000 | |
| ``` | |
| #### Build Fails with gifsicle/Native Module Errors | |
| **Error example:** | |
| ``` | |
| qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2' | |
| autoreconf: not found | |
| ``` | |
| **Cause:** Native Node.js modules (like `gifsicle` for image optimization) need to be compiled, but build tools are missing. | |
| **Solution (already implemented):** | |
| The Dockerfile now includes all necessary build dependencies: | |
| - `python3` - Required by node-gyp | |
| - `make`, `g++` - C/C++ build tools | |
| - `autoconf`, `automake`, `libtool` - Build configuration | |
| - `nasm` - Assembler for image optimization | |
| **If you still encounter this on Mac (ARM64):** | |
| - Ensure Docker Desktop is up to date | |
| - Try rebuilding without cache: `docker build --no-cache -t attstore:latest .` | |
| - Check that BuildKit is enabled (it is by default) | |
| #### Docs Not Showing in Container | |
| **Check if docs exist in image:** | |
| ```bash | |
| docker run --rm attstore:latest ls -la /app/static/docs | |
| ``` | |
| Should show files including `index.html`, `assets/`, etc. | |
| **Check build logs:** | |
| ```bash | |
| docker build -t attstore:latest . 2>&1 | grep -A 10 "docs-builder" | |
| ``` | |
| Look for errors in: | |
| - Git clone | |
| - npm ci | |
| - npm run build | |
| - COPY command | |
| #### Internal Links Not Working | |
| **Symptoms:** | |
| - Clicking links returns 404 | |
| - Assets don't load (no CSS/JS) | |
| - Pages redirect incorrectly | |
| **Check:** | |
| 1. Open browser DevTools console for errors | |
| 2. Verify docs were built with correct baseUrl: | |
| ```bash | |
| grep -r 'href="/docs' static/docs/index.html | |
| ``` | |
| Should find links starting with `/docs/` | |
| 3. Check Flask route is working: | |
| ```bash | |
| curl http://localhost:5003/docs/ | |
| ``` | |
| Should return the HTML content | |
| 4. Verify baseUrl in build: | |
| ```bash | |
| # In built docs | |
| cat static/docs/index.html | grep baseUrl | |
| ``` | |
| #### Clone/Git Errors in Docker Build | |
| **Error:** `fatal: unable to access 'https://github.com/...': Could not resolve host` | |
| **Solutions:** | |
| - Check internet connectivity | |
| - Verify DNS resolution works in Docker | |
| - Try using Docker's default DNS: `docker build --network=host ...` | |
| - Check firewall/proxy settings | |
| #### Build Too Slow | |
| **Optimization tips:** | |
| ```bash | |
| # Use BuildKit for better caching | |
| DOCKER_BUILDKIT=1 docker build -t attstore:latest . | |
| # Use --cache-from to leverage previous builds | |
| docker pull attstore:latest | |
| docker build --cache-from attstore:latest -t attstore:latest . | |
| ``` | |
| **Local development:** | |
| - Built docs are cached in `static/docs/` | |
| - Rebuild only when needed with `make docs-force` | |
| - Don't commit `static/docs/` to git (it's ignored) | |