================================================================================ SPHEREx DATA EXPLORER: CLOUD DEPLOYMENT & DATA ACCESS PLAN ================================================================================ Document: deploy.txt Target: Deployment architecture, Hugging Face dataset integration, and implementation plan Dataset Reference: https://huggingface.co/datasets/AkshithSurya/spherex-projects/tree/main ================================================================================ 1. EXECUTIVE ARCHITECTURE SUMMARY -------------------------------------------------------------------------------- The SPHEREx Data Explorer is NOT a static website. It is an astronomical web application combining a dynamic Python server backend (`s.py` and supporting modules) with a rich, interactive astronomical client (`index.html`, Aladin Lite v3, custom projections, HiPS sky viewer, and SIMBAD analytics). Core capabilities that require an active backend server: - Coordinate transformations and celestial cone searches (/api/observations/search). - Astronomical target resolution via Sesame, SIMBAD, and NED (/api/targets/resolve). - FITS header provenance extraction and WCS parameter parsing on demand. - HiPS sky survey mirroring, caching, and proxying (/hips/..., /mocserver...). - Dynamic astronomical cross-matching against SIMBAD TAP (/api/analysis/nearby). - FITS file streaming, range queries, and dynamic zip archive creation. - In-memory transient overlay blob processing with SIP distortion calculation. Attempting to deploy this as a static site (e.g. GitHub Pages, Vercel Static, or Hugging Face Spaces static SDK) breaks ~80% of the core scientific workflow. 2. THE 29 GB DATA REALITY & THE HUGGING FACE BRIDGE -------------------------------------------------------------------------------- The local SPHEREx dataset consists of: - 434 calibrated Level 2 FITS files. - 6 spectral bands: * SPHEREx-D1 (0.75 - 1.11 um): 56 files (~3.8 GB) * SPHEREx-D2 (1.11 - 1.64 um): 100 files (~6.7 GB) * SPHEREx-D3 (1.64 - 2.42 um): 59 files (~4.0 GB) * SPHEREx-D4 (2.42 - 3.82 um): 56 files (~3.8 GB) * SPHEREx-D5 (3.82 - 4.42 um): 100 files (~6.7 GB) * SPHEREx-D6 (4.42 - 5.00 um): 63 files (~4.3 GB) - Total raw storage: ~29 GB. THE ADVANTAGE: All 434 files are ALREADY hosted publicly in the Hugging Face dataset repository: https://huggingface.co/datasets/AkshithSurya/spherex-projects/tree/main Every single file can be accessed directly via HTTP GET: https://huggingface.co/datasets/AkshithSurya/spherex-projects/resolve/main/{band}/{filename} Furthermore, `footprints.json` in the codebase already contains the complete index: - Exact filenames and relative paths. - Central celestial coordinates (RA, Dec). - Sky coverage polygons for every single observation. This enables a "Zero-Disk" / "Lazy-Streaming" cloud architecture where the deployed application does NOT need to store 29 GB on container startup! 3. ARCHITECTURAL DATA ACCESS PATTERNS (HOW TO ACCESS THE FITS FILES) -------------------------------------------------------------------------------- There are three distinct patterns to handle the FITS files in production: -------------------------------------------------------------------------------- PATTERN A: SMART HYBRID LAZY-STREAMING & REDIRECT (RECOMMENDED FOR CLOUD) -------------------------------------------------------------------------------- How it works: 1. Boot Time: - The web server container boots with only the code, CSS, JS, and `footprints.json` (container image size: < 150 MB). Boot takes 2 seconds. 2. Band Browsing & Search: - The application serves band counts and footprints directly from `footprints.json`. - The interactive Aladin sky view shows all 434 observation footprints instantly. 3. FITS File Download (/files/{band}/{file} or /api/observations/{id}/file): - Check local cache directory. If absent, issue an HTTP 302 Redirect to: https://huggingface.co/datasets/AkshithSurya/spherex-projects/resolve/main/{band}/{file} - The client's browser downloads directly from Hugging Face's global CDN at full speed, consuming zero server bandwidth and zero container storage. 4. Header Provenance Extraction (/api/observations/{id}/provenance): - If the file is not in local cache, send an HTTP Range Request: `Range: bytes=0-65535` to Hugging Face. - Download only the first 64 KB (where FITS primary + extension headers reside). - Parse the WCS and metadata in memory without downloading the 40 MB image array. - Cache the parsed JSON provenance in `ScientificCache` (memory). 5. In-Server Analysis / Overlays: - If user requests server-side pixel manipulation, fetch the single required FITS file on demand and store it in an ephemeral LRU cache (e.g., /tmp/spherex_cache, max 2 GB). Advantages: - Free tier compatible on almost any cloud platform. - Blazing fast deployment (no 29 GB upload or long git clone). - Bandwidth for downloads is absorbed by Hugging Face's CDN. -------------------------------------------------------------------------------- PATTERN B: HUGGING FACE SPACES DOCKER (CO-LOCATED NATIVE) -------------------------------------------------------------------------------- How it works: - Deploy a Hugging Face Space using the Docker SDK in the same user account (`AkshithSurya`). - Declare the dataset dependency in the Space `README.md`: datasets: - AkshithSurya/spherex-projects - Hugging Face Spaces provides free 2 vCPU, 16 GB RAM, and 50 GB disk. - The Space and the Dataset reside in the same Hugging Face infrastructure. - Network transfers between HF Dataset and HF Space are internal, low-latency, and fast. - Can use Pattern A (Lazy Streaming) or lazily download files with `hf_hub_download` or `huggingface_hub`. Advantages: - Unified ecosystem (dataset + space under the same profile). - Free hardware allocation is generous (16 GB RAM, 50 GB disk). - Community discoverability on Hugging Face. -------------------------------------------------------------------------------- PATTERN C: DEDICATED VPS / CLOUD VM (FULL DISK MIRROR) -------------------------------------------------------------------------------- How it works: - Provision a Linux virtual server (DigitalOcean, Hetzner, AWS EC2, GCP) with >= 50 GB SSD. - Run: huggingface-cli download AkshithSurya/spherex-projects \ --repo-type dataset \ --local-dir spherex_data \ --local-dir-use-symlinks False - Run `s.py` as a systemd service behind Nginx with Let's Encrypt SSL. Advantages: - 100% offline self-contained once synced. - Zero dependency on external network after sync. Disadvantages: - Requires paying for 50+ GB cloud block storage ($5 - $15/month). 4. ESSENTIAL CODE REFACTORING CHECKLIST (BEFORE DEPLOYING) -------------------------------------------------------------------------------- Before `s.py` can run in a cloud container, 4 specific adjustments must be made: [CRITICAL FIX 1] Remove Localhost-Only Rejection - Current Behavior: In `s.py` (lines 1186-1212), `_require_local_request()` strictly checks if the HTTP `Host` header equals `localhost` or `127.0.0.1`. When deployed to the cloud, requests have hosts like `username-space.hf.space` or `app.onrender.com`. As currently coded, the server returns: `403 Forbidden: Local requests only` for every request! - Fix: Allow host headers when running in production: ```python ALLOW_REMOTE = os.environ.get("SPHEREX_ALLOW_REMOTE", "0") == "1" def _require_local_request(self, check_origin=False): if ALLOW_REMOTE: return True if not self._request_host_is_local() or (check_origin and not self._origin_is_local()): self.send_error(403, "Local requests only") return False return True ``` [CRITICAL FIX 2] Dynamic Port Binding ($PORT) - Current Behavior: `s.py` defaults to port 8000 and requires CLI args to change. Cloud hosts (Hugging Face Spaces uses 7860; Render/Railway use random $PORT). - Fix: In `s.py` line 2120: ```python target_port = int(os.environ.get("PORT", "8000")) ``` [CRITICAL FIX 3] Catalog Fallback in list_bands() - Current Behavior: `list_bands()` in `s.py` inspects `os.listdir(DATA_DIR)`. In a lightweight cloud container without 29 GB downloaded, it returns `{}`. - Fix: If `DATA_DIR` does not contain FITS files, parse `footprints.json`: ```python def list_bands(): # Check local disk first... # Fallback to footprints.json if local directory is empty: if not bands: fp_path = os.path.join(BASE_DIR, "footprints.json") if os.path.isfile(fp_path): with open(fp_path, "r", encoding="utf-8") as f: fp = json.load(f) for bname, items in fp.items(): bands[bname] = [item["file"] for item in items if "file" in item] return bands ``` [CRITICAL FIX 4] Hugging Face Stream & Redirect in /files/ Route - Current Behavior: `GET /files/{rel}` only looks at local disk `DATA_DIR`. - Fix: If local file is absent, redirect directly to Hugging Face or stream it: ```python if not os.path.isfile(full_path): # Redirect client directly to Hugging Face CDN: hf_url = f"https://huggingface.co/datasets/AkshithSurya/spherex-projects/resolve/main/{urllib.parse.quote(rel)}" self.send_response(302) self.send_header("Location", hf_url) self.send_header("Cache-Control", "public, max-age=86400") self.end_headers() return ``` [CRITICAL FIX 5] Whitelist Hugging Face in Security Allowlist - Current Behavior: `security.py` restricts outgoing HTTP calls to Caltech/CDS/NASA domains. - Fix: Add `"huggingface.co"` and `".huggingface.co"` to `ALLOWED_DOMAINS` and `ALLOWED_SUFFIXES`. 5. TOP DEPLOYMENT OPTIONS RANKED -------------------------------------------------------------------------------- ================================================================================ OPTION 1: HUGGING FACE SPACES (DOCKER SDK) - [RECOMMENDED] ================================================================================ Why this is the #1 choice: - Zero dollar hosting cost (Free Tier). - 2 vCPU, 16 GB RAM, 50 GB non-persistent container disk. - Directly co-located with `AkshithSurya/spherex-projects` dataset. - Built-in HTTPS with custom subdomain: `https://huggingface.co/spaces/AkshithSurya/spherex-explorer` - Public visibility and community astronomy sharing. How to deploy: 1. Go to https://huggingface.co/new-space 2. Space Name: `spherex-explorer` (or preferred name) 3. Space SDK: Choose "Docker" (Blank) 4. Repository visibility: Public 5. In the created Space repo, add the following files: - `Dockerfile` (see template below) - `README.md` with YAML header (see template below) - `requirements.txt` - Project source code (`s.py`, `index.html`, `elements.css`, `tokens.css`, `footprints.json`, `aladin_v3.js`, and supporting `.py` modules) 6. Commit & Push. Hugging Face automatically builds the Docker image and launches it. ================================================================================ OPTION 2: RENDER.COM / RAILWAY / FLY.IO (PAAS CONTAINER) ================================================================================ Why choose this: - Standard Git-push deployment workflow connected directly to a GitHub repository. - Automatic builds on every `git push`. - Uses the same Dockerfile or simple `python3 s.py`. - With Pattern A (Lazy Streaming), the container memory footprint is only ~100 MB. How to deploy (Render): 1. Push code to a GitHub repository (excluding `spherex_data/` via `.gitignore`). 2. Sign in to https://render.com -> New Web Service. 3. Select your GitHub repository. 4. Environment: `Python 3` or `Docker`. 5. Build Command: `pip install -r requirements.txt` 6. Start Command: `python3 s.py $PORT` 7. Environment Variables: SPHEREx_ALLOW_REMOTE = 1 PORT = 10000 ================================================================================ OPTION 3: DEDICATED CLOUD VM (UBUNTU 24.04 LTS ON HETZNER / AWS / DIGITALOCEAN) ================================================================================ Why choose this: - If you require persistent local storage of all 29 GB FITS files for low-latency offline processing or custom private access. - Specifications: 2 vCPU, 4 GB RAM, 60 GB SSD (~$6/month on Hetzner Cloud). How to deploy: 1. SSH into the server: `ssh root@your-server-ip` 2. Clone repo: `git clone /opt/spherex-explorer` 3. Install dependencies: apt update && apt install -y python3 python3-pip git-lfs nginx certbot python3-certbot-nginx 4. Download full 29 GB dataset from Hugging Face: pip3 install huggingface_hub python3 -c " from huggingface_hub import snapshot_download snapshot_download(repo_id='AkshithSurya/spherex-projects', repo_type='dataset', local_dir='/opt/spherex-explorer/spherex_data') " 5. Create systemd service `/etc/systemd/system/spherex.service`: [Unit] Description=SPHEREx Data Explorer Backend After=network.target [Service] Type=simple User=www-data WorkingDirectory=/opt/spherex-explorer Environment=SPHEREX_ALLOW_REMOTE=1 Environment=PORT=8000 ExecStart=/usr/bin/python3 /opt/spherex-explorer/s.py Restart=always RestartSec=3 [Install] WantedBy=multi-user.target 6. Enable and start: systemctl enable --now spherex 7. Configure Nginx reverse proxy to port 8000 with Certbot SSL. 6. READY-TO-USE CONFIGURATION TEMPLATES -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- [TEMPLATE 1] Hugging Face Spaces README.md -------------------------------------------------------------------------------- --- title: SPHEREx Data Explorer colorFrom: blue colorTo: indigo sdk: docker app_port: 7860 fullWidth: true datasets: - AkshithSurya/spherex-projects short_description: Interactive astronomical explorer for SPHEREx Level 2 FITS survey. --- # SPHEREx Data Explorer Explore calibrated Level 2 FITS observations across all six spectral bands (0.75 - 5.00 μm). FITS observations are streamed on demand from the [SPHEREx Project Dataset](https://huggingface.co/datasets/AkshithSurya/spherex-projects). -------------------------------------------------------------------------------- [TEMPLATE 2] Dockerfile (Production Ready for Hugging Face Spaces & PaaS) -------------------------------------------------------------------------------- FROM python:3.11-slim # Avoid buffering and bytecode issues ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PORT=7860 \ SPHEREX_ALLOW_REMOTE=1 WORKDIR /app # Install system dependencies if required RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ && rm -rf /var/lib/apt/lists/* # Copy requirements and install COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Create cache directories RUN mkdir -p /app/hips_cache /app/spherex_data /app/spherex_mosaics /tmp/spherex_cache && \ chmod -R 777 /app/hips_cache /app/spherex_data /app/spherex_mosaics /tmp/spherex_cache # Copy application files (exclude 29 GB local fits via .dockerignore) COPY . . # Hugging Face default port is 7860 EXPOSE 7860 # Launch server CMD ["python3", "s.py", "7860"] -------------------------------------------------------------------------------- [TEMPLATE 3] .dockerignore -------------------------------------------------------------------------------- .git .github __pycache__ *.pyc *.pyo .pytest_cache *.fits *.fits.gz *.tar *.zip spherex_data/* !spherex_data/.gitkeep hips_cache/* catalog_output/* cache_remote/* cache_previews/* server.log stitch_bands.log *.Identifier -------------------------------------------------------------------------------- [TEMPLATE 4] requirements.txt (Cloud Recommended) -------------------------------------------------------------------------------- # Scientific and astronomy libraries for full server capabilities astropy>=5.3.0 numpy>=1.24.0 scipy>=1.10.0 urllib3>=2.0.0 requests>=2.31.0 7. STEP-BY-STEP EXECUTION ROADMAP -------------------------------------------------------------------------------- PHASE 1: Code Preparation in Workspace 1. Update `s.py`: - Add `SPHEREX_ALLOW_REMOTE = os.environ.get("SPHEREX_ALLOW_REMOTE", "0") == "1"` check. - Add `int(os.environ.get("PORT", "8000"))` port fallback. - Add `footprints.json` fallback in `list_bands()`. - Add 302 Redirect to Hugging Face URL in `/files/{rel}` when file is absent locally. 2. Update `security.py`: - Add `"huggingface.co"` and `".huggingface.co"` to allowed domains. 3. Verify local tests pass: `python3 test_suite.py` PHASE 2: Git Packaging (Without 29 GB FITS) 1. Verify `.gitignore` includes `spherex_data/*.fits`. 2. Confirm `footprints.json`, `index.html`, `elements.css`, `tokens.css`, `aladin_v3.js`, and all python modules are tracked in git. 3. Total git repository size will be ~3.5 MB instead of 29 GB! PHASE 3: Hugging Face Space Deployment 1. Create Docker Space on Hugging Face (`AkshithSurya/spherex-explorer`). 2. Add `Dockerfile`, `requirements.txt`, and code. 3. Push to Hugging Face Space git remote. 4. Wait for Docker build (~2-3 minutes). 5. The app goes live at: `https://huggingface.co/spaces/AkshithSurya/spherex-explorer` PHASE 4: Verification Checklist [ ] Open public URL in browser. Confirm UI loads with no 403 errors. [ ] Confirm all 6 spectral band cards show correct file counts (56, 100, 59, 56, 100, 63). [ ] Verify Aladin Lite v3 renders sky surveys (DSS2, 2MASS, AllWISE). [ ] Click on any observation in the catalog table: - Footprint highlights on the sky map. - "Provenance" modal opens with correct WCS headers. - "Download FITS" initiates download directly from Hugging Face dataset CDN. [ ] Run Cone Search on target (e.g., "M 31" or "00h42m44s +41d16m09s"): - Coordinates resolve properly. - Matching SPHEREx observations are returned and listed. [ ] Test SIMBAD cross-match analysis tab. ================================================================================ END OF PLAN - SPHEREx CLOUD DEPLOYMENT ================================================================================