Spaces:
Running
Running
Download deploy.txt from AkshithSurya/spherex-projects: direct link, hf CLI and curl.
- Browser
- Download file 18.9 kB
-
https://huggingface.co/spaces/AkshithSurya/spherex-projects/resolve/main/deploy.txt
- Command line
-
hf download hf://spaces/AkshithSurya/spherex-projects/deploy.txt
-
curl -L -o deploy.txt https://huggingface.co/spaces/AkshithSurya/spherex-projects/resolve/main/deploy.txt
18.9 kB
| ================================================================================ | |
| SPHEREx DATA EXPLORER: CLOUD DEPLOYMENT & DATA ACCESS PLAN | |
| ================================================================================ | |
| Document: deploy.txt | |
| Target: Deployment architecture, Hugging Face dataset integration, and implementation plan | |
| Dataset Reference: https://huggingface.co/datasets/AkshithSurya/spherex-projects/tree/main | |
| ================================================================================ | |
| 1. EXECUTIVE ARCHITECTURE SUMMARY | |
| -------------------------------------------------------------------------------- | |
| The SPHEREx Data Explorer is NOT a static website. It is an astronomical web | |
| application combining a dynamic Python server backend (`s.py` and supporting modules) | |
| with a rich, interactive astronomical client (`index.html`, Aladin Lite v3, custom | |
| projections, HiPS sky viewer, and SIMBAD analytics). | |
| Core capabilities that require an active backend server: | |
| - Coordinate transformations and celestial cone searches (/api/observations/search). | |
| - Astronomical target resolution via Sesame, SIMBAD, and NED (/api/targets/resolve). | |
| - FITS header provenance extraction and WCS parameter parsing on demand. | |
| - HiPS sky survey mirroring, caching, and proxying (/hips/..., /mocserver...). | |
| - Dynamic astronomical cross-matching against SIMBAD TAP (/api/analysis/nearby). | |
| - FITS file streaming, range queries, and dynamic zip archive creation. | |
| - In-memory transient overlay blob processing with SIP distortion calculation. | |
| Attempting to deploy this as a static site (e.g. GitHub Pages, Vercel Static, or | |
| Hugging Face Spaces static SDK) breaks ~80% of the core scientific workflow. | |
| 2. THE 29 GB DATA REALITY & THE HUGGING FACE BRIDGE | |
| -------------------------------------------------------------------------------- | |
| The local SPHEREx dataset consists of: | |
| - 434 calibrated Level 2 FITS files. | |
| - 6 spectral bands: | |
| * SPHEREx-D1 (0.75 - 1.11 um): 56 files (~3.8 GB) | |
| * SPHEREx-D2 (1.11 - 1.64 um): 100 files (~6.7 GB) | |
| * SPHEREx-D3 (1.64 - 2.42 um): 59 files (~4.0 GB) | |
| * SPHEREx-D4 (2.42 - 3.82 um): 56 files (~3.8 GB) | |
| * SPHEREx-D5 (3.82 - 4.42 um): 100 files (~6.7 GB) | |
| * SPHEREx-D6 (4.42 - 5.00 um): 63 files (~4.3 GB) | |
| - Total raw storage: ~29 GB. | |
| THE ADVANTAGE: | |
| All 434 files are ALREADY hosted publicly in the Hugging Face dataset repository: | |
| https://huggingface.co/datasets/AkshithSurya/spherex-projects/tree/main | |
| Every single file can be accessed directly via HTTP GET: | |
| https://huggingface.co/datasets/AkshithSurya/spherex-projects/resolve/main/{band}/{filename} | |
| Furthermore, `footprints.json` in the codebase already contains the complete index: | |
| - Exact filenames and relative paths. | |
| - Central celestial coordinates (RA, Dec). | |
| - Sky coverage polygons for every single observation. | |
| This enables a "Zero-Disk" / "Lazy-Streaming" cloud architecture where the deployed | |
| application does NOT need to store 29 GB on container startup! | |
| 3. ARCHITECTURAL DATA ACCESS PATTERNS (HOW TO ACCESS THE FITS FILES) | |
| -------------------------------------------------------------------------------- | |
| There are three distinct patterns to handle the FITS files in production: | |
| -------------------------------------------------------------------------------- | |
| PATTERN A: SMART HYBRID LAZY-STREAMING & REDIRECT (RECOMMENDED FOR CLOUD) | |
| -------------------------------------------------------------------------------- | |
| How it works: | |
| 1. Boot Time: | |
| - The web server container boots with only the code, CSS, JS, and `footprints.json` | |
| (container image size: < 150 MB). Boot takes 2 seconds. | |
| 2. Band Browsing & Search: | |
| - The application serves band counts and footprints directly from `footprints.json`. | |
| - The interactive Aladin sky view shows all 434 observation footprints instantly. | |
| 3. FITS File Download (/files/{band}/{file} or /api/observations/{id}/file): | |
| - Check local cache directory. If absent, issue an HTTP 302 Redirect to: | |
| https://huggingface.co/datasets/AkshithSurya/spherex-projects/resolve/main/{band}/{file} | |
| - The client's browser downloads directly from Hugging Face's global CDN at full speed, | |
| consuming zero server bandwidth and zero container storage. | |
| 4. Header Provenance Extraction (/api/observations/{id}/provenance): | |
| - If the file is not in local cache, send an HTTP Range Request: | |
| `Range: bytes=0-65535` to Hugging Face. | |
| - Download only the first 64 KB (where FITS primary + extension headers reside). | |
| - Parse the WCS and metadata in memory without downloading the 40 MB image array. | |
| - Cache the parsed JSON provenance in `ScientificCache` (memory). | |
| 5. In-Server Analysis / Overlays: | |
| - If user requests server-side pixel manipulation, fetch the single required FITS | |
| file on demand and store it in an ephemeral LRU cache (e.g., /tmp/spherex_cache, | |
| max 2 GB). | |
| Advantages: | |
| - Free tier compatible on almost any cloud platform. | |
| - Blazing fast deployment (no 29 GB upload or long git clone). | |
| - Bandwidth for downloads is absorbed by Hugging Face's CDN. | |
| -------------------------------------------------------------------------------- | |
| PATTERN B: HUGGING FACE SPACES DOCKER (CO-LOCATED NATIVE) | |
| -------------------------------------------------------------------------------- | |
| How it works: | |
| - Deploy a Hugging Face Space using the Docker SDK in the same user account | |
| (`AkshithSurya`). | |
| - Declare the dataset dependency in the Space `README.md`: | |
| datasets: | |
| - AkshithSurya/spherex-projects | |
| - Hugging Face Spaces provides free 2 vCPU, 16 GB RAM, and 50 GB disk. | |
| - The Space and the Dataset reside in the same Hugging Face infrastructure. | |
| - Network transfers between HF Dataset and HF Space are internal, low-latency, and fast. | |
| - Can use Pattern A (Lazy Streaming) or lazily download files with `hf_hub_download` | |
| or `huggingface_hub`. | |
| Advantages: | |
| - Unified ecosystem (dataset + space under the same profile). | |
| - Free hardware allocation is generous (16 GB RAM, 50 GB disk). | |
| - Community discoverability on Hugging Face. | |
| -------------------------------------------------------------------------------- | |
| PATTERN C: DEDICATED VPS / CLOUD VM (FULL DISK MIRROR) | |
| -------------------------------------------------------------------------------- | |
| How it works: | |
| - Provision a Linux virtual server (DigitalOcean, Hetzner, AWS EC2, GCP) with >= 50 GB SSD. | |
| - Run: | |
| huggingface-cli download AkshithSurya/spherex-projects \ | |
| --repo-type dataset \ | |
| --local-dir spherex_data \ | |
| --local-dir-use-symlinks False | |
| - Run `s.py` as a systemd service behind Nginx with Let's Encrypt SSL. | |
| Advantages: | |
| - 100% offline self-contained once synced. | |
| - Zero dependency on external network after sync. | |
| Disadvantages: | |
| - Requires paying for 50+ GB cloud block storage ($5 - $15/month). | |
| 4. ESSENTIAL CODE REFACTORING CHECKLIST (BEFORE DEPLOYING) | |
| -------------------------------------------------------------------------------- | |
| Before `s.py` can run in a cloud container, 4 specific adjustments must be made: | |
| [CRITICAL FIX 1] Remove Localhost-Only Rejection | |
| - Current Behavior: | |
| In `s.py` (lines 1186-1212), `_require_local_request()` strictly checks if the | |
| HTTP `Host` header equals `localhost` or `127.0.0.1`. | |
| When deployed to the cloud, requests have hosts like `username-space.hf.space` | |
| or `app.onrender.com`. As currently coded, the server returns: | |
| `403 Forbidden: Local requests only` for every request! | |
| - Fix: | |
| Allow host headers when running in production: | |
| ```python | |
| ALLOW_REMOTE = os.environ.get("SPHEREX_ALLOW_REMOTE", "0") == "1" | |
| def _require_local_request(self, check_origin=False): | |
| if ALLOW_REMOTE: | |
| return True | |
| if not self._request_host_is_local() or (check_origin and not self._origin_is_local()): | |
| self.send_error(403, "Local requests only") | |
| return False | |
| return True | |
| ``` | |
| [CRITICAL FIX 2] Dynamic Port Binding ($PORT) | |
| - Current Behavior: | |
| `s.py` defaults to port 8000 and requires CLI args to change. | |
| Cloud hosts (Hugging Face Spaces uses 7860; Render/Railway use random $PORT). | |
| - Fix: | |
| In `s.py` line 2120: | |
| ```python | |
| target_port = int(os.environ.get("PORT", "8000")) | |
| ``` | |
| [CRITICAL FIX 3] Catalog Fallback in list_bands() | |
| - Current Behavior: | |
| `list_bands()` in `s.py` inspects `os.listdir(DATA_DIR)`. In a lightweight cloud | |
| container without 29 GB downloaded, it returns `{}`. | |
| - Fix: | |
| If `DATA_DIR` does not contain FITS files, parse `footprints.json`: | |
| ```python | |
| def list_bands(): | |
| # Check local disk first... | |
| # Fallback to footprints.json if local directory is empty: | |
| if not bands: | |
| fp_path = os.path.join(BASE_DIR, "footprints.json") | |
| if os.path.isfile(fp_path): | |
| with open(fp_path, "r", encoding="utf-8") as f: | |
| fp = json.load(f) | |
| for bname, items in fp.items(): | |
| bands[bname] = [item["file"] for item in items if "file" in item] | |
| return bands | |
| ``` | |
| [CRITICAL FIX 4] Hugging Face Stream & Redirect in /files/ Route | |
| - Current Behavior: | |
| `GET /files/{rel}` only looks at local disk `DATA_DIR`. | |
| - Fix: | |
| If local file is absent, redirect directly to Hugging Face or stream it: | |
| ```python | |
| if not os.path.isfile(full_path): | |
| # Redirect client directly to Hugging Face CDN: | |
| hf_url = f"https://huggingface.co/datasets/AkshithSurya/spherex-projects/resolve/main/{urllib.parse.quote(rel)}" | |
| self.send_response(302) | |
| self.send_header("Location", hf_url) | |
| self.send_header("Cache-Control", "public, max-age=86400") | |
| self.end_headers() | |
| return | |
| ``` | |
| [CRITICAL FIX 5] Whitelist Hugging Face in Security Allowlist | |
| - Current Behavior: | |
| `security.py` restricts outgoing HTTP calls to Caltech/CDS/NASA domains. | |
| - Fix: | |
| Add `"huggingface.co"` and `".huggingface.co"` to `ALLOWED_DOMAINS` and `ALLOWED_SUFFIXES`. | |
| 5. TOP DEPLOYMENT OPTIONS RANKED | |
| -------------------------------------------------------------------------------- | |
| ================================================================================ | |
| OPTION 1: HUGGING FACE SPACES (DOCKER SDK) - [RECOMMENDED] | |
| ================================================================================ | |
| Why this is the #1 choice: | |
| - Zero dollar hosting cost (Free Tier). | |
| - 2 vCPU, 16 GB RAM, 50 GB non-persistent container disk. | |
| - Directly co-located with `AkshithSurya/spherex-projects` dataset. | |
| - Built-in HTTPS with custom subdomain: `https://huggingface.co/spaces/AkshithSurya/spherex-explorer` | |
| - Public visibility and community astronomy sharing. | |
| How to deploy: | |
| 1. Go to https://huggingface.co/new-space | |
| 2. Space Name: `spherex-explorer` (or preferred name) | |
| 3. Space SDK: Choose "Docker" (Blank) | |
| 4. Repository visibility: Public | |
| 5. In the created Space repo, add the following files: | |
| - `Dockerfile` (see template below) | |
| - `README.md` with YAML header (see template below) | |
| - `requirements.txt` | |
| - Project source code (`s.py`, `index.html`, `elements.css`, `tokens.css`, | |
| `footprints.json`, `aladin_v3.js`, and supporting `.py` modules) | |
| 6. Commit & Push. Hugging Face automatically builds the Docker image and launches it. | |
| ================================================================================ | |
| OPTION 2: RENDER.COM / RAILWAY / FLY.IO (PAAS CONTAINER) | |
| ================================================================================ | |
| Why choose this: | |
| - Standard Git-push deployment workflow connected directly to a GitHub repository. | |
| - Automatic builds on every `git push`. | |
| - Uses the same Dockerfile or simple `python3 s.py`. | |
| - With Pattern A (Lazy Streaming), the container memory footprint is only ~100 MB. | |
| How to deploy (Render): | |
| 1. Push code to a GitHub repository (excluding `spherex_data/` via `.gitignore`). | |
| 2. Sign in to https://render.com -> New Web Service. | |
| 3. Select your GitHub repository. | |
| 4. Environment: `Python 3` or `Docker`. | |
| 5. Build Command: `pip install -r requirements.txt` | |
| 6. Start Command: `python3 s.py $PORT` | |
| 7. Environment Variables: | |
| SPHEREx_ALLOW_REMOTE = 1 | |
| PORT = 10000 | |
| ================================================================================ | |
| OPTION 3: DEDICATED CLOUD VM (UBUNTU 24.04 LTS ON HETZNER / AWS / DIGITALOCEAN) | |
| ================================================================================ | |
| Why choose this: | |
| - If you require persistent local storage of all 29 GB FITS files for low-latency | |
| offline processing or custom private access. | |
| - Specifications: 2 vCPU, 4 GB RAM, 60 GB SSD (~$6/month on Hetzner Cloud). | |
| How to deploy: | |
| 1. SSH into the server: `ssh root@your-server-ip` | |
| 2. Clone repo: `git clone <your-git-repo> /opt/spherex-explorer` | |
| 3. Install dependencies: | |
| apt update && apt install -y python3 python3-pip git-lfs nginx certbot python3-certbot-nginx | |
| 4. Download full 29 GB dataset from Hugging Face: | |
| pip3 install huggingface_hub | |
| python3 -c " | |
| from huggingface_hub import snapshot_download | |
| snapshot_download(repo_id='AkshithSurya/spherex-projects', repo_type='dataset', local_dir='/opt/spherex-explorer/spherex_data') | |
| " | |
| 5. Create systemd service `/etc/systemd/system/spherex.service`: | |
| [Unit] | |
| Description=SPHEREx Data Explorer Backend | |
| After=network.target | |
| [Service] | |
| Type=simple | |
| User=www-data | |
| WorkingDirectory=/opt/spherex-explorer | |
| Environment=SPHEREX_ALLOW_REMOTE=1 | |
| Environment=PORT=8000 | |
| ExecStart=/usr/bin/python3 /opt/spherex-explorer/s.py | |
| Restart=always | |
| RestartSec=3 | |
| [Install] | |
| WantedBy=multi-user.target | |
| 6. Enable and start: | |
| systemctl enable --now spherex | |
| 7. Configure Nginx reverse proxy to port 8000 with Certbot SSL. | |
| 6. READY-TO-USE CONFIGURATION TEMPLATES | |
| -------------------------------------------------------------------------------- | |
| -------------------------------------------------------------------------------- | |
| [TEMPLATE 1] Hugging Face Spaces README.md | |
| -------------------------------------------------------------------------------- | |
| --- | |
| title: SPHEREx Data Explorer | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| app_port: 7860 | |
| fullWidth: true | |
| datasets: | |
| - AkshithSurya/spherex-projects | |
| short_description: Interactive astronomical explorer for SPHEREx Level 2 FITS survey. | |
| --- | |
| # SPHEREx Data Explorer | |
| Explore calibrated Level 2 FITS observations across all six spectral bands (0.75 - 5.00 μm). | |
| FITS observations are streamed on demand from the [SPHEREx Project Dataset](https://huggingface.co/datasets/AkshithSurya/spherex-projects). | |
| -------------------------------------------------------------------------------- | |
| [TEMPLATE 2] Dockerfile (Production Ready for Hugging Face Spaces & PaaS) | |
| -------------------------------------------------------------------------------- | |
| FROM python:3.11-slim | |
| # Avoid buffering and bytecode issues | |
| ENV PYTHONUNBUFFERED=1 \ | |
| PYTHONDONTWRITEBYTECODE=1 \ | |
| PORT=7860 \ | |
| SPHEREX_ALLOW_REMOTE=1 | |
| WORKDIR /app | |
| # Install system dependencies if required | |
| RUN apt-get update && apt-get install -y --no-install-recommends \ | |
| curl \ | |
| && rm -rf /var/lib/apt/lists/* | |
| # Copy requirements and install | |
| COPY requirements.txt . | |
| RUN pip install --no-cache-dir -r requirements.txt | |
| # Create cache directories | |
| RUN mkdir -p /app/hips_cache /app/spherex_data /app/spherex_mosaics /tmp/spherex_cache && \ | |
| chmod -R 777 /app/hips_cache /app/spherex_data /app/spherex_mosaics /tmp/spherex_cache | |
| # Copy application files (exclude 29 GB local fits via .dockerignore) | |
| COPY . . | |
| # Hugging Face default port is 7860 | |
| EXPOSE 7860 | |
| # Launch server | |
| CMD ["python3", "s.py", "7860"] | |
| -------------------------------------------------------------------------------- | |
| [TEMPLATE 3] .dockerignore | |
| -------------------------------------------------------------------------------- | |
| .git | |
| .github | |
| __pycache__ | |
| *.pyc | |
| *.pyo | |
| .pytest_cache | |
| *.fits | |
| *.fits.gz | |
| *.tar | |
| *.zip | |
| spherex_data/* | |
| !spherex_data/.gitkeep | |
| hips_cache/* | |
| catalog_output/* | |
| cache_remote/* | |
| cache_previews/* | |
| server.log | |
| stitch_bands.log | |
| *.Identifier | |
| -------------------------------------------------------------------------------- | |
| [TEMPLATE 4] requirements.txt (Cloud Recommended) | |
| -------------------------------------------------------------------------------- | |
| # Scientific and astronomy libraries for full server capabilities | |
| astropy>=5.3.0 | |
| numpy>=1.24.0 | |
| scipy>=1.10.0 | |
| urllib3>=2.0.0 | |
| requests>=2.31.0 | |
| 7. STEP-BY-STEP EXECUTION ROADMAP | |
| -------------------------------------------------------------------------------- | |
| PHASE 1: Code Preparation in Workspace | |
| 1. Update `s.py`: | |
| - Add `SPHEREX_ALLOW_REMOTE = os.environ.get("SPHEREX_ALLOW_REMOTE", "0") == "1"` check. | |
| - Add `int(os.environ.get("PORT", "8000"))` port fallback. | |
| - Add `footprints.json` fallback in `list_bands()`. | |
| - Add 302 Redirect to Hugging Face URL in `/files/{rel}` when file is absent locally. | |
| 2. Update `security.py`: | |
| - Add `"huggingface.co"` and `".huggingface.co"` to allowed domains. | |
| 3. Verify local tests pass: | |
| `python3 test_suite.py` | |
| PHASE 2: Git Packaging (Without 29 GB FITS) | |
| 1. Verify `.gitignore` includes `spherex_data/*.fits`. | |
| 2. Confirm `footprints.json`, `index.html`, `elements.css`, `tokens.css`, | |
| `aladin_v3.js`, and all python modules are tracked in git. | |
| 3. Total git repository size will be ~3.5 MB instead of 29 GB! | |
| PHASE 3: Hugging Face Space Deployment | |
| 1. Create Docker Space on Hugging Face (`AkshithSurya/spherex-explorer`). | |
| 2. Add `Dockerfile`, `requirements.txt`, and code. | |
| 3. Push to Hugging Face Space git remote. | |
| 4. Wait for Docker build (~2-3 minutes). | |
| 5. The app goes live at: `https://huggingface.co/spaces/AkshithSurya/spherex-explorer` | |
| PHASE 4: Verification Checklist | |
| [ ] Open public URL in browser. Confirm UI loads with no 403 errors. | |
| [ ] Confirm all 6 spectral band cards show correct file counts (56, 100, 59, 56, 100, 63). | |
| [ ] Verify Aladin Lite v3 renders sky surveys (DSS2, 2MASS, AllWISE). | |
| [ ] Click on any observation in the catalog table: | |
| - Footprint highlights on the sky map. | |
| - "Provenance" modal opens with correct WCS headers. | |
| - "Download FITS" initiates download directly from Hugging Face dataset CDN. | |
| [ ] Run Cone Search on target (e.g., "M 31" or "00h42m44s +41d16m09s"): | |
| - Coordinates resolve properly. | |
| - Matching SPHEREx observations are returned and listed. | |
| [ ] Test SIMBAD cross-match analysis tab. | |
| ================================================================================ | |
| END OF PLAN - SPHEREx CLOUD DEPLOYMENT | |
| ================================================================================ | |