Spaces:
Configuration error
Configuration error
|
Download docs/DESIGN.md from thedataenginex/dex-demo: direct link, hf CLI and curl.
- Browser
- Download file 10.2 kB
-
https://huggingface.co/spaces/thedataenginex/dex-demo/resolve/main/docs/DESIGN.md
- Command line
-
hf download hf://spaces/thedataenginex/dex-demo/docs/DESIGN.md
-
curl -L -o DESIGN.md https://huggingface.co/spaces/thedataenginex/dex-demo/resolve/main/docs/DESIGN.md
10.2 kB
| # DEX Studio β Design Document | |
| ## 1. Purpose | |
| DEX Studio is a **local, Python-first, open-source UI application** that provides a single | |
| control plane for end-to-end data projects powered by [TheDataEngineX/DEX](https://github.com/TheDataEngineX/DEX). | |
| It does **not** fork or rebrand upstream DEX. It is a separate tool that connects to a local | |
| DEX engine instance and unifies workflows in one place: | |
| ``` | |
| Project Setup β Ingestion β Medallion Pipelines β ML/AI β Serving β Observability | |
| ``` | |
| ## 2. Architecture | |
| ``` | |
| ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β DEX Studio (NiceGUI) β | |
| β β | |
| β ββββββββββββ ββββββββββββββ ββββββββββββββββββββββββββββββββ β | |
| β β Pages β β Components β β State / Storage β β | |
| β β (views) β β (shared) β β (NiceGUI app.storage) β β | |
| β βββββββ¬ββββββ βββββββ¬βββββββ βββββββββββββββ¬βββββββββββββββ β | |
| β β β β β | |
| β βββββββ΄ββββββββββββββββ΄βββββββββββββββββββββββββ΄βββββββββββββββ β | |
| β β Studio Core β β | |
| β β ββββββββββββ ββββββββββββ ββββββββββββββ β β | |
| β β β Config β β Client β β Theme β β β | |
| β β β (YAML) β β (httpx) β β (CSS) β β β | |
| β β ββββββββββββ ββββββββββββ ββββββββββββββ β β | |
| β ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ β | |
| βββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββ | |
| β HTTP (localhost:8000) | |
| βΌ | |
| ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β DEX Engine (FastAPI) β | |
| β /health /ready /startup /metrics β | |
| β /api/v1/data/* /api/v1/warehouse/* /api/v1/system/* β | |
| β /api/v1/models /api/v1/models/{name} /api/v1/predict β | |
| ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| ``` | |
| ### Why NiceGUI + Native Mode | |
| | Requirement | NiceGUI Capability | | |
| |--------------------------------------|-------------------------------------------| | |
| | Python-first, no JS build step | β Pure Python components | | |
| | Real URL routing (`/quality`, `/models`) | β `@ui.page` decorator | | |
| | Websocket for live updates | β Built-in, auto-managed | | |
| | Desktop-native window | β `native=True` via pywebview | | |
| | Built on FastAPI | β Same stack as DEX engine | | |
| | Component model (cards, tables, tabs)| β Quasar + custom components | | |
| | Auth middleware (future) | β FastAPI middleware support | | |
| ### Why NOT Streamlit | |
| - No real URL routing (query params only) | |
| - No websocket support (polling via rerun loop) | |
| - No background tasks (script re-executes on every interaction) | |
| - Performance degrades with complex layouts | |
| - Can't embed into existing FastAPI infrastructure | |
| ## 3. Project Structure | |
| ``` | |
| dex-studio/ | |
| βββ pyproject.toml # Hatchling build, deps | |
| βββ .python-version # 3.11 | |
| βββ config/ | |
| β βββ default.yaml # Default config template | |
| βββ src/ | |
| β βββ dex_studio/ | |
| β βββ __init__.py # Package + version | |
| β βββ cli.py # CLI entry (dex-studio command) | |
| β βββ app.py # NiceGUI app bootstrap | |
| β βββ config.py # YAML config loading | |
| β βββ client.py # httpx DEX API client | |
| β βββ theme.py # CSS theme + colour palette | |
| β βββ components/ | |
| β β βββ sidebar.py # Navigation sidebar | |
| β β βββ status_card.py # Health status cards | |
| β β βββ metric_card.py # Metric display cards | |
| β β βββ page_layout.py # Sidebar + content wrapper | |
| β βββ pages/ | |
| β βββ overview.py # Dashboard (/) | |
| β βββ health.py # Health probes (/health) | |
| β βββ data_quality.py # Medallion quality (/quality) | |
| β βββ lineage.py # Lineage explorer (/lineage) | |
| β βββ ml_models.py # Model registry (/models) | |
| β βββ settings.py # Connection config (/settings) | |
| βββ tests/ | |
| β βββ conftest.py | |
| β βββ unit/ | |
| β βββ test_config.py | |
| β βββ test_client.py | |
| β βββ test_cli.py | |
| βββ docs/ | |
| βββ DESIGN.md # This document | |
| ``` | |
| ## 4. Configuration | |
| Config is loaded with this priority (highest wins): | |
| 1. CLI arguments (`--url`, `--token`, `--theme`) | |
| 1. Environment variables (`DEX_STUDIO_API_URL`, `DEX_STUDIO_API_TOKEN`, β¦) | |
| 1. Project-local `.dex-studio.yaml` | |
| 1. User-level `~/.dex-studio/config.yaml` | |
| 1. Built-in defaults | |
| ```yaml | |
| # ~/.dex-studio/config.yaml | |
| api_url: "http://localhost:8000" | |
| api_token: null | |
| timeout: 10.0 | |
| theme: dark | |
| poll_interval: 5.0 | |
| window_width: 1400 | |
| window_height: 900 | |
| ``` | |
| ## 5. Lockstep Development Model | |
| Each Studio page drives corresponding DEX engine API endpoints. | |
| When Studio needs data that DEX doesn't expose yet, the engine API must be extended first. | |
| | Studio Page | DEX Engine Endpoints Required | Status | | |
| |-----------------|-------------------------------------------------------------|------------| | |
| | Overview | `GET /`, `/health`, `/startup`, `/api/v1/data/quality`, `/api/v1/system/config` | β Available | | |
| | Health | `GET /health`, `/ready`, `/startup` | β Available | | |
| | Data Quality | `GET /api/v1/data/quality`, `/api/v1/data/quality/{layer}`, `/api/v1/data/sources`, `/api/v1/warehouse/layers` | β Available | | |
| | Lineage | `GET /api/v1/warehouse/lineage/{event_id}` | β Available | | |
| | ML Models | `GET /api/v1/models`, `/api/v1/models/{name}`, `POST /api/v1/predict` | π§ Router exists but was unmounted β **fixed in lockstep** | | |
| | Settings | (local only β no API calls) | β N/A | | |
| ### Lockstep Rule | |
| > **No Studio page without a DEX API endpoint. No DEX API endpoint without a Studio page.** | |
| Future pages (pipelines, ingestion, drift detection) require new DEX engine endpoints. | |
| Those endpoints and pages are developed together in coordinated PRs across both repos. | |
| ## 6. DEX Engine Changes (v0.3.x Lockstep) | |
| ### ML Router Mounted | |
| The ML router (`ml_router` in `dataenginex.api.routers.ml`) existed but was never | |
| `include_router()`'d in the CareerDEX app. Fixed by adding: | |
| ```python | |
| from dataenginex.api.routers.ml import ml_router | |
| app.include_router(ml_router) | |
| ``` | |
| This activates 3 previously dead endpoints: | |
| - `POST /api/v1/predict` | |
| - `GET /api/v1/models` | |
| - `GET /api/v1/models/{name}` | |
| ## 7. Technology Choices | |
| | Choice | Rationale | | |
| |---------------------------|----------------------------------------------------------| | |
| | **NiceGUI** | Python-first, real routing, websockets, built on FastAPI | | |
| | **pywebview (native mode)** | Desktop-native window without Electron overhead | | |
| | **httpx** | Async HTTP client, connection pooling, same API as requests | | |
| | **Pydantic** | Config validation, matches DEX engine patterns | | |
| | **PyYAML** | Config file format | | |
| | **Hatchling** | Same build backend as DEX engine | | |
| | **Separate repo** | Different release cadence, users, and dependency trees | | |
| ## 8. Phased Roadmap | |
| ### Phase 0 β Foundation (v0.1.0) β current | |
| - Project scaffold + CI | |
| - httpx client with health check | |
| - YAML config system (file + env + CLI) | |
| - 6 pages: Overview, Health, Quality, Lineage, ML Models, Settings | |
| - DEX engine lockstep: mount ML router | |
| ### Phase 1 β Data Operations (v0.2.0) | |
| - Pipeline trigger page (needs `POST /api/v1/pipelines/run` on DEX engine) | |
| - Ingestion management (needs `POST /api/v1/data/ingest`) | |
| - Live pipeline log streaming (NiceGUI websocket) | |
| ### Phase 2 β ML Workflows (v0.3.0) | |
| - Experiment comparison page | |
| - Drift detection alerts (needs `GET /api/v1/ml/drift`) | |
| - Model promotion workflow | |
| ### Phase 3 β Observability (v0.4.0) | |
| - Prometheus metrics charts (needs structured `GET /api/v1/metrics` JSON endpoint) | |
| - Log viewer (needs `GET /api/v1/logs`) | |
| - Trace explorer (needs `GET /api/v1/traces`) | |
| ### Phase 4 β Project Management (v0.5.0) | |
| - Project scaffolder (`dex-studio init`) | |
| - Config editor (edit DEX engine config from Studio) | |
| - Multi-instance support (connect to multiple DEX engines) | |
| ## 9. Non-Goals (v0.1.0) | |
| - **Cloud deployment** β Studio is local-first; no hosted SaaS version | |
| - **Multi-user auth** β single-user desktop app | |
| - **Data editing** β read-only dashboard; mutations are pipeline triggers only | |
| - **Replacing Grafana** β Studio shows DEX-specific views, not generic metrics dashboards | |
| - **Package on PyPI** β distribute via GitHub releases for now | |