Spaces:
Configuration error
Download docs/DESIGN.md from thedataenginex/dex-demo: direct link, hf CLI and curl.
- Browser
- Download file 10.2 kB
-
https://huggingface.co/spaces/thedataenginex/dex-demo/resolve/main/docs/DESIGN.md
- Command line
-
hf download hf://spaces/thedataenginex/dex-demo/docs/DESIGN.md
-
curl -L -o DESIGN.md https://huggingface.co/spaces/thedataenginex/dex-demo/resolve/main/docs/DESIGN.md
DEX Studio β Design Document
1. Purpose
DEX Studio is a local, Python-first, open-source UI application that provides a single control plane for end-to-end data projects powered by TheDataEngineX/DEX.
It does not fork or rebrand upstream DEX. It is a separate tool that connects to a local DEX engine instance and unifies workflows in one place:
Project Setup β Ingestion β Medallion Pipelines β ML/AI β Serving β Observability
2. Architecture
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DEX Studio (NiceGUI) β
β β
β ββββββββββββ ββββββββββββββ ββββββββββββββββββββββββββββββββ β
β β Pages β β Components β β State / Storage β β
β β (views) β β (shared) β β (NiceGUI app.storage) β β
β βββββββ¬ββββββ βββββββ¬βββββββ βββββββββββββββ¬βββββββββββββββ β
β β β β β
β βββββββ΄ββββββββββββββββ΄βββββββββββββββββββββββββ΄βββββββββββββββ β
β β Studio Core β β
β β ββββββββββββ ββββββββββββ ββββββββββββββ β β
β β β Config β β Client β β Theme β β β
β β β (YAML) β β (httpx) β β (CSS) β β β
β β ββββββββββββ ββββββββββββ ββββββββββββββ β β
β ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββ
β HTTP (localhost:8000)
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DEX Engine (FastAPI) β
β /health /ready /startup /metrics β
β /api/v1/data/* /api/v1/warehouse/* /api/v1/system/* β
β /api/v1/models /api/v1/models/{name} /api/v1/predict β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Why NiceGUI + Native Mode
| Requirement | NiceGUI Capability |
|---|---|
| Python-first, no JS build step | β Pure Python components |
Real URL routing (/quality, /models) |
β
@ui.page decorator |
| Websocket for live updates | β Built-in, auto-managed |
| Desktop-native window | β
native=True via pywebview |
| Built on FastAPI | β Same stack as DEX engine |
| Component model (cards, tables, tabs) | β Quasar + custom components |
| Auth middleware (future) | β FastAPI middleware support |
Why NOT Streamlit
- No real URL routing (query params only)
- No websocket support (polling via rerun loop)
- No background tasks (script re-executes on every interaction)
- Performance degrades with complex layouts
- Can't embed into existing FastAPI infrastructure
3. Project Structure
dex-studio/
βββ pyproject.toml # Hatchling build, deps
βββ .python-version # 3.11
βββ config/
β βββ default.yaml # Default config template
βββ src/
β βββ dex_studio/
β βββ __init__.py # Package + version
β βββ cli.py # CLI entry (dex-studio command)
β βββ app.py # NiceGUI app bootstrap
β βββ config.py # YAML config loading
β βββ client.py # httpx DEX API client
β βββ theme.py # CSS theme + colour palette
β βββ components/
β β βββ sidebar.py # Navigation sidebar
β β βββ status_card.py # Health status cards
β β βββ metric_card.py # Metric display cards
β β βββ page_layout.py # Sidebar + content wrapper
β βββ pages/
β βββ overview.py # Dashboard (/)
β βββ health.py # Health probes (/health)
β βββ data_quality.py # Medallion quality (/quality)
β βββ lineage.py # Lineage explorer (/lineage)
β βββ ml_models.py # Model registry (/models)
β βββ settings.py # Connection config (/settings)
βββ tests/
β βββ conftest.py
β βββ unit/
β βββ test_config.py
β βββ test_client.py
β βββ test_cli.py
βββ docs/
βββ DESIGN.md # This document
4. Configuration
Config is loaded with this priority (highest wins):
- CLI arguments (
--url,--token,--theme) - Environment variables (
DEX_STUDIO_API_URL,DEX_STUDIO_API_TOKEN, β¦) - Project-local
.dex-studio.yaml - User-level
~/.dex-studio/config.yaml - Built-in defaults
# ~/.dex-studio/config.yaml
api_url: "http://localhost:8000"
api_token: null
timeout: 10.0
theme: dark
poll_interval: 5.0
window_width: 1400
window_height: 900
5. Lockstep Development Model
Each Studio page drives corresponding DEX engine API endpoints. When Studio needs data that DEX doesn't expose yet, the engine API must be extended first.
| Studio Page | DEX Engine Endpoints Required | Status |
|---|---|---|
| Overview | GET /, /health, /startup, /api/v1/data/quality, /api/v1/system/config |
β Available |
| Health | GET /health, /ready, /startup |
β Available |
| Data Quality | GET /api/v1/data/quality, /api/v1/data/quality/{layer}, /api/v1/data/sources, /api/v1/warehouse/layers |
β Available |
| Lineage | GET /api/v1/warehouse/lineage/{event_id} |
β Available |
| ML Models | GET /api/v1/models, /api/v1/models/{name}, POST /api/v1/predict |
π§ Router exists but was unmounted β fixed in lockstep |
| Settings | (local only β no API calls) | β N/A |
Lockstep Rule
No Studio page without a DEX API endpoint. No DEX API endpoint without a Studio page.
Future pages (pipelines, ingestion, drift detection) require new DEX engine endpoints. Those endpoints and pages are developed together in coordinated PRs across both repos.
6. DEX Engine Changes (v0.3.x Lockstep)
ML Router Mounted
The ML router (ml_router in dataenginex.api.routers.ml) existed but was never
include_router()'d in the CareerDEX app. Fixed by adding:
from dataenginex.api.routers.ml import ml_router
app.include_router(ml_router)
This activates 3 previously dead endpoints:
POST /api/v1/predictGET /api/v1/modelsGET /api/v1/models/{name}
7. Technology Choices
| Choice | Rationale |
|---|---|
| NiceGUI | Python-first, real routing, websockets, built on FastAPI |
| pywebview (native mode) | Desktop-native window without Electron overhead |
| httpx | Async HTTP client, connection pooling, same API as requests |
| Pydantic | Config validation, matches DEX engine patterns |
| PyYAML | Config file format |
| Hatchling | Same build backend as DEX engine |
| Separate repo | Different release cadence, users, and dependency trees |
8. Phased Roadmap
Phase 0 β Foundation (v0.1.0) β current
- Project scaffold + CI
- httpx client with health check
- YAML config system (file + env + CLI)
- 6 pages: Overview, Health, Quality, Lineage, ML Models, Settings
- DEX engine lockstep: mount ML router
Phase 1 β Data Operations (v0.2.0)
- Pipeline trigger page (needs
POST /api/v1/pipelines/runon DEX engine) - Ingestion management (needs
POST /api/v1/data/ingest) - Live pipeline log streaming (NiceGUI websocket)
Phase 2 β ML Workflows (v0.3.0)
- Experiment comparison page
- Drift detection alerts (needs
GET /api/v1/ml/drift) - Model promotion workflow
Phase 3 β Observability (v0.4.0)
- Prometheus metrics charts (needs structured
GET /api/v1/metricsJSON endpoint) - Log viewer (needs
GET /api/v1/logs) - Trace explorer (needs
GET /api/v1/traces)
Phase 4 β Project Management (v0.5.0)
- Project scaffolder (
dex-studio init) - Config editor (edit DEX engine config from Studio)
- Multi-instance support (connect to multiple DEX engines)
9. Non-Goals (v0.1.0)
- Cloud deployment β Studio is local-first; no hosted SaaS version
- Multi-user auth β single-user desktop app
- Data editing β read-only dashboard; mutations are pipeline triggers only
- Replacing Grafana β Studio shows DEX-specific views, not generic metrics dashboards
- Package on PyPI β distribute via GitHub releases for now