# Development Guide This document provides a comprehensive guide for setting up a local development environment, understanding the project structure, and extending the Visual Notebook with new operations, chart types, and frontend components. ## Prerequisites Before setting up the development environment, ensure you have the following tools installed: | Tool | Version | Purpose | |------|---------|---------| | Python | 3.11+ | Backend runtime | | Node.js | 20+ | Frontend build and development | | npm | 10+ | Frontend package management | | uv | Latest | Python package management (replaces pip/venv) | | Git | 2.40+ | Version control | ### Installing uv ```bash # macOS / Linux curl -LsSf https://astral.sh/uv/install.sh | sh # Windows powershell -c "irm https://astral.sh/uv/install.ps1 | iex" ``` ## Local Development Setup ### Backend Setup 1. **Navigate to the project root**: ```bash cd /path/to/va_project ``` 2. **Create a virtual environment and install dependencies**: ```bash uv sync ``` This command reads `pyproject.toml`, resolves dependencies, creates a virtual environment in `.venv/`, and installs all packages including development dependencies (pytest, httpx). 3. **Run the backend server**: ```bash uv run uvicorn visual_notebook.main:app --reload --host 0.0.0.0 --port 8000 ``` The `--reload` flag enables automatic server restart when source files change. 4. **Verify the server is running**: ```bash curl http://localhost:8000/health # Expected: {"status":"ok","version":"0.1.0"} ``` 5. **Access the API documentation**: Open `http://localhost:8000/docs` in a browser to view the Swagger UI generated by FastAPI. ### Frontend Setup 1. **Navigate to the frontend directory**: ```bash cd frontend ``` 2. **Install dependencies**: ```bash npm install ``` 3. **Start the development server**: ```bash npm run dev ``` The Vite dev server starts at `http://localhost:5173` with hot module replacement (HMR). API requests to `/api` and `/health` are automatically proxied to the backend at `http://localhost:8000` as configured in `vite.config.ts`. 4. **Build for production**: ```bash npm run build ``` The production build outputs to `frontend/dist/` with code-split chunks for ECharts, React, and vendor libraries. ### Running Both Together For the best development experience, run the backend and frontend in separate terminal sessions: - Terminal 1: `uv run uvicorn visual_notebook.main:app --reload --port 8000` - Terminal 2: `cd frontend && npm run dev` Visit `http://localhost:5173` for the full application with live reloading on both frontend and backend changes. ## Project Structure Overview ``` va_project/ pyproject.toml # Root workspace config Dockerfile # Combined build (frontend + backend) Dockerfile.backend # Backend-only build docker-compose.yml # Development Docker Compose datasets/ # Uploaded CSV files checkpoints/ # Session checkpoint files static/ # Built frontend (production) backend/ pyproject.toml # Backend dependencies visual_notebook/ main.py # FastAPI application entry config.py # Settings from environment session.py # Session and SessionStore api/ routes.py # API endpoint definitions schemas.py # Pydantic request/response models deps.py # Dependency injection domain/ dataframe.py # DataFrameRef wrapper history.py # History tree (Memento) operations.py # Operation registry checkpoints.py # Checkpoint persistence validators.py # Validation helpers data/ csv_loader.py # CSV loading and export schema.py # Schema inference types.py # Column type classification viz/ _common.py # Shared chart utilities histogram.py # Histogram computation scatter.py # Scatter plot computation bar_topn.py # Top-N bar chart timeline.py # Timeline/line chart heatmap.py # Heatmap computation map.py # Map computation tests/ test_history.py # History tests test_operations.py # Operations tests frontend/ package.json # Frontend dependencies vite.config.ts # Vite configuration tailwind.config.js # Tailwind CSS theme tsconfig.json # TypeScript configuration index.html # HTML entry point src/ main.tsx # React entry point App.tsx # Root component with ErrorBoundary api/ client.ts # Axios instance and API functions hooks.ts # React hooks for data fetching types.ts # TypeScript type definitions store/ session.ts # Session state store notebook.ts # Notebook cell store ui.ts # UI toggle store history.ts # Undo/redo state store components/ Header.tsx # Top navigation bar NotebookPage.tsx # Main notebook layout UploadPanel.tsx # CSV upload dialog TableCellView.tsx # Table data display ChartCellView.tsx # ECharts rendering SchemaView.tsx # Column schema display ManipulationPanel.tsx # Operation controls VisualizationPanel.tsx # Chart configuration TablePreview.tsx # Table row preview MapCanvas.tsx # Map point visualization ErrorToast.tsx # Error notification lib/ chartTypes.ts # Chart type definitions csv.ts # CSV utility functions format.ts # Number/formatting helpers styles/ index.css # Global styles and Tailwind docs/ # Documentation (this directory) ``` ## Adding a New Operation Follow these steps to add a new data operation (e.g., `deduplicate`) to the system: ### Step 1: Define the OperationDef Add a new entry to the `OPERATIONS` dictionary in `backend/visual_notebook/domain/operations.py`: ```python "deduplicate": OperationDef( key="deduplicate", label="Remove Duplicates", description="Remove duplicate rows based on specified columns", param_schema={ "type": "object", "required": ["columns"], "properties": { "columns": {"type": "array", "items": {"type": "string"}}, "keep": {"type": "string", "enum": ["first", "last", "none"], "default": "first"}, }, }, validate=_validate_deduplicate, apply=_apply_deduplicate, ), ``` ### Step 2: Implement the Validate Function ```python def _validate_deduplicate(params: dict, schema: list[dict]) -> dict: columns = params.get("columns", []) if not columns: raise ValueError("'columns' list is required") params.setdefault("keep", "first") return params ``` The validate function should check that all required parameters are present and have valid values. It receives the raw user params and the current table schema. ### Step 3: Implement the Apply Function Provide both DuckDB and Polars implementations: ```python def _apply_deduplicate(ref: DataFrameRef, params: dict) -> DataFrameRef: columns = params["columns"] keep = params.get("keep", "first") if ref.backend == "duckdb" and ref.name and ref.con: new_name = f"_dedup_{uuid.uuid4().hex[:6]}" cols = ", ".join(f'"{c}"' for c in columns) if keep == "first": sql = f''' SELECT DISTINCT ON ({cols}) * FROM "{ref.name}" ORDER BY {cols} ''' elif keep == "last": sql = f''' SELECT DISTINCT ON ({cols}) * FROM "{ref.name}" ORDER BY {cols} DESC ''' else: sql = f''' SELECT * FROM "{ref.name}" GROUP BY {cols} HAVING COUNT(*) = 1 ''' ref.con.execute(f'CREATE OR REPLACE TABLE "{new_name}" AS {sql}') return DataFrameRef(backend="duckdb", name=new_name, con=ref.con, source_path=ref.source_path) elif ref.backend == "polars" and ref.lf is not None: lf = ref.lf.unique(subset=columns, keep=keep) return DataFrameRef(backend="polars", lf=lf, source_path=ref.source_path) raise ValueError("Invalid DataFrameRef") ``` ### Step 4: Add Tests Add test cases to `backend/visual_notebook/tests/test_operations.py`: ```python def test_deduplicate_validation(): op = OPERATIONS["deduplicate"] with pytest.raises(ValueError): op.validate({}, []) result = op.validate({"columns": ["a", "b"]}, []) assert result["keep"] == "first" def test_deduplicate_in_registry(): assert "deduplicate" in OPERATIONS ``` ### Step 5: Verify 1. Run the tests: `uv run pytest backend/visual_notebook/tests/ -v` 2. Start the server and verify the operation appears in `GET /api/ops`. 3. Test the operation via the Swagger UI at `http://localhost:8000/docs`. ## Adding a New Chart Type Follow these steps to add a new chart type (e.g., `pie`) to the system: ### Step 1: Create the Computation Module Create a new file `backend/visual_notebook/viz/pie.py`: ```python """Pie chart generation.""" from __future__ import annotations from typing import Any import duckdb from visual_notebook.viz._common import PALETTE, base_chart, title, tooltip_style, palette_colors def compute_pie( con: duckdb.DuckDBPyConnection, table_name: str, category_col: str, value_col: str | None = None, n: int = 10, ) -> dict[str, Any]: """Compute pie chart data and return ECharts spec.""" if value_col: result = con.execute(f''' SELECT "{category_col}" as cat, SUM("{value_col}") as val FROM "{table_name}" WHERE "{category_col}" IS NOT NULL GROUP BY cat ORDER BY val DESC LIMIT {n} ''').fetchall() else: result = con.execute(f''' SELECT "{category_col}" as cat, COUNT(*) as val FROM "{table_name}" WHERE "{category_col}" IS NOT NULL GROUP BY cat ORDER BY val DESC LIMIT {n} ''').fetchall() data = [{"name": str(r[0]), "value": float(r[1])} for r in result] colors = palette_colors(len(data)) chart = base_chart() chart["title"] = title(f"Distribution of {category_col}") chart["tooltip"] = {**tooltip_style(), "trigger": "item", "formatter": "{b}: {c} ({d}%)"} chart["legend"] = {"orient": "vertical", "left": "left", "top": "middle"} chart["series"] = [{ "type": "pie", "radius": ["40%", "70%"], "data": data, "emphasis": {"itemStyle": {"shadowBlur": 10, "shadowOffsetX": 0, "shadowColor": "rgba(0,0,0,0.5)"}}, }] return chart ``` ### Step 2: Register in the Route Handler Add the import and handler in `backend/visual_notebook/api/routes.py`: ```python from visual_notebook.viz.pie import compute_pie # In the create_chart endpoint: elif req.chart_type == "pie": spec = compute_pie(con, table, params["category_col"], params.get("value_col"), params.get("n", 10)) ``` ### Step 3: Add Frontend Support Update `frontend/src/lib/chartTypes.ts` to include the new chart type, and add a configuration form in the `VisualizationPanel` component. ## Adding a New Frontend Component ### Step 1: Create the Component File Create a new file in `frontend/src/components/`: ```typescript // StatsPanel.tsx import React from 'react'; import { useSessionStore } from '../store/session'; export function StatsPanel() { const { columns, rowCount } = useSessionStore(); return (

Dataset Statistics

Total Rows {rowCount.toLocaleString()}
Total Columns {columns.length}
); } ``` ### Step 2: Use Tailwind Classes Follow the project's Tailwind conventions: - Use `brand-*` colors for primary actions and accents. - Use `gray-*` colors for neutral text, borders, and backgrounds. - Use `text-sm` for body text, `text-xs` for secondary labels. - Use `rounded-lg` for card corners, `shadow` for elevation. - Use `flex` and `grid` for layout, with `gap-*` for spacing. ### Step 3: Integrate into the Application Import and render the component in the appropriate parent: ```typescript // In NotebookPage.tsx import { StatsPanel } from './StatsPanel'; // Render conditionally {showStatsPanel && } ``` ### Step 4: Add a UI Toggle If the component should be togglable, add a flag to `useUIStore`: ```typescript showStatsPanel: boolean; toggleStatsPanel: () => void; ``` ## Testing ### Backend Tests The backend uses pytest with test files in `backend/visual_notebook/tests/`. **Run all tests**: ```bash uv run pytest backend/visual_notebook/tests/ -v ``` **Run a specific test file**: ```bash uv run pytest backend/visual_notebook/tests/test_operations.py -v ``` **Run with coverage** (if pytest-cov is installed): ```bash uv run pytest backend/visual_notebook/tests/ -v --cov=visual_notebook ``` **Test structure**: - `test_history.py`: Tests for commit, undo, redo, branch, and tree serialization. - `test_operations.py`: Tests for the operations registry, parameter validation, and default values. ### Frontend Testing The frontend currently does not have automated tests configured. For future implementation, consider: - **Vitest**: Fast unit testing framework compatible with Vite. - **React Testing Library**: Component testing with DOM assertions. - **Playwright**: End-to-end testing for API integration. ## Code Style and Conventions ### Python (Backend) - **Line length**: 100 characters maximum. - **Type hints**: Use type hints on all function signatures. Import `from __future__ import annotations` for modern type syntax. - **Docstrings**: Use triple-quoted docstrings for all public functions and classes. - **Naming**: `snake_case` for functions and variables, `PascalCase` for classes. - **Imports**: Standard library first, then third-party, then local. Use absolute imports within the package. - **f-strings**: Use f-strings for SQL query construction (the SQL injection risk is mitigated by server-generated table names and column names from the schema). - **Error handling**: Raise `ValueError` in validators, `HTTPException` in route handlers. ### TypeScript (Frontend) - **Strict mode**: TypeScript strict mode is enabled. All variables must have explicit or inferred types. - **Functional components**: Use function declarations with explicit return types. - **Naming**: `camelCase` for variables and functions, `PascalCase` for components and types. - **Imports**: Use relative paths for local modules, package names for third-party libraries. - **Hooks**: Custom hooks in `api/hooks.ts` follow the `use[X]` naming pattern and return `{ action, loading, error }`. - **CSS**: Use Tailwind utility classes exclusively. Avoid inline styles except for dynamic values (chart dimensions, SVG coordinates). ## Git Workflow ### Branch Naming - `feature/[description]`: New features - `fix/[description]`: Bug fixes - `docs/[description]`: Documentation changes - `refactor/[description]`: Code refactoring ### Commit Messages Follow conventional commit format: ``` type(scope): description [optional body] ``` Types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore` Examples: - `feat(operations): add deduplicate operation` - `fix(history): correct undo pointer after branch` - `docs(api): update endpoint descriptions` ### Pull Request Process 1. Create a feature branch from `main`. 2. Implement changes with tests. 3. Run the full test suite: `uv run pytest`. 4. Ensure the frontend builds: `cd frontend && npm run build`. 5. Open a pull request with a description of the changes and any manual testing performed. 6. Request review from a maintainer. 7. Address review feedback and re-run tests. 8. Merge when approved and all checks pass.