Spaces:

caid-technologies
/

parti

Runtime error

App Files Files Community

parti / FORMA_ALIGNMENT.md

hudien

Clarify review navigation and document Forma compatibility gaps

ca30643 verified about 2 months ago

preview code

Raw History Blame Contribute Delete

6.37 kB

# Parti / Forma alignment assessment

Assessed 2026-09-09. **Partial integration, not demonstrated end-to-end compatibility.** This assessment is based on source and model metadata, not a new training run, GPU inference benchmark, Forma deployment test, or physical build.

## Findings

| Layer | Assessment | Evidence and gap |
| --- | --- | --- |
| Provider integration | Partial | Forma explicitly selects `caid-technologies/parti-base` through its RunPod-specific branch. Naming the provider is not an output-contract or quality guarantee. |
| Dataset | Not demonstrated | Parti prepares grouped train/validation/test ChatML tasks A–F around its own canonical project representation. No Forma HardwareIR-target training task or public dataset-to-released-weights provenance was established in this review. The Hub dataset listing for the organization was empty; that does not exclude private or externally stored data. |
| Training recipe | Suitable family; Forma-specific alignment unproven | The inspected recipe uses Qwen2.5-3B-Instruct, QLoRA rank/alpha 32, response-only loss, 8,192-token training windows and 6,144-token validation windows. Overlength samples are dropped. These settings alone do not show the released model was trained on Forma's contract. |
| Released model | Text-generation integration is plausible | The model card describes merged Qwen2.5-3B-Instruct weights. The config identifies Qwen2ForCausalLM and a 32,768-position context. Parti pins revision `95304fd1d24142313b3acfdb4151a20bd5e411f0`; lineage from the current recipe to those weights remains unverified. |
| Output contract | Not drop-in compatible | Parti Mode B returns `project`, `requirements`, `components`, `relationships`, `fabrication`, `instructions`, `sourcing`, `validation`. Forma HardwareIR v0.2 uses part definitions and instances, BOM, nets, buses, pin mappings, assembly, mechanical/CAD and power fields. Conversion needs semantic validation, not just field renaming. |
| Prompt fidelity | Material adapter gap | Forma's dedicated Parti branch requests a concise seed with a 500-token budget, then builds a fixed ESP32/DHT22/BMP280/OLED/battery design and environmental-monitor requirements. A title change does not preserve arbitrary requested functions, power constraints, or parts. |
| Real-life proof | Not established | Rendering, source links, schema checks and model-generated validation are not electrical simulation, build evidence, safety approval or a manufacturing release. |

## Important adapter details

Forma's `_request_parti_base_seed` requests a flat project name, summary and role hints, not Parti's trained full-project JSON. Its dedicated path uses 500 tokens, whereas the model card recommends at least 6,000 for full JSON and this Space defaults to 6,144. A smaller seed contract could be useful, but it needs its own training/evaluation evidence. The inspected seed request accepts image parameters but does not include them in the model messages; this does not establish image understanding.

The Space's generation time cap can still yield incomplete JSON. Incomplete or faulty output is surfaced for correction, never silently replaced with the example. Imported verification claims and test results are not accepted as independent evidence.

## What would establish alignment

1. Choose one supported integration: validated canonical Parti-to-HardwareIR conversion, or an explicitly trained and evaluated Forma seed contract. Preserve the original brief and flag missing facts; do not replace requests with fixed templates.
2. Map part identities/instances, quantities, net endpoints, pin IDs, buses, assembly dependencies and power rails. Validate against Forma's actual models, including referential integrity and deterministic BOM aggregation. Keep unknown CAD, firmware and electrical facts unresolved.
3. Publish a versioned dataset manifest, schema version, project-group split IDs, filtering counts, hashes and license/provenance. Link the exact training run, recipe and evaluation artifacts to the released model revision.
4. Evaluate held-out projects end to end: valid schema, requested-feature retention, exact constraint retention, no substituted parts, pin/reference integrity, budget compliance, unsupported-input failure, and malformed/truncated-output handling. Report denominators and failures, not just aggregate model confidence.
5. Separately validate physical builds with exact board/firmware revisions, qualified review, measured evidence and acceptance criteria. Recorded PASS values in Parti remain self-reported.

## Interface changes in this update

- Clear describe/import → review → export journey and a no-GPU example entry point.
- Plain editable JSON import and a one-click path from the current draft back to correction.
- Successful draft creation/import returns to Overview; review actions scroll to their output.
- Selected tests show their method and pass condition beside the observation controls.
- Role guidance is expandable; controls have larger touch targets and compact mobile spacing.
- Exports explicitly identify the Forma compatibility boundary. Existing validation, provenance, raw output and draft-only warnings remain intact.

No training data, weights, Forma source, or physical verification results were changed by this UI update.

## Sources

- [Forma dedicated Parti adapter and orchestration](https://github.com/caid-technologies/Forma-OSS/blob/e600c4886201d52261442804edd18f5ef7adf91a/forma_core/agents/orchestrator.py), especially `_request_parti_base_seed` and `_generate_parti_base_project`.
- [Forma project and HardwareIR models](https://github.com/caid-technologies/Forma-OSS/blob/e600c4886201d52261442804edd18f5ef7adf91a/forma_core/workspaces/projects/models.py).
- [Parti training recipe](https://github.com/caid-technologies/Parti-Base/blob/fb5c257/train_local.py) and [training preparation](https://github.com/caid-technologies/Parti-Base/blob/fb5c257/prepare_training.py).
- [Released model card](https://huggingface.co/caid-technologies/parti-base/blob/main/README.md) and [configuration](https://huggingface.co/caid-technologies/parti-base/blob/main/config.json) inspected at assessment time; card/config metadata alone are not a reproducible training record.
- [Current Space implementation](https://huggingface.co/spaces/caid-technologies/parti/tree/main).

	# Parti / Forma alignment assessment

	Assessed 2026-09-09. Partial integration, not demonstrated end-to-end compatibility. This assessment is based on source and model metadata, not a new training run, GPU inference benchmark, Forma deployment test, or physical build.

	## Findings

	\| Layer \| Assessment \| Evidence and gap \|
	\| --- \| --- \| --- \|
	\| Provider integration \| Partial \| Forma explicitly selects `caid-technologies/parti-base` through its RunPod-specific branch. Naming the provider is not an output-contract or quality guarantee. \|
	\| Dataset \| Not demonstrated \| Parti prepares grouped train/validation/test ChatML tasks A–F around its own canonical project representation. No Forma HardwareIR-target training task or public dataset-to-released-weights provenance was established in this review. The Hub dataset listing for the organization was empty; that does not exclude private or externally stored data. \|
	\| Training recipe \| Suitable family; Forma-specific alignment unproven \| The inspected recipe uses Qwen2.5-3B-Instruct, QLoRA rank/alpha 32, response-only loss, 8,192-token training windows and 6,144-token validation windows. Overlength samples are dropped. These settings alone do not show the released model was trained on Forma's contract. \|
	\| Released model \| Text-generation integration is plausible \| The model card describes merged Qwen2.5-3B-Instruct weights. The config identifies Qwen2ForCausalLM and a 32,768-position context. Parti pins revision `95304fd1d24142313b3acfdb4151a20bd5e411f0`; lineage from the current recipe to those weights remains unverified. \|
	\| Output contract \| Not drop-in compatible \| Parti Mode B returns `project`, `requirements`, `components`, `relationships`, `fabrication`, `instructions`, `sourcing`, `validation`. Forma HardwareIR v0.2 uses part definitions and instances, BOM, nets, buses, pin mappings, assembly, mechanical/CAD and power fields. Conversion needs semantic validation, not just field renaming. \|
	\| Prompt fidelity \| Material adapter gap \| Forma's dedicated Parti branch requests a concise seed with a 500-token budget, then builds a fixed ESP32/DHT22/BMP280/OLED/battery design and environmental-monitor requirements. A title change does not preserve arbitrary requested functions, power constraints, or parts. \|
	\| Real-life proof \| Not established \| Rendering, source links, schema checks and model-generated validation are not electrical simulation, build evidence, safety approval or a manufacturing release. \|

	## Important adapter details

	Forma's `_request_parti_base_seed` requests a flat project name, summary and role hints, not Parti's trained full-project JSON. Its dedicated path uses 500 tokens, whereas the model card recommends at least 6,000 for full JSON and this Space defaults to 6,144. A smaller seed contract could be useful, but it needs its own training/evaluation evidence. The inspected seed request accepts image parameters but does not include them in the model messages; this does not establish image understanding.

	The Space's generation time cap can still yield incomplete JSON. Incomplete or faulty output is surfaced for correction, never silently replaced with the example. Imported verification claims and test results are not accepted as independent evidence.

	## What would establish alignment

	1. Choose one supported integration: validated canonical Parti-to-HardwareIR conversion, or an explicitly trained and evaluated Forma seed contract. Preserve the original brief and flag missing facts; do not replace requests with fixed templates.
	2. Map part identities/instances, quantities, net endpoints, pin IDs, buses, assembly dependencies and power rails. Validate against Forma's actual models, including referential integrity and deterministic BOM aggregation. Keep unknown CAD, firmware and electrical facts unresolved.
	3. Publish a versioned dataset manifest, schema version, project-group split IDs, filtering counts, hashes and license/provenance. Link the exact training run, recipe and evaluation artifacts to the released model revision.
	4. Evaluate held-out projects end to end: valid schema, requested-feature retention, exact constraint retention, no substituted parts, pin/reference integrity, budget compliance, unsupported-input failure, and malformed/truncated-output handling. Report denominators and failures, not just aggregate model confidence.
	5. Separately validate physical builds with exact board/firmware revisions, qualified review, measured evidence and acceptance criteria. Recorded PASS values in Parti remain self-reported.

	## Interface changes in this update

	- Clear describe/import → review → export journey and a no-GPU example entry point.
	- Plain editable JSON import and a one-click path from the current draft back to correction.
	- Successful draft creation/import returns to Overview; review actions scroll to their output.
	- Selected tests show their method and pass condition beside the observation controls.
	- Role guidance is expandable; controls have larger touch targets and compact mobile spacing.
	- Exports explicitly identify the Forma compatibility boundary. Existing validation, provenance, raw output and draft-only warnings remain intact.

	No training data, weights, Forma source, or physical verification results were changed by this UI update.

	## Sources

	- [Forma dedicated Parti adapter and orchestration](https://github.com/caid-technologies/Forma-OSS/blob/e600c4886201d52261442804edd18f5ef7adf91a/forma_core/agents/orchestrator.py), especially `_request_parti_base_seed` and `_generate_parti_base_project`.
	- [Forma project and HardwareIR models](https://github.com/caid-technologies/Forma-OSS/blob/e600c4886201d52261442804edd18f5ef7adf91a/forma_core/workspaces/projects/models.py).
	- [Parti training recipe](https://github.com/caid-technologies/Parti-Base/blob/fb5c257/train_local.py) and [training preparation](https://github.com/caid-technologies/Parti-Base/blob/fb5c257/prepare_training.py).
	- [Released model card](https://huggingface.co/caid-technologies/parti-base/blob/main/README.md) and [configuration](https://huggingface.co/caid-technologies/parti-base/blob/main/config.json) inspected at assessment time; card/config metadata alone are not a reproducible training record.
	- [Current Space implementation](https://huggingface.co/spaces/caid-technologies/parti/tree/main).