| # OpenSysOne |
|
|
| An experimental natural-language decision scorer built by tuning Qwen3-4B. |
| Give it context, a question and possible answers; it returns a probability for |
| each answer. The project includes a browser playground and a Jev-compatible API. |
|
|
| **Credit:** OpenSysOne is inspired by [Jev](https://typesafe.ai/), TypeSafe.ai's |
| System One model for structured decisions with probabilities. Credit goes to the |
| TypeSafe team for motivating this independent experimental implementation. |
|
|
| **Training and evaluation are complete.** Start with the |
| [accuracy and speed report](results/report.md), [browser playground guide](docs/usage/playground.md), |
| or [API guide](docs/usage/jev-api.md). The calibrated model and publication files |
| are available in [andyshu/opensysone on Hugging Face](https://huggingface.co/andyshu/opensysone). |
|
|
| ## Results |
|
|
| | Reserved evaluation | Examples | Trained 4B | Pretrained per-option verifier | |
| | --- | ---: | ---: | ---: | |
| | Original four-family test | 2,042 | **92.90%** | 84.48% | |
| | Social IQA family holdout | 768 | **72.92%** | 70.31% | |
|
|
| Current inference has an accuracy–speed tradeoff. On one idle Spark in FP32, |
| one four-choice question with a 768-token state takes **3.710 s** for the trained |
| scorer, **3.177 s** for the pretrained per-option verifier and **0.818 s** for a |
| pretrained joint answer-label prompt. On a separate matched 320-example sample, |
| the trained and joint-label methods score **89.06%** and **86.25%**. These warm |
| measurements exclude model loading and HTTP. See the report for all workloads, |
| calibration scores, confidence intervals and limits. |
|
|
| The selected weights are Spark B step 1,500, retained identically at expanded |
| branch step 0. A separate 510-example calibration split fits temperature 1.745822. |
| The model uses custom rank-8 adapters and a scalar head; it requires the pinned |
| pretrained base and the project's reconstruction code. It is not a generic |
| Transformers or PEFT checkpoint. Social IQA was excluded from our fine-tuning; |
| exposure during base-model pretraining is unknown. |
|
|
| ## Repository layout |
|
|
| | Location | Purpose | |
| | --- | --- | |
| | [docs/](docs/README.md) | Usage guides, research notes and operational records | |
| | [results/](results/README.md) | Final report and charts, plus preserved historical evidence | |
| | [examples/](examples/) | Sample API request | |
| | [web/](web/) | Browser playground assets | |
| | [scripts/](scripts/) | Training, evaluation, verification and archival tools | |
| | [tests/](tests/) | CPU and harness checks | |
| | `experiment.py`, `training_model.py`, `selection.py` | Training, reconstruction and checkpoint selection | |
| | `jev_harness.py`, `playground.py` | API and interactive model comparison | |
| | `smoke_train.py`, `decision_model.py`, `smoke_data.py` | Original 0.5B smoke and shared-prefix experiments | |
|
|
| ## Try it on the existing GX10 installation |
|
|
| The GUI uses port **7466**. On the Mac, run `ssh -N -L 7466:127.0.0.1:7466 gx10` |
| and open **http://localhost:7466**. Select **Qwen3 4B · Selected** for the calibrated |
| release. The local API is on loopback port **18081**: |
|
|
| ```bash |
| curl --fail-with-body http://127.0.0.1:18081/v1/systemone \ |
| -H 'Content-Type: application/json' --data-binary @examples/jev_request.json |
| ``` |
|
|
| For another machine, read the [reconstruction requirements](https://huggingface.co/andyshu/opensysone/blob/main/model/README.md) |
| and [reproduction guide](https://huggingface.co/andyshu/opensysone/blob/main/docs/reproduce.md). The checkpoint records |
| its pinned local base-model path; downloading the adapter alone is insufficient. |
| Hosted Jev requests require `TYPESAFE_API_KEY`; no hosted Jev benchmark is claimed. |
|
|
| ## Development and continuation |
|
|
| The existing isolated environment on GX10 can run the CPU suite: |
|
|
| ```bash |
| ~/ai/envs/opensysone/bin/python -m unittest discover -s tests -v |
| ``` |
|
|
| Read [HANDOVER.md](HANDOVER.md), [PLAN.md](PLAN.md) and [RESULTS.md](RESULTS.md) |
| before continuing experiments. Model weights, optimizer state and runtime logs |
| live outside this source tree under `~/ai/models/opensysone` and |
| `~/ai/opensysone/runs`. Dataset/source pins and checkpoint provenance accompany |
| the archived results. No additional training is active or scheduled by this |
| publication cleanup. |
|
|
