File size: 4,221 Bytes
2d5c26a 294f8ea 2d5c26a 294f8ea 2d5c26a 75c3d90 294f8ea 2d5c26a 294f8ea 2d5c26a 294f8ea 2d5c26a 294f8ea 2d5c26a 294f8ea | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | # OpenSysOne
An experimental natural-language decision scorer built by tuning Qwen3-4B.
Give it context, a question and possible answers; it returns a probability for
each answer. The project includes a browser playground and a Jev-compatible API.
**Credit:** OpenSysOne is inspired by [Jev](https://typesafe.ai/), TypeSafe.ai's
System One model for structured decisions with probabilities. Credit goes to the
TypeSafe team for motivating this independent experimental implementation.
**Training and evaluation are complete.** Start with the
[accuracy and speed report](results/report.md), [browser playground guide](docs/usage/playground.md),
or [API guide](docs/usage/jev-api.md). The calibrated model and publication files
are available in [andyshu/opensysone on Hugging Face](https://huggingface.co/andyshu/opensysone).
## Results
| Reserved evaluation | Examples | Trained 4B | Pretrained per-option verifier |
| --- | ---: | ---: | ---: |
| Original four-family test | 2,042 | **92.90%** | 84.48% |
| Social IQA family holdout | 768 | **72.92%** | 70.31% |
Current inference has an accuracy–speed tradeoff. On one idle Spark in FP32,
one four-choice question with a 768-token state takes **3.710 s** for the trained
scorer, **3.177 s** for the pretrained per-option verifier and **0.818 s** for a
pretrained joint answer-label prompt. On a separate matched 320-example sample,
the trained and joint-label methods score **89.06%** and **86.25%**. These warm
measurements exclude model loading and HTTP. See the report for all workloads,
calibration scores, confidence intervals and limits.
The selected weights are Spark B step 1,500, retained identically at expanded
branch step 0. A separate 510-example calibration split fits temperature 1.745822.
The model uses custom rank-8 adapters and a scalar head; it requires the pinned
pretrained base and the project's reconstruction code. It is not a generic
Transformers or PEFT checkpoint. Social IQA was excluded from our fine-tuning;
exposure during base-model pretraining is unknown.
## Repository layout
| Location | Purpose |
| --- | --- |
| [docs/](docs/README.md) | Usage guides, research notes and operational records |
| [results/](results/README.md) | Final report and charts, plus preserved historical evidence |
| [examples/](examples/) | Sample API request |
| [web/](web/) | Browser playground assets |
| [scripts/](scripts/) | Training, evaluation, verification and archival tools |
| [tests/](tests/) | CPU and harness checks |
| `experiment.py`, `training_model.py`, `selection.py` | Training, reconstruction and checkpoint selection |
| `jev_harness.py`, `playground.py` | API and interactive model comparison |
| `smoke_train.py`, `decision_model.py`, `smoke_data.py` | Original 0.5B smoke and shared-prefix experiments |
## Try it on the existing GX10 installation
The GUI uses port **7466**. On the Mac, run `ssh -N -L 7466:127.0.0.1:7466 gx10`
and open **http://localhost:7466**. Select **Qwen3 4B · Selected** for the calibrated
release. The local API is on loopback port **18081**:
```bash
curl --fail-with-body http://127.0.0.1:18081/v1/systemone \
-H 'Content-Type: application/json' --data-binary @examples/jev_request.json
```
For another machine, read the [reconstruction requirements](https://huggingface.co/andyshu/opensysone/blob/main/model/README.md)
and [reproduction guide](https://huggingface.co/andyshu/opensysone/blob/main/docs/reproduce.md). The checkpoint records
its pinned local base-model path; downloading the adapter alone is insufficient.
Hosted Jev requests require `TYPESAFE_API_KEY`; no hosted Jev benchmark is claimed.
## Development and continuation
The existing isolated environment on GX10 can run the CPU suite:
```bash
~/ai/envs/opensysone/bin/python -m unittest discover -s tests -v
```
Read [HANDOVER.md](HANDOVER.md), [PLAN.md](PLAN.md) and [RESULTS.md](RESULTS.md)
before continuing experiments. Model weights, optimizer state and runtime logs
live outside this source tree under `~/ai/models/opensysone` and
`~/ai/opensysone/runs`. Dataset/source pins and checkpoint provenance accompany
the archived results. No additional training is active or scheduled by this
publication cleanup.
|