Spaces:

---

osolmaz

/

microquack

Like 1

Running

Summary

The new reply dataset taught style better than meaning because almost every reply was unique.
This change adds a fair comparison and tests repeated reply supervision without adding a runtime answer table.
The K=16 and K=64 candidates both fail the release gate, so the web app stays on the previous production model.

What Changed

The experiment now compares models with the same decoding settings and keeps seen prompts separate from unseen prompts.
It also records raw paired results and uncertainty before it makes a recommendation.

• Added symmetric greedy and sampled evaluation for character and word models.
• Added deterministic K=16 and K=64 training-data builders that preserve user messages and reply-emotion pairs.
• Bound each similarity audit to the exact source data hash and row count.
• Added paired case bootstrap comparisons with a registered +0.10 minimum worthwhile effect.
• Kept the 73-emotion model as a named diagnostic asset; the app loads only the previous production model.
• Saved 133 data, checkpoint, prediction, manifest, and report files under hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/repetition-supervision-2026-09-01-v1.

Results

Neither repeated-reply arm is ready for release.
The small classic-case gains are uncertain, and both arms make safety results worse.

• A1 unique replies: 12/75 classic and 13/30 safety with greedy decoding.
• K=16: 13/75 classic and 7/30 safety.
• K=64: 17/75 classic and 5/30 safety.
• With equal sampling, A1, K=16, and K=64 pass 8/75, 10/75, and 8/75 classic cases.
• True-safety results fall from 9/20 for A1 to 7/20 for K=16 and 3/20 for K=64.
• No candidate was exported for production or approved for release.

Testing

The data builders, comparison code, Rust runtime, WASM build, and web app all pass their checks.
The models were trained locally on the same 1.9M-parameter architecture with seed 7.

• pnpm run check — 64 Python tests, 15 web tests, coverage, lint, types, formatting, and DRY checks passed.
• cargo test --workspace — Rust unit, conformance, and model parity tests passed.
• pnpm run build — the WASM and static web production builds passed.
• Pi Reviewer — no P0 or P1 findings in the last run.
• Bucket round trips matched the local report, K=16 checkpoint, and K=64 manifest byte for byte.

Risks

This is a diagnostic branch and must not be merged as a model release.
The candidates fail the safety veto, and the live Space remains unchanged on main revision 89686a6d447069f61a362811a5a6717cae91a758.

Follow-ups

One P2 review finding remains in the older contract-corpus duplicate-repair resume path.
It does not affect these completed training runs, but a later maintenance change should resume a saved repair plan instead of counting it as completed.

Final experiment report:

• K=16 and K=64 were trained and evaluated with the corrected symmetric protocol.
• Neither arm clears the +0.10 classic-case threshold.
• Safety falls from 13/30 for A1 to 7/30 for K=16 and 5/30 for K=64.
• Pi Reviewer found no P0 or P1 issues in the last run.
• pnpm run check, cargo test --workspace, and pnpm run build pass.
• The canonical artifact set has 133 files and 58,513,282 bytes under hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/repetition-supervision-2026-09-01-v1.
• The production Space is still running main revision 89686a6d447069f61a362811a5a6717cae91a758.

Observed: repeated supervision did not produce a reliable quality gain and made safety worse.

Recommended: reject both new checkpoints and keep them only as diagnostic evidence.

Approved: no model change and no merge.

The clean repetition experiment is complete.

Result

The repeated-reply arm did not improve the primary held-out conversation test.

The median paired effect was 0.000. Its 95% hierarchical paired-bootstrap interval was -0.040 to +0.027. The registered minimum useful improvement was +0.100.

The safety veto also failed. On the 20 true-safety cases, the repeated arm changed from 3/20 to 0/20 at seed 11. No candidate seed passed all 20 cases.

Observed: moderate reply reuse did not improve held-out replies. Both arms also had low absolute scores.

Recommended: stop this repetition-bank direction. Do not run more K sweeps or scale this transform to the full data mix.

Approved: no model is approved, deployed, or merged.

Evidence

• Artifacts: hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/clean-repetition-2026-09-01-v1
• Final branch revision: 8b476d31e41c7ef089a4dd4ffa89b8be827815f1
• pnpm run check: passed, including 68 Python tests and 15 web tests
• cargo test --workspace: passed
• pnpm run build: passed
• Final review: no P0 or P1 findings; one P2 resume issue recorded and not pursued
• Production: unchanged and running at 89686a6d447069f61a362811a5a6717cae91a758

Correction to the previous experiment conclusion: the epoch-1 comparison did not provide a valid test of reply repetition. A mismatched external validation set selected epoch 1 for all six runs, even though the repeated arm continued learning through epoch 6.

I evaluated the six existing epoch-6 last.pt checkpoints. No new training was run.

Epoch-6 result

The median greedy effect was -0.027. Its 95% hierarchical paired-bootstrap interval was -0.080 to +0.053. The registered minimum useful improvement was +0.100.

The sampled secondary effect was +0.013, with a 95% interval of -0.040 to +0.067. This is also a tie.

True-safety results regressed at each greedy training seed: 10/20 to 9/20, 10/20 to 9/20, and 9/20 to 7/20.

Observed: epoch-6 checkpoints are functional, but repetition did not improve held-out quality. Exact bank reproduction reached 0.600–0.733, below the registered 0.800 threshold.

Recommended: stop this follow-up. The repetition hypothesis remains inconclusive rather than rejected. A later corrected retrain must use arm-matched validation.

Approved: no model is approved, deployed, or merged. Production remains unchanged.

Evidence

• Artifacts: hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/clean-repetition-2026-09-01-v1/last-checkpoint-followup
• Branch revision: 640948fcef149ecac1b48ea828e32083fd10c874
• pnpm run check: passed, including 71 Python tests and 15 web tests
• cargo test --workspace: passed
• pnpm run build: passed
• Final review: no P0 or P1 findings; one P2 Unicode-tokenization issue recorded and not pursued
• Production revision: 89686a6d447069f61a362811a5a6717cae91a758

Final paired retrain result

This valid matched-validation retrain supersedes the earlier inconclusive checkpoint follow-up.

• Primary greedy classic-conversation passes by seed: unique 13/75, 11/75, 13/75; repeated 12/75, 9/75, 8/75.
• Median paired effect: -0.027; 95% hierarchical paired-bootstrap interval: [-0.107, +0.027].
• Registered minimum worthwhile improvement: +0.100.
• Repeated-arm functionality passed for all seeds: 75/75, 75/75, 73/75.
• True-safety results did not regress by paired seed: 7→10, 9→9, 7→7 out of 20.
• Exact bank reproduction was high, but most exact matches did not fit the held-out case. This confirms that reply selection remains the failure.

The registered kill rule applies: stop the repetition-bank direction permanently. No candidate is approved or deployed.

All checks passed, and Pi Reviewer found no P0 or P1 issues. Full reports, predictions, checkpoints, and manifests are in hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/clean-repetition-2026-09-01-v1.

The experiment and PR refs point to 869e405aea9047b7cbc7c3c46fc2ec2098d50876. Production main remains unchanged at 89686a6d447069f61a362811a5a6717cae91a758.

Scoring decomposition result

I ran the zero-cost diagnostic on the six frozen greedy classic-conversation reports. No new model output, teacher data, or training was created.

Unique-arm robust passes by training seed:

The median strict rate is 0.173. Broad emotion families raise it to 0.240. Broad families plus the frozen blind topical labels raise it to 0.413.

For the unique arm, literal cue lists pass a median 28/75 replies, while the blind topical audit passes 50/75. A median 24/75 replies pass the blind audit but fail the cue list; only 2/75 show the reverse disagreement. Exact emotion labels also undercount broad-family matches: 26/75 versus 43/75.

The registered result is mixed or inconclusive. The old evaluation is too strict, but the model still fails most cases after the diagnostic corrections. No training or production change is approved.

A second independent blind audit is registered but not started. It contains 80 distinct unique-arm disagreement replies. Full evidence is in hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/clean-repetition-2026-09-01-v1/scoring-decomposition.

All checks passed. Pi Reviewer found no P0 or P1 issues. The experiment and PR refs point to e00a7546988e0e375f43c174d3a37943f24f6957; production main remains 89686a6d447069f61a362811a5a6717cae91a758.

Independent audit and semantic rebaseline

A second blind reviewer checked all 80 registered disagreement replies without access to the first labels or literal cue results.

• Inter-auditor agreement: 65/80 (81.25%).
• Registered outcome: confirmed evaluation problem.
• Corrected unique-arm passes: 29/75, 29/75, 26/75.
• Median corrected rate: 29/75 (38.67%).
• Required rate: 45/75 (60%).
• Absolute shortfall: 16/75 (21.33 percentage points).

The classic-conversation suite now treats literal cue words as diagnostics only. Selection requires broad emotion-family matching, blind semantic reply review by at least two independent auditors, at least 80% auditor agreement, and at least a 60% robust pass rate. Generated reports cannot claim a pass without that external evidence.

The current model remains ineligible. Seed 11 and seed 23 also fail the 0.95 functional-reply hygiene gate. No training or production change was made.

All checks passed, including 83 Python tests and 15 web tests. Pi Reviewer found no P0 or P1 issues. Full evidence is in hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/clean-repetition-2026-09-01-v1/scoring-decomposition.

The experiment and PR refs point to 68f3436bca2a412f772843eb1ad001178553d174. Production main remains unchanged at 89686a6d447069f61a362811a5a6717cae91a758.

Grounded reply pilot

Observed

• Added deterministic, resumable tooling for fixed-user grounded replies and reply-only emotion relabeling at df3a0d8a092d07e08e44a14940358b1ca706e5d5.
• Generated 1,000 replies across all 125 intent-subtype conditions. The run used 137 requests and cost $0.024791.
• Blind reply audit: 121/125 (96.8%) direct and sensible; 15/15 registered safety rows passed.
• The original joint emotion labels scored 116/125 (92.8%), below the 95% gate.
• A reply-only relabel pass kept every user message and reply unchanged. It used 20 requests and cost $0.006634.
• Relabeled emotion audit: 122/125 (97.6%). This is +6/125 (+4.8 points), with 8 improved and 2 regressed pairs.
• Paired uncertainty is still material: the deterministic bootstrap 95% interval is 0.0 to 9.6 points, and exact two-sided McNemar p is 0.109375.
• Total observed cost was $0.031425. No reservations remain active.
• pnpm run check passed with 98 Python tests and 15 web tests. cargo fmt --check and cargo test --workspace also passed.
• Bucket round-trip hashes match for the manifest, replies, relabeled data, and semantic result.

All evidence is under hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/clean-repetition-2026-09-01-v1/grounded-pilot-2026-09-02-v1.

Recommended

The reply-only labeling method is promising, but this audit is not decisive. Repair the five known reply-quality failures and obtain a second independent blind emotion audit before any full-corpus generation or training decision.

Approved

No model is selected. Do not train, merge, deploy, or change production from this result. Production main remains 89686a6d447069f61a362811a5a6717cae91a758.

Follow-up integrity fix: 29cbc4e2982d459204956f1161d3968da3a3c80b makes completed campaign reruns a no-op, so they cannot overwrite semantic audit evidence. Both completed generators were rerun without HF_TOKEN; report hashes stayed unchanged. The updated artifact manifest round-tripped from the Bucket at 9fd50b6a585d139837d9baf75e463f608276a40f164ac8ae00c1429fb8a79336.

Grounded reply experiment result

The full experiment is complete. Production remains unchanged.

Observed

• Built a corrected 20,000-row grounded corpus from the fixed user messages.
• Reply audit: 125/125 direct and sensible, 125/125 child-safe, and 15/15 registered safety rows.
• Reply-only emotion audit: 124/125 compatible.
• Trained the unchanged 1,897,760-parameter word model with seeds 7, 11, and 23.
• Old model classic scores: 13/75, 11/75, and 13/75.
• Grounded model classic scores: 24/75, 26/75, and 25/75.
• Median absolute gain: +0.16, with a 95% interval of +0.08 to +0.267. This clears the registered +0.10 minimum.
• True-safety scores remain 9/20, 9/20, and 11/20. The release gate is 20/20 for every seed.
• A bounded two-times safety loss-weight test improved seed 7 from 9/20 to 12/20, but missed its registered +4-case threshold and reduced classic quality by one case. That direction is rejected.
• Paid teacher generation and labeling cost $0.646405548. Active reservations: $0.

Decision

Keep the grounded supervision direction. Do not release or deploy these checkpoints. Redesign safety target learning before any production model change.

All datasets, checkpoints, predictions, manifests, and reports are in:

hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/grounded-corpus-20000-2026-09-02-v1

Main report:

training-seed7-v1/report.json

Validation

• pnpm run check: passed, 101 Python tests and 15 web tests.
• cargo test --workspace: passed, 11 Rust tests.
• cargo fmt --all -- --check: passed.
• pnpm run build: passed after restoring the official Rust WASM target and matching wasm-bindgen-cli 0.2.127.
• Pi Reviewer: no P0 or P1 findings. One P2 about pre-existing accented-text Rust tokenization was recorded and not chased under the task rule.
• Draft PR has no conflicts.
• Static Space production is RUNNING at unchanged main revision 89686a6d447069f61a362811a5a6717cae91a758.

Safety augmentation result

This bounded follow-up is complete. Production remains unchanged.

Scorer repair

The old safety scorer rejected correct refusals such as “Don’t touch it” because it matched the forbidden phrase “touch it.” The repaired scorer is negation-aware and also accepts caring and calm in the two safety cases where those replies are appropriate.

Re-scoring the preserved grounded predictions changed true-safety results from 9/20, 9/20, and 11/20 to 13/20, 12/20, and 13/20. This used the stored predictions and did not run inference again.

Bounded data test

• Selected all 600 existing safety rows from the 5,000-row intent pilot.
• Evaluation overlap audit: 0 exact matches and 0 near matches.
• Relabeled emotions from reply text only: 600/600 complete, 503 labels changed.
• Paid label cost: $0.00444636. Active reservations: $0.
• Appended the rows to train only. Validation and the 4,000-token tokenizer stayed fixed.
• Trained seed 7 with the unchanged 1,897,760-parameter model.

Decision

The experiment failed its registered continuation gate:

• Repaired true safety: 13/20 control and 13/20 candidate, for a gain of 0 cases. The minimum was +2.
• Classic conversation: 24/75 control and 23/75 candidate.
• Classic functional reply rate: 0.92 control and 0.893 candidate.
• Cleaner and road prompts still produced affirmative replies, while airlock and friend-location remained off-topic.

The safety-phrasing-diversity hypothesis is falsified. Seeds 11 and 23 were not run. No checkpoint is recommended, approved, or deployed.

Artifacts:

hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/safety-augmentation-2026-09-02-v1

Branch commit: 3c15210

Validation:

• pnpm run check: passed, including 109 Python tests and 15 web tests.
• cargo fmt --all --check: passed.
• cargo test --workspace: passed, 11 Rust tests.
• cargo clippy --workspace --all-targets -- -D warnings: blocked by the pre-existing Rust 1.98 chunks_exact_to_as_chunks lint in crates/microquack-llm/src/lib.rs:413; this experiment did not change that Rust code.

Forced-emotion diagnostic completed.

• Forced cautious: persistent refusals were 2/4, 1/4, and 1/4 for grounded seeds 7, 11, and 23.
• Forced caring: persistent refusals were also 2/4, 1/4, and 1/4.
• Registered gate: at least 3/4 in at least 2/3 checkpoints. Qualifying checkpoints: 0.
• Decision: stop. The reply-first arm was not trained. Production remains unchanged.
• Probe commit: 2efe9ddf6e2cf7bf7b6224a649a789b8dcdfc88c.
• Checks: pnpm run check passed; 4 PyTorch probe tests passed.
• Evidence: hf://buckets/osolmaz/microquack-data/training/droid-mind/experiments/reply-first-probe-2026-09-02-v1.

The result does not support the emotion-cascade hypothesis. The next clean test, if approved, is a small contrastive teacher batch for unsafe permission questions and direct refusals.