teckedd/gha-twi-understanding-r4

Ghana Health AI: Twi R4

Availability: This Hugging Face release is a model card only. It contains no downloadable adapter, base weights, training examples or inference endpoint.

In the matched 11 November comparison, R4 picked the published English meaning in both answer orders for 890 of 1,092 Twi sentences (base: 636). In independent conversation histories it recalled both the changed and unchanged fact in 23/24 Twi scenarios and 18/24 English scenarios. The former measures agreement with fallible references; the latter uses four authored templates and does not establish general conversational usefulness. See the side-by-side results and protocol.

R4 is a private research LoRA adapter continued from the final R3 response adapter. Its intended study is Twi conversational instruction-following and meaning retention, with English retention. It is not a standalone base model, a speech model, a current-information service, or a clinically validated system. The live public application remains on R3 during this comparison.

This is a companion card. The original, hash-sealed PEFT adapter README is preserved unchanged; its autogenerated placeholders are not the research report. No public model or corpus upload was made in this pass. Distribution remains subject to upstream model and source-data rights; this card grants no additional data redistribution rights.

Model Identity

Component	Identity
Base	McGill-NLP/AfriqueQwen3.5-9B-50Langs-Instruct-v1
Base revision	693f12e444a30971ca53889128d7bade9491ecb3
Base weight SHA256	6c03034e336dc916d46b353493bc30b3244f83b80d487be83d6ad0999af15af6
Initial R3 adapter SHA256	5cc0537b21674231b640ae9109a08ffa6fab426c80ac1969705dfac9c9e6b874
R4 final adapter SHA256	1c18338e732f7461901403f5a50c876d64712c21285cf24b2f78a86530d87aca
R4 first-pass adapter SHA256	f7ca7d04f0fe089100394c8f3b41f2b703ce3b39214bd4535d324b4de0cf3631
Training run	understanding_20261109T103659Z

Full local readback hashes match the saved remote adapter bytes. Base weights, tokenizer vocabulary and chat template were not changed. The runtime verifies the selected adapter tensors rather than relying on a UI model name.

Data and Exposure

Dataset version: 20261109-r4-v3. Pack SHA256: b71682b47512506f9621faba7370f9f2d941437acc44e4dbe6c2716c0bf50cc7.

Training component	Distinct views	Presentations	Effective target tokens
Responses	355	710	97,122
English retention	152	152	32,370
Meaning	535	535	16,183
Twi language	600	600	16,098
Total	1,642	1,997	161,773

There are 1,544 dataset records: 257 response records and 1,287 replay records. Response records expand to deduplicated assistant-prefix views while preserving their complete preceding context. Records, training views and presentations are different counts, and none should be described as independent new conversations.

The response views comprise 229 preserved R3 views, 67 new targets from 33 project-authored histories, and 59 user-spelling variants. The latter replace selected lowercase epsilon characters with 3 only in user turns; assistant targets are unchanged. They add no new independent answers. Eight duplicate prefix views were removed. Two of 35 candidate histories were held out of admission over unresolved meaning. Twelve replay candidates were excluded by protected-content checks. Previous holdouts and source relationships remain bound in the pack's overlap audit.

New histories are agent-translated and agent-checked derivatives of authored English frames. They are not recordings of natural conversations and do not have native-human or clinical certification. Original wording, amendments and review decisions remain available in private research storage. New plus augmented targets account for approximately 7.4% of response-target tokens. This is a small continuation experiment, not full-corpus training.

Training

• Frozen parameters: 8,953,803,264. Trainable adapter parameters: 43,278,336.
• Rank-16 LoRA, alpha 32, dropout 0.05; exact R3 adapter values at initialization.
• Learning rate 1e-5, gradient accumulation 16, warmup fraction 0.05.
• Two response passes with finite, no-replacement replay. Actual target-token mixture: 60.04% responses, 20.01% English, 10.00% meaning, 9.95% Twi language.
• 125 committed optimizer updates; checkpoint after the first pass at update 63.
• 315,357 input tokens and 161,773 effective shifted target tokens consumed.
• All 1,642 unique encoded rows were checked against the native tokenizer. Context is masked for chat supervision, with one supervised final chat EOS; language-learning rows use causal next-token supervision. No truncation.
• New optimizer and scheduler for the continuation; saved-state restoration at the pass boundary verifies model, optimizer, scheduler, position and RNG equality.
• Private Modal H100 compute. Recorded worker elapsed time: 1,658.58 seconds. This is not total end-to-end time including preparation, downloads and evaluation.

Pinned runtime: torch 2.13.0, transformers 5.17.0, peft 0.20.0, accelerate 1.15.0, numpy 2.4.2, triton 3.7.1, flash-linear-attention 0.5.2. The reference causal-convolution implementation is used consistently; its performance warning is documented, and kernels were not changed between candidates.

Evaluation Protocol

The saved development set contains 46 fixed-context cases plus 29 turns across 10 independent dialogue rollouts. It includes Twi, English, code-switching, negation, quantities, corrections, language requests and health-meaning checks. Previously observed failures are development evidence, not blind final-test cases.

Compare the base, R3 final, R4 final and R4 first-pass snapshot in raw mode. Separately compare base, R3 and R4 final with the same repetition guard. Later rollout turns use each model's own replies; fixed-context prompts are identical. The maximum new-token budget and decoding contract are held constant.

Report EOS completion, truncation, repeated four-word sequences, consecutive identical rollout replies, eight strict JSON controls and token-weighted language loss on 343 held-out text rows. A low perplexity or valid JSON is not proof of faithful meaning, useful advice or clinical safety. Guard-triggered pauses are not counted as EOS success. Exact raw outputs, input histories, stop reasons, checkpoint identities and runtime proofs remain available.

Both detached evaluation runs completed: 1,672 raw slots and 225 guarded slots. The raw count includes 300 chat requests and 1,372 language-loss measurements. Finalized slots include explicit errors and blocked follow-ups, not only successes.

Raw development diagnostic	Base	R3	R4 final	R4 first pass
EOS replies / 75 planned	71	67	73	71
Generated, verified replies	75	73	75	75
Replies containing repeated four-word sequences	5/75	12/73	8/75	9/75
Exact consecutive rollout replies / compared pairs	4/19	3/17	0/19	2/19
Strict JSON literal matches with EOS	6/8	8/8	8/8	8/8
Held-out Twi perplexity	15.4532	5.1590	5.1968	5.2031

All four language-loss evaluations cover the same 343 rows and 9,315 predicted tokens. R4 final has approximately 0.73% higher perplexity than R3: a small worsening, not a retention improvement. Raw R3 has one follow-up input error after a Unicode-truncated reply and one downstream blocked turn. Both remain in the evidence; they are not assigned semantic scores. These results should not be pooled with earlier runs using the same development set.

With identical repetition guards, EOS counts are 71/75 base, 69/75 R3 and 73/75 R4; guard pauses are 1/75, 5/75 and 1/75 respectively. All 225 guarded requests produce verified outputs. Later own-history inputs can differ from raw mode because a guard changes the prior reply. A paused response is not EOS success; raw and guarded measurements are kept separate.

Agent inspection finds improved distinction between a call date and an arrival date, concise corrected orange quantities, and a Twi summary preserving mother, left knee and onset. These are limited case-level findings. Incorrect Twi arithmetic, unsupported initial quantities, weak planning and a meaning-reversing shopping phrase remain. Matched English health summaries still invent mobility limitations. A separate live name-retention follow-up succeeds for R3 and fails for R4. The first-pass model gives a better final next action in one English planning case, so an extra pass is not uniformly beneficial.

Final R4 is exposed as the completed continuation for local review, not selected as universally best or promoted to production. Both checkpoints are retained. No calibrated native-human semantic score, clinical rating or blind final-test improvement is claimed. The public execution record explains denominators and the private HTML report preserves exact questions, histories and outputs.

Limitations and Use

R4 can echo a request, misinterpret Twi, fail a follow-up, confuse arithmetic, invent facts and repeat. A deterministic repetition guard limits loops; it does not repair meaning or certify the remaining answer. No other LLM rewrites the responses. No current web information or real commerce actions are connected in this comparison.

Use the local three-way UI for research assessment and saved feedback. Do not use the model alone for diagnosis, treatment decisions, legal facts, purchases, or other consequential actions. No model-wide semantic, fairness or clinical safety rate has been established. Testing is development-focused and the small, agent-generated addition does not establish dialect coverage or population-wide performance.

Evidence and Reproduction

See the execution record for the private artifact paths, receipts, source ledger, checksums and local comparison URL. Training inputs, completed execution metadata, tokenizer bundle and exact code snapshots are versioned. Private artifacts are stored on the existing Modal research volume; local readbacks and review history stay on the owner's device. Public GitHub holds implementation and documentation, not the private corpus.