Barbados / README.md
Remove outdated review report and documentation reference
cc518ac verified 13 days ago
23.7 kB
---
language:
  - en
pipeline_tag: text-generation
tags:
  - research
  - experimental
  - mixture-of-experts
  - looped-transformer
  - custom-code
datasets:
  - HuggingFaceFW/fineweb-edu
  - TheFusionCube/Fable-5-CoT-Traces
license: apache-2.0
library_name: pytorch
---
# Barbados — a looped MoE research prototype

**This is a research prototype. It is not a useful general-purpose assistant.**
Its saved generations are repetitive, off task and sometimes incorrect. Useful
instruction following, coding, broad reasoning gains and acceleration are
unestablished. This release documents a completed training trajectory and its
measured cache–quality tradeoff.

The [latest MELT findings](reports/barbados-melt-followup-2026-10-19.md)
add selective loop-gate resets, actual attention mass, K/V distances and a
native-bypass baseline. They separate completed measurements from the pending
gates-only training experiment.

Barbados is a **112,237,056-parameter causal language model** with a backbone
trained from random initialization, three recurrent passes, MELT-style KV sharing
and small visible-response fine-tuning. This independent research has no
affiliation with or endorsement by Mistral AI.
Barbados is named after the [Barbados threadsnake](https://science.psu.edu/news/worlds-smallest-snake-found-barbados).

## Architecture

| Component | Implemented setting |
| --- | --- |
| Physical stack | Six blocks, hidden width 384 |
| Recurrence | Three complete passes; 18 block visits with shared attention/expert weights |
| Attention | Grouped-query attention: six query heads, three KV heads, head dimension 64; RoPE |
| Experts per block | 24 routed experts and two always-active shared experts; intermediate width 384 |
| Expert selection | Four routed experts plus both shared experts per token; separate learned routers per loop |
| Tokenizer | Reused SmolLM tokenizer, vocabulary 49,152, EOS ID 0 |
| Context | At most 1,023 tokens, including formatted prompt and continuation |
| Parameter count | 110,465,280 backbone parameters plus 1,771,776 latent-memory gate parameters |

Sparse expert selection does not skip recurrent passes: all three loops execute.
The geometry and native recipe derive from the public
[LOOM implementation at commit `a2ee5a7`](https://github.com/hed-ucas/LOOM/tree/a2ee5a7ecd38b083d9c049d97271ece38f4bce4c),
with reduced size, loop count, global batch and training exposure. This is a
budget derivative, not a published LOOM/Ouro run reproduction. Looping, sparse
experts and MELT sharing are prior methods; no invention is claimed here.

Each physical attention layer has one latent-memory gate shared across its
three visits: six gates stored in 18 learned tensors. For normalized attention
input `u`, loop one initializes `m = u`. Later loops compute
`z = sigmoid(Wu*u + Wm*m_previous + b)` and
`m = z*m_previous + (1-z)*u`. Queries use `u`; keys and values use `m`.
Gate initialization is `Wu = Wm = 0`, `b = -4`.

Fixed two-unit CPU diagnostics measured FP32 mean retention of **1.02–1.66%
FULL**, **1.90–4.22% SHARED**, and **1.94–4.49% after response SFT**, versus
1.80% at initialization. FULL pre-activation means shifted below −4 in all
12 layer/loop cells; SHARED and the child shifted above −4 in ten cells and
below in two, with wider distributions. Both mean and spread matter.

A completed policy-by-gate 2×2 on the **pre-SFT SHARED@32 parent** and
32,065 pilot targets finds a conditional benefit under SHARED and a cost under
FULL. Resetting the gates raises SHARED CPU FP32 CE **3.724062 → 3.729816**,
but lowers FULL CE **3.708169 → 3.695786**. Learned gates help SHARED in
chunks 3–20 and hurt FULL in every chunk; the policy interaction is
**+0.018137 nat/target**, with exact chunk-zero controls. This supports
sharing-dependent compensation on this coadapted backbone. It does not
reproduce training with fixed gates or identify a unique adaptation mechanism.
The final SFT child's gate contribution has not been ablated. Selectively
resetting the final visit (loop 2, zero-based) removes most of the conditional
SHARED benefit, while resetting loop 1 removes more FULL cost; the effects are
nonadditive. Actual earlier-loop
SHARED attention on old positions is **13.16% in chunk 2**, versus **58.81% of
eligible keys**, in the fixed 17-unit cohort. Learned gates reduce old attention
mass. Pooled keys become slightly closer, while values become slightly farther
apart under SHARED, so uniform K/V convergence is not an established mechanism.
See the [new measurements and figures](reports/barbados-melt-followup-2026-10-19.md),
[policy control](reports/barbados-policy-control-v1.md) and
[earlier gate statistics](reports/barbados-controlled-evaluation-v1.md).

The **SHARED** policy retains a complete final-loop KV prefix and the latest
50 positions for each earlier loop, separately at each physical layer. An
earlier-loop query at position `i` uses final-loop KV for positions
`j < max(0, i-50)` and its own loop's KV for more recent positions. All selected
positions enter one causal softmax. Chunk size and window are both 50. Chunks
commit after all loops finish; old final-loop states therefore come from
completed chunks. Sharing operates across loops at the same layer.

Training keeps full gradients through all loops and selected earlier chunks;
history is not detached. Residual carry and latent slots reset between chunks;
explicit KV preserves token history. Cache savings do not imply equivalent
autograd-memory savings.

## Training lineage

| Stage | Retained training | Initialization and objective |
| --- | --- | --- |
| Native backbone | 7,217 updates; 920,770,653 next-token targets | Random weights, seed 0; final-loop causal CE |
| MELT-style adaptation | 32 updates; 4,087,238 replay targets **per arm** | Identical gated native states for FULL and SHARED; teacher-free final-loop CE |
| Released SHARED child | Three epochs; 21 updates; 147,597 supervised response-target exposures | Saved SHARED parent; fresh FP32 Adam; response-only CE |

**No pretrained foundation weights or teacher were used for the backbone.**
Scratch pretraining uses one prepared pass over FineWeb-Edu documents from two
shards of revision `87f09149ef4734204d70ed1d046ddc9ca3f2b8f9`, component-based
train/development separation and approximate deduplication. It uses the author
LeCun initializer, microbatch 16, accumulation eight, global batch 128,
AdamW `(0.9, 0.95)`, peak LR `4e-4`, weight decay `0.1`, clip norm 1 and a
10% warmup / 80% stable / 10% decay schedule to 10% of peak. Native capacity
factor is 1.5. Adaptation and causal evaluation disable capacity dropping to
avoid future-token competition for expert slots. The released source's
configured auxiliary z-loss remains disconnected from the active objective.

The MELT comparison replays the same FineWeb-Edu prefix for both arms. FULL
retains distinct complete histories for all loops; SHARED uses the policy above.
Both use active gates, full gradients and the same 32-update budget, with zero
skipped updates. This is a reduced, teacher-free CE adaptation of
[MELT](https://arxiv.org/abs/2605.07721v3), **not its complete teacher-supervision
objective, sequence length or training budget**. BF16 parameter views remain
active through forward and backward, with FP32 masters, gradients, optimizer
states and router projection arithmetic.

## Small Fable visible-response SFT

The final stage uses public, uploader-attributed responses from
[TheFusionCube/Fable-5-CoT-Traces](https://huggingface.co/datasets/TheFusionCube/Fable-5-CoT-Traces),
revision `3624e3a8aab36a89dc3b89aebef0e034a64ea04e`. **The generating model's
identity is not independently authenticated; these are visible answers, not
authenticated hidden/private chain-of-thought.** Selection excludes decoys,
ambiguous metadata, truncation, unsuitable tasks and overlength examples.
Answers remain verbatim, with unverified correctness. The duplicate armand0e
chat conversion adds no data.

| Split | Unique examples | Task families | Input tokens per epoch | Supervised targets per epoch |
| --- | ---: | ---: | ---: | ---: |
| Train | 68 | 19 | 54,651 | 49,199 |
| Validation | 5 | 3 | 4,041 | 3,663 |
| Test | 12 | 3 | 9,369 | 8,455 |

Question families, linked sessions and normalized duplicate payloads are grouped
before splitting. Prompt and response are separately tokenized, concatenated,
and followed by EOS. Only response tokens and EOS receive CE labels; prompt-state
gradients remain. The exact wrapper is:

```text
### Context
{prompt}
### Response
```

Three epochs reuse the **same 68 training examples**: 163,953 input-token and
147,597 supervised-target exposures. All 92 FP32 Adam states reached step 21
with zero skipped updates.
Backbone/gate peak learning rates are `2e-5`/`1e-4`, with warmup/cosine decay,
an 8,192-input-token accumulation threshold, with whole-example overshoot and
retained epoch tails, and loss normalized by the actual response-target count.
There is no online teacher or KL objective.

### Matched CPU continuation control

Two fresh CPU BF16 arms completed the same fixed 21-update schedule from the
pre-SFT SHARED parent: original Fable responses versus geometry/mask-matched
FineWeb-Edu replay. Each reused 68 examples for three epochs, **163,953 input
tokens and 147,597 supervised targets**. Lengths, prefix masks, EOS, update
boundaries, optimizer settings and execution were matched. No updates were
skipped, and final checkpoints were selected before PIQA scoring.

| Experimental CPU child | PIQA correct / 1,838 | Accuracy |
| --- | ---: | ---: |
| Fable responses | **1,072** | **58.32%** |
| FineWeb replay | 1,054 | 57.34% |

The single predeclared primary comparison gives **+18 answers / +0.979 percentage
points**, with **40 wins / 22 losses**, exact two-sided paired **p=0.0300**.
FineWeb finished +1 over the saved SHARED parent; the new Fable child finished
+19. This supports a data-source difference in this matched run, without
identifying which property of the examples caused it. It does not establish a
formatting-only mechanism or broader reasoning gains. The experiment uses one
training seed and previously evaluated public validation; this is not pristine
confirmatory evidence. Replay is mask-matched, rather than conventional
all-token continued pretraining. See the
[complete protocol, results and limitations](reports/barbados-policy-control-v1.md).

**These experimental children remain local; the released weights are unchanged.**
The released GPU-trained child's 1,070 count below remains its measured result.

### Continuation replication progress

Two additional fixed endpoints have now completed, including the previously
interrupted Fable order-3 run. Their full CPU FP32 PIQA scores are:

| CPU continuation endpoint | PIQA correct / 1,838 |
| --- | ---: |
| Original Fable order 0 | 1,072 |
| Additional Fable order 3 | 1,066 |
| Original deterministic FineWeb replay | 1,054 |
| New random FineWeb span draw 1 | 1,058 |

These are descriptive checkpoint results. Three more random replay draws and
Fable order 6 remain unstarted, so the planned training-variation study is
incomplete. The two Fable orders share the same 68 responses. No new
significance claim or best-seed weight selection is made; experimental weights
remain local. See the [complete follow-up](reports/barbados-melt-followup-2026-10-19.md).

### Gates-only follow-up: qualification complete, training pending

The next controlled intervention freezes all **110,465,280 backbone parameters
and every buffer**, including router bias, and trains only the **1,771,776 gate
parameters**. Disposable CPU checks found exact checkpointed-versus-direct
BF16 gate gradients and routing on two 128-token training prefixes, verified
one gate-only optimizer update, and preserved the frozen state and gate-bypass
function. A fresh-process resume rehearsal reproduced model, optimizer and RNG
state exactly. These are implementation checks, not evidence of improved quality.

Full-length CPU profiling measured **14.07 seconds for 1×1,023 targets** and
**59.76 seconds for 8×1,023 targets**, at approximately 2.83 and 11.07 GiB peak
process RSS. The largest 12×1,023 group was skipped by the conservative RAM
admission rule; it did not fail with an observed out-of-memory error. **No
retained gates-only training has run**. The intended run retains the original
32 updates and 4,087,238 replay targets. See the
[CPU qualification and profiling report](reports/barbados-gates-only-progress.md).

A subsequent disposable **5070 Ti Laptop GPU** profile completed one original
127,600-target update with a batch cap of four in **421.21 seconds**, peaking at
**4.91 GiB allocated**. This gives a bare 32-update extrapolation of 3.74 hours,
before overhead and sustained-load effects. Larger batches were skipped by the
conservative VRAM rule, not observed to OOM. No trained endpoint was retained.

On two 128-token TRAIN prefixes, default CUDA repeats changed 105/4,608 expert
sets and produced different gate gradients. Under deterministic execution,
checkpointed repeats and checkpointed/direct passes had **bit-identical losses,
all 18 gate gradients and all 108 routing calls**. This is a sampled numerical
control on native update-zero weights, not proof of the historical PIQA flip's
cause. A subsequent deterministic original 127,600-target update completed
in **321.10 seconds**, peaking at **4.91 GiB allocated**. All 18 gate tensors
moved at the prescribed first LR; fixed-input gate changes were nonzero.
The entire native state was restored afterward: **retained updates remain zero**.
This extrapolates to 2.85 hours for 32 updates before overhead; it is not a full-run
measurement or a held-out quality result. CUDA checkpoint-resume remains untested.
See the [first-step results](evidence/melt-findings-v2/notes/barbados-gate-step-results-v1.md)
and [earlier GPU controls](reports/barbados-cuda-profile-v1.md).

## Evaluation and the measured tradeoff

### Full PIQA validation, CPU FP32

All **1,838 original validation items** were scored once per saved arm under a
frozen CPU FP32 protocol: mean complete-answer NLL, no EOS, capacity off,
equal-length batches up to eight without padding and answer-only readout.
This is a custom evaluation, not an exact reproduction of the earlier GPU BF16
counts. The previously reserved 1,338 rows are now consumed; no PIQA examples
were used for new training or tuning.

| Arm | Correct | Accuracy | Wilson 95% interval |
| --- | ---: | ---: | ---: |
| Native, gates disabled | 1065/1838 | 57.94% | 55.67–60.18% |
| FULL@32 | 1062/1838 | 57.78% | 55.51–60.02% |
| SHARED@32 | 1053/1838 | 57.29% | 55.02–59.53% |
| Released response-SFT child | 1070/1838 | 58.22% | 55.95–60.45% |

Native exceeds the majority-position control (928/1838) and all four recorded
answer-length controls (870–968/1838). Its label-association Fisher p is
**6.43e-12**; the paired majority comparison gives **2.69e-6**. This supports
task-relevant predictive signal beyond those simple controls, without isolating
physical reasoning or resolving pretraining contamination.

The child wins 37 and loses 20 cases versus SHARED: **+0.925 percentage points**,
with raw paired two-sided **p=0.0331** and **Holm-adjusted p=0.198864** across
the six exploratory pairwise comparisons. **None of those six comparisons
survives correction at familywise 0.05.** Against native it wins 36 and loses
31 (raw **p=0.6254**), without an established improvement. Continued training
and the response-data intervention remain confounded in the historical result.
These results do not establish general instruction or reasoning gains, or a
formatting-only mechanism. The [new report](reports/barbados-policy-control-v1.md)
contains the complete Holm table and the matched continuation design.

Eight batch-one checks per arm found no prediction changes and maximum mean-NLL
difference **9.54e-7**; that is a sample check, not a global bound. Most inputs
(83.30% of candidates) fit in the first 50-token chunk, so this is a weak test
of older-chunk KV sharing. Full results, paired comparisons and controls are in
the [controlled evaluation report](reports/barbados-controlled-evaluation-v1.md).

### General-language DEV and cache tradeoff

Matched evaluation scores **1,021,448 primary DEV targets across 923 components**
with capacity disabled. Four units / 3,442 targets overlapping earlier evaluated
components are excluded. Absence of pretraining overlap is unestablished.

| Saved arm, before response SFT | Primary CE, nats/target |
| --- | ---: |
| Native, gates disabled | 3.729070 |
| FULL after adaptation | 3.728203 |
| SHARED after adaptation | 3.785575 |

SHARED minus FULL is **+0.057371 nat/target**, paired component-bootstrap 95%
interval **[+0.051666, +0.063394]**: about **5.9% higher perplexity**. Bootstrap
uses 2,000 draws, without training-seed uncertainty. FULL–native also changes
gates and training exposure, preventing isolation of a MELT mechanism effect.
The original 64-item PIQA diagnostic and its exploratory significance tests
are documented in the [controlled report](reports/barbados-controlled-evaluation-v1.md).
The full 1,838-item evaluation above is the primary PIQA result on this card.

The saved **32,065-target pilot** also has update-zero scores: SHARED–FULL CE
gap fell **0.115440 → 0.049794 nats** over 32 updates (PPL penalty **12.24% →
5.11%**). The larger cohort's **5.9% is a measurement after 32 updates**, not an
upper bound or a converged method result. Intermediate heldout scores were not
saved, so convergence was not assessed. The cohorts cannot form one curve.

At batch one and 976 cached positions, an earlier pilot measured persistent
BF16 KV storage of **13,492,224 bytes FULL versus 4,958,208 bytes SHARED: 63.251%
less**. In that pilot, inference allocator peaks fell only **0.891%**. In the
separate larger DEV workload, SHARED's peak allocated memory was **12.445%**
lower, while its measured forward time was **2.083% longer**. These are different
measurements; none is a 63% reduction in total GPU memory, and no speedup is
demonstrated.

SFT validation response CE decreased **5.669114 → 5.327462** on five examples;
final test response CE was **5.504149** on 12 examples. These tiny holdouts do
not establish broad generalization. A historical pre-SFT GPU discrepancy on
one near-tie PIQA item remains unresolved and is not attributed to SFT;
[the controlled report](reports/barbados-controlled-evaluation-v1.md) preserves
the counts, margins and arithmetic diagnostics. The larger general-language
DEV score was not rerun on the SFT child.

New CPU BF16 routing diagnostics also show that changing explicit accumulation
order changes expert selections and candidate NLLs (maximum **0.0538 nat/token**
between matched ordered sums). Two identical default replays match exactly;
all four tested conditions have identical prediction vectors. The original
GPU routing traces were not saved: historical causation and any general ±1-item
noise bound remain unestablished.

Four fixed prompts ran once on both parent and child with CPU FP32 greedy
decoding, a 32-token cap and no rerolls:

1. “What does a mixture-of-experts layer do in a language model?”
2. “Write a Python function add(a, b) that returns their sum.”
3. “What is 17 + 25? Give the answer.”
4. “Explain in one sentence why Python list membership is usually slower than set membership for many items.”

All eight continuations reached the cap and failed their requests. The child
repeats “the sum of the sum” for the coding prompt and writes `17 + 25 = 17`.
[All eight unedited outputs](reports/loom-finish-qualitative-results-v1.md) and the
[full evaluation report](reports/loom-finish-results-v1.md) accompany the release.

## Running the model

This uses the bundled custom runtime. It is **not currently compatible with
`AutoModelForCausalLM`, the Transformers pipeline or hosted inference APIs**.
From the release repository root, install the pinned dependencies in a virtual
environment, then use the supplied CLI:

Use Python 3.12. Install the Hugging Face CLI separately from the pinned model
runtime dependencies:

```bash
python3.12 -m venv .venv-barbados
. .venv-barbados/bin/activate
pip install huggingface_hub
hf download Vykos/Barbados --local-dir Barbados
cd Barbados
pip install -r bundle/requirements.txt
CUDA_VISIBLE_DEVICES='' python -B bundle/run.py \
  --prompt "What is a mixture-of-experts layer?" --max-new-tokens 32
```

The CLI inserts the wrapper and emits JSON. Default inference is CPU FP32.
Optional `--device cuda` requires the pinned Torch 2.13.0
CUDA 13.0 build and BF16 support; it uses BF16 parameter views with FP32 master
state and routing. Run in a fresh process. Keep the `bundle/` inventory intact:
the runtime verifies files and hashes. Source, tokenizer and weights are bundled;
training data, optimizer and cloud access are unnecessary. Relocated CPU
execution matched the source reference without changing states; the public
distribution also passed an independent [relocated CPU check](release/licensed-cpu-verification.json).
These two-token checks verify delivery, not generation quality. Exact GPU
training resume was not rehearsed.

## Scope, provenance and licensing

Intended use is research on recurrence, sparse routing, differentiable KV sharing
and small-budget training. Limitations include one seed, short context, limited
training, tiny SFT splits, incomplete contamination coverage, unresolved PIQA
disagreement and poor generations. Architectural superiority and useful
capability are unestablished. Do not rely on its factual or coding advice.

The immutable bundle contains source attribution, training summaries, data
receipts and hash-bound lineage. Its final packed state SHA-256 is
`3df93e528bd1e821fc437b75aa3afb9bc6b129ffe7375b8192b915ddeef1f23d`.
Acknowledgements: the LOOM authors for the released backbone/training code;
the Ouro and MELT authors for related recurrent-model research; Hugging Face
contributors for FineWeb-Edu and the SmolLM tokenizer; and the public response
dataset's uploader. These acknowledgements do not imply endorsement.

The uploader releases the original Barbados model weights and project-authored
runtime under **Apache-2.0**; see [LICENSE](LICENSE) and [NOTICE](NOTICE).
Bundled upstream components retain their respective licenses and notices.
Training datasets are not redistributed. FineWeb-Edu attribution and its ODC-By /
CommonCrawl terms, the reused tokenizer's Apache-2.0 provenance and the Fable
uploader's MIT declaration are documented in NOTICE. The Fable declaration does
not authenticate teacher identity or establish rights in every response.

This is a separately packaged and licensed public distribution of the original
trained artifact. Original manifest SHA-256:
`6947858ef8baa2db08db06d80b58217ef246d66558af7819be1784954c41f64b`.
Public bundle manifest SHA-256:
`7d5956fb54d733b0a419facf5c589f0a88c98cda2fbbaf40e8632d88d75a6151`.
The distribution adds comment-only change/license notices to seven source files
and rebinds two runtime hash constants and their receipts. All 34 Python source
ASTs are unchanged; all 19 other source files, model weights and tokenizer bytes
are identical. Hash guards remain active; the inference algorithm is unchanged.
[The release audit](release/licensing-rebind-audit.json) maps every original and
public payload hash and preserves the exact original manifests alongside it.
These distribution edits do not retrain the model or change its measured results.