Quazim0t0 commited on
Commit
9e49e5f
ยท
verified ยท
1 Parent(s): 46d9b28

Update org card: honest base-pair metrics (99.8% routing, +0.089 base-pair, no domain ahead yet) + build log of what went right/wrong

Browse files
Files changed (1) hide show
  1. README.md +62 -18
README.md CHANGED
@@ -23,12 +23,11 @@ each unit handling what it's good at and passing the rest along. That's exactly
23
 
24
  - **Each link is one small specialist** โ€” a dense ~74M model trained on a *single* domain. It is
25
  excellent at its own data and (deliberately) surprised by everything else.
26
- - **The router is the connector between links.** When an input arrives, it travels down the chain:
27
- every specialist reports how *surprised* it is (bits/base) and exposes its hidden state, and a tiny
28
- learned router hands the work to the link that's most at home with it.
29
  - **The chain grows link by link.** Because the specialists are trained *separately*, you can chain a
30
- new domain on without retraining the others โ€” add a link, extend the router, done. Remove or upgrade
31
- a single link the same way.
32
  - **One link runs per query.** Only the routed specialist computes, so a chain of four ~74M experts
33
  costs ~74M of compute per token โ€” roughly **7ร— cheaper** than a 500M monolith of comparable scope.
34
 
@@ -49,28 +48,73 @@ Each specialist is grown by **interleaving two steps**, per domain:
49
  clone โ€” the specialization is what makes routing work.
50
 
51
  We iterate those two steps until each link is as strong as its capacity allows, then train the
52
- **router**: a small head that reads every specialist's surprise plus a compressed view of its hidden
53
- state and predicts the home domain โ€” recovering bias-corrections a plain "lowest-perplexity-wins" rule
54
- misses.
55
-
56
- This is, in lineage, a **cluster Branch-Train-Merge (cBTM) mixture of domain experts** โ€” independent
57
- experts + perplexity-aware routing โ€” with iterative distillation from a larger teacher layered on top.
58
 
59
  ---
60
 
61
  ## ๐Ÿงฌ Current project โ€” DaisyChain Genomics
62
 
63
  Four DNA/RNA specialists (**eukaryote ยท prokaryote ยท mRNA ยท mRNA-splice**, ~74M each, **โ‰ˆ295M total โ€”
64
- under 500M**), each distilled per-domain from a 500M genomic foundation model, behind a learned router.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
- | | |
67
- |---|---|
68
- | **Routing accuracy** (held-out) | **94.8%** |
69
- | **Active params / query** | ~74M (one specialist) |
70
- | **vs the 500M teacher** | within ~6% likelihood; closing with training |
71
 
72
  - ๐Ÿ“ฆ **Model:** [`DaisyChainAI/daisychain-genomics`](https://huggingface.co/DaisyChainAI/daisychain-genomics)
73
- - ๐ŸŽฎ **Live demo:** [`Daisychain-Genomics-Demo`](https://huggingface.co/spaces/DaisyChainAI/Daisychain-Genomics-Demo) โ€” paste DNA, watch the chain light up specialist-by-specialist and route in real time.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
 
75
  More links on the chain โ€” and more chains โ€” coming. ๐ŸŒผ
76
 
 
23
 
24
  - **Each link is one small specialist** โ€” a dense ~74M model trained on a *single* domain. It is
25
  excellent at its own data and (deliberately) surprised by everything else.
26
+ - **The router is the connector between links.** When an input arrives, every specialist reports how
27
+ *surprised* it is (bits/base) and exposes its hidden state, and a tiny learned router hands the work
28
+ to the link that's most at home with it.
29
  - **The chain grows link by link.** Because the specialists are trained *separately*, you can chain a
30
+ new domain on without retraining the others โ€” add a link, extend the router, done.
 
31
  - **One link runs per query.** Only the routed specialist computes, so a chain of four ~74M experts
32
  costs ~74M of compute per token โ€” roughly **7ร— cheaper** than a 500M monolith of comparable scope.
33
 
 
48
  clone โ€” the specialization is what makes routing work.
49
 
50
  We iterate those two steps until each link is as strong as its capacity allows, then train the
51
+ **router**. In lineage this is a **cluster Branch-Train-Merge (cBTM)** mixture of domain experts โ€”
52
+ independent experts + perplexity-aware routing โ€” with iterative distillation from a larger teacher.
 
 
 
 
53
 
54
  ---
55
 
56
  ## ๐Ÿงฌ Current project โ€” DaisyChain Genomics
57
 
58
  Four DNA/RNA specialists (**eukaryote ยท prokaryote ยท mRNA ยท mRNA-splice**, ~74M each, **โ‰ˆ295M total โ€”
59
+ under 500M**), each distilled per-domain from **[Carbon-500M](https://huggingface.co/HuggingFaceBio/Carbon-500M)**
60
+ behind a learned router. Carbon's domain mixture (50% eukaryotic / 25% mRNA / 10% splice / 15% bacterial)
61
+ maps one-to-one onto our four specialists.
62
+
63
+ ### Where it actually stands (measured on Carbon's own base-pair / FNS metric)
64
+
65
+ We score likelihood the way Carbon does โ€” marginalizing each 6-mer into six per-base distributions and
66
+ taking mean per-base log-prob (`score_sequence`). Our implementation reproduces Carbon's `compute_bp_probs`
67
+ to **6e-08**, so these are apples-to-apples.
68
+
69
+ | | DaisyChain | Carbon-500M |
70
+ |---|---|---|
71
+ | **Routing accuracy** (held-out) | **99.8%** | โ€” |
72
+ | **Likelihood โ€” base-pair bits/base** (โ†“) | **1.876** | **1.787** |
73
+ | Seq-recovery, eukaryote (FNS, โ†‘) | 31.5% | 38.9% |
74
+ | Seq-recovery, bacteria (FNS, โ†‘) | 40.9% | 54.1% |
75
+ | Active params / query | ~74M (one specialist) | 500M |
76
 
77
+ **Honest standing: ~+0.089 bits/base behind, and no single domain beats Carbon yet.** The gap is
78
+ concentrated in mRNA and bacterial DNA (Carbon's strongest domains); eukaryote and splice are closest.
79
+ Note Carbon-500M is itself a *draft model*, explicitly "not designed to be competitive on downstream
80
+ benchmarks" โ€” so it's a fair, achievable target, not the 3B/8B flagships.
 
81
 
82
  - ๐Ÿ“ฆ **Model:** [`DaisyChainAI/daisychain-genomics`](https://huggingface.co/DaisyChainAI/daisychain-genomics)
83
+ - ๐ŸŽฎ **Live demo:** [`Daisychain-Genomics-Demo`](https://huggingface.co/spaces/DaisyChainAI/Daisychain-Genomics-Demo) โ€” paste DNA, watch the chain light up specialist-by-specialist and route in real time, then generate with Carbon's base-pair (FNS) decoder.
84
+
85
+ ---
86
+
87
+ ## ๐Ÿ““ Build log โ€” what we got right, and what we got wrong
88
+
89
+ We build in the open, mistakes included. This project's honest history:
90
+
91
+ **What worked**
92
+ - **Per-domain specialists + a learned router** reached **99.8%** held-out routing โ€” one ~74M model active per query.
93
+ - **Snapshot-then-pick-best** distillation: snapshot every few thousand steps, deploy the snapshot with the
94
+ best *held-out* score, never the last one. This caught over-distillation (models that memorize the distill
95
+ cache and regress on held-out data) and made every round regression-guarded.
96
+ - **Re-fitting the router after every specialist swap.** Router features are coupled to the checkpoints;
97
+ skipping the re-fit once produced a fake "regression" that was pure routing drift.
98
+ - **FNS per-base distillation targets** โ€” distilling the teacher's *base-pair* marginals, not the 4096-way
99
+ 6-mer distribution, gave the small students a tractable, base-pair-correct objective.
100
+
101
+ **What we got wrong (and corrected)**
102
+ - **We reported the wrong metric for days.** We measured likelihood as **6-mer cross-entropy** (a softer proxy)
103
+ instead of Carbon's **base-pair (FNS)** score. The proxy flattered us: it showed ~+0.043 behind and even
104
+ "splice beats Carbon." On Carbon's actual metric the gap is **+0.089 and no domain is ahead.** We re-baselined
105
+ the entire project history on the real metric.
106
+ - **We measured sequence recovery with the wrong decoder** (6-mer argmax) instead of Carbon's **FNS base-level
107
+ argmax**. Re-measuring with their decoder changed the numbers (and actually *raised* our bacteria recovery).
108
+ - **An early eval had a frame-alignment bug** โ€” feeding a context length not divisible by 6 knocked our 6-mer
109
+ model out of phase and produced an impossible near-zero recovery. Fixed by aligning context to the 6-mer grid.
110
+ - **Decoding took several wrong turns** before matching Carbon: greedy with no repetition control (collapsed to
111
+ homopolymers), then top-k sampling (trapped on low-complexity GC/AT loops), before adopting Carbon's actual
112
+ **base-pair FNS decoder** (top-p at the 6-mer level โ†’ per-base selection).
113
+ - **One training round improved the proxy while regressing the real metric** (an early mRNA distill-only pass)
114
+ โ€” invisible on 6-mer CE, obvious on base-pair. A later base+distill round fixed it.
115
+
116
+ **The lesson:** *measure the way the baseline measures, or you aren't comparing anything.* A stricter, honest
117
+ evaluation didn't sink the project โ€” it pointed to exactly which domains to attack and which "wins" were illusions.
118
 
119
  More links on the chain โ€” and more chains โ€” coming. ๐ŸŒผ
120