Spaces:
Running
Running
Update org card: honest base-pair metrics (99.8% routing, +0.089 base-pair, no domain ahead yet) + build log of what went right/wrong
Browse files
README.md
CHANGED
|
@@ -23,12 +23,11 @@ each unit handling what it's good at and passing the rest along. That's exactly
|
|
| 23 |
|
| 24 |
- **Each link is one small specialist** โ a dense ~74M model trained on a *single* domain. It is
|
| 25 |
excellent at its own data and (deliberately) surprised by everything else.
|
| 26 |
-
- **The router is the connector between links.** When an input arrives,
|
| 27 |
-
|
| 28 |
-
|
| 29 |
- **The chain grows link by link.** Because the specialists are trained *separately*, you can chain a
|
| 30 |
-
new domain on without retraining the others โ add a link, extend the router, done.
|
| 31 |
-
a single link the same way.
|
| 32 |
- **One link runs per query.** Only the routed specialist computes, so a chain of four ~74M experts
|
| 33 |
costs ~74M of compute per token โ roughly **7ร cheaper** than a 500M monolith of comparable scope.
|
| 34 |
|
|
@@ -49,28 +48,73 @@ Each specialist is grown by **interleaving two steps**, per domain:
|
|
| 49 |
clone โ the specialization is what makes routing work.
|
| 50 |
|
| 51 |
We iterate those two steps until each link is as strong as its capacity allows, then train the
|
| 52 |
-
**router**
|
| 53 |
-
|
| 54 |
-
misses.
|
| 55 |
-
|
| 56 |
-
This is, in lineage, a **cluster Branch-Train-Merge (cBTM) mixture of domain experts** โ independent
|
| 57 |
-
experts + perplexity-aware routing โ with iterative distillation from a larger teacher layered on top.
|
| 58 |
|
| 59 |
---
|
| 60 |
|
| 61 |
## ๐งฌ Current project โ DaisyChain Genomics
|
| 62 |
|
| 63 |
Four DNA/RNA specialists (**eukaryote ยท prokaryote ยท mRNA ยท mRNA-splice**, ~74M each, **โ295M total โ
|
| 64 |
-
under 500M**), each distilled per-domain from
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
| **vs the 500M teacher** | within ~6% likelihood; closing with training |
|
| 71 |
|
| 72 |
- ๐ฆ **Model:** [`DaisyChainAI/daisychain-genomics`](https://huggingface.co/DaisyChainAI/daisychain-genomics)
|
| 73 |
-
- ๐ฎ **Live demo:** [`Daisychain-Genomics-Demo`](https://huggingface.co/spaces/DaisyChainAI/Daisychain-Genomics-Demo) โ paste DNA, watch the chain light up specialist-by-specialist and route in real time.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
More links on the chain โ and more chains โ coming. ๐ผ
|
| 76 |
|
|
|
|
| 23 |
|
| 24 |
- **Each link is one small specialist** โ a dense ~74M model trained on a *single* domain. It is
|
| 25 |
excellent at its own data and (deliberately) surprised by everything else.
|
| 26 |
+
- **The router is the connector between links.** When an input arrives, every specialist reports how
|
| 27 |
+
*surprised* it is (bits/base) and exposes its hidden state, and a tiny learned router hands the work
|
| 28 |
+
to the link that's most at home with it.
|
| 29 |
- **The chain grows link by link.** Because the specialists are trained *separately*, you can chain a
|
| 30 |
+
new domain on without retraining the others โ add a link, extend the router, done.
|
|
|
|
| 31 |
- **One link runs per query.** Only the routed specialist computes, so a chain of four ~74M experts
|
| 32 |
costs ~74M of compute per token โ roughly **7ร cheaper** than a 500M monolith of comparable scope.
|
| 33 |
|
|
|
|
| 48 |
clone โ the specialization is what makes routing work.
|
| 49 |
|
| 50 |
We iterate those two steps until each link is as strong as its capacity allows, then train the
|
| 51 |
+
**router**. In lineage this is a **cluster Branch-Train-Merge (cBTM)** mixture of domain experts โ
|
| 52 |
+
independent experts + perplexity-aware routing โ with iterative distillation from a larger teacher.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
---
|
| 55 |
|
| 56 |
## ๐งฌ Current project โ DaisyChain Genomics
|
| 57 |
|
| 58 |
Four DNA/RNA specialists (**eukaryote ยท prokaryote ยท mRNA ยท mRNA-splice**, ~74M each, **โ295M total โ
|
| 59 |
+
under 500M**), each distilled per-domain from **[Carbon-500M](https://huggingface.co/HuggingFaceBio/Carbon-500M)**
|
| 60 |
+
behind a learned router. Carbon's domain mixture (50% eukaryotic / 25% mRNA / 10% splice / 15% bacterial)
|
| 61 |
+
maps one-to-one onto our four specialists.
|
| 62 |
+
|
| 63 |
+
### Where it actually stands (measured on Carbon's own base-pair / FNS metric)
|
| 64 |
+
|
| 65 |
+
We score likelihood the way Carbon does โ marginalizing each 6-mer into six per-base distributions and
|
| 66 |
+
taking mean per-base log-prob (`score_sequence`). Our implementation reproduces Carbon's `compute_bp_probs`
|
| 67 |
+
to **6e-08**, so these are apples-to-apples.
|
| 68 |
+
|
| 69 |
+
| | DaisyChain | Carbon-500M |
|
| 70 |
+
|---|---|---|
|
| 71 |
+
| **Routing accuracy** (held-out) | **99.8%** | โ |
|
| 72 |
+
| **Likelihood โ base-pair bits/base** (โ) | **1.876** | **1.787** |
|
| 73 |
+
| Seq-recovery, eukaryote (FNS, โ) | 31.5% | 38.9% |
|
| 74 |
+
| Seq-recovery, bacteria (FNS, โ) | 40.9% | 54.1% |
|
| 75 |
+
| Active params / query | ~74M (one specialist) | 500M |
|
| 76 |
|
| 77 |
+
**Honest standing: ~+0.089 bits/base behind, and no single domain beats Carbon yet.** The gap is
|
| 78 |
+
concentrated in mRNA and bacterial DNA (Carbon's strongest domains); eukaryote and splice are closest.
|
| 79 |
+
Note Carbon-500M is itself a *draft model*, explicitly "not designed to be competitive on downstream
|
| 80 |
+
benchmarks" โ so it's a fair, achievable target, not the 3B/8B flagships.
|
|
|
|
| 81 |
|
| 82 |
- ๐ฆ **Model:** [`DaisyChainAI/daisychain-genomics`](https://huggingface.co/DaisyChainAI/daisychain-genomics)
|
| 83 |
+
- ๐ฎ **Live demo:** [`Daisychain-Genomics-Demo`](https://huggingface.co/spaces/DaisyChainAI/Daisychain-Genomics-Demo) โ paste DNA, watch the chain light up specialist-by-specialist and route in real time, then generate with Carbon's base-pair (FNS) decoder.
|
| 84 |
+
|
| 85 |
+
---
|
| 86 |
+
|
| 87 |
+
## ๐ Build log โ what we got right, and what we got wrong
|
| 88 |
+
|
| 89 |
+
We build in the open, mistakes included. This project's honest history:
|
| 90 |
+
|
| 91 |
+
**What worked**
|
| 92 |
+
- **Per-domain specialists + a learned router** reached **99.8%** held-out routing โ one ~74M model active per query.
|
| 93 |
+
- **Snapshot-then-pick-best** distillation: snapshot every few thousand steps, deploy the snapshot with the
|
| 94 |
+
best *held-out* score, never the last one. This caught over-distillation (models that memorize the distill
|
| 95 |
+
cache and regress on held-out data) and made every round regression-guarded.
|
| 96 |
+
- **Re-fitting the router after every specialist swap.** Router features are coupled to the checkpoints;
|
| 97 |
+
skipping the re-fit once produced a fake "regression" that was pure routing drift.
|
| 98 |
+
- **FNS per-base distillation targets** โ distilling the teacher's *base-pair* marginals, not the 4096-way
|
| 99 |
+
6-mer distribution, gave the small students a tractable, base-pair-correct objective.
|
| 100 |
+
|
| 101 |
+
**What we got wrong (and corrected)**
|
| 102 |
+
- **We reported the wrong metric for days.** We measured likelihood as **6-mer cross-entropy** (a softer proxy)
|
| 103 |
+
instead of Carbon's **base-pair (FNS)** score. The proxy flattered us: it showed ~+0.043 behind and even
|
| 104 |
+
"splice beats Carbon." On Carbon's actual metric the gap is **+0.089 and no domain is ahead.** We re-baselined
|
| 105 |
+
the entire project history on the real metric.
|
| 106 |
+
- **We measured sequence recovery with the wrong decoder** (6-mer argmax) instead of Carbon's **FNS base-level
|
| 107 |
+
argmax**. Re-measuring with their decoder changed the numbers (and actually *raised* our bacteria recovery).
|
| 108 |
+
- **An early eval had a frame-alignment bug** โ feeding a context length not divisible by 6 knocked our 6-mer
|
| 109 |
+
model out of phase and produced an impossible near-zero recovery. Fixed by aligning context to the 6-mer grid.
|
| 110 |
+
- **Decoding took several wrong turns** before matching Carbon: greedy with no repetition control (collapsed to
|
| 111 |
+
homopolymers), then top-k sampling (trapped on low-complexity GC/AT loops), before adopting Carbon's actual
|
| 112 |
+
**base-pair FNS decoder** (top-p at the 6-mer level โ per-base selection).
|
| 113 |
+
- **One training round improved the proxy while regressing the real metric** (an early mRNA distill-only pass)
|
| 114 |
+
โ invisible on 6-mer CE, obvious on base-pair. A later base+distill round fixed it.
|
| 115 |
+
|
| 116 |
+
**The lesson:** *measure the way the baseline measures, or you aren't comparing anything.* A stricter, honest
|
| 117 |
+
evaluation didn't sink the project โ it pointed to exactly which domains to attack and which "wins" were illusions.
|
| 118 |
|
| 119 |
More links on the chain โ and more chains โ coming. ๐ผ
|
| 120 |
|