Refine release highlights and Figure 1 caption
#1
by lewtun HF Staff - opened
- index.html +31 -19
index.html
CHANGED
|
@@ -122,6 +122,7 @@
|
|
| 122 |
/* collapsible boxes inside the recipe */
|
| 123 |
details.box { border: 1px solid var(--rule); border-radius: 6px; padding: 0 16px; margin: 24px 0; }
|
| 124 |
details.box:last-of-type { border-bottom: 1px solid var(--rule); }
|
|
|
|
| 125 |
details.box summary { font-weight: 600; padding: 14px 30px 14px 0; }
|
| 126 |
details.box summary::after { top: 10px; }
|
| 127 |
details.box[open] summary { border-bottom: 1px solid var(--rule); margin-bottom: 14px; }
|
|
@@ -232,19 +233,18 @@
|
|
| 232 |
</div>
|
| 233 |
<p class="hl-lead"><b>TL;DR</b></p>
|
| 234 |
<ul class="clean highlights">
|
| 235 |
-
<li><b>Best open model.</b> Ahead of every open model on
|
| 236 |
-
<li><b>Frontier performance, 1,000× cheaper.</b>
|
| 237 |
-
<li><b>Best on two tasks.</b> Ahead of all models tested, closed ones included, on coding-variant effects and next-base prediction.</li>
|
| 238 |
<li><b>Open and local.</b> Apache 2.0, runs on your own hardware.</li>
|
| 239 |
<li><b>DNA and English in one model.</b> Reads and writes both natively.</li>
|
| 240 |
<li><b>Integrates into agents.</b> Works as a DNA sub-agent that general-purpose agents can delegate sequence reading to. <span class="tbd">add result: Kimi K3 alone vs Kimi K3 + Carbon-X</span></li>
|
| 241 |
</ul>
|
| 242 |
-
<p>Averaged over five DNA benchmarks, Carbon-X lands next to GPT-6.1 Sol (66.7% vs 67.5%) at about 1/1,800 of the cost per problem, and
|
| 243 |
<!-- HERO FIGURE: native SVG, drawn by the script at the end of the page -->
|
| 244 |
<figure class="hero">
|
| 245 |
-
<div class="figtitle">The best open model for DNA
|
| 246 |
<div class="chart" id="fig1"><noscript><img src="assets/fig1-score-vs-cost.png" width="1135" height="696" alt="Average DNA benchmark score against cost per problem relative to Carbon-X."></noscript></div>
|
| 247 |
-
<figcaption><b>Figure 1.</b>
|
| 248 |
</figure>
|
| 249 |
<p>The average hides where each model is strong. Carbon-X is the best model we tested, open or closed, at predicting the effects of mutations in protein-coding regions and at predicting the next bases of a sequence. Frontier models still lead on the other three tasks: complex variants, single-base variant effects and DNA understanding.</p>
|
| 250 |
<!-- Data: carbon-x-blog-figures/benchmark-scores.html (leaderboard snapshot 2026-10-21T13:31:46Z). Each model uses the
|
|
@@ -264,8 +264,22 @@
|
|
| 264 |
</table></div>
|
| 265 |
<figcaption><b>Table 1.</b> DNA benchmark scores (%). Each model is shown at the reasoning effort with its highest average score; the best score on each benchmark is shaded.</figcaption>
|
| 266 |
</figure>
|
| 267 |
-
<
|
| 268 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 269 |
</header>
|
| 270 |
|
| 271 |
<main class="col">
|
|
@@ -273,21 +287,19 @@
|
|
| 273 |
|
| 274 |
<section>
|
| 275 |
<h2>Combining DNA and Language</h2>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 276 |
<figure>
|
| 277 |
<div class="lineage" role="img" aria-label="Three rows. Carbon takes DNA tokens and outputs DNA tokens. Qwen3 takes text tokens and outputs text tokens. Carbon-X takes mixed text and DNA tokens and outputs mixed text and DNA tokens.">
|
| 278 |
-
<div class="row"><div class="io"><span class="tk dna">ATGCGC</span><span class="tk dna">TAGCTA</span><span class="tk dna">ACGTAC</span></div><span class="arr">→</span><div class="model">Carbon</div><span class="arr">→</span><div class="io"><span class="tk dna">GCTAGC</span><span class="tk dna">TCGATC</span></div></div>
|
| 279 |
-
<div class="row"><div class="io"><span class="tk txt">What</span><span class="tk txt">is</span><span class="tk txt">DNA</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Qwen3</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">molecule</span><span class="tk txt">that</span><span class="tk txt">…</span></div></div>
|
| 280 |
<div class="row cx"><div class="io"><span class="tk txt">What</span><span class="tk txt">does</span><span class="tk dna">TATAAA</span><span class="tk dna">GCTAGC</span><span class="tk txt">do</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Carbon-X</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">promoter:</span><span class="tk dna">TATAAA</span><span class="tk txt">…</span></div></div>
|
| 281 |
</div>
|
| 282 |
-
<figcaption><b>Figure 2.</b> The evolution from pure DNA models like Carbon and pure text models such as
|
| 283 |
</figure>
|
| 284 |
-
<p>
|
| 285 |
-
<p>The picture is very different with DNA: it’s not a language constructed by us, so our understanding is very limited. We hope that training models on DNA will allow them to learn patterns we already understand and uncover new insights into its structure and function.</p>
|
| 286 |
-
<p>DNA encodes biological instructions in an alphabet of four letters. Genomic foundation models, such as our earlier <a class="inline" href="https://huggingface.co/HuggingFaceBio/Carbon-3B">Carbon-3B</a>, learn patterns from large collections of DNA sequences, just as language models learn patterns from text. Researchers can use them to score sequences, estimate the effects of mutations, or predict which DNA bases come next. However, these models do not speak natural language: they cannot communicate any deeper or novel insights into the DNA structure.</p>
|
| 287 |
-
<p>But DNA is not a self-contained object, so a pure DNA model cannot fully model an organism’s shape or inner workings from its bases alone. Its effect is coupled to the laws of nature which are intricate and require observation of the world. Over centuries, humans have made such observations and built a written body of knowledge about how nature works, including the role of DNA in biology.</p>
|
| 288 |
-
<p>In contrast to genomic foundation models, language models trained mainly on text can explain biological concepts, reason to experiments and discuss findings from scientific literature. But they learn biology largely through descriptions and explanations, with little direct training on DNA sequences themselves.</p>
|
| 289 |
-
<p>Bringing these strengths together would let a model connect patterns learned from DNA with knowledge accumulated through decades of experiments, recorded in papers and databases. Researchers could then ask questions about a specific sequence and discuss its biological significance in natural language.</p>
|
| 290 |
-
<p>Earlier work has explored this connection by treating DNA as a separate modality, pairing a DNA encoder with a language-model decoder <a class="inline" href="https://arxiv.org/abs/2505.23579">[1]</a>. With Carbon-X, we instead train a single 14B decoder on both DNA and English, reaching performance competitive with frontier LLMs on several DNA benchmarks.</p>
|
| 291 |
<figure>
|
| 292 |
<div class="arch-wrap"><div class="arch" role="img" aria-label="Left to right. The input 'Describe this sequence: ACGTACGATTCA' is split by a hybrid tokenizer into Qwen3 BPE text tokens and DNA 6-mer tokens wrapped in dna tags. Each token gets a vector from one shared embedding matrix: Qwen3's text embeddings plus new DNA embeddings. The vectors pass through the Qwen3-14B decoder layers and LM head, which output an interleaved sequence of text and DNA tokens.">
|
| 293 |
|
|
@@ -333,7 +345,7 @@
|
|
| 333 |
|
| 334 |
<section>
|
| 335 |
<h2>Predicting the Effect of Mutations with Carbon-X</h2>
|
| 336 |
-
<p>Given an 8,192-base window of human DNA with a single A→G change, Carbon-X correctly identifies the variant as pathogenic and explains why: the change disrupts the AG at a splice site.
|
| 337 |
<figure>
|
| 338 |
<iframe class="fig" src="figures/vep-comparison.html" title="One variant, every model: NF1 c.6007-2A>G" loading="lazy"></iframe>
|
| 339 |
<figcaption><b>Figure 4.</b> One variant, every model: NF1 c.6007-2A>G.</figcaption>
|
|
|
|
| 122 |
/* collapsible boxes inside the recipe */
|
| 123 |
details.box { border: 1px solid var(--rule); border-radius: 6px; padding: 0 16px; margin: 24px 0; }
|
| 124 |
details.box:last-of-type { border-bottom: 1px solid var(--rule); }
|
| 125 |
+
figure:has(+ details.box) { margin-bottom: 14px; }
|
| 126 |
details.box summary { font-weight: 600; padding: 14px 30px 14px 0; }
|
| 127 |
details.box summary::after { top: 10px; }
|
| 128 |
details.box[open] summary { border-bottom: 1px solid var(--rule); margin-bottom: 14px; }
|
|
|
|
| 233 |
</div>
|
| 234 |
<p class="hl-lead"><b>TL;DR</b></p>
|
| 235 |
<ul class="clean highlights">
|
| 236 |
+
<li><b>Best open model.</b> Ahead of every open model on DNA benchmarks.</li>
|
| 237 |
+
<li><b>Frontier performance, 1,000× cheaper.</b> Within a point of GPT-6.1 Sol on our DNA benchmarks.</li>
|
|
|
|
| 238 |
<li><b>Open and local.</b> Apache 2.0, runs on your own hardware.</li>
|
| 239 |
<li><b>DNA and English in one model.</b> Reads and writes both natively.</li>
|
| 240 |
<li><b>Integrates into agents.</b> Works as a DNA sub-agent that general-purpose agents can delegate sequence reading to. <span class="tbd">add result: Kimi K3 alone vs Kimi K3 + Carbon-X</span></li>
|
| 241 |
</ul>
|
| 242 |
+
<p>Averaged over five DNA benchmarks (our DNA intelligence index), Carbon-X lands next to GPT-6.1 Sol (66.7% vs 67.5%) at about 1/1,800 of the cost per problem, and 27 points above Qwen3-14B, the model we started from.</p>
|
| 243 |
<!-- HERO FIGURE: native SVG, drawn by the script at the end of the page -->
|
| 244 |
<figure class="hero">
|
| 245 |
+
<div class="figtitle">The best open model for DNA understanding</div>
|
| 246 |
<div class="chart" id="fig1"><noscript><img src="assets/fig1-score-vs-cost.png" width="1135" height="696" alt="Average DNA benchmark score against cost per problem relative to Carbon-X."></noscript></div>
|
| 247 |
+
<figcaption><b>Figure 1.</b> Each model is shown at the reasoning effort with its highest average score. Costs use standard API pricing; for Carbon-X, measured throughput per H200-hour at $5/hour.</figcaption>
|
| 248 |
</figure>
|
| 249 |
<p>The average hides where each model is strong. Carbon-X is the best model we tested, open or closed, at predicting the effects of mutations in protein-coding regions and at predicting the next bases of a sequence. Frontier models still lead on the other three tasks: complex variants, single-base variant effects and DNA understanding.</p>
|
| 250 |
<!-- Data: carbon-x-blog-figures/benchmark-scores.html (leaderboard snapshot 2026-10-21T13:31:46Z). Each model uses the
|
|
|
|
| 264 |
</table></div>
|
| 265 |
<figcaption><b>Table 1.</b> DNA benchmark scores (%). Each model is shown at the reasoning effort with its highest average score; the best score on each benchmark is shaded.</figcaption>
|
| 266 |
</figure>
|
| 267 |
+
<!-- TO CHECK before publishing: FrontierVEP metric (plain vs balanced accuracy), whether "up to 8,192 bases"
|
| 268 |
+
holds for the whole benchmark, and whether the decontamination sentence is still accurate. -->
|
| 269 |
+
<details class="box">
|
| 270 |
+
<summary>About the five benchmarks <span class="tbd">double-check: FrontierVEP metric, 8,192-base length, decontamination</span></summary>
|
| 271 |
+
<div class="dbody">
|
| 272 |
+
<ul class="clean">
|
| 273 |
+
<li><b>BioReason Coding:</b> compare a reference and a mutated protein-coding sequence and predict whether the mutation is harmful. Balanced accuracy.</li>
|
| 274 |
+
<li><b>BioReason Non-SNV:</b> predict whether changes other than single-base swaps, such as insertions and deletions, are benign or harmful. Balanced accuracy.</li>
|
| 275 |
+
<li><b>GenomeQA:</b> identify biological features, such as regions that control gene activity, and classify sequences by organism group. Average of six tasks.</li>
|
| 276 |
+
<li><b>NextBase:</b> predict the next bases of a DNA sequence. Per-base accuracy.</li>
|
| 277 |
+
<li><b>FrontierVEP:</b> given a long reference sequence (up to 8,192 bases) and a single-base change, predict whether it is benign or pathogenic. Accuracy.</li>
|
| 278 |
+
</ul>
|
| 279 |
+
<p>BioReason and GenomeQA come from the literature; FrontierVEP and NextBase were built by us. All five are decontaminated against the training data (DNA overlap, text overlap and homology checks).</p>
|
| 280 |
+
</div>
|
| 281 |
+
</details>
|
| 282 |
+
<p>We built Carbon-X by teaching Qwen3-14B DNA as a new modality during mid-training, then post-training it to answer questions about sequences. Below we explain how.</p>
|
| 283 |
</header>
|
| 284 |
|
| 285 |
<main class="col">
|
|
|
|
| 287 |
|
| 288 |
<section>
|
| 289 |
<h2>Combining DNA and Language</h2>
|
| 290 |
+
<p>When we train language models on text, we expect them to learn its underlying patterns. At best, they learn its structure as well as the most skilled humans, but we don’t expect them to find patterns unknown to us.</p>
|
| 291 |
+
<p>DNA is different: it is not a language we constructed, and our understanding of it is limited. Training models on DNA could let them learn the patterns we already know and uncover new ones in its structure and function.</p>
|
| 292 |
+
<p>DNA encodes biological instructions in an alphabet of four letters. Genomic foundation models, such as our earlier <a class="inline" href="https://huggingface.co/HuggingFaceBio/Carbon-3B">Carbon-3B</a>, learn patterns from large collections of DNA sequences, just as language models learn patterns from text. Researchers use them to score sequences, estimate the effects of mutations, or predict which bases come next, but these models cannot explain a prediction or answer a question about a sequence.</p>
|
| 293 |
+
<p>Language models trained mainly on text can explain biological concepts, reason about experiments and discuss findings from the literature. But they learn biology largely through descriptions, with little direct exposure to DNA itself. Bringing the two together would let a model connect patterns in DNA with decades of knowledge recorded in papers and databases, so researchers can ask about a specific sequence and discuss what it means.</p>
|
| 294 |
<figure>
|
| 295 |
<div class="lineage" role="img" aria-label="Three rows. Carbon takes DNA tokens and outputs DNA tokens. Qwen3 takes text tokens and outputs text tokens. Carbon-X takes mixed text and DNA tokens and outputs mixed text and DNA tokens.">
|
| 296 |
+
<div class="row"><div class="io"><span class="tk dna">ATGCGC</span><span class="tk dna">TAGCTA</span><span class="tk dna">ACGTAC</span></div><span class="arr">→</span><div class="model">Carbon-3B</div><span class="arr">→</span><div class="io"><span class="tk dna">GCTAGC</span><span class="tk dna">TCGATC</span></div></div>
|
| 297 |
+
<div class="row"><div class="io"><span class="tk txt">What</span><span class="tk txt">is</span><span class="tk txt">DNA</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Qwen3-14B</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">molecule</span><span class="tk txt">that</span><span class="tk txt">…</span></div></div>
|
| 298 |
<div class="row cx"><div class="io"><span class="tk txt">What</span><span class="tk txt">does</span><span class="tk dna">TATAAA</span><span class="tk dna">GCTAGC</span><span class="tk txt">do</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Carbon-X</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">promoter:</span><span class="tk dna">TATAAA</span><span class="tk txt">…</span></div></div>
|
| 299 |
</div>
|
| 300 |
+
<figcaption><b>Figure 2.</b> The evolution from pure DNA models like Carbon-3B and pure text models such as Qwen3-14B to the hybrid Carbon-X model.</figcaption>
|
| 301 |
</figure>
|
| 302 |
+
<p>Earlier work connected the two by pairing a separate DNA encoder with a language-model decoder <a class="inline" href="https://arxiv.org/abs/2505.23579">[1]</a>. Carbon-X instead trains a single 14B decoder on both DNA and English: DNA is simply another set of tokens in the same model.</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 303 |
<figure>
|
| 304 |
<div class="arch-wrap"><div class="arch" role="img" aria-label="Left to right. The input 'Describe this sequence: ACGTACGATTCA' is split by a hybrid tokenizer into Qwen3 BPE text tokens and DNA 6-mer tokens wrapped in dna tags. Each token gets a vector from one shared embedding matrix: Qwen3's text embeddings plus new DNA embeddings. The vectors pass through the Qwen3-14B decoder layers and LM head, which output an interleaved sequence of text and DNA tokens.">
|
| 305 |
|
|
|
|
| 345 |
|
| 346 |
<section>
|
| 347 |
<h2>Predicting the Effect of Mutations with Carbon-X</h2>
|
| 348 |
+
<p>Given an 8,192-base window of human DNA with a single A→G change, Carbon-X correctly identifies the variant as pathogenic and explains why: the change disrupts the AG at a splice site. Of the other models, only GPT-6.1 Sol at maximum effort got it right; Opus 5.5 reached the right label for the wrong reason, and the rest answered benign, were blocked, or ran out of tokens.</p>
|
| 349 |
<figure>
|
| 350 |
<iframe class="fig" src="figures/vep-comparison.html" title="One variant, every model: NF1 c.6007-2A>G" loading="lazy"></iframe>
|
| 351 |
<figcaption><b>Figure 4.</b> One variant, every model: NF1 c.6007-2A>G.</figcaption>
|