Refine release highlights and Figure 1 caption

#1
by lewtun HF Staff - opened
Files changed (1) hide show
  1. index.html +31 -19
index.html CHANGED
@@ -122,6 +122,7 @@
122
  /* collapsible boxes inside the recipe */
123
  details.box { border: 1px solid var(--rule); border-radius: 6px; padding: 0 16px; margin: 24px 0; }
124
  details.box:last-of-type { border-bottom: 1px solid var(--rule); }
 
125
  details.box summary { font-weight: 600; padding: 14px 30px 14px 0; }
126
  details.box summary::after { top: 10px; }
127
  details.box[open] summary { border-bottom: 1px solid var(--rule); margin-bottom: 14px; }
@@ -232,19 +233,18 @@
232
  </div>
233
  <p class="hl-lead"><b>TL;DR</b></p>
234
  <ul class="clean highlights">
235
- <li><b>Best open model.</b> Ahead of every open model on all five DNA benchmarks.</li>
236
- <li><b>Frontier performance, 1,000× cheaper.</b> 66.7% vs GPT-6.1 Sol’s 67.5%.</li>
237
- <li><b>Best on two tasks.</b> Ahead of all models tested, closed ones included, on coding-variant effects and next-base prediction.</li>
238
  <li><b>Open and local.</b> Apache 2.0, runs on your own hardware.</li>
239
  <li><b>DNA and English in one model.</b> Reads and writes both natively.</li>
240
  <li><b>Integrates into agents.</b> Works as a DNA sub-agent that general-purpose agents can delegate sequence reading to. <span class="tbd">add result: Kimi K3 alone vs Kimi K3 + Carbon-X</span></li>
241
  </ul>
242
- <p>Averaged over five DNA benchmarks, Carbon-X lands next to GPT-6.1 Sol (66.7% vs 67.5%) at about 1/1,800 of the cost per problem, and well ahead of every other open model, including Qwen3-14B, the model we started from.</p>
243
  <!-- HERO FIGURE: native SVG, drawn by the script at the end of the page -->
244
  <figure class="hero">
245
- <div class="figtitle">The best open model for DNA intelligence</div>
246
  <div class="chart" id="fig1"><noscript><img src="assets/fig1-score-vs-cost.png" width="1135" height="696" alt="Average DNA benchmark score against cost per problem relative to Carbon-X."></noscript></div>
247
- <figcaption><b>Figure 1.</b> Carbon-X outperforms leading open models across five tasks of DNA understanding and generation, at a fraction of the cost. Each comparison model is shown at the reasoning effort (low/max) that delivers its highest average score. Cost estimates are based on standard API pricing, while for Carbon-X we calculate the cost by using the model’s throughput per H200-hour and $5/hour H200 price.</figcaption>
248
  </figure>
249
  <p>The average hides where each model is strong. Carbon-X is the best model we tested, open or closed, at predicting the effects of mutations in protein-coding regions and at predicting the next bases of a sequence. Frontier models still lead on the other three tasks: complex variants, single-base variant effects and DNA understanding.</p>
250
  <!-- Data: carbon-x-blog-figures/benchmark-scores.html (leaderboard snapshot 2026-10-21T13:31:46Z). Each model uses the
@@ -264,8 +264,22 @@
264
  </table></div>
265
  <figcaption><b>Table 1.</b> DNA benchmark scores (%). Each model is shown at the reasoning effort with its highest average score; the best score on each benchmark is shaded.</figcaption>
266
  </figure>
267
- <p><span class="tbd">collapsed box to explain the evals?</span></p>
268
- <p>We built Carbon-X by teaching Qwen3-14B DNA as a new modality during mid-training, then post-training it to answer questions about sequences. The weights are available under Apache 2.0; the rest of this post explains how.</p>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
269
  </header>
270
 
271
  <main class="col">
@@ -273,21 +287,19 @@
273
 
274
  <section>
275
  <h2>Combining DNA and Language</h2>
 
 
 
 
276
  <figure>
277
  <div class="lineage" role="img" aria-label="Three rows. Carbon takes DNA tokens and outputs DNA tokens. Qwen3 takes text tokens and outputs text tokens. Carbon-X takes mixed text and DNA tokens and outputs mixed text and DNA tokens.">
278
- <div class="row"><div class="io"><span class="tk dna">ATGCGC</span><span class="tk dna">TAGCTA</span><span class="tk dna">ACGTAC</span></div><span class="arr">→</span><div class="model">Carbon</div><span class="arr">→</span><div class="io"><span class="tk dna">GCTAGC</span><span class="tk dna">TCGATC</span></div></div>
279
- <div class="row"><div class="io"><span class="tk txt">What</span><span class="tk txt">is</span><span class="tk txt">DNA</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Qwen3</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">molecule</span><span class="tk txt">that</span><span class="tk txt">…</span></div></div>
280
  <div class="row cx"><div class="io"><span class="tk txt">What</span><span class="tk txt">does</span><span class="tk dna">TATAAA</span><span class="tk dna">GCTAGC</span><span class="tk txt">do</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Carbon-X</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">promoter:</span><span class="tk dna">TATAAA</span><span class="tk txt">…</span></div></div>
281
  </div>
282
- <figcaption><b>Figure 2.</b> The evolution from pure DNA models like Carbon and pure text models such as Qwen to the hybrid Carbon-X model.</figcaption>
283
  </figure>
284
- <p>When we train language models on texts we expect them to learn the underlying patterns. In the best case we hope that they learn the structure as well as the very best humans, but we don’t expect them to learn patterns unknown to us.</p>
285
- <p>The picture is very different with DNA: it’s not a language constructed by us, so our understanding is very limited. We hope that training models on DNA will allow them to learn patterns we already understand and uncover new insights into its structure and function.</p>
286
- <p>DNA encodes biological instructions in an alphabet of four letters. Genomic foundation models, such as our earlier <a class="inline" href="https://huggingface.co/HuggingFaceBio/Carbon-3B">Carbon-3B</a>, learn patterns from large collections of DNA sequences, just as language models learn patterns from text. Researchers can use them to score sequences, estimate the effects of mutations, or predict which DNA bases come next. However, these models do not speak natural language: they cannot communicate any deeper or novel insights into the DNA structure.</p>
287
- <p>But DNA is not a self-contained object, so a pure DNA model cannot fully model an organism’s shape or inner workings from its bases alone. Its effect is coupled to the laws of nature which are intricate and require observation of the world. Over centuries, humans have made such observations and built a written body of knowledge about how nature works, including the role of DNA in biology.</p>
288
- <p>In contrast to genomic foundation models, language models trained mainly on text can explain biological concepts, reason to experiments and discuss findings from scientific literature. But they learn biology largely through descriptions and explanations, with little direct training on DNA sequences themselves.</p>
289
- <p>Bringing these strengths together would let a model connect patterns learned from DNA with knowledge accumulated through decades of experiments, recorded in papers and databases. Researchers could then ask questions about a specific sequence and discuss its biological significance in natural language.</p>
290
- <p>Earlier work has explored this connection by treating DNA as a separate modality, pairing a DNA encoder with a language-model decoder <a class="inline" href="https://arxiv.org/abs/2505.23579">[1]</a>. With Carbon-X, we instead train a single 14B decoder on both DNA and English, reaching performance competitive with frontier LLMs on several DNA benchmarks.</p>
291
  <figure>
292
  <div class="arch-wrap"><div class="arch" role="img" aria-label="Left to right. The input 'Describe this sequence: ACGTACGATTCA' is split by a hybrid tokenizer into Qwen3 BPE text tokens and DNA 6-mer tokens wrapped in dna tags. Each token gets a vector from one shared embedding matrix: Qwen3's text embeddings plus new DNA embeddings. The vectors pass through the Qwen3-14B decoder layers and LM head, which output an interleaved sequence of text and DNA tokens.">
293
 
@@ -333,7 +345,7 @@
333
 
334
  <section>
335
  <h2>Predicting the Effect of Mutations with Carbon-X</h2>
336
- <p>Given an 8,192-base window of human DNA with a single A→G change, Carbon-X correctly identifies the variant as pathogenic and explains why: the change disrupts the AG at a splice site. Most frontier models answered benign, some after reasoning about the wrong gene, while others were blocked or ran out of tokens.</p>
337
  <figure>
338
  <iframe class="fig" src="figures/vep-comparison.html" title="One variant, every model: NF1 c.6007-2A>G" loading="lazy"></iframe>
339
  <figcaption><b>Figure 4.</b> One variant, every model: NF1 c.6007-2A&gt;G.</figcaption>
 
122
  /* collapsible boxes inside the recipe */
123
  details.box { border: 1px solid var(--rule); border-radius: 6px; padding: 0 16px; margin: 24px 0; }
124
  details.box:last-of-type { border-bottom: 1px solid var(--rule); }
125
+ figure:has(+ details.box) { margin-bottom: 14px; }
126
  details.box summary { font-weight: 600; padding: 14px 30px 14px 0; }
127
  details.box summary::after { top: 10px; }
128
  details.box[open] summary { border-bottom: 1px solid var(--rule); margin-bottom: 14px; }
 
233
  </div>
234
  <p class="hl-lead"><b>TL;DR</b></p>
235
  <ul class="clean highlights">
236
+ <li><b>Best open model.</b> Ahead of every open model on DNA benchmarks.</li>
237
+ <li><b>Frontier performance, 1,000× cheaper.</b> Within a point of GPT-6.1 Sol on our DNA benchmarks.</li>
 
238
  <li><b>Open and local.</b> Apache 2.0, runs on your own hardware.</li>
239
  <li><b>DNA and English in one model.</b> Reads and writes both natively.</li>
240
  <li><b>Integrates into agents.</b> Works as a DNA sub-agent that general-purpose agents can delegate sequence reading to. <span class="tbd">add result: Kimi K3 alone vs Kimi K3 + Carbon-X</span></li>
241
  </ul>
242
+ <p>Averaged over five DNA benchmarks (our DNA intelligence index), Carbon-X lands next to GPT-6.1 Sol (66.7% vs 67.5%) at about 1/1,800 of the cost per problem, and 27 points above Qwen3-14B, the model we started from.</p>
243
  <!-- HERO FIGURE: native SVG, drawn by the script at the end of the page -->
244
  <figure class="hero">
245
+ <div class="figtitle">The best open model for DNA understanding</div>
246
  <div class="chart" id="fig1"><noscript><img src="assets/fig1-score-vs-cost.png" width="1135" height="696" alt="Average DNA benchmark score against cost per problem relative to Carbon-X."></noscript></div>
247
+ <figcaption><b>Figure 1.</b> Each model is shown at the reasoning effort with its highest average score. Costs use standard API pricing; for Carbon-X, measured throughput per H200-hour at $5/hour.</figcaption>
248
  </figure>
249
  <p>The average hides where each model is strong. Carbon-X is the best model we tested, open or closed, at predicting the effects of mutations in protein-coding regions and at predicting the next bases of a sequence. Frontier models still lead on the other three tasks: complex variants, single-base variant effects and DNA understanding.</p>
250
  <!-- Data: carbon-x-blog-figures/benchmark-scores.html (leaderboard snapshot 2026-10-21T13:31:46Z). Each model uses the
 
264
  </table></div>
265
  <figcaption><b>Table 1.</b> DNA benchmark scores (%). Each model is shown at the reasoning effort with its highest average score; the best score on each benchmark is shaded.</figcaption>
266
  </figure>
267
+ <!-- TO CHECK before publishing: FrontierVEP metric (plain vs balanced accuracy), whether "up to 8,192 bases"
268
+ holds for the whole benchmark, and whether the decontamination sentence is still accurate. -->
269
+ <details class="box">
270
+ <summary>About the five benchmarks <span class="tbd">double-check: FrontierVEP metric, 8,192-base length, decontamination</span></summary>
271
+ <div class="dbody">
272
+ <ul class="clean">
273
+ <li><b>BioReason Coding:</b> compare a reference and a mutated protein-coding sequence and predict whether the mutation is harmful. Balanced accuracy.</li>
274
+ <li><b>BioReason Non-SNV:</b> predict whether changes other than single-base swaps, such as insertions and deletions, are benign or harmful. Balanced accuracy.</li>
275
+ <li><b>GenomeQA:</b> identify biological features, such as regions that control gene activity, and classify sequences by organism group. Average of six tasks.</li>
276
+ <li><b>NextBase:</b> predict the next bases of a DNA sequence. Per-base accuracy.</li>
277
+ <li><b>FrontierVEP:</b> given a long reference sequence (up to 8,192 bases) and a single-base change, predict whether it is benign or pathogenic. Accuracy.</li>
278
+ </ul>
279
+ <p>BioReason and GenomeQA come from the literature; FrontierVEP and NextBase were built by us. All five are decontaminated against the training data (DNA overlap, text overlap and homology checks).</p>
280
+ </div>
281
+ </details>
282
+ <p>We built Carbon-X by teaching Qwen3-14B DNA as a new modality during mid-training, then post-training it to answer questions about sequences. Below we explain how.</p>
283
  </header>
284
 
285
  <main class="col">
 
287
 
288
  <section>
289
  <h2>Combining DNA and Language</h2>
290
+ <p>When we train language models on text, we expect them to learn its underlying patterns. At best, they learn its structure as well as the most skilled humans, but we don’t expect them to find patterns unknown to us.</p>
291
+ <p>DNA is different: it is not a language we constructed, and our understanding of it is limited. Training models on DNA could let them learn the patterns we already know and uncover new ones in its structure and function.</p>
292
+ <p>DNA encodes biological instructions in an alphabet of four letters. Genomic foundation models, such as our earlier <a class="inline" href="https://huggingface.co/HuggingFaceBio/Carbon-3B">Carbon-3B</a>, learn patterns from large collections of DNA sequences, just as language models learn patterns from text. Researchers use them to score sequences, estimate the effects of mutations, or predict which bases come next, but these models cannot explain a prediction or answer a question about a sequence.</p>
293
+ <p>Language models trained mainly on text can explain biological concepts, reason about experiments and discuss findings from the literature. But they learn biology largely through descriptions, with little direct exposure to DNA itself. Bringing the two together would let a model connect patterns in DNA with decades of knowledge recorded in papers and databases, so researchers can ask about a specific sequence and discuss what it means.</p>
294
  <figure>
295
  <div class="lineage" role="img" aria-label="Three rows. Carbon takes DNA tokens and outputs DNA tokens. Qwen3 takes text tokens and outputs text tokens. Carbon-X takes mixed text and DNA tokens and outputs mixed text and DNA tokens.">
296
+ <div class="row"><div class="io"><span class="tk dna">ATGCGC</span><span class="tk dna">TAGCTA</span><span class="tk dna">ACGTAC</span></div><span class="arr">→</span><div class="model">Carbon-3B</div><span class="arr">→</span><div class="io"><span class="tk dna">GCTAGC</span><span class="tk dna">TCGATC</span></div></div>
297
+ <div class="row"><div class="io"><span class="tk txt">What</span><span class="tk txt">is</span><span class="tk txt">DNA</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Qwen3-14B</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">molecule</span><span class="tk txt">that</span><span class="tk txt">…</span></div></div>
298
  <div class="row cx"><div class="io"><span class="tk txt">What</span><span class="tk txt">does</span><span class="tk dna">TATAAA</span><span class="tk dna">GCTAGC</span><span class="tk txt">do</span><span class="tk txt">?</span></div><span class="arr">→</span><div class="model">Carbon-X</div><span class="arr">→</span><div class="io"><span class="tk txt">A</span><span class="tk txt">promoter:</span><span class="tk dna">TATAAA</span><span class="tk txt">…</span></div></div>
299
  </div>
300
+ <figcaption><b>Figure 2.</b> The evolution from pure DNA models like Carbon-3B and pure text models such as Qwen3-14B to the hybrid Carbon-X model.</figcaption>
301
  </figure>
302
+ <p>Earlier work connected the two by pairing a separate DNA encoder with a language-model decoder <a class="inline" href="https://arxiv.org/abs/2505.23579">[1]</a>. Carbon-X instead trains a single 14B decoder on both DNA and English: DNA is simply another set of tokens in the same model.</p>
 
 
 
 
 
 
303
  <figure>
304
  <div class="arch-wrap"><div class="arch" role="img" aria-label="Left to right. The input 'Describe this sequence: ACGTACGATTCA' is split by a hybrid tokenizer into Qwen3 BPE text tokens and DNA 6-mer tokens wrapped in dna tags. Each token gets a vector from one shared embedding matrix: Qwen3's text embeddings plus new DNA embeddings. The vectors pass through the Qwen3-14B decoder layers and LM head, which output an interleaved sequence of text and DNA tokens.">
305
 
 
345
 
346
  <section>
347
  <h2>Predicting the Effect of Mutations with Carbon-X</h2>
348
+ <p>Given an 8,192-base window of human DNA with a single A→G change, Carbon-X correctly identifies the variant as pathogenic and explains why: the change disrupts the AG at a splice site. Of the other models, only GPT-6.1 Sol at maximum effort got it right; Opus 5.5 reached the right label for the wrong reason, and the rest answered benign, were blocked, or ran out of tokens.</p>
349
  <figure>
350
  <iframe class="fig" src="figures/vep-comparison.html" title="One variant, every model: NF1 c.6007-2A>G" loading="lazy"></iframe>
351
  <figcaption><b>Figure 4.</b> One variant, every model: NF1 c.6007-2A&gt;G.</figcaption>