lewtun HF Staff Codex commited on
Commit
372d19f
·
1 Parent(s): d93fcd2

Tighten hybrid tokenizer introduction wording

Browse files

Co-authored-by: Codex <codex@openai.com>

Files changed (1) hide show
  1. index.html +1 -1
index.html CHANGED
@@ -361,7 +361,7 @@
361
  <h2>How we built Carbon-X</h2>
362
  <p>We built Carbon-X by adapting <a class="inline" href="https://huggingface.co/Qwen/Qwen3-14B">Qwen3-14B</a> to DNA through midtraining, supervised fine-tuning, and reinforcement learning. Three challenges shaped the recipe: learning from genomes without losing language ability, building reliable supervision where existing models were weak teachers, and turning sequence knowledge into better answers using verifiable rewards.</p>
363
  <h3>Learning DNA without losing language</h3>
364
- <p><strong>Teaching Qwen3 DNA required balancing new capabilities against the language skills it already had.</strong> We started with a hybrid tokenizer, extending Qwen3’s vocabulary with 4,096 six-base DNA tokens, plus tokens for sequence boundaries and partial sequences, while preserving its original tokenizer for text. This gives genomic sequences a consistent representation instead of the irregular fragments produced by text tokenization. Six-base tokens also encode DNA more compactly than text-oriented BPE tokenizers, using roughly a third as many tokens as Qwen3’s tokenizer.</p>
365
  <p>The design builds on our earlier <a class="inline" href="https://www.biorxiv.org/content/10.64898/2026.05.22.727119v1">Carbon-3B</a> results, where six-base tokenization outperformed byte-pair encoding, and follows <a class="inline" href="https://arxiv.org/abs/2502.07527">NatureLM</a>’s approach to combining biological sequences with text. Carbon-X processes both token types within a single model and can generate responses containing DNA and natural language.</p>
366
  <figure>
367
  <iframe class="fig" src="figures/tokenizer.html" title="Qwen3 tokenizer vs Carbon-X tokenizer" loading="lazy"></iframe>
 
361
  <h2>How we built Carbon-X</h2>
362
  <p>We built Carbon-X by adapting <a class="inline" href="https://huggingface.co/Qwen/Qwen3-14B">Qwen3-14B</a> to DNA through midtraining, supervised fine-tuning, and reinforcement learning. Three challenges shaped the recipe: learning from genomes without losing language ability, building reliable supervision where existing models were weak teachers, and turning sequence knowledge into better answers using verifiable rewards.</p>
363
  <h3>Learning DNA without losing language</h3>
364
+ <p><strong>Teaching Qwen3 DNA required balancing new capabilities against the language skills it already had.</strong> We started with a hybrid tokenizer, extending Qwen3’s vocabulary with 4,096 six-base DNA tokens, plus tokens for sequence boundaries and partial sequences. This gives genomic sequences a consistent representation instead of the irregular fragments produced by text tokenization. Six-base tokens also encode DNA more compactly than BPE tokenizers, using roughly a third as many tokens as Qwen3’s tokenizer.</p>
365
  <p>The design builds on our earlier <a class="inline" href="https://www.biorxiv.org/content/10.64898/2026.05.22.727119v1">Carbon-3B</a> results, where six-base tokenization outperformed byte-pair encoding, and follows <a class="inline" href="https://arxiv.org/abs/2502.07527">NatureLM</a>’s approach to combining biological sequences with text. Carbon-X processes both token types within a single model and can generate responses containing DNA and natural language.</p>
366
  <figure>
367
  <iframe class="fig" src="figures/tokenizer.html" title="Qwen3 tokenizer vs Carbon-X tokenizer" loading="lazy"></iframe>