Tighten hybrid tokenizer introduction wording
Browse filesCo-authored-by: Codex <codex@openai.com>
- index.html +1 -1
index.html
CHANGED
|
@@ -361,7 +361,7 @@
|
|
| 361 |
<h2>How we built Carbon-X</h2>
|
| 362 |
<p>We built Carbon-X by adapting <a class="inline" href="https://huggingface.co/Qwen/Qwen3-14B">Qwen3-14B</a> to DNA through midtraining, supervised fine-tuning, and reinforcement learning. Three challenges shaped the recipe: learning from genomes without losing language ability, building reliable supervision where existing models were weak teachers, and turning sequence knowledge into better answers using verifiable rewards.</p>
|
| 363 |
<h3>Learning DNA without losing language</h3>
|
| 364 |
-
<p><strong>Teaching Qwen3 DNA required balancing new capabilities against the language skills it already had.</strong> We started with a hybrid tokenizer, extending Qwen3’s vocabulary with 4,096 six-base DNA tokens, plus tokens for sequence boundaries and partial sequences
|
| 365 |
<p>The design builds on our earlier <a class="inline" href="https://www.biorxiv.org/content/10.64898/2026.05.22.727119v1">Carbon-3B</a> results, where six-base tokenization outperformed byte-pair encoding, and follows <a class="inline" href="https://arxiv.org/abs/2502.07527">NatureLM</a>’s approach to combining biological sequences with text. Carbon-X processes both token types within a single model and can generate responses containing DNA and natural language.</p>
|
| 366 |
<figure>
|
| 367 |
<iframe class="fig" src="figures/tokenizer.html" title="Qwen3 tokenizer vs Carbon-X tokenizer" loading="lazy"></iframe>
|
|
|
|
| 361 |
<h2>How we built Carbon-X</h2>
|
| 362 |
<p>We built Carbon-X by adapting <a class="inline" href="https://huggingface.co/Qwen/Qwen3-14B">Qwen3-14B</a> to DNA through midtraining, supervised fine-tuning, and reinforcement learning. Three challenges shaped the recipe: learning from genomes without losing language ability, building reliable supervision where existing models were weak teachers, and turning sequence knowledge into better answers using verifiable rewards.</p>
|
| 363 |
<h3>Learning DNA without losing language</h3>
|
| 364 |
+
<p><strong>Teaching Qwen3 DNA required balancing new capabilities against the language skills it already had.</strong> We started with a hybrid tokenizer, extending Qwen3’s vocabulary with 4,096 six-base DNA tokens, plus tokens for sequence boundaries and partial sequences. This gives genomic sequences a consistent representation instead of the irregular fragments produced by text tokenization. Six-base tokens also encode DNA more compactly than BPE tokenizers, using roughly a third as many tokens as Qwen3’s tokenizer.</p>
|
| 365 |
<p>The design builds on our earlier <a class="inline" href="https://www.biorxiv.org/content/10.64898/2026.05.22.727119v1">Carbon-3B</a> results, where six-base tokenization outperformed byte-pair encoding, and follows <a class="inline" href="https://arxiv.org/abs/2502.07527">NatureLM</a>’s approach to combining biological sequences with text. Carbon-X processes both token types within a single model and can generate responses containing DNA and natural language.</p>
|
| 366 |
<figure>
|
| 367 |
<iframe class="fig" src="figures/tokenizer.html" title="Qwen3 tokenizer vs Carbon-X tokenizer" loading="lazy"></iframe>
|