appvoid commited on
Commit
f7534c8
·
verified ·
1 Parent(s): 1e700b9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -10
README.md CHANGED
@@ -3,13 +3,16 @@ library_name: transformers
3
  pipeline_tag: text-generation
4
  tags: [qwen3, byte-level, pretraining]
5
  ---
6
- # Qwen90M Byte
7
- 24 vanilla Qwen3 layers, width 640, MLP 1440, 10 query / 2 KV heads, head dimension 64.
8
- 90,148,352 tied-embedding parameters. 259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258.
9
- Trained from scratch; this is a base model, not a Qwen instruction checkpoint.
10
-
11
- Load the model with `AutoModelForCausalLM.from_pretrained(repo)`.
12
- Load its custom byte tokenizer with `AutoTokenizer.from_pretrained(repo, trust_remote_code=True)`.
13
- Training records begin with BOS; prepend `tokenizer.bos_token_id` to encoded generation prompts.
14
- See training_config.json and coverage.json for the mixture and consumed data statistics.
15
- Model weights are FP32 masters trained under BF16 autocast. Use dtype=torch.bfloat16 for GPU inference.
 
 
 
 
3
  pipeline_tag: text-generation
4
  tags: [qwen3, byte-level, pretraining]
5
  ---
6
+ # void.1
7
+
8
+ ![logo](https://huggingface.co/dotlabs/void.1/resolve/main/void.1-logo.png)
9
+
10
+ # details
11
+
12
+ 90,148,352 tied-embedding parameters.
13
+
14
+ 259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258.
15
+
16
+ pretrained from scratch on cortex (our internal symbolic kernel), rewrite, climbmix, openmath and ultrafineweb.
17
+
18
+ we used revision `92c601560cc2179e3bfc33e067fdd3bf9254e163` for evaluations, for more details on the evals, please have a look at the official notebook.