appvoid commited on
Commit
e4eef4a
·
verified ·
1 Parent(s): 572d5ed

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +24 -5
README.md CHANGED
@@ -9,12 +9,31 @@ tags: [qwen3, byte-level, pretraining]
9
 
10
  # details
11
 
12
- state of the art byte-level language model pretrained from scratch on rewrite, climbmix, openmath, ultrafineweb and cortex (our internal symbolic kernel).
13
-
14
- 90,148,352 tied-embedding parameters.
15
 
16
  259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258.
17
 
18
- we used revision `92c601560cc2179e3bfc33e067fdd3bf9254e163` (step 900,000) for evaluations.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
- for more details on the evals, please have a look at the official notebook.
 
9
 
10
  # details
11
 
12
+ State of the art, small language model pretrained from scratch on rewrite, climbmix, openmath, ultrafineweb and cortex (our internal symbolic kernel).
 
 
13
 
14
  259 tokens: bytes 0–255, PAD 256, BOS 257, EOS 258.
15
 
16
+ ## Benchmark Results
17
+
18
+ Model: `appvoid/void-byte` · Checkpoint: Step 900,000
19
+
20
+ | Benchmark | Score |
21
+ | ---------------------------- | ------- |
22
+ | HellaSwag | 38.68% |
23
+ | PIQA | 67.46% |
24
+ | ARC-Easy | 47.31% |
25
+ | ARC-Challenge | 28.16% |
26
+ | ArithMark-3 | 44.80% |
27
+ | Intelligence Index | 23.9182 |
28
+ | Benchmark Max Length | 2,048 |
29
+
30
+
31
+ | Property | Value |
32
+ | ------------ | ------------------------------------------------------------- |
33
+ | Parameters | 90,148,352 - tied-embedding |
34
+ | Architecture | Byte-level |
35
+ | Revision | `92c601560cc2179e3bfc33e067fdd3bf9254e163` |
36
+
37
+ We used the revision on step 900,000 for evaluations which trained for around 120 billion bytes which is around 30 to 35 billion bpe tokens.
38
 
39
+ For more details on the evals, please have a look at the official notebook.