Text Generation

Add pipeline tag and project/code links

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +7 -2
README.md CHANGED
@@ -1,12 +1,17 @@
1
  ---
2
- license: apache-2.0
3
  datasets:
4
  - HuggingFaceFW/fineweb-edu
 
 
5
  ---
 
6
  Checkpoints evaluated in the paper "Graph Machine: Towards Better Pretraining via Edges" (https://arxiv.org/abs/2609.02881):
7
 
8
  We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing.
9
  Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1).
10
  Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing.
11
  We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens.
12
- With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
 
 
 
 
1
  ---
 
2
  datasets:
3
  - HuggingFaceFW/fineweb-edu
4
+ license: apache-2.0
5
+ pipeline_tag: text-generation
6
  ---
7
+
8
  Checkpoints evaluated in the paper "Graph Machine: Towards Better Pretraining via Edges" (https://arxiv.org/abs/2609.02881):
9
 
10
  We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing.
11
  Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1).
12
  Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing.
13
  We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens.
14
+ With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
15
+
16
+ Project page: https://lintaihou.com/research/graph-machine-2
17
+ Code: https://github.com/lintaihou/gm2