User01110 commited on
Commit
e78ca88
·
verified ·
1 Parent(s): c148ec7

Remove former organization references from model card

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -20,7 +20,7 @@ language:
20
 
21
  <div align="center">
22
 
23
- # SupraCMA 8M
24
 
25
  ### Channel-Mixing Attention at 7.85 million parameters.
26
 
@@ -36,7 +36,7 @@ A compact generalist language model built to study channel mixing at small scale
36
 
37
  ## Overview
38
 
39
- SupraCMA 8M is a **7,849,161-parameter** Channel-Mixing Attention generalist
40
  causal language model trained for 40,000 steps. It pairs a compact nine-layer
41
  architecture with the unmodified native GPT-S 4,096-token tokenizer.
42
 
@@ -57,7 +57,7 @@ CMA in the feed-forward position. For every token, CMA reshapes the 288-channel
57
  hidden state into **12 chunks of 24**, routes between those chunks with **3
58
  heads**, applies a signed blend and SiLU gate, then projects back to 288 channels.
59
 
60
- ![CMA 8M architecture](https://huggingface.co/SupraLabs/cma-8M/resolve/main/assets/cma-architecture.svg?v=5)
61
 
62
  ## Quick start
63
 
@@ -66,7 +66,7 @@ heads**, applies a signed blend and SiLU gate, then projects back to 288 channel
66
  import torch
67
  from transformers import AutoModelForCausalLM, AutoTokenizer
68
 
69
- repo_id = "SupraLabs/SupraCMA-8M"
70
 
71
  tokenizer = AutoTokenizer.from_pretrained(
72
  repo_id,
@@ -135,7 +135,7 @@ Across the logged validation checkpoints, normalized bits per byte (BPB)
135
  decreased from **3.7838** to **1.3520**, while validation loss decreased from
136
  **8.3686** to **2.9903**.
137
 
138
- ![CMA 8M training trajectory](https://huggingface.co/SupraLabs/cma-8M/resolve/main/assets/training-trajectory.svg?v=5)
139
 
140
  ## Evaluation
141
 
 
20
 
21
  <div align="center">
22
 
23
+ # CMA 8M
24
 
25
  ### Channel-Mixing Attention at 7.85 million parameters.
26
 
 
36
 
37
  ## Overview
38
 
39
+ CMA 8M is a **7,849,161-parameter** Channel-Mixing Attention generalist
40
  causal language model trained for 40,000 steps. It pairs a compact nine-layer
41
  architecture with the unmodified native GPT-S 4,096-token tokenizer.
42
 
 
57
  hidden state into **12 chunks of 24**, routes between those chunks with **3
58
  heads**, applies a signed blend and SiLU gate, then projects back to 288 channels.
59
 
60
+ ![CMA 8M architecture](https://huggingface.co/User01110/CMA-8M/resolve/main/assets/cma-architecture.svg?v=5)
61
 
62
  ## Quick start
63
 
 
66
  import torch
67
  from transformers import AutoModelForCausalLM, AutoTokenizer
68
 
69
+ repo_id = "User01110/CMA-8M"
70
 
71
  tokenizer = AutoTokenizer.from_pretrained(
72
  repo_id,
 
135
  decreased from **3.7838** to **1.3520**, while validation loss decreased from
136
  **8.3686** to **2.9903**.
137
 
138
+ ![CMA 8M training trajectory](https://huggingface.co/User01110/CMA-8M/resolve/main/assets/training-trajectory.svg?v=5)
139
 
140
  ## Evaluation
141