Instructions to use CyanMonkey/Test-Subject-3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use CyanMonkey/Test-Subject-3 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="CyanMonkey/Test-Subject-3")# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Test-Subject-3", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use CyanMonkey/Test-Subject-3 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "CyanMonkey/Test-Subject-3" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "CyanMonkey/Test-Subject-3", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/CyanMonkey/Test-Subject-3
- SGLang
How to use CyanMonkey/Test-Subject-3 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "CyanMonkey/Test-Subject-3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "CyanMonkey/Test-Subject-3", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "CyanMonkey/Test-Subject-3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "CyanMonkey/Test-Subject-3", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use CyanMonkey/Test-Subject-3 with Docker Model Runner:
docker model run hf.co/CyanMonkey/Test-Subject-3
round 1 step 500 (1.75%)
Browse files- README.md +15 -15
- config.json +6 -6
- model.safetensors +1 -1
- progress.json +11 -11
- train_config.json +2 -2
- training_meta.json +53 -44
README.md
CHANGED
|
@@ -13,12 +13,12 @@ tags:
|
|
| 13 |
|
| 14 |
# Oleander-150M
|
| 15 |
|
| 16 |
-
**Training checkpoint** (`
|
| 17 |
|
| 18 |
```text
|
| 19 |
-
steps
|
| 20 |
-
tokens
|
| 21 |
-
phase tst (TST bag training ends at step
|
| 22 |
```
|
| 23 |
|
| 24 |
Trained with the Oleander marimo trainer (molab): hybrid **GDN2** (gated delta-rule, gated-delta-net-2) and **gated GQA** layers with **block attention residuals**, MuonEq-R + AdamW, WSD schedule.
|
|
@@ -27,17 +27,17 @@ Trained with the Oleander marimo trainer (molab): hybrid **GDN2** (gated delta-r
|
|
| 27 |
|
| 28 |
| | |
|
| 29 |
|---|---|
|
| 30 |
-
| Step | **
|
| 31 |
-
| % complete (steps) | **
|
| 32 |
-
| Compute tokens seen | **
|
| 33 |
-
| Raw tokens seen (TST bags unpacked) |
|
| 34 |
-
| Cluster tokens (all 1 worker(s)) |
|
| 35 |
| Phase | `tst` |
|
| 36 |
-
| Val loss / ppl | **
|
| 37 |
-
| Train loss (latest) | 5.
|
| 38 |
-
| Throughput |
|
| 39 |
-
| Session elapsed | 0d
|
| 40 |
-
| Checkpoint written | 2026-11-
|
| 41 |
|
| 42 |
## How to load
|
| 43 |
|
|
@@ -98,6 +98,6 @@ Oleander-150M (28L = 7 x [GDN2 · GDN2 · GDN2 · GQA] · d=576 · SwiGLU(1600)
|
|
| 98 |
|
| 99 |
## Notes
|
| 100 |
|
| 101 |
-
- This is a **work-in-progress** checkpoint at
|
| 102 |
- The tokenizer comes from the base model the run started from; the vocabulary is a plain BPE model and is not extended during training.
|
| 103 |
- `training_meta.json` is regenerated on every push and is the authoritative record of how far training got.
|
|
|
|
| 13 |
|
| 14 |
# Oleander-150M
|
| 15 |
|
| 16 |
+
**Training checkpoint** (`round 1`) — step **500 / 28,500** · **1.75%** of the planned run.
|
| 17 |
|
| 18 |
```text
|
| 19 |
+
steps ░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1.75% 500 / 28,500
|
| 20 |
+
tokens ░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1.75% 262.14M / 14.94B compute
|
| 21 |
+
phase tst (TST bag training ends at step 9,000)
|
| 22 |
```
|
| 23 |
|
| 24 |
Trained with the Oleander marimo trainer (molab): hybrid **GDN2** (gated delta-rule, gated-delta-net-2) and **gated GQA** layers with **block attention residuals**, MuonEq-R + AdamW, WSD schedule.
|
|
|
|
| 27 |
|
| 28 |
| | |
|
| 29 |
|---|---|
|
| 30 |
+
| Step | **500** / 28,500 |
|
| 31 |
+
| % complete (steps) | **1.75%** |
|
| 32 |
+
| Compute tokens seen | **262.14M** / 14.94B (1.75%) |
|
| 33 |
+
| Raw tokens seen (TST bags unpacked) | 1.57B / 38.54B |
|
| 34 |
+
| Cluster tokens (all 1 worker(s)) | 262.14M / 14.94B compute |
|
| 35 |
| Phase | `tst` |
|
| 36 |
+
| Val loss / ppl | **6.3458** / **570.11** |
|
| 37 |
+
| Train loss (latest) | 5.7024 |
|
| 38 |
+
| Throughput | 76,417 compute tok/s · 0.128 steps/s |
|
| 39 |
+
| Session elapsed | 0d 01h 10m · ETA 2d 12h 38m |
|
| 40 |
+
| Checkpoint written | 2026-11-14T19:44:14Z |
|
| 41 |
|
| 42 |
## How to load
|
| 43 |
|
|
|
|
| 98 |
|
| 99 |
## Notes
|
| 100 |
|
| 101 |
+
- This is a **work-in-progress** checkpoint at 1.75% of the planned run, not a finished model: expect repetition and incoherence, especially with few-shot prompts.
|
| 102 |
- The tokenizer comes from the base model the run started from; the vocabulary is a plain BPE model and is not extended during training.
|
| 103 |
- `training_meta.json` is regenerated on every push and is the authoritative record of how far training got.
|
config.json
CHANGED
|
@@ -113,12 +113,12 @@
|
|
| 113 |
},
|
| 114 |
"model_name": "Oleander-150M",
|
| 115 |
"training": {
|
| 116 |
-
"step":
|
| 117 |
-
"kind": "
|
| 118 |
-
"max_steps":
|
| 119 |
-
"percent_complete":
|
| 120 |
-
"tokens_seen":
|
| 121 |
-
"compute_tokens_seen":
|
| 122 |
},
|
| 123 |
"transformers_version": "5.14.1",
|
| 124 |
"torch_dtype": "float32"
|
|
|
|
| 113 |
},
|
| 114 |
"model_name": "Oleander-150M",
|
| 115 |
"training": {
|
| 116 |
+
"step": 500,
|
| 117 |
+
"kind": "round 1",
|
| 118 |
+
"max_steps": 28500,
|
| 119 |
+
"percent_complete": 1.754,
|
| 120 |
+
"tokens_seen": 1572864000,
|
| 121 |
+
"compute_tokens_seen": 262144000
|
| 122 |
},
|
| 123 |
"transformers_version": "5.14.1",
|
| 124 |
"torch_dtype": "float32"
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 599970860
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:26885fc07d17789bef17e9d5bf7386ca750bd6d550ba418423272159432fe842
|
| 3 |
size 599970860
|
progress.json
CHANGED
|
@@ -1,15 +1,15 @@
|
|
| 1 |
{
|
| 2 |
-
"step":
|
| 3 |
-
"max_steps":
|
| 4 |
-
"percent_complete":
|
| 5 |
-
"tokens_seen":
|
| 6 |
-
"compute_tokens_seen":
|
| 7 |
-
"compute_tokens_total":
|
| 8 |
-
"percent_tokens":
|
| 9 |
-
"val_loss":
|
| 10 |
-
"val_ppl":
|
| 11 |
"params_M": 149.976,
|
| 12 |
-
"pushed_at": "2026-11-
|
| 13 |
"architecture": "Oleander-150M",
|
| 14 |
"features": [
|
| 15 |
"GDN2 (gated delta net 2) x21: 9h x 64, conv4",
|
|
@@ -25,5 +25,5 @@
|
|
| 25 |
],
|
| 26 |
"optimizer": "muon_adamw",
|
| 27 |
"phase": "tst",
|
| 28 |
-
"kind": "
|
| 29 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"step": 500,
|
| 3 |
+
"max_steps": 28500,
|
| 4 |
+
"percent_complete": 1.754,
|
| 5 |
+
"tokens_seen": 1572864000,
|
| 6 |
+
"compute_tokens_seen": 262144000,
|
| 7 |
+
"compute_tokens_total": 14942208000,
|
| 8 |
+
"percent_tokens": 1.754,
|
| 9 |
+
"val_loss": 6.345822848379612,
|
| 10 |
+
"val_ppl": 570.1063078163132,
|
| 11 |
"params_M": 149.976,
|
| 12 |
+
"pushed_at": "2026-11-14T19:44:14Z",
|
| 13 |
"architecture": "Oleander-150M",
|
| 14 |
"features": [
|
| 15 |
"GDN2 (gated delta net 2) x21: 9h x 64, conv4",
|
|
|
|
| 25 |
],
|
| 26 |
"optimizer": "muon_adamw",
|
| 27 |
"phase": "tst",
|
| 28 |
+
"kind": "round 1"
|
| 29 |
}
|
train_config.json
CHANGED
|
@@ -155,10 +155,10 @@
|
|
| 155 |
"profile": true
|
| 156 |
},
|
| 157 |
"diloco": {
|
| 158 |
-
"enabled":
|
| 159 |
"is_leader": true,
|
| 160 |
"worker_id": 0,
|
| 161 |
-
"n_workers":
|
| 162 |
"inner_steps": 500,
|
| 163 |
"outer_lr": 0.7,
|
| 164 |
"outer_momentum": 0.9,
|
|
|
|
| 155 |
"profile": true
|
| 156 |
},
|
| 157 |
"diloco": {
|
| 158 |
+
"enabled": true,
|
| 159 |
"is_leader": true,
|
| 160 |
"worker_id": 0,
|
| 161 |
+
"n_workers": 1,
|
| 162 |
"inner_steps": 500,
|
| 163 |
"outer_lr": 0.7,
|
| 164 |
"outer_momentum": 0.9,
|
training_meta.json
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
{
|
| 2 |
"format": "oleander-training-meta/1",
|
| 3 |
-
"generated_at": "2026-11-
|
| 4 |
-
"kind": "
|
| 5 |
-
"step":
|
| 6 |
-
"tokens_seen":
|
| 7 |
-
"pushed_at": "2026-11-
|
| 8 |
"params_M": 149.976,
|
| 9 |
"architecture": "Oleander-150M",
|
| 10 |
"features": [
|
|
@@ -20,55 +20,55 @@
|
|
| 20 |
"TST (token superposition training, bag 6)"
|
| 21 |
],
|
| 22 |
"optimizer": "muon_adamw",
|
| 23 |
-
"val_loss":
|
| 24 |
-
"val_ppl":
|
| 25 |
"run": {
|
| 26 |
"model_name": "Oleander-150M",
|
| 27 |
"architecture_detail": "Oleander-150M (28L = 7 x [GDN2 \u00b7 GDN2 \u00b7 GDN2 \u00b7 GQA] \u00b7 d=576 \u00b7 SwiGLU(1600) \u00b7 tied 32,770 vocab)",
|
| 28 |
-
"session_id": "
|
| 29 |
"repo_id": "CyanMonkey/Test-Subject-3",
|
| 30 |
-
"started_at": "2026-11-
|
| 31 |
-
"session_elapsed_s":
|
| 32 |
-
"last_update_at": "2026-11-
|
| 33 |
-
"steps_per_second": 0.
|
| 34 |
-
"compute_tokens_per_second":
|
| 35 |
-
"eta": "
|
| 36 |
-
"checkpoint_kind": "
|
| 37 |
},
|
| 38 |
"progress": {
|
| 39 |
-
"step":
|
| 40 |
-
"max_steps":
|
| 41 |
-
"step_fraction": 0.
|
| 42 |
-
"percent_complete":
|
| 43 |
-
"tokens_fraction": 0.
|
| 44 |
-
"percent_tokens":
|
| 45 |
"phase": "tst",
|
| 46 |
-
"phase_fraction": 0.
|
| 47 |
-
"tst_switch_step":
|
| 48 |
"tst_switch_applied": false
|
| 49 |
},
|
| 50 |
"tokens": {
|
| 51 |
-
"compute_tokens_seen":
|
| 52 |
-
"compute_tokens_total":
|
| 53 |
-
"raw_tokens_seen":
|
| 54 |
-
"raw_tokens_total":
|
| 55 |
"raw_over_compute_ratio": 6.0,
|
| 56 |
-
"samples_seen":
|
| 57 |
-
"rows_consumed":
|
| 58 |
"compute_tokens_per_step": 524288,
|
| 59 |
"cluster": {
|
| 60 |
"n_workers": 1,
|
| 61 |
-
"compute_tokens_seen_all_workers":
|
| 62 |
-
"raw_tokens_seen_all_workers":
|
| 63 |
-
"compute_tokens_total_planned_all_workers":
|
| 64 |
-
"raw_tokens_total_planned_all_workers":
|
| 65 |
}
|
| 66 |
},
|
| 67 |
"metrics": {
|
| 68 |
-
"train_loss": 5.
|
| 69 |
-
"val_loss":
|
| 70 |
-
"val_ppl":
|
| 71 |
-
"grad_norm": 0.
|
| 72 |
"lr_mult": 1.0
|
| 73 |
},
|
| 74 |
"optimization": {
|
|
@@ -98,13 +98,13 @@
|
|
| 98 |
"stable_until": 0.825,
|
| 99 |
"decay_shape": "1-sqrt",
|
| 100 |
"min_lr_ratio": 0.0,
|
| 101 |
-
"decay_start_step":
|
| 102 |
},
|
| 103 |
"tst": {
|
| 104 |
"enabled": true,
|
| 105 |
"bag_size": 6,
|
| 106 |
"phase_fraction": 0.3,
|
| 107 |
-
"switch_step":
|
| 108 |
}
|
| 109 |
},
|
| 110 |
"model": {
|
|
@@ -151,7 +151,7 @@
|
|
| 151 |
"doc_masking": "auto",
|
| 152 |
"shuffle_block_rows": 192,
|
| 153 |
"seed": 42,
|
| 154 |
-
"epochs_consumed_this_worker":
|
| 155 |
},
|
| 156 |
"hardware": {
|
| 157 |
"device": "cuda",
|
|
@@ -162,10 +162,10 @@
|
|
| 162 |
"transformers": "5.14.1",
|
| 163 |
"gpu": "NVIDIA RTX PRO 6000 Blackwell Server Edition",
|
| 164 |
"vram_gb": 101.98,
|
| 165 |
-
"peak_gb":
|
| 166 |
},
|
| 167 |
"checkpoint": {
|
| 168 |
-
"kind": "
|
| 169 |
"weights_file": "model.safetensors",
|
| 170 |
"dtype": "float32",
|
| 171 |
"config_file": "config.json",
|
|
@@ -186,7 +186,16 @@
|
|
| 186 |
]
|
| 187 |
},
|
| 188 |
"diloco": {
|
| 189 |
-
"enabled":
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 190 |
},
|
| 191 |
-
"extra": {
|
|
|
|
|
|
|
| 192 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"format": "oleander-training-meta/1",
|
| 3 |
+
"generated_at": "2026-11-14T19:44:14Z",
|
| 4 |
+
"kind": "round 1",
|
| 5 |
+
"step": 500,
|
| 6 |
+
"tokens_seen": 1572864000,
|
| 7 |
+
"pushed_at": "2026-11-14T19:44:14Z",
|
| 8 |
"params_M": 149.976,
|
| 9 |
"architecture": "Oleander-150M",
|
| 10 |
"features": [
|
|
|
|
| 20 |
"TST (token superposition training, bag 6)"
|
| 21 |
],
|
| 22 |
"optimizer": "muon_adamw",
|
| 23 |
+
"val_loss": 6.345822848379612,
|
| 24 |
+
"val_ppl": 570.1063078163132,
|
| 25 |
"run": {
|
| 26 |
"model_name": "Oleander-150M",
|
| 27 |
"architecture_detail": "Oleander-150M (28L = 7 x [GDN2 \u00b7 GDN2 \u00b7 GDN2 \u00b7 GQA] \u00b7 d=576 \u00b7 SwiGLU(1600) \u00b7 tied 32,770 vocab)",
|
| 28 |
+
"session_id": "20261114_181817_2de909",
|
| 29 |
"repo_id": "CyanMonkey/Test-Subject-3",
|
| 30 |
+
"started_at": "2026-11-14T18:33:29Z",
|
| 31 |
+
"session_elapsed_s": 4245.2,
|
| 32 |
+
"last_update_at": "2026-11-14T19:44:13Z",
|
| 33 |
+
"steps_per_second": 0.12827,
|
| 34 |
+
"compute_tokens_per_second": 76416.7,
|
| 35 |
+
"eta": "2d 12h 38m",
|
| 36 |
+
"checkpoint_kind": "round 1"
|
| 37 |
},
|
| 38 |
"progress": {
|
| 39 |
+
"step": 500,
|
| 40 |
+
"max_steps": 28500,
|
| 41 |
+
"step_fraction": 0.017544,
|
| 42 |
+
"percent_complete": 1.754,
|
| 43 |
+
"tokens_fraction": 0.017544,
|
| 44 |
+
"percent_tokens": 1.754,
|
| 45 |
"phase": "tst",
|
| 46 |
+
"phase_fraction": 0.055556,
|
| 47 |
+
"tst_switch_step": 9000,
|
| 48 |
"tst_switch_applied": false
|
| 49 |
},
|
| 50 |
"tokens": {
|
| 51 |
+
"compute_tokens_seen": 262144000,
|
| 52 |
+
"compute_tokens_total": 14942208000,
|
| 53 |
+
"raw_tokens_seen": 1572864000,
|
| 54 |
+
"raw_tokens_total": 38535168000,
|
| 55 |
"raw_over_compute_ratio": 6.0,
|
| 56 |
+
"samples_seen": 1536000,
|
| 57 |
+
"rows_consumed": 1536000,
|
| 58 |
"compute_tokens_per_step": 524288,
|
| 59 |
"cluster": {
|
| 60 |
"n_workers": 1,
|
| 61 |
+
"compute_tokens_seen_all_workers": 262144000,
|
| 62 |
+
"raw_tokens_seen_all_workers": 1572864000,
|
| 63 |
+
"compute_tokens_total_planned_all_workers": 14942208000,
|
| 64 |
+
"raw_tokens_total_planned_all_workers": 38535168000
|
| 65 |
}
|
| 66 |
},
|
| 67 |
"metrics": {
|
| 68 |
+
"train_loss": 5.702352523803711,
|
| 69 |
+
"val_loss": 6.345822848379612,
|
| 70 |
+
"val_ppl": 570.1063078163132,
|
| 71 |
+
"grad_norm": 0.34867212176322937,
|
| 72 |
"lr_mult": 1.0
|
| 73 |
},
|
| 74 |
"optimization": {
|
|
|
|
| 98 |
"stable_until": 0.825,
|
| 99 |
"decay_shape": "1-sqrt",
|
| 100 |
"min_lr_ratio": 0.0,
|
| 101 |
+
"decay_start_step": 23512
|
| 102 |
},
|
| 103 |
"tst": {
|
| 104 |
"enabled": true,
|
| 105 |
"bag_size": 6,
|
| 106 |
"phase_fraction": 0.3,
|
| 107 |
+
"switch_step": 9000
|
| 108 |
}
|
| 109 |
},
|
| 110 |
"model": {
|
|
|
|
| 151 |
"doc_masking": "auto",
|
| 152 |
"shuffle_block_rows": 192,
|
| 153 |
"seed": 42,
|
| 154 |
+
"epochs_consumed_this_worker": 0.1041
|
| 155 |
},
|
| 156 |
"hardware": {
|
| 157 |
"device": "cuda",
|
|
|
|
| 162 |
"transformers": "5.14.1",
|
| 163 |
"gpu": "NVIDIA RTX PRO 6000 Blackwell Server Edition",
|
| 164 |
"vram_gb": 101.98,
|
| 165 |
+
"peak_gb": 39.36
|
| 166 |
},
|
| 167 |
"checkpoint": {
|
| 168 |
+
"kind": "round 1",
|
| 169 |
"weights_file": "model.safetensors",
|
| 170 |
"dtype": "float32",
|
| 171 |
"config_file": "config.json",
|
|
|
|
| 186 |
]
|
| 187 |
},
|
| 188 |
"diloco": {
|
| 189 |
+
"enabled": true,
|
| 190 |
+
"worker_id": 0,
|
| 191 |
+
"n_workers": 1,
|
| 192 |
+
"inner_steps": 500,
|
| 193 |
+
"outer_lr": 0.7,
|
| 194 |
+
"outer_momentum": 0.9,
|
| 195 |
+
"is_leader": true,
|
| 196 |
+
"support_repo_id": "CyanMonkey/TestSubject-3-support"
|
| 197 |
},
|
| 198 |
+
"extra": {
|
| 199 |
+
"diloco_round": 1
|
| 200 |
+
}
|
| 201 |
}
|