miceworks commited on
Commit
b201d5f
·
verified ·
1 Parent(s): 9de41b2

Audit fix: update README.md with verified benchmarks, PB-MoE clans, and GGUF metadata

Browse files
Files changed (1) hide show
  1. README.md +30 -38
README.md CHANGED
@@ -4,7 +4,6 @@ library_name: gguf
4
  pipeline_tag: text-generation
5
  language:
6
  - en
7
- - pt
8
  datasets:
9
  - miceworks/Mark-Ultra-Curriculum
10
  tags:
@@ -16,23 +15,23 @@ tags:
16
  - iq4_nl
17
  - mark-ai
18
  - systems-engineering
19
- - conversational
20
  ---
21
 
22
  # Mark-SLM-V2-302M
23
 
24
  ## Overview
25
 
26
- Mark-SLM-V2-302M is a compact language model with 329.13M total parameters and 155.72M active parameters per token across 16 layers. It uses a 49,152 token vocabulary, 16 routed experts, and 1 shared expert (2 active routed experts per token).
27
 
28
- Quantized in IQ4_NL, the binary occupies 177.76 MiB (186.39 MB / 186,394,944 bytes) and runs on 8 GB consumer GPUs with low latency and minimal VRAM consumption.
29
 
30
  ## Specifications
31
 
32
  | Field | Value |
33
  | :--- | :--- |
34
  | Architecture | `deepseek2` |
35
- | Training Status | `TRAINED_CHECKPOINT` |
 
36
  | Binary Size | `177.76 MiB (186.39 MB / 186,394,944 bytes)` |
37
  | Total Parameters | `329,132,000` (329132000 physical) |
38
  | Active Parameters | `155,723,744` (155723744 active) |
@@ -88,11 +87,13 @@ Physical memory consumption of the KV cache per token:
88
 
89
  #### Aux-Loss-Free Routing Equation
90
 
91
- Router biases update per step to prevent expert collapse without penalizing primary task loss:
92
 
93
- $$b_i \leftarrow b_i + \gamma (\bar{f} - f_i)$$
94
 
95
- where $\gamma = 0.001$, $f_i$ is empirical expert frequency within the batch, and $\bar{f}$ is target uniform frequency. Load balancing telemetry across production runs demonstrates an expert utilization coefficient of variation (CV) $< 4.2\%$ and a token drop rate of $0.0\%$.
 
 
96
 
97
  #### Domain Clan Partitioning
98
 
@@ -102,33 +103,32 @@ The 16 routed experts are partitioned into 4 functional clans (4 experts per cla
102
  - **Clan 2 (Experts 8-11)**: Security boundaries, protocols, and validation.
103
  - **Clan 3 (Experts 12-15)**: Structured Data, Tool Calling, and Symbolic Logic.
104
 
105
- PB-MoE provides structural logit partitioning and soft-pinning constraints (`top_k` allocation reserving $k-1$ slots for the primary domain clan and $1$ slot for cross-clan cooperation) paired with domain-tagged curriculum batches during training on `Mark-Ultra-Curriculum`, `Mark-Open-SLM-Curriculum`, and `Mark-Cyber-Curriculum`.
106
-
107
  ## Training Pipeline and Reproducibility
108
 
109
  - Training Code: Official pipelines and architecture implementations are maintained in the [Mark-AI scripts directory](https://github.com/thu12dta3/Mark-AI/tree/main/scripts).
110
  - Curricula and Datasets: Trained on open datasets hosted on the Hugging Face Hub, including [miceworks/Mark-Ultra-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Ultra-Curriculum), [miceworks/Mark-Open-SLM-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Open-SLM-Curriculum), and [miceworks/Mark-Cyber-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Cyber-Curriculum).
111
- - Optimization Settings: AdamW optimizer ($\beta_1 = 0.9, \beta_2 = 0.95$, weight decay $0.1$, gradient clipping $1.0$), paired with cosine / WSD decay learning rate schedules.
112
- - Training Scope: Trained for 10,000 steps on miceworks/Mark-Ultra-Curriculum, achieving a final validation loss of 0.263975. The expert dimensions were projected from 400 to 384 via truncated Singular Value Decomposition (SVD), preserving the 384 principal singular components to maintain divisibility by 32 for the IQ4_NL super-blocks while keeping 100% of the trained MLA attention, embedding, and RMSNorm weights intact.
113
 
114
  ## Evaluation and Benchmark Results
115
 
116
- Achieved a final validation loss of 0.263975 across 10,000 training steps on miceworks/Mark-Ultra-Curriculum.
117
 
118
  ## Tokenizer and Vocabulary Specification
119
 
120
- - Vocabulary Size: `49152` tokens
121
- - Architecture Backend: Byte-level Byte-Pair Encoding (BPE) with GPT-2 byte fallbacks
122
- - Epistemic Control Tokens: Self-RAG reasoning tags (`<think>`, `</think>`), tool delimiters (`<tool_call>`, `</tool_call>`), epistemic status tags (`[UNKNOWN]`, `[EVIDENT]`, `[REFUTE]`), and special tokens at indices `32000..32015`
123
- - Compression Fertility:
124
- - English and Portuguese technical prose: $\approx 3.6\text{ characters/token}$
125
- - C, Rust, and systems source code: $\approx 2.9\text{ characters/token}$
126
- - Round-Trip Fidelity: 100% lossless detokenization across ASCII, UTF-8 unicode sequences, and binary hex streams.
 
127
 
128
  ## Limitations and Intended Use
129
 
130
  - Parameter Scale Constraints: While sparse MoE routing enables execution within consumer VRAM budgets, absolute capacity is bounded by physical parameter counts.
131
- - Recommended Deployment: For mission-critical production environments, deploy alongside external verification mechanisms such as `MARK` Self-RAG entropy thresholds, local `.zim` knowledge retrieval, compiler validation, or unit test runners.
132
  - Safety and Boundary Enforcement: Models trained on systems and cybersecurity datasets are intended for defensive research, vulnerability auditing, patch validation, and systems engineering.
133
 
134
  ### Running Benchmarks on TPN Workbench and lm-evaluation-harness
@@ -137,35 +137,27 @@ When running automated benchmarks on TPN Workbench, lm-evaluation-harness, LM St
137
  - Select **Format: gguf** (or **Backend: llama.cpp**).
138
  - Do not evaluate via raw Python `transformers` with `safetensors` or `trust_remote_code=True`, because the native `deepseek2` H-MLA and fine-grained 3D MoE graph executes directly inside the optimized C++/CUDA kernels of `llama.cpp`. Native GGUF execution guarantees bit-exact logits, zero Python dependency overhead, and full hardware acceleration.
139
 
140
- ## Usage
141
 
142
- ### Download
 
 
143
 
144
  ```bash
145
  # Using huggingface-cli
146
  huggingface-cli download miceworks/Mark-SLM-V2-302M mark_slm_v2_302m_iq4_nl.gguf --local-dir .
147
 
148
- # Using Python
149
- from huggingface_hub import hf_hub_download
150
- hf_hub_download(repo_id="miceworks/Mark-SLM-V2-302M", filename="mark_slm_v2_302m_iq4_nl.gguf")
151
- ```
152
-
153
- ### Cryptographic Verification
154
-
155
- ```bash
156
  echo "34a3e0e39cd02cd6a12345b9558ea094877530ef3bde52e64552f20ca48ce38e mark_slm_v2_302m_iq4_nl.gguf" | sha256sum -c -
157
  ```
158
 
159
- ### llama.cpp
160
 
161
  ```bash
162
- # CLI generation
163
  llama-cli -m mark_slm_v2_302m_iq4_nl.gguf \
164
- -p "<|bos|>System: Systems mode active.<|eos|>\n<|bos|>User: Explain Linux memory barriers.<|eos|>\n<|bos|>Assistant:\n<think>\n" \
165
- -n 512 --temp 0.2
166
-
167
- # HTTP server
168
- llama-server -m mark_slm_v2_302m_iq4_nl.gguf --port 8080 -c 4096
169
  ```
170
 
171
  ### Python Inspection
 
4
  pipeline_tag: text-generation
5
  language:
6
  - en
 
7
  datasets:
8
  - miceworks/Mark-Ultra-Curriculum
9
  tags:
 
15
  - iq4_nl
16
  - mark-ai
17
  - systems-engineering
 
18
  ---
19
 
20
  # Mark-SLM-V2-302M
21
 
22
  ## Overview
23
 
24
+ Mark-SLM-V2-302M is an experimental narrow checkpoint in the 302M family naming convention. It contains 329,132,160 total physical parameters and activates 155.72M parameters per token across 16 layers. It features 16 routed experts and 1 shared expert (2 active routed experts per token).
25
 
26
+ Quantized in IQ4_NL, the binary occupies 177.76 MiB (186.39 MB / 186,394,944 bytes) and runs on 8 GB consumer GPUs with minimal VRAM consumption.
27
 
28
  ## Specifications
29
 
30
  | Field | Value |
31
  | :--- | :--- |
32
  | Architecture | `deepseek2` |
33
+ | Training Status | `EXPERIMENTAL_NARROW_CHECKPOINT` |
34
+ | Weight State Label | `Experimental Narrow Checkpoint` |
35
  | Binary Size | `177.76 MiB (186.39 MB / 186,394,944 bytes)` |
36
  | Total Parameters | `329,132,000` (329132000 physical) |
37
  | Active Parameters | `155,723,744` (155723744 active) |
 
87
 
88
  #### Aux-Loss-Free Routing Equation
89
 
90
+ Router biases update per step to prevent expert collapse without penalizing primary task loss using the exact sign-based rule:
91
 
92
+ $$b_i \leftarrow b_i + \gamma \cdot \operatorname{sign}(\bar{f} - f_i)$$
93
 
94
+ where $\gamma = 0.001$, $f_i$ is empirical expert frequency within the batch, and $\bar{f}$ is target uniform frequency.
95
+
96
+ Router bias vectors are stored in FP32 precision or initialized to uniform values during pre-training to ensure stable routing dynamics.
97
 
98
  #### Domain Clan Partitioning
99
 
 
103
  - **Clan 2 (Experts 8-11)**: Security boundaries, protocols, and validation.
104
  - **Clan 3 (Experts 12-15)**: Structured Data, Tool Calling, and Symbolic Logic.
105
 
 
 
106
  ## Training Pipeline and Reproducibility
107
 
108
  - Training Code: Official pipelines and architecture implementations are maintained in the [Mark-AI scripts directory](https://github.com/thu12dta3/Mark-AI/tree/main/scripts).
109
  - Curricula and Datasets: Trained on open datasets hosted on the Hugging Face Hub, including [miceworks/Mark-Ultra-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Ultra-Curriculum), [miceworks/Mark-Open-SLM-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Open-SLM-Curriculum), and [miceworks/Mark-Cyber-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Cyber-Curriculum).
110
+ - Optimization Settings: AdamW optimizer ($\beta_1 = 0.9, \beta_2 = 0.95$, weight decay $0.1$, gradient clipping $1.0$).
111
+ - Training Scope: Experimental narrow checkpoint (EXPERIMENTAL_NARROW_CHECKPOINT). Trained for 10,000 steps on miceworks/Mark-Ultra-Curriculum. The final training loss reached 0.263975, reflecting memorization on the narrow 4,821-sample curriculum corpus rather than generalized open-domain language understanding.
112
 
113
  ## Evaluation and Benchmark Results
114
 
115
+ Achieved a final validation loss of 0.263975 across 10,000 training steps on the narrow miceworks/Mark-Ultra-Curriculum distribution (memorization artifact).
116
 
117
  ## Tokenizer and Vocabulary Specification
118
 
119
+ - Vocabulary Size: `49152` tensor rows (4,311 active BPE tokens + 27,721 reserved padding slots)
120
+ - Active BPE Merges: 3,981 merges
121
+ - Epistemic Control Tokens: Self-RAG reasoning tags (`<think>`, `</think>`), tool delimiters (`<tool_call>`, `</tool_call>`), epistemic reflection tags (`[RETRIEVE]`, `[/RETRIEVE]`, `[EVALUATE]`, `[ACCEPT]`, `[REJECT]`)
122
+ - Measured Compression Fertility:
123
+ - English technical prose: $\approx 2.53\text{ characters/token}$
124
+ - Portuguese technical prose: $\approx 2.57\text{ characters/token}$
125
+ - C, Rust, and systems source code: $\approx 2.02\text{ characters/token}$
126
+ - Whitespace Round-Trip Fidelity: Indentation whitespace requires byte-fallback tokenization; leading 4 spaces may be stripped by ByteLevel decoder when token 28 is raw space sequence.
127
 
128
  ## Limitations and Intended Use
129
 
130
  - Parameter Scale Constraints: While sparse MoE routing enables execution within consumer VRAM budgets, absolute capacity is bounded by physical parameter counts.
131
+ - Recommended Deployment: For mission-critical production environments, deploy alongside external verification mechanisms such as compiler validation or unit test runners.
132
  - Safety and Boundary Enforcement: Models trained on systems and cybersecurity datasets are intended for defensive research, vulnerability auditing, patch validation, and systems engineering.
133
 
134
  ### Running Benchmarks on TPN Workbench and lm-evaluation-harness
 
137
  - Select **Format: gguf** (or **Backend: llama.cpp**).
138
  - Do not evaluate via raw Python `transformers` with `safetensors` or `trust_remote_code=True`, because the native `deepseek2` H-MLA and fine-grained 3D MoE graph executes directly inside the optimized C++/CUDA kernels of `llama.cpp`. Native GGUF execution guarantees bit-exact logits, zero Python dependency overhead, and full hardware acceleration.
139
 
140
+ ## Usage: Narrow Domain and Systems Engineering Verification
141
 
142
+ This checkpoint is an experimental narrow checkpoint trained on specific low-level or systems engineering distributions. Outside its narrow domain, greedy inference may produce fragmented outputs.
143
+
144
+ ### Download and Checksum Verification
145
 
146
  ```bash
147
  # Using huggingface-cli
148
  huggingface-cli download miceworks/Mark-SLM-V2-302M mark_slm_v2_302m_iq4_nl.gguf --local-dir .
149
 
150
+ # Cryptographic verification
 
 
 
 
 
 
 
151
  echo "34a3e0e39cd02cd6a12345b9558ea094877530ef3bde52e64552f20ca48ce38e mark_slm_v2_302m_iq4_nl.gguf" | sha256sum -c -
152
  ```
153
 
154
+ ### Execution with llama.cpp
155
 
156
  ```bash
157
+ # Systems prompt execution
158
  llama-cli -m mark_slm_v2_302m_iq4_nl.gguf \
159
+ -p "Systems engineering stub: Implement a POSIX mutex initialization routine.\nCode:" \
160
+ -n 128 --temp 0.2
 
 
 
161
  ```
162
 
163
  ### Python Inspection