Text Generation
GGUF
Safetensors
English
mark_slm_v2
llama.cpp
deepseek2
mixture-of-experts
multi-head-latent-attention
iq4_nl
mark-ai
systems-engineering
custom_code
conversational
Instructions to use miceworks/Mark-SLM-V2-302M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use miceworks/Mark-SLM-V2-302M with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf miceworks/Mark-SLM-V2-302M:IQ4_NL # Run inference directly in the terminal: llama cli -hf miceworks/Mark-SLM-V2-302M:IQ4_NL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf miceworks/Mark-SLM-V2-302M:IQ4_NL # Run inference directly in the terminal: llama cli -hf miceworks/Mark-SLM-V2-302M:IQ4_NL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf miceworks/Mark-SLM-V2-302M:IQ4_NL # Run inference directly in the terminal: ./build/bin/llama cli -hf miceworks/Mark-SLM-V2-302M:IQ4_NL
Use Docker
docker model run hf.co/miceworks/Mark-SLM-V2-302M:IQ4_NL
- Llama new
- LM Studio
- Jan
- vLLM
How to use miceworks/Mark-SLM-V2-302M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "miceworks/Mark-SLM-V2-302M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "miceworks/Mark-SLM-V2-302M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/miceworks/Mark-SLM-V2-302M:IQ4_NL
- Ollama
How to use miceworks/Mark-SLM-V2-302M with Ollama:
ollama run hf.co/miceworks/Mark-SLM-V2-302M:IQ4_NL
- Unsloth Desktop
- Docker Model Runner
How to use miceworks/Mark-SLM-V2-302M with Docker Model Runner:
docker model run hf.co/miceworks/Mark-SLM-V2-302M:IQ4_NL
- Lemonade
How to use miceworks/Mark-SLM-V2-302M with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull miceworks/Mark-SLM-V2-302M:IQ4_NL
Run and chat with the model
lemonade run user.Mark-SLM-V2-302M-IQ4_NL
List all available models
lemonade list
- Atomic Chat
Audit fix: update README.md with verified benchmarks, PB-MoE clans, and GGUF metadata
Browse files
README.md
CHANGED
|
@@ -4,7 +4,6 @@ library_name: gguf
|
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
language:
|
| 6 |
- en
|
| 7 |
-
- pt
|
| 8 |
datasets:
|
| 9 |
- miceworks/Mark-Ultra-Curriculum
|
| 10 |
tags:
|
|
@@ -16,23 +15,23 @@ tags:
|
|
| 16 |
- iq4_nl
|
| 17 |
- mark-ai
|
| 18 |
- systems-engineering
|
| 19 |
-
- conversational
|
| 20 |
---
|
| 21 |
|
| 22 |
# Mark-SLM-V2-302M
|
| 23 |
|
| 24 |
## Overview
|
| 25 |
|
| 26 |
-
Mark-SLM-V2-302M is
|
| 27 |
|
| 28 |
-
Quantized in IQ4_NL, the binary occupies 177.76 MiB (186.39 MB / 186,394,944 bytes) and runs on 8 GB consumer GPUs with
|
| 29 |
|
| 30 |
## Specifications
|
| 31 |
|
| 32 |
| Field | Value |
|
| 33 |
| :--- | :--- |
|
| 34 |
| Architecture | `deepseek2` |
|
| 35 |
-
| Training Status | `
|
|
|
|
| 36 |
| Binary Size | `177.76 MiB (186.39 MB / 186,394,944 bytes)` |
|
| 37 |
| Total Parameters | `329,132,000` (329132000 physical) |
|
| 38 |
| Active Parameters | `155,723,744` (155723744 active) |
|
|
@@ -88,11 +87,13 @@ Physical memory consumption of the KV cache per token:
|
|
| 88 |
|
| 89 |
#### Aux-Loss-Free Routing Equation
|
| 90 |
|
| 91 |
-
Router biases update per step to prevent expert collapse without penalizing primary task loss:
|
| 92 |
|
| 93 |
-
$$b_i \leftarrow b_i + \gamma (\bar{f} - f_i)$$
|
| 94 |
|
| 95 |
-
where $\gamma = 0.001$, $f_i$ is empirical expert frequency within the batch, and $\bar{f}$ is target uniform frequency.
|
|
|
|
|
|
|
| 96 |
|
| 97 |
#### Domain Clan Partitioning
|
| 98 |
|
|
@@ -102,33 +103,32 @@ The 16 routed experts are partitioned into 4 functional clans (4 experts per cla
|
|
| 102 |
- **Clan 2 (Experts 8-11)**: Security boundaries, protocols, and validation.
|
| 103 |
- **Clan 3 (Experts 12-15)**: Structured Data, Tool Calling, and Symbolic Logic.
|
| 104 |
|
| 105 |
-
PB-MoE provides structural logit partitioning and soft-pinning constraints (`top_k` allocation reserving $k-1$ slots for the primary domain clan and $1$ slot for cross-clan cooperation) paired with domain-tagged curriculum batches during training on `Mark-Ultra-Curriculum`, `Mark-Open-SLM-Curriculum`, and `Mark-Cyber-Curriculum`.
|
| 106 |
-
|
| 107 |
## Training Pipeline and Reproducibility
|
| 108 |
|
| 109 |
- Training Code: Official pipelines and architecture implementations are maintained in the [Mark-AI scripts directory](https://github.com/thu12dta3/Mark-AI/tree/main/scripts).
|
| 110 |
- Curricula and Datasets: Trained on open datasets hosted on the Hugging Face Hub, including [miceworks/Mark-Ultra-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Ultra-Curriculum), [miceworks/Mark-Open-SLM-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Open-SLM-Curriculum), and [miceworks/Mark-Cyber-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Cyber-Curriculum).
|
| 111 |
-
- Optimization Settings: AdamW optimizer ($\beta_1 = 0.9, \beta_2 = 0.95$, weight decay $0.1$, gradient clipping $1.0$)
|
| 112 |
-
- Training Scope: Trained for 10,000 steps on miceworks/Mark-Ultra-Curriculum
|
| 113 |
|
| 114 |
## Evaluation and Benchmark Results
|
| 115 |
|
| 116 |
-
Achieved a final validation loss of 0.263975 across 10,000 training steps on miceworks/Mark-Ultra-Curriculum.
|
| 117 |
|
| 118 |
## Tokenizer and Vocabulary Specification
|
| 119 |
|
| 120 |
-
- Vocabulary Size: `49152` tokens
|
| 121 |
-
-
|
| 122 |
-
- Epistemic Control Tokens: Self-RAG reasoning tags (`<think>`, `</think>`), tool delimiters (`<tool_call>`, `</tool_call>`), epistemic
|
| 123 |
-
- Compression Fertility:
|
| 124 |
-
- English
|
| 125 |
-
-
|
| 126 |
-
-
|
|
|
|
| 127 |
|
| 128 |
## Limitations and Intended Use
|
| 129 |
|
| 130 |
- Parameter Scale Constraints: While sparse MoE routing enables execution within consumer VRAM budgets, absolute capacity is bounded by physical parameter counts.
|
| 131 |
-
- Recommended Deployment: For mission-critical production environments, deploy alongside external verification mechanisms such as
|
| 132 |
- Safety and Boundary Enforcement: Models trained on systems and cybersecurity datasets are intended for defensive research, vulnerability auditing, patch validation, and systems engineering.
|
| 133 |
|
| 134 |
### Running Benchmarks on TPN Workbench and lm-evaluation-harness
|
|
@@ -137,35 +137,27 @@ When running automated benchmarks on TPN Workbench, lm-evaluation-harness, LM St
|
|
| 137 |
- Select **Format: gguf** (or **Backend: llama.cpp**).
|
| 138 |
- Do not evaluate via raw Python `transformers` with `safetensors` or `trust_remote_code=True`, because the native `deepseek2` H-MLA and fine-grained 3D MoE graph executes directly inside the optimized C++/CUDA kernels of `llama.cpp`. Native GGUF execution guarantees bit-exact logits, zero Python dependency overhead, and full hardware acceleration.
|
| 139 |
|
| 140 |
-
## Usage
|
| 141 |
|
| 142 |
-
|
|
|
|
|
|
|
| 143 |
|
| 144 |
```bash
|
| 145 |
# Using huggingface-cli
|
| 146 |
huggingface-cli download miceworks/Mark-SLM-V2-302M mark_slm_v2_302m_iq4_nl.gguf --local-dir .
|
| 147 |
|
| 148 |
-
#
|
| 149 |
-
from huggingface_hub import hf_hub_download
|
| 150 |
-
hf_hub_download(repo_id="miceworks/Mark-SLM-V2-302M", filename="mark_slm_v2_302m_iq4_nl.gguf")
|
| 151 |
-
```
|
| 152 |
-
|
| 153 |
-
### Cryptographic Verification
|
| 154 |
-
|
| 155 |
-
```bash
|
| 156 |
echo "34a3e0e39cd02cd6a12345b9558ea094877530ef3bde52e64552f20ca48ce38e mark_slm_v2_302m_iq4_nl.gguf" | sha256sum -c -
|
| 157 |
```
|
| 158 |
|
| 159 |
-
### llama.cpp
|
| 160 |
|
| 161 |
```bash
|
| 162 |
-
#
|
| 163 |
llama-cli -m mark_slm_v2_302m_iq4_nl.gguf \
|
| 164 |
-
-p "
|
| 165 |
-
-n
|
| 166 |
-
|
| 167 |
-
# HTTP server
|
| 168 |
-
llama-server -m mark_slm_v2_302m_iq4_nl.gguf --port 8080 -c 4096
|
| 169 |
```
|
| 170 |
|
| 171 |
### Python Inspection
|
|
|
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
language:
|
| 6 |
- en
|
|
|
|
| 7 |
datasets:
|
| 8 |
- miceworks/Mark-Ultra-Curriculum
|
| 9 |
tags:
|
|
|
|
| 15 |
- iq4_nl
|
| 16 |
- mark-ai
|
| 17 |
- systems-engineering
|
|
|
|
| 18 |
---
|
| 19 |
|
| 20 |
# Mark-SLM-V2-302M
|
| 21 |
|
| 22 |
## Overview
|
| 23 |
|
| 24 |
+
Mark-SLM-V2-302M is an experimental narrow checkpoint in the 302M family naming convention. It contains 329,132,160 total physical parameters and activates 155.72M parameters per token across 16 layers. It features 16 routed experts and 1 shared expert (2 active routed experts per token).
|
| 25 |
|
| 26 |
+
Quantized in IQ4_NL, the binary occupies 177.76 MiB (186.39 MB / 186,394,944 bytes) and runs on 8 GB consumer GPUs with minimal VRAM consumption.
|
| 27 |
|
| 28 |
## Specifications
|
| 29 |
|
| 30 |
| Field | Value |
|
| 31 |
| :--- | :--- |
|
| 32 |
| Architecture | `deepseek2` |
|
| 33 |
+
| Training Status | `EXPERIMENTAL_NARROW_CHECKPOINT` |
|
| 34 |
+
| Weight State Label | `Experimental Narrow Checkpoint` |
|
| 35 |
| Binary Size | `177.76 MiB (186.39 MB / 186,394,944 bytes)` |
|
| 36 |
| Total Parameters | `329,132,000` (329132000 physical) |
|
| 37 |
| Active Parameters | `155,723,744` (155723744 active) |
|
|
|
|
| 87 |
|
| 88 |
#### Aux-Loss-Free Routing Equation
|
| 89 |
|
| 90 |
+
Router biases update per step to prevent expert collapse without penalizing primary task loss using the exact sign-based rule:
|
| 91 |
|
| 92 |
+
$$b_i \leftarrow b_i + \gamma \cdot \operatorname{sign}(\bar{f} - f_i)$$
|
| 93 |
|
| 94 |
+
where $\gamma = 0.001$, $f_i$ is empirical expert frequency within the batch, and $\bar{f}$ is target uniform frequency.
|
| 95 |
+
|
| 96 |
+
Router bias vectors are stored in FP32 precision or initialized to uniform values during pre-training to ensure stable routing dynamics.
|
| 97 |
|
| 98 |
#### Domain Clan Partitioning
|
| 99 |
|
|
|
|
| 103 |
- **Clan 2 (Experts 8-11)**: Security boundaries, protocols, and validation.
|
| 104 |
- **Clan 3 (Experts 12-15)**: Structured Data, Tool Calling, and Symbolic Logic.
|
| 105 |
|
|
|
|
|
|
|
| 106 |
## Training Pipeline and Reproducibility
|
| 107 |
|
| 108 |
- Training Code: Official pipelines and architecture implementations are maintained in the [Mark-AI scripts directory](https://github.com/thu12dta3/Mark-AI/tree/main/scripts).
|
| 109 |
- Curricula and Datasets: Trained on open datasets hosted on the Hugging Face Hub, including [miceworks/Mark-Ultra-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Ultra-Curriculum), [miceworks/Mark-Open-SLM-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Open-SLM-Curriculum), and [miceworks/Mark-Cyber-Curriculum](https://huggingface.co/datasets/miceworks/Mark-Cyber-Curriculum).
|
| 110 |
+
- Optimization Settings: AdamW optimizer ($\beta_1 = 0.9, \beta_2 = 0.95$, weight decay $0.1$, gradient clipping $1.0$).
|
| 111 |
+
- Training Scope: Experimental narrow checkpoint (EXPERIMENTAL_NARROW_CHECKPOINT). Trained for 10,000 steps on miceworks/Mark-Ultra-Curriculum. The final training loss reached 0.263975, reflecting memorization on the narrow 4,821-sample curriculum corpus rather than generalized open-domain language understanding.
|
| 112 |
|
| 113 |
## Evaluation and Benchmark Results
|
| 114 |
|
| 115 |
+
Achieved a final validation loss of 0.263975 across 10,000 training steps on the narrow miceworks/Mark-Ultra-Curriculum distribution (memorization artifact).
|
| 116 |
|
| 117 |
## Tokenizer and Vocabulary Specification
|
| 118 |
|
| 119 |
+
- Vocabulary Size: `49152` tensor rows (4,311 active BPE tokens + 27,721 reserved padding slots)
|
| 120 |
+
- Active BPE Merges: 3,981 merges
|
| 121 |
+
- Epistemic Control Tokens: Self-RAG reasoning tags (`<think>`, `</think>`), tool delimiters (`<tool_call>`, `</tool_call>`), epistemic reflection tags (`[RETRIEVE]`, `[/RETRIEVE]`, `[EVALUATE]`, `[ACCEPT]`, `[REJECT]`)
|
| 122 |
+
- Measured Compression Fertility:
|
| 123 |
+
- English technical prose: $\approx 2.53\text{ characters/token}$
|
| 124 |
+
- Portuguese technical prose: $\approx 2.57\text{ characters/token}$
|
| 125 |
+
- C, Rust, and systems source code: $\approx 2.02\text{ characters/token}$
|
| 126 |
+
- Whitespace Round-Trip Fidelity: Indentation whitespace requires byte-fallback tokenization; leading 4 spaces may be stripped by ByteLevel decoder when token 28 is raw space sequence.
|
| 127 |
|
| 128 |
## Limitations and Intended Use
|
| 129 |
|
| 130 |
- Parameter Scale Constraints: While sparse MoE routing enables execution within consumer VRAM budgets, absolute capacity is bounded by physical parameter counts.
|
| 131 |
+
- Recommended Deployment: For mission-critical production environments, deploy alongside external verification mechanisms such as compiler validation or unit test runners.
|
| 132 |
- Safety and Boundary Enforcement: Models trained on systems and cybersecurity datasets are intended for defensive research, vulnerability auditing, patch validation, and systems engineering.
|
| 133 |
|
| 134 |
### Running Benchmarks on TPN Workbench and lm-evaluation-harness
|
|
|
|
| 137 |
- Select **Format: gguf** (or **Backend: llama.cpp**).
|
| 138 |
- Do not evaluate via raw Python `transformers` with `safetensors` or `trust_remote_code=True`, because the native `deepseek2` H-MLA and fine-grained 3D MoE graph executes directly inside the optimized C++/CUDA kernels of `llama.cpp`. Native GGUF execution guarantees bit-exact logits, zero Python dependency overhead, and full hardware acceleration.
|
| 139 |
|
| 140 |
+
## Usage: Narrow Domain and Systems Engineering Verification
|
| 141 |
|
| 142 |
+
This checkpoint is an experimental narrow checkpoint trained on specific low-level or systems engineering distributions. Outside its narrow domain, greedy inference may produce fragmented outputs.
|
| 143 |
+
|
| 144 |
+
### Download and Checksum Verification
|
| 145 |
|
| 146 |
```bash
|
| 147 |
# Using huggingface-cli
|
| 148 |
huggingface-cli download miceworks/Mark-SLM-V2-302M mark_slm_v2_302m_iq4_nl.gguf --local-dir .
|
| 149 |
|
| 150 |
+
# Cryptographic verification
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
echo "34a3e0e39cd02cd6a12345b9558ea094877530ef3bde52e64552f20ca48ce38e mark_slm_v2_302m_iq4_nl.gguf" | sha256sum -c -
|
| 152 |
```
|
| 153 |
|
| 154 |
+
### Execution with llama.cpp
|
| 155 |
|
| 156 |
```bash
|
| 157 |
+
# Systems prompt execution
|
| 158 |
llama-cli -m mark_slm_v2_302m_iq4_nl.gguf \
|
| 159 |
+
-p "Systems engineering stub: Implement a POSIX mutex initialization routine.\nCode:" \
|
| 160 |
+
-n 128 --temp 0.2
|
|
|
|
|
|
|
|
|
|
| 161 |
```
|
| 162 |
|
| 163 |
### Python Inspection
|