Instructions to use domyn/Domyn-Edge-SFT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use domyn/Domyn-Edge-SFT with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="domyn/Domyn-Edge-SFT", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("domyn/Domyn-Edge-SFT", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use domyn/Domyn-Edge-SFT with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "domyn/Domyn-Edge-SFT" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "domyn/Domyn-Edge-SFT", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/domyn/Domyn-Edge-SFT
- SGLang
How to use domyn/Domyn-Edge-SFT with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "domyn/Domyn-Edge-SFT" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "domyn/Domyn-Edge-SFT", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "domyn/Domyn-Edge-SFT" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "domyn/Domyn-Edge-SFT", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use domyn/Domyn-Edge-SFT with Docker Model Runner:
docker model run hf.co/domyn/Domyn-Edge-SFT
RoPE fp32 + gradient checkpointing + SDPA attention backend
Browse filesThree fixes to `modeling_domynedge.py`. Default CUDA `varlen` path untouched (SDPA is opt-in): no GPU regression. Checks: CPU, tiny random config, no weights (torch 2.15).
## 1. RoPE precision — resolves the `TODO` in `DomynEdgeRotaryEmbedding.forward`
- **Found:** angles round to bf16 at large positions — under `autocast(bfloat16)` the matmul runs bf16, and after `model.bfloat16()` the `inv_freq` buffer is bf16 (both noted in the TODO).
- **Why:** cos/sin drift at long context. Real val rows (bf16): prob shift vs fp32 ~8% at <1k, ~53% at 4–8k; accuracy 53% → 27% at 4–8k.
- **Fix:** disable autocast, recompute `inv_freq` in fp32 in `forward` (as the TODO proposes).
- **Proved:** cos error at pos 7000 = **0.0** (fix) vs **2.0** (old).
## 2. Gradient checkpointing — `supports_gradient_checkpointing` was `False`
- **Found:** flag `False` → `gradient_checkpointing_enable()` is a no-op.
- **Why:** 16,384-token training needs ~161 GB of activations — won't fit one 141 GB GPU without recompute.
- **Fix:** layers are stateless → set flag, route layer loop through `_gradient_checkpointing_func` when training.
- **Proved:** gradients identical to non-checkpointed (max diff **0.0**).
## 3. SDPA attention — `_supports_sdpa` was `False`
- **Found:** attention hard-wired to CUDA `torch.nn.attention.varlen`; `attn_implementation` ignored, so no portable path.
- **Why:** can't run without the flash/varlen kernel; no fp32-capable attention.
- **Fix:** opt-in `attn_implementation="sdpa"` branch — same GQA + per-layer sliding window + padding mask via `scaled_dot_product_attention`; default path unchanged.
- **Proved:** matches independent masked-attention reference to **1.5e-8**; full forward runs on CPU with finite logits (impossible on varlen). GPU `varlen`-vs-SDPA parity left to confirm on GPU.
## Hardware
| | before | after |
|---|---|---|
| H200 / A100 (flash varlen) | ✅ | ✅ unchanged + gradient checkpointing |
| T4 / older GPU (no varlen) | ❌ | ✅ `sdpa` |
| CPU | ❌ | ✅ `sdpa` (separately tested) |
- modeling_domynedge.py +72 -21
|
@@ -218,18 +218,19 @@ class DomynEdgeRotaryEmbedding(torch.nn.Module):
|
|
| 218 |
|
| 219 |
@torch.no_grad()
|
| 220 |
def forward(self, x, position_ids):
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
#
|
| 225 |
-
#
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
|
|
|
|
| 233 |
|
| 234 |
return cos, sin
|
| 235 |
|
|
@@ -322,6 +323,11 @@ class DomynEdgeAttention(torch.nn.Module):
|
|
| 322 |
cos, sin = position_embeddings
|
| 323 |
q, k = self.apply_rope(q, k, cos, sin, unsqueeze_dim=2)
|
| 324 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 325 |
if past_key_values is not None:
|
| 326 |
k, v = k.transpose(1, 2), v.transpose(1, 2)
|
| 327 |
k, v = past_key_values.update(k, v, self.layer_idx)
|
|
@@ -355,6 +361,44 @@ class DomynEdgeAttention(torch.nn.Module):
|
|
| 355 |
attn_output = attn_output.view(batch_size, sequence_length, self.config.num_attention_heads * self.head_dim)
|
| 356 |
return self.o_proj(attn_output)
|
| 357 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 358 |
|
| 359 |
class DomynEdgeDecoderLayer(torch.nn.Module):
|
| 360 |
def __init__(self, config: DomynEdgeConfig, layer_idx: int):
|
|
@@ -398,9 +442,9 @@ class DomynEdgePreTrainedModel(transformers.PreTrainedModel):
|
|
| 398 |
|
| 399 |
config: DomynEdgeConfig
|
| 400 |
base_model_prefix = "model"
|
| 401 |
-
supports_gradient_checkpointing =
|
| 402 |
_supports_flash_attn = True
|
| 403 |
-
_supports_sdpa =
|
| 404 |
_supports_flex_attn = False
|
| 405 |
|
| 406 |
_can_compile_fullgraph = True
|
|
@@ -425,6 +469,7 @@ class DomynEdgeModel(DomynEdgePreTrainedModel):
|
|
| 425 |
)
|
| 426 |
self.norm = torch.nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
| 427 |
self.rotary_emb = DomynEdgeRotaryEmbedding(config=config)
|
|
|
|
| 428 |
|
| 429 |
# Initialize weights and apply final processing
|
| 430 |
self.post_init()
|
|
@@ -469,13 +514,19 @@ class DomynEdgeModel(DomynEdgePreTrainedModel):
|
|
| 469 |
)
|
| 470 |
|
| 471 |
for decoder_layer in self.layers:
|
| 472 |
-
|
| 473 |
-
|
| 474 |
-
|
| 475 |
-
|
| 476 |
-
|
| 477 |
-
|
| 478 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 479 |
|
| 480 |
hidden_states = self.norm(hidden_states)
|
| 481 |
return transformers.modeling_outputs.BaseModelOutputWithPast(
|
|
|
|
| 218 |
|
| 219 |
@torch.no_grad()
|
| 220 |
def forward(self, x, position_ids):
|
| 221 |
+
# The angles must stay exact at large positions. Two cases would otherwise lose precision:
|
| 222 |
+
# (1) under torch.autocast(bfloat16) the matmul runs in bf16 despite the `.float()` casts; (2) after
|
| 223 |
+
# model.to(torch.bfloat16) / model.bfloat16() the `inv_freq` buffer itself is cast to bf16. We therefore
|
| 224 |
+
# disable autocast and recompute `inv_freq` in fp32 (tiny vector) instead of reading the buffer, so neither
|
| 225 |
+
# case can round the positions off. Matches standard HF rotaries, which also disable autocast here.
|
| 226 |
+
with torch.autocast(device_type=x.device.type, enabled=False):
|
| 227 |
+
inv_freq, _ = self.compute_default_rope_parameters(self.config, x.device)
|
| 228 |
+
inv_freq_expanded = inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
|
| 229 |
+
position_ids_expanded = position_ids[:, None, :].float()
|
| 230 |
+
freqs = (inv_freq_expanded @ position_ids_expanded).transpose(1, 2)
|
| 231 |
+
emb = torch.cat((freqs, freqs), dim=-1)
|
| 232 |
+
cos = emb.cos() * self.attention_scaling
|
| 233 |
+
sin = emb.sin() * self.attention_scaling
|
| 234 |
|
| 235 |
return cos, sin
|
| 236 |
|
|
|
|
| 323 |
cos, sin = position_embeddings
|
| 324 |
q, k = self.apply_rope(q, k, cos, sin, unsqueeze_dim=2)
|
| 325 |
|
| 326 |
+
# Opt-in SDPA path for hosts without the CUDA varlen/flash kernel (CPU, older GPUs) and for an
|
| 327 |
+
# fp32-capable attention. Enabled only with attn_implementation="sdpa"; the default path below is unchanged.
|
| 328 |
+
if getattr(self.config, "_attn_implementation", "flash_attention_2") == "sdpa":
|
| 329 |
+
return self._sdpa_forward(q, k, v, attention_mask, past_key_values, batch_size, sequence_length)
|
| 330 |
+
|
| 331 |
if past_key_values is not None:
|
| 332 |
k, v = k.transpose(1, 2), v.transpose(1, 2)
|
| 333 |
k, v = past_key_values.update(k, v, self.layer_idx)
|
|
|
|
| 361 |
attn_output = attn_output.view(batch_size, sequence_length, self.config.num_attention_heads * self.head_dim)
|
| 362 |
return self.o_proj(attn_output)
|
| 363 |
|
| 364 |
+
def _sdpa_forward(self, q, k, v, attention_mask, past_key_values, batch_size, sequence_length):
|
| 365 |
+
"""SDPA fallback matching the varlen path: grouped-query attention with this layer's sliding window and the
|
| 366 |
+
padding mask, built as an additive bias so it runs on any backend (including CPU/fp32)."""
|
| 367 |
+
if past_key_values is not None:
|
| 368 |
+
k, v = k.transpose(1, 2), v.transpose(1, 2)
|
| 369 |
+
k, v = past_key_values.update(k, v, self.layer_idx)
|
| 370 |
+
k, v = k.transpose(1, 2), v.transpose(1, 2)
|
| 371 |
+
|
| 372 |
+
q = q.transpose(1, 2) # [b, n_heads, q_len, d]
|
| 373 |
+
k = k.transpose(1, 2) # [b, n_kv, k_len, d]
|
| 374 |
+
v = v.transpose(1, 2)
|
| 375 |
+
q_len, k_len = q.shape[2], k.shape[2]
|
| 376 |
+
|
| 377 |
+
# Absolute positions so decode (q_len=1, k_len>q_len) and prefill both work.
|
| 378 |
+
qpos = torch.arange(k_len - q_len, k_len, device=q.device)[:, None]
|
| 379 |
+
kpos = torch.arange(k_len, device=q.device)[None, :]
|
| 380 |
+
left, right = self.window_size # [-1, -1] = full attention; [w, 0] = causal sliding window
|
| 381 |
+
allowed = torch.ones(q_len, k_len, dtype=torch.bool, device=q.device)
|
| 382 |
+
if right >= 0:
|
| 383 |
+
allowed &= (kpos - qpos) <= right
|
| 384 |
+
if left >= 0:
|
| 385 |
+
allowed &= (qpos - kpos) <= left
|
| 386 |
+
mask = allowed[None, None] # [1, 1, q_len, k_len]
|
| 387 |
+
if attention_mask is not None:
|
| 388 |
+
mask = mask & attention_mask[:, None, None, :].to(torch.bool)
|
| 389 |
+
bias = torch.zeros(mask.shape, dtype=q.dtype, device=q.device).masked_fill(~mask, float("-inf"))
|
| 390 |
+
# A fully-masked query row (e.g. a padding position) would make softmax produce NaN; neutralize it — those
|
| 391 |
+
# positions are discarded downstream.
|
| 392 |
+
bias = bias.masked_fill(torch.isneginf(bias).all(dim=-1, keepdim=True), 0.0)
|
| 393 |
+
|
| 394 |
+
attn_output = torch.nn.functional.scaled_dot_product_attention(
|
| 395 |
+
q, k, v, attn_mask=bias, scale=self.scaling, enable_gqa=self.enable_gqa
|
| 396 |
+
)
|
| 397 |
+
attn_output = attn_output.transpose(1, 2).reshape(
|
| 398 |
+
batch_size, sequence_length, self.config.num_attention_heads * self.head_dim
|
| 399 |
+
)
|
| 400 |
+
return self.o_proj(attn_output)
|
| 401 |
+
|
| 402 |
|
| 403 |
class DomynEdgeDecoderLayer(torch.nn.Module):
|
| 404 |
def __init__(self, config: DomynEdgeConfig, layer_idx: int):
|
|
|
|
| 442 |
|
| 443 |
config: DomynEdgeConfig
|
| 444 |
base_model_prefix = "model"
|
| 445 |
+
supports_gradient_checkpointing = True
|
| 446 |
_supports_flash_attn = True
|
| 447 |
+
_supports_sdpa = True
|
| 448 |
_supports_flex_attn = False
|
| 449 |
|
| 450 |
_can_compile_fullgraph = True
|
|
|
|
| 469 |
)
|
| 470 |
self.norm = torch.nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
| 471 |
self.rotary_emb = DomynEdgeRotaryEmbedding(config=config)
|
| 472 |
+
self.gradient_checkpointing = False
|
| 473 |
|
| 474 |
# Initialize weights and apply final processing
|
| 475 |
self.post_init()
|
|
|
|
| 514 |
)
|
| 515 |
|
| 516 |
for decoder_layer in self.layers:
|
| 517 |
+
if self.gradient_checkpointing and self.training:
|
| 518 |
+
# Layers are stateless, so activations can be recomputed in backward (no KV cache during training).
|
| 519 |
+
hidden_states = self._gradient_checkpointing_func(
|
| 520 |
+
decoder_layer.__call__, hidden_states, attention_mask, past_key_values, position_embeddings
|
| 521 |
+
)
|
| 522 |
+
else:
|
| 523 |
+
hidden_states = decoder_layer(
|
| 524 |
+
hidden_states,
|
| 525 |
+
attention_mask=attention_mask,
|
| 526 |
+
position_embeddings=position_embeddings,
|
| 527 |
+
past_key_values=past_key_values,
|
| 528 |
+
**kwargs,
|
| 529 |
+
)
|
| 530 |
|
| 531 |
hidden_states = self.norm(hidden_states)
|
| 532 |
return transformers.modeling_outputs.BaseModelOutputWithPast(
|