MoE Sovereign Database Engineering & Data-Infrastructure Expert 4B (moe-expert-datainfra-4b)

License: Apache 2.0 Base Model: Qwen3.5-4B


Model Summary

moe-expert-datainfra-4b is a LoRA fine-tune (recipe 2) of the text-decoder of Qwen3.5-4B, specialized as the datainfra domain expert within the MoE Sovereign compound-AI system.

You are a database engineering, query optimization, and data-infrastructure expert (moe-expert-datainfra-4b) specialized in PostgreSQL, DuckDB, and ClickHouse. Give concrete, execution-ready SQL, EXPLAIN ANALYZE-based query-plan diagnosis, safe zero-downtime schema migrations, and precise index recommendations. Verify syntax before answering.

Base Architecture

Qwen3.5-4B is a hybrid linear-attention / full-attention decoder (not a plain Transformer): 32 layers (8 full-attention, 24 linear/Mamba-style), hidden size 2,560, 248,320-token vocabulary, native 262,144-token context window.

Training Configuration (recipe 2)

Parameter Value
Method LoRA (rank 32, alpha 64, dropout 0.05), targeting q/k/v/o_proj, gate/up/down_proj and the linear-attention projections in_proj_qkv, in_proj_z, out_proj
Trainable parameters 60,948,480 of 4,266,699,776 (1.43%)
Loss Assistant answer only (prompt tokens masked), non-thinking mode (empty <think> block in the prompt)
Epochs 2 (253 optimizer steps)
Effective batch size 32 (micro-batch 1 x 8 GPUs x grad-accum 4)
Learning rate 1e-4, cosine, 3% warmup
Training sequence length up to 6,144 tokens (longest example 4,553; nothing truncated)
Optimizer sharding DeepSpeed ZeRO-2, bf16
Compute EuroHPC LUMI-G, 8x AMD Instinct MI250X GCDs, ROCm, 1 h 40 min
Training examples 4,040: 3,232 datainfra instruction/response pairs (filtered from the 3,386 of the previous version: no secrets, placeholders or <think> blocks, no duplicates) plus 808 capability examples (tool calls, prompt-injection resistance, structured output, instruction following, German) so that general abilities are kept
Held-out 100 datainfra prompts never trained on

Observed Training Trajectory

Held-out loss: 0.696 (epoch 0.2) -> 0.651 (0.6) -> 0.622 (epoch 2). Training loss ended at about 0.55. These are language-modelling losses, not task accuracy.

Evaluation (N02-M60, 8 GB Tesla M60, Q4_K_M, KV q4_0, context 262,144 fully in VRAM)

Same tasks and conditions for all models; deterministic checkers (executed code and SQL, schema checks, exact values). Few tasks per domain (6), so differences of one task are noise.

this version v1 (previous) Qwen3.5-4B base
Domain probe (9 domains x 6 tasks) 43/54 = 0.80 34/54 = 0.63 44/54 = 0.81
datainfra domain 6/6 3/6 6/6
Capability probe (tool calls, injection, structured output, instruction following, thinking, German, long context) 29/33 = 0.88 14/33 = 0.42 33/33 = 1.00
Tool calls 10/12 0/12 12/12

What this means: recipe 2 repairs the damage of recipe 1 (tool calls, thinking mode, prompt-injection resistance) and keeps the datainfra domain at the base model's level. It does not beat the base model on these probes; the datainfra tasks are already saturated by the base model, so the probe cannot show a gain there. Known weaknesses: it guesses a missing tool argument instead of asking (2/2 tasks), summaries run too long, and it can invent citations and article numbers (research, governance). Treat answers as drafts to verify.

Held-out language-modelling loss: 0.696 (epoch 0.2) -> 0.622 (epoch 2).

Prompt Format

ChatML with an empty thinking block (the model was trained in non-thinking mode):

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
<think>

</think>

{response}<|im_end|>

System Prompt

You are a database engineering, query optimization, and data-infrastructure expert (moe-expert-datainfra-4b) specialized in PostgreSQL, DuckDB, and ClickHouse. Give concrete, execution-ready SQL, EXPLAIN ANALYZE-based query-plan diagnosis, safe zero-downtime schema migrations, and precise index recommendations. Verify syntax before answering.

Available Formats

File Notes
moe-expert-datainfra-4b-f16.gguf Unquantized reference (f16)
moe-expert-datainfra-4b-Q8_0.gguf Higher-fidelity reference quantization
moe-expert-datainfra-4b-Q4_K_M.gguf Recommended for single/multi-GPU deployment

SHA256SUMS lists the checksums. The previous version (recipe 1) stays available under the git tag v1.

Hardware Guidance

Native 262,144-token context usable in full on multi-GPU pools with q4_0-quantized KV-cache and Flash Attention (Ampere/Turing+). On single 8GB GPUs cap num_ctx to what fits in VRAM and use f16 KV-cache where Flash Attention is not supported (Maxwell-generation GPUs).

Ollama Modelfile

FROM ./moe-expert-datainfra-4b-Q4_K_M.gguf
SYSTEM """You are a database engineering, query optimization, and data-infrastructure expert (moe-expert-datainfra-4b) specialized in PostgreSQL, DuckDB, and ClickHouse. Give concrete, execution-ready SQL, EXPLAIN ANALYZE-based query-plan diagnosis, safe zero-downtime schema migrations, and precise index recommendations. Verify syntax before answering."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
<think>

</think>

{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 32768

Limitations

  • Does not execute code/queries/tools itself; outputs should be validated against the actual system before use.
  • Specialized for its domain; general-purpose conversation is out of scope.
  • Trained in non-thinking mode; behaviour with thinking enabled is not trained.

License

Apache 2.0, inherited from the Qwen3.5-4B base model.

Downloads last month
161
GGUF
Model size
4B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for h3rb3rn/moe-expert-datainfra-4b

Finetuned
Qwen/Qwen3.5-4B
Adapter
(1,112)
this model