issai/foggen-gemma3-1b

FogGen (Gemma-3-1b-it): cross-family R14 endpoint

The Gemma-architecture port of issai/foggen. Same 14-round self-evolving recipe, same cloud teacher, same seven MCQ domains; the edge family is the only change.

This checkpoint exists to test whether the FogGen recipe transfers across model family, not just across scale. The canonical 0.6B Qwen3-based endpoint lives at issai/foggen and is the deployment model. This Gemma variant demonstrates the recipe is not Qwen-specific: a different architecture trained with the same protocol still routes better than random routing in every round.

For the system overview, training pipeline, and routing protocol, see the issai/foggen model card; only the differences are documented here.

Recipe

• Cloud teacher: Qwen3-30B-A3B-Instruct-2507
• 7 domain rotation, same domain order
• 14 sequential SFT rounds (R0 → R14)
• LoRA r=16, α=32, all-linear, bf16, 2 epochs, lr=5e-5
• Same confidence buckets and same FogGen output format
• For R0 the 1,800-question calibration buffer is re-labeled from scratch with the raw Gemma-3-1b-it base (N=8 at T=0.7)

The only change is the edge backbone (google/gemma-3-1b-it in place of Qwen/Qwen3-0.6B). Note: the relabeled buffer's bucket distribution is sharply bimodal for Gemma-3-1b-it (mostly low-confidence and high-confidence rows, almost no middle-bucket mass), unlike the more balanced Qwen distribution. The recipe is robust to this; new-domain pools contribute enough middle-bucket exposure to keep the calibration vocabulary from collapsing.

Results

R14 at tau = 0.5 on the seven evaluation sets. System accuracy counts the edge answer on kept queries and the cloud answer (Qwen3-30B-A3B-Instruct-2507) on routed queries. Random routing sends the same share of queries to the cloud at random. Scores use the corrected answer keys of issai/foggen-data.

Domain	Cloud only	Edge only	Random routing	FogGen	Routed to cloud
Finance	92.3%	52.7%	65.3%	73.9%	31.8%
Science	93.6%	38.2%	72.6%	83.8%	62.1%
Coding	93.5%	55.8%	64.4%	70.7%	22.8%
Law	87.7%	50.4%	60.7%	68.0%	27.5%
Math	95.5%	32.2%	73.2%	79.7%	64.7%
Kazakh culture	96.2%	76.1%	79.0%	83.9%	14.7%
Medical	88.9%	44.0%	65.1%	74.3%	47.0%
Mean	92.5%	49.9%	68.6%	76.3%	38.6%

FogGen is 7.7 points above random routing at R14 and 7.5 points on average over rounds 1 to 14. The mean expected calibration error is 18.7 at R14. Against the stored keys of the first release, R14 reaches 62.0% against 57.8% for random routing.

Earlier versions of this card credited routed queries with the cloud's average accuracy over the whole set and used the stored keys.

Quick demo

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("issai/foggen-gemma3-1b", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("issai/foggen-gemma3-1b")

SYSTEM = """You are a self-aware multiple-choice assistant.

Rules:
- First, assess your confidence in solving this question.
- Then give your answer.
- Output format:
  Confidence: <0.0|0.25|0.5|0.75|1.0>
  Final answer: <OPTION_LETTER>"""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "<your MCQ here>"},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The routing decision (route_query helper, threshold τ) is identical to the issai/foggen card.

License

Inherits the Gemma Terms of Use from google/gemma-3-1b-it.

Citation

Paper coming soon.