Configuration Parsing Warning:In adapter_config.json: "peft.base_model_name_or_path" must be a string

gpt-oss-20b Transcript Formatter β€” LoRA adapter (distilled student)

A LoRA adapter for openai/gpt-oss-20b that turns a raw speech-to-text transcript into a clean, formatted transcript β€” fixing punctuation, casing, fillers, disfluencies, homophones, ITN, garbled proper nouns / phonetic URLs-emails, and spoken layout commands β€” while preserving the speaker's words and passing through anything that is already correct.

This is the distilled 20B student of Akash-Sakala/gpt-oss-120b-transcript-formatter-lora. It reproduces the teacher's formatting behaviour at a fraction of the cost: direct final channel, no chain-of-thought, and a 20B base that fits on a single large-VRAM GPU.

This repo ships only the LoRA adapter (~hundreds of MB), not the merged 20B. Merge it on demand against the public base β€” see Usage.

Model details

Base model openai/gpt-oss-20b (MoE, harmony format)
Adapter LoRA rank 32 (Apache-2.0, same as base)
Renderer / format gpt_oss_no_sysprompt (harmony); direct final channel, no chain-of-thought
Task ASR transcript formatting (single-turn: system + raw transcript β†’ formatted transcript)
Languages English (multi-dialect: en-US/GB/AU/IN/CA/IE/ZA)
Teacher gpt-oss-120b-transcript-formatter-lora (RL-final)
Trained with Tinker API + tinker-cookbook
Hyperparameters LoRA rank 32 Β· lr 2e-4, cosine Β· 2 epochs Β· batch 64 Β· max_len 4096 Β· AdamW (Ξ²1 0.9, Ξ²2 0.95, Ξ΅ 1e-8) Β· seed 42

Training approach β€” distillation

The student is trained by distilling the 120B teacher onto the 20B base (Phase-1 offline SFT):

  1. Teacher generation. The RL-final 120B teacher produces gold formatted outputs over the transcript-formatter-curriculum prompts (all 21 categories, L0–L5 + RL-align).
  2. Student SFT. gpt-oss-20b is fine-tuned (LoRA rank 32, lr 2e-4 cosine, 2 epochs, batch 64, max_len 4096, AdamW Ξ²=(0.9, 0.95)) to match the teacher's outputs on the direct final channel β€” the reasoning trace is dropped, so the student answers immediately without chain-of-thought.
  3. Combined corpus. All curriculum layers are pooled into a single 18,505 (input β†’ gold) pair SFT mix (priority-ordered so every category and the passthrough negative-traps are represented), rather than trained stage by stage. Layer composition:
Layer Adds Pairs
L0 basics (punctuation, capitals, passthrough) 2,442
L1 surface (filler, homophone, ITN, comma, date) 2,400
L2 disfluency (stutter, false-start, backtrack) 4,122
L3 artifacts (proper nouns, URLs/emails) 4,220
L4 layout (emails, lists, symbols) 2,157
L5 corrective (over-format fix, gold cleanup) 2,974
RL-align archetype-verifiable prompts 190

21 categories total, 18,505 pairs. Full dataset: Akash-Sakala/transcript-formatter-curriculum.

Results

Held-out evaluation over 2,384 rows across all 21 categories, scored with the same two-round protocol as the teacher: deterministic Round 1 (EM / ContentOK / CER), then an independent Claude judge re-adjudicates Round-1 fails to credit cases where the model is correct but the gold label is unrealistic ("adjusted accuracy").

Metric 20B student 120B teacher
Judge-adjusted accuracy 99.5% 99.9%
Round-1 PASS 98.2% 98.4%
Exact Match 95.2% 94.8%
ContentOK (no hallucination/omission) 98.3% 98.5%
CER / WER 0.0008 / 0.0020 0.0011 / 0.0033
FalseEdit (passthrough safety) 1.3% 0.9%

Student numbers measured over 2,384 held-out rows (all 21 categories), Round-1 judge claude-sonnet-4-6; 44 Round-1 fails re-adjudicated β†’ 33 rescued, 11 confirmed fails. Source: final_eval_all_layers_20B_p1.json / final_eval_round1_20B_p1.json. The distillation hits teacher parity (99.5% vs 99.9% adjusted acc) at 20B with no reasoning β€” cheaper and faster to serve, and the student's EM/CER/WER slightly beat the teacher.

EM understates quality for the layout categories (many valid paragraph/list renderings); read ContentOK / adjusted accuracy / CER, not EM.

Usage

⚠ The system prompt is required (the model was trained with it verbatim).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-20b", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "Akash-Sakala/gpt-oss-20b-transcript-formatter-lora")
tok = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")

SYSTEM_PROMPT = "..."  # the verbatim training system prompt
messages = [{"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": "i spoke to aisha she confirmed the PR will be merged by friday"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=4096, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=False))
# β†’ "I spoke to Aisha. She confirmed the PR will be merged by Friday."

Merge once for serving (vLLM/SGLang):

# Option A β€” PEFT
merged = PeftModel.from_pretrained(base, "Akash-Sakala/gpt-oss-20b-transcript-formatter-lora").merge_and_unload()
merged.save_pretrained("./gpt-oss-20b-transcript-formatter-merged")

# Option B β€” Tinker cookbook
from tinker_cookbook import weights
weights.build_hf_model(
    base_model="openai/gpt-oss-20b",
    adapter_path="./adapter",            # this repo, downloaded locally
    output_path="./merged",
    dtype="bfloat16",
)

Hardware: gpt-oss-20b is a 20B MoE β€” far cheaper than the 120B teacher. Greedy decoding (temperature 0) for deterministic output.

Intended use & limitations

  • Use: post-processing ASR/Whisper output into readable text (dictation, meeting notes, voice memos, emails).
  • Preserves words. It edits format, not meaning β€” it does not paraphrase, summarise, answer, or invent content; ambiguous cases pass through unchanged.
  • Limitations: English only; rare/novel proper-noun spellings can be missed; paragraph segmentation of free-dictated emails is subjective; not a general assistant (it only formats transcripts).
  • License: Apache-2.0, inherited from the base openai/gpt-oss-20b.

Citation / provenance

Distilled with the Tinker API from teacher gpt-oss-120b-transcript-formatter-lora. Student base openai/gpt-oss-20b, LoRA rank 32, lr 2e-4 cosine, 2 epochs, batch 64, direct-final (no reasoning), renderer gpt_oss_no_sysprompt. Phase-1 distillation run. Tinker checkpoint: ac4a6812-1d65-507a-9f93-080fab31f7e4:train:0/sampler_weights/final. See the dataset card for the full curriculum.

Downloads last month
7
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Akash-Sakala/gpt-oss-20b-transcript-formatter-lora

Adapter
(492)
this model

Dataset used to train Akash-Sakala/gpt-oss-20b-transcript-formatter-lora