doofz

/

THX-01

Like 80

Hi! Thanks for open-sourcing THX-01, the thx01 library and the RLCD recipe. I used them to build laya-indic, a model focused on Indian languages and Hinglish (Apache-2.0).

• Built on mmBERT-base, not THX-01's weights. It uses your DecisionModel architecture and RLCD objective, trained from mmBERT-base, with continued pretraining on Sangraha (AI4Bharat). The data is public, permissively licensed sets plus labels from open-weights Qwen3.
• Credit. I evaluated it with a harness adapted from your bench/tasks.py, and the card credits THX-01 for that, the architecture and the training method.
• Results. It does well on Indic intent and Hinglish, while THX-01 is still stronger on AG News and English/European SIB-200. The comparison table is on the model card.

Two things I ran into that might be useful to you:

1. The model field. Outputs always say "model": "THX-01", even for other checkpoints loaded with thx01.load(...), because the name is hardcoded in agent.py, decide.py and server.py. Reading model_name from rl_agent_config.json, with "THX-01" as the default, would let derived models identify themselves.
2. NaNs on H100. With torch 2.11, the cuDNN SDPA backend gave NaN gradients on some padded batches with a ModernBERT encoder. torch.backends.cuda.enable_cudnn_sdp(False) fixed it, in case others training with thx01 hit the same thing.

Thanks again. The library made this much easier.