shibatch/tinyqwen35dense2m

Tiny Qwen3.5 Dense 2M

This is a 2,068,944-parameter, text-only Qwen3.5 dense language model trained from scratch on one epoch of the full TinyStories training corpus. It uses Qwen3_5ForCausalLM: three Gated DeltaNet linear-attention layers followed by one full-attention layer, with a dense gated MLP in each layer. It is not an official Qwen model, has no original Qwen weights, and is not intended as a general-purpose assistant.

Files

• hf/: Hugging Face float32 checkpoint and tokenizer
• gguf/tinyqwen35dense2m-Q4_K_M.gguf: llama.cpp Q4_K_M checkpoint
• example_generate.py: local Transformers generation example
• convert_to_gguf.py: reproducible HF-to-GGUF conversion and quantization
• artifact_metadata.json: training arguments and final evaluation
• eval_text_generation.json: final sampled continuations
• qwen3_5_config_dump.json: expanded model configuration

Optimizer checkpoints and the full training log are intentionally omitted from the distribution package.

Architecture

architecture: Qwen3_5ForCausalLM
model_type: qwen3_5_text
parameter_count: 2,068,944
model_vocab_size: 1,024
tokenizer_size: 1,004
hidden_size: 256
intermediate_size: 256
num_hidden_layers: 4
layer_types: [linear_attention, linear_attention, linear_attention, full_attention]
num_attention_heads: 8
num_key_value_heads: 1
head_dim: 32
linear_num_key_heads: 4
linear_key_head_dim: 32
linear_num_value_heads: 8
linear_value_head_dim: 32
linear_conv_kernel_dim: 4
tie_word_embeddings: true
max_position_embeddings: 1,024

The small tokenizer is a custom byte-level BPE shared with a local Tiny Qwen3-Next experiment, not the original Qwen3.5 tokenizer. The 1,024-row embedding matrix includes unused padding rows. For this text-only model, the explicit M-RoPE sections [4, 0, 0] provide llama.cpp with the required metadata; all position axes are equivalent for text tokens.

Training

The model was trained with float32 parameters and BF16 autocast on an NVIDIA GeForce RTX 5060 Ti. Flash Linear Attention accelerated the Gated DeltaNet training path. Stories were packed as BOS + story + EOS into contiguous 256-token blocks without padding. An independent 1% of the corpus was held out for validation.

selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
training_blocks: 2,441,089
validation_blocks: 24,608
dataset_fraction: 1.0
epochs: 1.0
batch_size: 32
block_size: 256
training_steps: 76,284
tokens_processed: 624,918,528
training_time_including_preprocessing: about 25m 45s
optimizer: AdamW
peak_learning_rate: 4.0e-4
warmup_steps: 1,000
scheduler: warmup + cosine decay to 10% of peak
weight_decay: 0
gradient_clip: 1.0

The training entry point in the source workspace is train_qwen3_5_dense.py. Exact arguments and final metrics are recorded in artifact_metadata.json.

Evaluation

The final validation loss over 16 batches was 1.4809 (perplexity 4.3969), down from 1.7282 at step 10,000. The final 500-step training loss averaged 1.4735. The complete metrics and generated samples are in artifact_metadata.json and eval_text_generation.json. One final sample starts:

There was a little girl named Lily. She loved to play in the woods near her home. One day, she found a big leaf on the ground. She picked it up and held it close.

These are training diagnostics, not general language-model benchmarks. The model is small, so repetition, contradictions, and abrupt endings remain possible.

Transformers usage

Install the inference requirements and run:

pip install -r requirements.txt
python example_generate.py

Transformers 5.15.0 and flash-linear-attention 0.6.0 were used for training. The optional causal-conv1d package was not installed; Transformers' PyTorch convolution path handled that part of the model.

llama.cpp usage

llama-completion -m gguf/tinyqwen35dense2m-Q4_K_M.gguf \
  -p 'Once upon' -n 100 -t 4 -no-cnv

Q4_K_M is a quantization preset: the file mixes Q4_K and higher-precision tensor types. This GGUF includes 27 Q4_K tensors among its 55 tensors. To regenerate it from hf/ with the Debian llama.cpp tools installed, run python convert_to_gguf.py. For a llama.cpp checkout, pass --converter /path/to/llama.cpp/convert_hf_to_gguf.py. The wrapper validates the custom byte-level BPE before providing equivalent gpt-2 pre-tokenizer metadata, which the upstream converter does not recognize from this tokenizer's probe hash. The intermediate F16 GGUF is removed after conversion.

License and attribution

MIT License; see LICENSE (Naoki Shibata). TinyStories is by Ronen Eldan and Yuanzhi Li. See the TinyStories dataset and the Transformers Qwen3.5 documentation.