--- library_name: transformers pipeline_tag: text-generation language: - en license: mit datasets: - roneneldan/TinyStories tags: - qwen3.5 - gated-deltanet - hybrid-attention - dense - tinystories - tiny-model --- # Tiny Qwen3.5 Dense 2M This is a **2,068,944-parameter, text-only Qwen3.5 dense** language model trained from scratch on one epoch of the full TinyStories training corpus. It uses `Qwen3_5ForCausalLM`: three Gated DeltaNet linear-attention layers followed by one full-attention layer, with a dense gated MLP in each layer. It is not an official Qwen model, has no original Qwen weights, and is not intended as a general-purpose assistant. ## Files - `hf/`: Hugging Face float32 checkpoint and tokenizer - `gguf/tinyqwen35dense2m-Q4_K_M.gguf`: llama.cpp Q4_K_M checkpoint - `example_generate.py`: local Transformers generation example - `convert_to_gguf.py`: reproducible HF-to-GGUF conversion and quantization - `artifact_metadata.json`: training arguments and final evaluation - `eval_text_generation.json`: final sampled continuations - `qwen3_5_config_dump.json`: expanded model configuration Optimizer checkpoints and the full training log are intentionally omitted from the distribution package. ## Architecture ```yaml architecture: Qwen3_5ForCausalLM model_type: qwen3_5_text parameter_count: 2,068,944 model_vocab_size: 1,024 tokenizer_size: 1,004 hidden_size: 256 intermediate_size: 256 num_hidden_layers: 4 layer_types: [linear_attention, linear_attention, linear_attention, full_attention] num_attention_heads: 8 num_key_value_heads: 1 head_dim: 32 linear_num_key_heads: 4 linear_key_head_dim: 32 linear_num_value_heads: 8 linear_value_head_dim: 32 linear_conv_kernel_dim: 4 tie_word_embeddings: true max_position_embeddings: 1,024 ``` The small tokenizer is a custom byte-level BPE shared with a local Tiny Qwen3-Next experiment, not the original Qwen3.5 tokenizer. The 1,024-row embedding matrix includes unused padding rows. For this text-only model, the explicit M-RoPE sections `[4, 0, 0]` provide llama.cpp with the required metadata; all position axes are equivalent for text tokens. ## Training The model was trained with float32 parameters and BF16 autocast on an NVIDIA GeForce RTX 5060 Ti. Flash Linear Attention accelerated the Gated DeltaNet training path. Stories were packed as `BOS + story + EOS` into contiguous 256-token blocks without padding. An independent 1% of the corpus was held out for validation. ```yaml selected_stories: 2,119,489 training_stories: 2,098,294 validation_stories: 21,195 training_blocks: 2,441,089 validation_blocks: 24,608 dataset_fraction: 1.0 epochs: 1.0 batch_size: 32 block_size: 256 training_steps: 76,284 tokens_processed: 624,918,528 training_time_including_preprocessing: about 25m 45s optimizer: AdamW peak_learning_rate: 4.0e-4 warmup_steps: 1,000 scheduler: warmup + cosine decay to 10% of peak weight_decay: 0 gradient_clip: 1.0 ``` The training entry point in the source workspace is `train_qwen3_5_dense.py`. Exact arguments and final metrics are recorded in `artifact_metadata.json`. ## Evaluation The final validation loss over 16 batches was **1.4809** (perplexity **4.3969**), down from 1.7282 at step 10,000. The final 500-step training loss averaged 1.4735. The complete metrics and generated samples are in `artifact_metadata.json` and `eval_text_generation.json`. One final sample starts: > There was a little girl named Lily. She loved to play in the woods near > her home. One day, she found a big leaf on the ground. She picked it up > and held it close. These are training diagnostics, not general language-model benchmarks. The model is small, so repetition, contradictions, and abrupt endings remain possible. ## Transformers usage Install the inference requirements and run: ```bash pip install -r requirements.txt python example_generate.py ``` Transformers 5.15.0 and `flash-linear-attention` 0.6.0 were used for training. The optional `causal-conv1d` package was not installed; Transformers' PyTorch convolution path handled that part of the model. ## llama.cpp usage ```bash llama-completion -m gguf/tinyqwen35dense2m-Q4_K_M.gguf \ -p 'Once upon' -n 100 -t 4 -no-cnv ``` Q4_K_M is a quantization *preset*: the file mixes Q4_K and higher-precision tensor types. This GGUF includes 27 Q4_K tensors among its 55 tensors. To regenerate it from `hf/` with the Debian llama.cpp tools installed, run `python convert_to_gguf.py`. For a llama.cpp checkout, pass `--converter /path/to/llama.cpp/convert_hf_to_gguf.py`. The wrapper validates the custom byte-level BPE before providing equivalent `gpt-2` pre-tokenizer metadata, which the upstream converter does not recognize from this tokenizer's probe hash. The intermediate F16 GGUF is removed after conversion. ## License and attribution MIT License; see `LICENSE` (Naoki Shibata). TinyStories is by Ronen Eldan and Yuanzhi Li. See the [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories) and the [Transformers Qwen3.5 documentation](https://huggingface.co/docs/transformers/model_doc/qwen3_5).