tinyqwen35dense2m / README.md
shibatch's picture
Upload folder using huggingface_hub
0bd3079 verified
|
Raw History Blame Contribute Delete
5.11 kB
---
library_name: transformers
pipeline_tag: text-generation
language:
- en
license: mit
datasets:
- roneneldan/TinyStories
tags:
- qwen3.5
- gated-deltanet
- hybrid-attention
- dense
- tinystories
- tiny-model
---
# Tiny Qwen3.5 Dense 2M
This is a **2,068,944-parameter, text-only Qwen3.5 dense** language model
trained from scratch on one epoch of the full TinyStories training corpus.
It uses `Qwen3_5ForCausalLM`: three Gated DeltaNet linear-attention layers
followed by one full-attention layer, with a dense gated MLP in each layer.
It is not an official Qwen model, has no original Qwen weights, and is not
intended as a general-purpose assistant.
## Files
- `hf/`: Hugging Face float32 checkpoint and tokenizer
- `gguf/tinyqwen35dense2m-Q4_K_M.gguf`: llama.cpp Q4_K_M checkpoint
- `example_generate.py`: local Transformers generation example
- `convert_to_gguf.py`: reproducible HF-to-GGUF conversion and quantization
- `artifact_metadata.json`: training arguments and final evaluation
- `eval_text_generation.json`: final sampled continuations
- `qwen3_5_config_dump.json`: expanded model configuration
Optimizer checkpoints and the full training log are intentionally omitted
from the distribution package.
## Architecture
```yaml
architecture: Qwen3_5ForCausalLM
model_type: qwen3_5_text
parameter_count: 2,068,944
model_vocab_size: 1,024
tokenizer_size: 1,004
hidden_size: 256
intermediate_size: 256
num_hidden_layers: 4
layer_types: [linear_attention, linear_attention, linear_attention, full_attention]
num_attention_heads: 8
num_key_value_heads: 1
head_dim: 32
linear_num_key_heads: 4
linear_key_head_dim: 32
linear_num_value_heads: 8
linear_value_head_dim: 32
linear_conv_kernel_dim: 4
tie_word_embeddings: true
max_position_embeddings: 1,024
```
The small tokenizer is a custom byte-level BPE shared with a local Tiny
Qwen3-Next experiment, not the original Qwen3.5 tokenizer. The 1,024-row
embedding matrix includes unused padding rows. For this text-only model,
the explicit M-RoPE sections `[4, 0, 0]` provide llama.cpp with the
required metadata; all position axes are equivalent for text tokens.
## Training
The model was trained with float32 parameters and BF16 autocast on an NVIDIA
GeForce RTX 5060 Ti. Flash Linear Attention accelerated the Gated DeltaNet
training path. Stories were packed as `BOS + story + EOS` into contiguous
256-token blocks without
padding. An independent 1% of the corpus was held out for validation.
```yaml
selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
training_blocks: 2,441,089
validation_blocks: 24,608
dataset_fraction: 1.0
epochs: 1.0
batch_size: 32
block_size: 256
training_steps: 76,284
tokens_processed: 624,918,528
training_time_including_preprocessing: about 25m 45s
optimizer: AdamW
peak_learning_rate: 4.0e-4
warmup_steps: 1,000
scheduler: warmup + cosine decay to 10% of peak
weight_decay: 0
gradient_clip: 1.0
```
The training entry point in the source workspace is
`train_qwen3_5_dense.py`. Exact arguments and final metrics are recorded
in `artifact_metadata.json`.
## Evaluation
The final validation loss over 16 batches was **1.4809** (perplexity
**4.3969**), down from 1.7282 at step 10,000. The final 500-step
training loss averaged 1.4735. The complete metrics and generated samples
are in `artifact_metadata.json` and `eval_text_generation.json`. One final
sample starts:
> There was a little girl named Lily. She loved to play in the woods near
> her home. One day, she found a big leaf on the ground. She picked it up
> and held it close.
These are training diagnostics, not general language-model benchmarks.
The model is small, so repetition, contradictions, and abrupt endings
remain possible.
## Transformers usage
Install the inference requirements and run:
```bash
pip install -r requirements.txt
python example_generate.py
```
Transformers 5.15.0 and `flash-linear-attention` 0.6.0 were used for
training. The optional `causal-conv1d` package was not installed;
Transformers' PyTorch convolution path handled that part of the model.
## llama.cpp usage
```bash
llama-completion -m gguf/tinyqwen35dense2m-Q4_K_M.gguf \
-p 'Once upon' -n 100 -t 4 -no-cnv
```
Q4_K_M is a quantization *preset*: the file mixes Q4_K and
higher-precision tensor types. This GGUF includes 27 Q4_K tensors among
its 55 tensors. To regenerate it from `hf/` with the
Debian llama.cpp tools installed, run `python convert_to_gguf.py`.
For a llama.cpp checkout, pass
`--converter /path/to/llama.cpp/convert_hf_to_gguf.py`.
The wrapper validates the custom byte-level BPE before providing
equivalent `gpt-2` pre-tokenizer metadata, which the upstream converter
does not recognize from this tokenizer's probe hash. The intermediate
F16 GGUF is removed after conversion.
## License and attribution
MIT License; see `LICENSE` (Naoki Shibata). TinyStories is by Ronen Eldan
and Yuanzhi Li. See the
[TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories)
and the [Transformers Qwen3.5 documentation](https://huggingface.co/docs/transformers/model_doc/qwen3_5).