Text Generation
Transformers
Safetensors
GGUF
English
qwen3.5
gated-deltanet
hybrid-attention
dense
tinystories
tiny-model
conversational
Instructions to use shibatch/tinyqwen35dense2m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shibatch/tinyqwen35dense2m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="shibatch/tinyqwen35dense2m")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("shibatch/tinyqwen35dense2m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use shibatch/tinyqwen35dense2m with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf shibatch/tinyqwen35dense2m:Q4_K_M # Run inference directly in the terminal: llama cli -hf shibatch/tinyqwen35dense2m:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf shibatch/tinyqwen35dense2m:Q4_K_M # Run inference directly in the terminal: llama cli -hf shibatch/tinyqwen35dense2m:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf shibatch/tinyqwen35dense2m:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf shibatch/tinyqwen35dense2m:Q4_K_M
Use Docker
docker model run hf.co/shibatch/tinyqwen35dense2m:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use shibatch/tinyqwen35dense2m with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "shibatch/tinyqwen35dense2m" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinyqwen35dense2m", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/shibatch/tinyqwen35dense2m:Q4_K_M
- SGLang
How to use shibatch/tinyqwen35dense2m with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "shibatch/tinyqwen35dense2m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinyqwen35dense2m", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "shibatch/tinyqwen35dense2m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinyqwen35dense2m", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use shibatch/tinyqwen35dense2m with Ollama:
ollama run hf.co/shibatch/tinyqwen35dense2m:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use shibatch/tinyqwen35dense2m with Docker Model Runner:
docker model run hf.co/shibatch/tinyqwen35dense2m:Q4_K_M
- Lemonade
How to use shibatch/tinyqwen35dense2m with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull shibatch/tinyqwen35dense2m:Q4_K_M
Run and chat with the model
lemonade run user.tinyqwen35dense2m-Q4_K_M
List all available models
lemonade list
- Atomic Chat
|
Download README.md from shibatch/tinyqwen35dense2m: direct link, hf CLI and curl.
- Browser
- Download file 5.11 kB
-
https://huggingface.co/shibatch/tinyqwen35dense2m/resolve/main/README.md
- Command line
-
hf download hf://shibatch/tinyqwen35dense2m/README.md
-
curl -L -o README.md https://huggingface.co/shibatch/tinyqwen35dense2m/resolve/main/README.md
5.11 kB
| library_name: transformers | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| license: mit | |
| datasets: | |
| - roneneldan/TinyStories | |
| tags: | |
| - qwen3.5 | |
| - gated-deltanet | |
| - hybrid-attention | |
| - dense | |
| - tinystories | |
| - tiny-model | |
| # Tiny Qwen3.5 Dense 2M | |
| This is a **2,068,944-parameter, text-only Qwen3.5 dense** language model | |
| trained from scratch on one epoch of the full TinyStories training corpus. | |
| It uses `Qwen3_5ForCausalLM`: three Gated DeltaNet linear-attention layers | |
| followed by one full-attention layer, with a dense gated MLP in each layer. | |
| It is not an official Qwen model, has no original Qwen weights, and is not | |
| intended as a general-purpose assistant. | |
| ## Files | |
| - `hf/`: Hugging Face float32 checkpoint and tokenizer | |
| - `gguf/tinyqwen35dense2m-Q4_K_M.gguf`: llama.cpp Q4_K_M checkpoint | |
| - `example_generate.py`: local Transformers generation example | |
| - `convert_to_gguf.py`: reproducible HF-to-GGUF conversion and quantization | |
| - `artifact_metadata.json`: training arguments and final evaluation | |
| - `eval_text_generation.json`: final sampled continuations | |
| - `qwen3_5_config_dump.json`: expanded model configuration | |
| Optimizer checkpoints and the full training log are intentionally omitted | |
| from the distribution package. | |
| ## Architecture | |
| ```yaml | |
| architecture: Qwen3_5ForCausalLM | |
| model_type: qwen3_5_text | |
| parameter_count: 2,068,944 | |
| model_vocab_size: 1,024 | |
| tokenizer_size: 1,004 | |
| hidden_size: 256 | |
| intermediate_size: 256 | |
| num_hidden_layers: 4 | |
| layer_types: [linear_attention, linear_attention, linear_attention, full_attention] | |
| num_attention_heads: 8 | |
| num_key_value_heads: 1 | |
| head_dim: 32 | |
| linear_num_key_heads: 4 | |
| linear_key_head_dim: 32 | |
| linear_num_value_heads: 8 | |
| linear_value_head_dim: 32 | |
| linear_conv_kernel_dim: 4 | |
| tie_word_embeddings: true | |
| max_position_embeddings: 1,024 | |
| ``` | |
| The small tokenizer is a custom byte-level BPE shared with a local Tiny | |
| Qwen3-Next experiment, not the original Qwen3.5 tokenizer. The 1,024-row | |
| embedding matrix includes unused padding rows. For this text-only model, | |
| the explicit M-RoPE sections `[4, 0, 0]` provide llama.cpp with the | |
| required metadata; all position axes are equivalent for text tokens. | |
| ## Training | |
| The model was trained with float32 parameters and BF16 autocast on an NVIDIA | |
| GeForce RTX 5060 Ti. Flash Linear Attention accelerated the Gated DeltaNet | |
| training path. Stories were packed as `BOS + story + EOS` into contiguous | |
| 256-token blocks without | |
| padding. An independent 1% of the corpus was held out for validation. | |
| ```yaml | |
| selected_stories: 2,119,489 | |
| training_stories: 2,098,294 | |
| validation_stories: 21,195 | |
| training_blocks: 2,441,089 | |
| validation_blocks: 24,608 | |
| dataset_fraction: 1.0 | |
| epochs: 1.0 | |
| batch_size: 32 | |
| block_size: 256 | |
| training_steps: 76,284 | |
| tokens_processed: 624,918,528 | |
| training_time_including_preprocessing: about 25m 45s | |
| optimizer: AdamW | |
| peak_learning_rate: 4.0e-4 | |
| warmup_steps: 1,000 | |
| scheduler: warmup + cosine decay to 10% of peak | |
| weight_decay: 0 | |
| gradient_clip: 1.0 | |
| ``` | |
| The training entry point in the source workspace is | |
| `train_qwen3_5_dense.py`. Exact arguments and final metrics are recorded | |
| in `artifact_metadata.json`. | |
| ## Evaluation | |
| The final validation loss over 16 batches was **1.4809** (perplexity | |
| **4.3969**), down from 1.7282 at step 10,000. The final 500-step | |
| training loss averaged 1.4735. The complete metrics and generated samples | |
| are in `artifact_metadata.json` and `eval_text_generation.json`. One final | |
| sample starts: | |
| > There was a little girl named Lily. She loved to play in the woods near | |
| > her home. One day, she found a big leaf on the ground. She picked it up | |
| > and held it close. | |
| These are training diagnostics, not general language-model benchmarks. | |
| The model is small, so repetition, contradictions, and abrupt endings | |
| remain possible. | |
| ## Transformers usage | |
| Install the inference requirements and run: | |
| ```bash | |
| pip install -r requirements.txt | |
| python example_generate.py | |
| ``` | |
| Transformers 5.15.0 and `flash-linear-attention` 0.6.0 were used for | |
| training. The optional `causal-conv1d` package was not installed; | |
| Transformers' PyTorch convolution path handled that part of the model. | |
| ## llama.cpp usage | |
| ```bash | |
| llama-completion -m gguf/tinyqwen35dense2m-Q4_K_M.gguf \ | |
| -p 'Once upon' -n 100 -t 4 -no-cnv | |
| ``` | |
| Q4_K_M is a quantization *preset*: the file mixes Q4_K and | |
| higher-precision tensor types. This GGUF includes 27 Q4_K tensors among | |
| its 55 tensors. To regenerate it from `hf/` with the | |
| Debian llama.cpp tools installed, run `python convert_to_gguf.py`. | |
| For a llama.cpp checkout, pass | |
| `--converter /path/to/llama.cpp/convert_hf_to_gguf.py`. | |
| The wrapper validates the custom byte-level BPE before providing | |
| equivalent `gpt-2` pre-tokenizer metadata, which the upstream converter | |
| does not recognize from this tokenizer's probe hash. The intermediate | |
| F16 GGUF is removed after conversion. | |
| ## License and attribution | |
| MIT License; see `LICENSE` (Naoki Shibata). TinyStories is by Ronen Eldan | |
| and Yuanzhi Li. See the | |
| [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories) | |
| and the [Transformers Qwen3.5 documentation](https://huggingface.co/docs/transformers/model_doc/qwen3_5). | |