| --- |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - llama |
| - quartz |
| - causal-lm |
| - conversational |
| - safetensors |
| - gguf |
| - llama.cpp |
| pipeline_tag: text-generation |
| language: |
| - en |
| --- |
| |
| # Quartz-M1-100M-Instruct |
|
|
| Quartz-M1-100M-Instruct is a lightweight, ultra-fast 100M parameter language model based on the LLaMA architecture. It is designed for low-latency edge deployment, local chat, and high-throughput inference (reaching **400–485 tokens/second** on consumer GPUs like the RTX 3060). |
|
|
| --- |
|
|
| ## Model Files & Formats |
|
|
| | File | Format / Quant | Size | Recommended Use | |
| | :--- | :---: | :---: | :--- | |
| | `model.safetensors` | Safetensors (BF16/FP16) | ~202 MB | PyTorch, Hugging Face Transformers | |
| | `Quartz-M1-100M-Instruct-f16.gguf` | GGUF FP16 | ~202 MB | Full precision, llama.cpp, LM Studio, Ollama | |
| | `Quartz-M1-100M-Instruct-Q8_0.gguf` | GGUF Q8_0 | ~107 MB | Near-lossless 8-bit quantization | |
| | `Quartz-M1-100M-Instruct-Q4_K_M.gguf` | GGUF Q4_K_M | ~65 MB | Ultra-compact 4-bit quantization | |
| |
| --- |
| |
| ## Performance Benchmarks |
| |
| * **RTX 3060 (12GB) with CUDA**: |
| * **Text Generation:** ~400 – 485 Tokens/Second (TPS) |
| * **Prompt Processing:** ~23,000 – 33,000 Tokens/Second |
| * **VRAM Usage:** < 250 MB |
| |
| --- |
| |
| ## Quickstart |
| |
| ### 1. Using `llama.cpp` CLI |
| ```bash |
| llama-cli -m Quartz-M1-100M-Instruct-f16.gguf -ngl 99 -p "Explain quantum computing simply." -n 128 |
| ``` |
| |
| ### 2. Using `transformers` (Python) |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "honvert/Quartz-M1-100M-Instruct" |
|
|
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto") |
|
|
| messages = [ |
| {"role": "user", "content": "What are three key habits of effective developers?"} |
| ] |
| |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate(**inputs, max_new_tokens=150, temperature=0.7, top_p=0.9) |
| print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
| |
| --- |
| |
| ## Architecture & Parameters |
| * **Parameters:** 101,343,744 (101M) |
| * **Hidden Size:** 512 |
| * **Layers:** 24 |
| * **Attention Heads:** 8 (KV Heads: 4 - Grouped Query Attention) |
| * **Context Window:** 1,024 tokens |
| |
