Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,204 @@
+---
+license: apache-2.0
+library_name: transformers
+language:
+  - en
+pipeline_tag: text-generation
+tags:
+  - moe
+  - mixture-of-experts
+  - from-scratch
+  - instruction-tuned
+  - custom_code
+vertex_family: quartz
+vertex_name: Quartz Micro Preview
+vertex_base: VertexAGI/quartz-micro-preview-base
+vertex_params: 1.03B
+vertex_purpose: Full-parameter instruction tuning of the from-scratch pretrain
+vertex_result: "Full-parameter SFT on databricks-dolly-15k, 1,317 steps. Modest, mixed improvement over the base checkpoint: clearly better at grounded/context QA, still weak at strict format- and length-following."
+vertex_status: released
+---
+
+# Quartz Micro Preview
+
+The instruction-tuned release of [Quartz Micro Preview
+Base](https://huggingface.co/VertexAGI/quartz-micro-preview-base) — the
+same ~1B-parameter mixture-of-experts model, trained completely from
+scratch on a single consumer GPU, now given a full-parameter supervised
+fine-tune so it follows instructions instead of just completing text.
+
+This is a **preview**, not a finished product. It is small on every axis
+by design, and its evaluation below is reported honestly, including where
+it still falls short — that's this project's standing practice, not a
+disclaimer added after the fact.
+
+## What changed vs. the base checkpoint
+
+- **Full-parameter SFT**, not LoRA — the base model has zero
+  instruction-following behavior, so a light adapter touch isn't the
+  right tool; every parameter was updated.
+- **Dataset:** [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k),
+  human-written (not distilled from another company's model), formatted as
+  `### Instruction:\n...\n\n### Response:\n` (with an added `### Context:\n`
+  block for context-grounded examples), tokenized with the pretrain's own
+  custom 32K byte-level BPE tokenizer. 14,051 of 15,011 examples kept after
+  filtering to the model's 2,048-token context.
+- Only the response tokens are supervised (prompt tokens masked to
+  `label=-100`); optimizer is 8-bit AdamW (bitsandbytes) with full gradient
+  checkpointing, matching the pretrain's own training stack.
+- This is the **second** SFT attempt (v2): the first pass underperformed,
+  so this run used a higher learning rate and a fresh optimizer state
+  rather than continuing from the first run's optimizer. It completed
+  1,317 steps on a single GTX 1660 Ti (6GB VRAM).
+
+## Architecture
+
+Unchanged from the base checkpoint — DeepSeek-style fine-grained
+mixture-of-experts:
+
+| | |
+|---|---|
+| Total parameters | 1,031.0M (~1.03B) |
+| Active parameters / token | 394.0M |
+| Hidden size | 1,024 |
+| Layers | 20 (first 2 dense, rest MoE) |
+| Attention | 16 query heads / 4 KV heads (GQA), head dim 64 |
+| Context length | 2,048 tokens |
+| Routed experts | 24 (6 active per token) |
+| Shared experts | 2 (always active) |
+| Expert FFN size | 640 (fine-grained segmentation) |
+| Router | top-6 of 24, 0.01-weighted load-balancing loss |
+| Vocabulary | 32,000 tokens |
+| Tied embeddings | yes |
+
+## Held-out evaluation: base vs. this SFT checkpoint
+
+Ten hand-authored prompts, not sampled from dolly-15k or any training
+data, spanning open QA, closed/context QA, summarization, brainstorming,
+classification, extraction, general QA, instruction-following, and
+creative writing. Full transcript below is unedited.
+
+**Honest summary: a modest, mixed improvement — not a clean win.**
+
+The clearest gain is grounded, context-based QA. The weakest spot is
+still strict instruction-following (format, length). Both checkpoints
+remain largely incoherent past the first sentence or two, which tracks
+with the model's size — that's not something this SFT pass was expected
+to fix.
+
+**Win — closed QA (uses given context correctly):**
+> Prompt: *Context: The Golden Gate Bridge is 2,737 meters long and was
+> completed in 1937.\nQuestion: How long is the Golden Gate Bridge?*
+>
+> Base: ignores the context entirely, wanders into an unrelated story about
+> "the Banana" process and the Coast Guard.
+>
+> SFT: *"The Golden Gate Bridge is 2,737 meters long and has an area of
+> approximately 2.4 million square miles."* — correctly extracts and
+> restates the fact (the second clause is invented, but the requested
+> answer is right).
+
+**Partial win — brainstorming (attempts the requested list format):**
+> Prompt: *List three ideas for reducing food waste in a household kitchen.*
+>
+> Base: no list structure, generic unrelated advice about water and diet.
+>
+> SFT: *"1. Make sure you have all food waste in your kitchen. \n2.f leaf
+> food and water should be discharged from local resources. \n3. Focus on
+> local food waste..."* — correctly produces a numbered list (the content
+> is still weak/garbled).
+
+**Partial win — topical relevance:**
+> Prompt: *Why do cats purr?*
+>
+> Base: drifts immediately into an unrelated riff on "the human body,"
+> never mentions cats again.
+>
+> SFT: *"Finding the right balance between breast and chest health is
+> hard... So my recommendation is to keep your cat's skin healthy and
+> strong..."* — stays anchored to "cat" throughout, base does not.
+
+**No win — strict instruction-following:**
+> Prompt: *Write a short two-sentence greeting for a birthday card.*
+>
+> Base: an unrelated bulleted list of vague social advice.
+>
+> SFT: *"The free meeting is open for all people... The national party has
+> the following 82 seats: The Democratic Party..."* — completely
+> unrelated to the prompt, and no better than base here.
+
+**No win — classification:**
+> Prompt: *Classify the following as either a fruit or a vegetable:
+> broccoli*
+>
+> Base: never answers the classification, rambles about diet generally.
+>
+> SFT: *"broccoli, broccoli, cabbage,iant cucumber, cucumber, ormanus...
+> This fruit is most closely associated with pears and cucumber."* — also
+> fails to answer, with garbled sub-word artifacts not seen in the base
+> output.
+
+## Files
+
+- `model.safetensors` — model weights, fp32 (optimizer state dropped)
+- `configuration_quartz.py`, `modeling_quartz.py` — `transformers`-compatible
+  `PretrainedConfig`/`PreTrainedModel` wrapper (`QuartzMoEConfig`,
+  `QuartzForCausalLM`), wired up via `auto_map` in `config.json`
+- `model.py`, `config.py` — the original plain-PyTorch model class and
+  architecture config, used by `load_model.py`
+- `config.json` — architecture config in HF's expected format
+- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` — a
+  standard `transformers` fast tokenizer
+- `tokenizer/vocab.json`, `tokenizer/merges.txt` — the custom tokenizer's
+  raw vocab/merges, used by `load_model.py`'s plain-PyTorch path. Only
+  compatible with this model's weights.
+- `load_model.py` — minimal working example using the plain-PyTorch path
+
+## Usage
+
+**Via `transformers` (recommended):**
+
+```bash
+pip install transformers torch
+```
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "VertexAGI/quartz-micro-preview", trust_remote_code=True
+)
+tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview")
+
+prompt = "### Instruction:\nWhy do cats purr?\n\n### Response:\n"
+ids = tok(prompt, return_tensors="pt")
+out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.7)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+`trust_remote_code=True` is required — this is a bespoke architecture
+(DeepSeek-style fine-grained MoE), not one of `transformers`' built-in
+model types. No KV-cache support yet, so `generate()` recomputes attention
+over the full sequence each step.
+
+**Plain PyTorch (no `transformers` dependency):**
+
+```bash
+pip install torch safetensors tokenizers
+python load_model.py
+```
+
+## Limitations
+
+Same size-driven limitations as the base checkpoint (factual
+unreliability, weak general knowledge, no safety fine-tuning), plus
+SFT-specific ones documented honestly above: strict format/length
+instructions (e.g. "write exactly two sentences") are not reliably
+followed, and generation occasionally produces garbled sub-word artifacts
+the base checkpoint doesn't. Treat this as a proof-of-concept instruction
+tune on a genuinely tiny model, not a general-purpose assistant.
+
+---
+
+Built by [Vertex AGI](https://vertexagi.vercel.app). Every model we ship —
+weights, not just claims.