Present Astheral v2 capabilities, measured comparisons and deterministic inference defaults

README.md

@@ -12,47 +12,61 @@ tags:
 - llama
 - instruct
 - reasoning
+- qlora
 ---
 
 # Astheral
 
-Built with Llama. Astheral is SEE7's 8.03B assistant with instruct and thinking modes, custom system instructions, and deterministic inference defaults.
-
-Version 2 combines a broad instruction/coding/math training stage with a native BF16 repair stage for structured output, complete code, and multi-step balances. Rank-32 adapters update attention and feed-forward projections. Training and inference verification run remotely on a 24 GB NVIDIA A10G.
-
-Exact training stages, source revisions, data, adapter weights, merged Safetensors and measured comparisons are included. Use inference.py with --mode instruct or --mode thinking. Use --system to supply your own instructions and --sample for creative variation.
-
-Derived from Nous Research DeepHermes 3 8B Preview and Meta Llama 3.1. See LICENSE, UPSTREAM_LLAMA3_LICENSE, NOTICE and data_sources.json.
-
-## Verified core results
-
-{
-  "regression": {
-    "passed": 2,
-    "total": 2
-  },
-  "fresh": {
-    "passed": 36,
-    "total": 36
-  },
-  "exact": {
-    "passed": 10,
-    "total": 10
-  },
-  "json": {
-    "passed": 4,
-    "total": 4
-  },
-  "number": {
-    "passed": 16,
-    "total": 16
-  },
-  "code": {
-    "passed": 6,
-    "total": 6
-  },
-  "confirmation": {
-    "passed": 16,
-    "total": 16
-  }
-}
+**Built with Llama.** Astheral is SEE7's 8.03B assistant for instruction following, coding, concise explanations and fictional roleplay.
+
+## Instruct and thinking
+
+Choose **instruct** for direct answers or **thinking** for a worked solution before the final answer. Both modes use the same trained weights. The included inference script selects the mode and accepts your own system instructions with --system.
+
+## What's changed in v2
+
+- Balanced training across general instruction tasks, formatting, text editing, Python, and multi-step mathematics.
+- Rank-32 adapters across attention and feed-forward projections.
+- Explicit training to account for all groups in counting problems and preserve requested output structure.
+- Short, complete worked solutions in thinking mode.
+- A saved, merged BF16 checkpoint plus the separately downloadable adapter.
+
+Training used **4,080 examples**, **1084 optimizer steps**, and **83,886,080 trainable adapter parameters** on a remote **NVIDIA A10G**. The original DeepHermes weights provide the starting model.
+
+## Measured results
+
+Both models used identical prompts and deterministic generation settings: up to 384 new tokens for instruct and 1,024 for thinking. The core suite contains 14 formatting tasks, 16 numerical reasoning tasks and six executable Python tasks.
+
+| Check | DeepHermes base | Astheral v2 |
+|---|---:|---:|
+| Formatting | 9/14 | 14/14 |
+| Numerical reasoning | 16/16 | 16/16 |
+| Executable Python | 4/6 | 6/6 |
+| Core checks total | 29/36 | 36/36 |
+| Known regression cases | 1/2 | 2/2 |
+
+The two regression cases cover the alpha/omega line format and counting the remaining red and blue balls together. Full prompts, answers, checker code and run metrics are available in evaluate.py, base_outputs.json, astheral_merged_outputs.json and evaluation.json. Six additional samples cover roleplay, prompt design, authorized networking explanations, uncertainty, debugging and summarization.
+
+Default mode formatting was verified in **60/60 responses**: instruct responses omit thinking tags; thinking responses contain a completed thinking block.
+
+The saved checkpoint also passed **16/16 additional confirmation cases** covering new CSV records, literal copying, fraction-based balances, time conversion and complete Python functions. Their prompts and answers are in confirmation_outputs.json.
+
+## Run on a remote GPU
+
+Install a CUDA-enabled PyTorch build and requirements.txt, then download inference.py from this repository.
+
+    python inference.py "Explain Python generators." --mode instruct
+    python inference.py "A box has 23 red and 17 blue beads; 6 red are removed. How many remain?" --mode thinking
+    python inference.py "Write a fictional starship announcement." --mode instruct --load-in-4bit
+
+Use --system "Your instructions" to replace the default assistant instructions. Decoding is deterministic by default; add --sample for creative variation. BF16 weights occupy about 16 GB; --load-in-4bit loads the weights in NF4 for lower GPU memory use. The script loads SEE7/Astheral directly. Dockerfile includes the CUDA runtime and compiler dependencies for the 4-bit backend.
+
+## Files and reproducibility
+
+The repository includes merged Safetensors weights, the adapter folder, train.py, prepare_data.py, the exact training and validation examples, trainer logs, data source revisions, output comparisons and artifact hashes. The merged checkpoint was saved, reloaded, and evaluated before release.
+
+## Training data and attribution
+
+Training combines original Astheral examples with selected components of HuggingFaceTB/smoltalk (Apache 2.0), the openai/gsm8k training split (MIT), and bigcode/self-oss-instruct-sc2-exec-filter-50k (ODC Attribution). See data_sources.json, DATA.md, DATA_LICENSE files and NOTICE.
+
+Astheral is fine-tuned from NousResearch/DeepHermes-3-Llama-3-8B-Preview, revision 53d902c7dcfa4c749a3b455149064f82af2e2549, built on Meta Llama 3.1 8B. The project is Astheral; the distributed model name is Llama-3.1-Astheral. See LICENSE and UPSTREAM_LLAMA3_LICENSE.

artifact_manifest.json

@@ -52,8 +52,8 @@
     "sha256": "e5a038e1955ffd829ccb9ec510bba1c6a990cf9d2c3c7a3f502c5110a75a9528"
   },
   "generation_config.json": {
-    "bytes": 264,
-    "sha256": "600226d04de07399e276b27693db6f84e40473b6005a0ae9bf74a1b504f3584e"
+    "bytes": 305,
+    "sha256": "79db16a2918647878da86e0e215e8fc461e2918e6cbdafa84c2f8b8356b57c34"
   },
   "model-00001-of-00005.safetensors": {
     "bytes": 3986791344,
@@ -144,8 +144,8 @@
     "sha256": "56f15e83ef3031f8b64c77df07320c599c41eb95410ca9a94d00b2f6446e7073"
   },
   "inference.py": {
-    "bytes": 2390,
-    "sha256": "d7058ae7b4f67fb2aa6d83b738e3a3a66399b0f73528f2af4e7105d2366b3cfd"
+    "bytes": 2741,
+    "sha256": "c4cfc3f09840a024f6464a2114b3ecf9cf08766a624c5031c39937b3e40c324e"
   },
   "requirements.txt": {
     "bytes": 117,
@@ -156,8 +156,8 @@
     "sha256": "fda689d55ae71fc8ca8151b131078b9663843ef965b5d69cad8d2618235edef5"
   },
   "README.md": {
-    "bytes": 1531,
-    "sha256": "12e4b56ad743a412e60b566f92cf03efa68d5dbb0cdc49cc4f5b73805c73ca7f"
+    "bytes": 4350,
+    "sha256": "f65acc0e71be56b27cab016ae9ea695921b5ec9736d2f4f0d08ab503211a503a"
   },
   "adapter/README.md": {
     "bytes": 5242,

generation_config.json

@@ -1,7 +1,7 @@
 {
   "_from_model_config": true,
   "bos_token_id": 128000,
-  "do_sample": true,
+  "do_sample": false,
   "eos_token_id": [
     128001,
     128008,
@@ -9,7 +9,9 @@
   ],
   "max_length": 131072,
   "pad_token_id": 128004,
-  "temperature": 0.6,
-  "top_p": 0.9,
-  "transformers_version": "4.56.2"
-}
+  "temperature": 1.0,
+  "top_p": 1.0,
+  "transformers_version": "4.56.2",
+  "top_k": 50,
+  "max_new_tokens": 1024
+}

inference.py

@@ -2,7 +2,7 @@
 import argparse,json,torch
 from pathlib import Path
 from huggingface_hub import hf_hub_download
-from transformers import AutoTokenizer,AutoModelForCausalLM,BitsAndBytesConfig,set_seed
+from transformers import AutoConfig,AutoTokenizer,AutoModelForCausalLM,BitsAndBytesConfig,set_seed
 p=argparse.ArgumentParser()
 p.add_argument("prompt")
 p.add_argument("--mode",choices=["instruct","thinking"],default="instruct")
@@ -21,7 +21,13 @@ system=(args.system if args.system is not None else settings["system"])+("\n"+se
 tok=AutoTokenizer.from_pretrained(args.model)
 dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
 quant=BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type="nf4",bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=dtype) if args.load_in_4bit else None
-model=AutoModelForCausalLM.from_pretrained(args.model,torch_dtype=dtype,device_map={"":"cuda:0"},quantization_config=quant)
+config=AutoConfig.from_pretrained(args.model)
+saved_quant=getattr(config,"quantization_config",None)
+if quant is None and isinstance(saved_quant,dict) and saved_quant.get("load_in_4bit"):
+    quant=BitsAndBytesConfig.from_dict(saved_quant)
+    quant.bnb_4bit_compute_dtype=dtype
+    config.quantization_config=quant.to_dict()
+model=AutoModelForCausalLM.from_pretrained(args.model,config=config,torch_dtype=dtype,device_map={"":"cuda:0"},quantization_config=quant)
 inputs=tok.apply_chat_template([{"role":"system","content":system},{"role":"user","content":args.prompt}],tokenize=True,add_generation_prompt=True,return_dict=True,return_tensors="pt").to(model.device)
 with torch.inference_mode():
     generation={"max_new_tokens":args.max_new_tokens,"do_sample":args.sample,"pad_token_id":128004,"eos_token_id":128009}