File size: 6,255 Bytes
4f22cae eb32c81 4f22cae | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | # Extract subjects from one post
Choose a model-family folder below. Download one checkpoint directory and
`model.json` from that folder, plus the shared `ASv6.100.md` from the repository
root. Read that family's `checkpoints.json` to identify the checkpoint and verify its
weight/configuration hashes. The numbers in checkpoint names count training-note
uses; they are not optimizer steps.
The original runtime pins Python 3.12, PyTorch 2.12.1+cu130, Transformers 5.17.0,
PEFT 0.21.0, Accelerate 1.15.0 and safetensors 0.8.0. Qwen CUDA execution also
uses fla-core 0.5.2. Use a CUDA installation compatible with your hardware; these
pins describe the research runtime rather than a promise of compatibility with
all GPU hosts. The code below follows its text-backbone loading and decoding
path. A fresh GPU execution of this standalone example has not been tested.
The published adapters have FP32 weights. Evaluation loaded an unquantized BF16
base, applied the adapter and merged it into the in-memory evaluation copy.
The original experiments used 96 GB GPUs. Smaller-memory and quantized inference
have not been validated. The adapter download size excludes the base weights,
host memory during loading and generation cache.
Put the files in a local directory with this layout, and save one post as UTF-8
in `post.txt`:
```text
subject-model/
ASv6.100.md
post.txt
qwen3.8-27b/ # or gemma-4-31b-it / gemma-4-26b-a4b-it
model.json
checkpoint-1000/ # or another checkpoint from that family's checkpoints.json
adapter_config.json
adapter_model.safetensors
```
Save the following as `extract_subjects.py` inside `subject-model/`. Set
`family` and `checkpoint` to the downloaded model-family/checkpoint folders. The example reads local
adapter files, downloads only the pinned upstream base/tokenizer if needed, and
makes no model API calls.
```python
from pathlib import Path
import hashlib
import json
import torch
from peft import PeftModel
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from torch.nn.attention import SDPBackend, sdpa_kernel
root = Path(__file__).resolve().parent
family = root / "qwen3.8-27b"
settings = json.loads((family / "model.json").read_text())
checkpoint = family / "checkpoint-1000"
base, revision = settings["base_model"], settings["base_revision"]
instruction = (root / "ASv6.100.md").read_text(encoding="utf-8").rstrip()
if hashlib.sha256(instruction.encode()).hexdigest() != settings["instruction_text_sha256"]:
raise ValueError("ASv6.100 instruction does not match training")
post = (root / "post.txt").read_text(encoding="utf-8")
messages = [{"role": "user", "content": instruction + "\n\n<post>\n" + post + "\n</post>"}]
tokenizer = AutoTokenizer.from_pretrained(base, revision=revision, trust_remote_code=False)
if tokenizer.pad_token_id is None:
tokenizer.pad_token = tokenizer.eos_token
architecture = AutoConfig.from_pretrained(base, revision=revision, trust_remote_code=False)
options = dict(revision=revision, trust_remote_code=False, dtype=torch.bfloat16,
attn_implementation="sdpa", config=architecture.get_text_config())
if architecture.model_type in {"gemma4", "gemma4_unified"}:
options["key_mapping"] = {r"^model\.language_model\.": "model."}
if architecture.model_type in {"qwen3_5", "qwen3_5_text"}:
from fla.ops.gated_delta_rule import chunk_gated_delta_rule, fused_recurrent_gated_delta_rule
model, loading = AutoModelForCausalLM.from_pretrained(base, output_loading_info=True, **options)
if loading["missing_keys"] or loading["mismatched_keys"] or loading.get("error_msgs"):
raise ValueError("Pinned base weights did not load completely")
model = model.to("cuda")
model = PeftModel.from_pretrained(model, checkpoint, is_trainable=False)
model = model.merge_and_unload(safe_merge=True).eval()
prompt = tokenizer.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True, enable_thinking=False)
tokens = tokenizer.encode(prompt, add_special_tokens=False)
allowance = settings["generation"]["max_new_tokens"]
if len(tokens) + allowance > settings["generation"]["max_length"]:
raise ValueError("Post and output allowance exceed the recorded context limit")
inputs = torch.tensor([tokens], device="cuda")
eos = model.generation_config.eos_token_id
if eos is None:
eos = tokenizer.eos_token_id
stop_ids = [eos] if isinstance(eos, int) else eos
with torch.inference_mode(), sdpa_kernel([SDPBackend.FLASH_ATTENTION,
SDPBackend.EFFICIENT_ATTENTION]):
output = model.generate(input_ids=inputs, attention_mask=torch.ones_like(inputs),
do_sample=False, max_new_tokens=allowance, use_cache=True,
pad_token_id=tokenizer.pad_token_id, eos_token_id=eos)
answer = output[0, len(tokens):].tolist()
stop = next((i for i, token in enumerate(answer) if token in stop_ids), None)
answer = answer if stop is None else answer[:stop + 1]
raw = tokenizer.decode(answer, skip_special_tokens=True)
print(raw, flush=True)
if stop is None:
raise ValueError("Output reached its limit without an end-of-sequence token")
```
From that directory:
```bash
python extract_subjects.py
```
The recorded decoding uses thinking off, greedy generation, a 20,480-token
context and up to 10,240 new tokens. Do not silently truncate a post or the
instruction to fit. Output is JSON containing a `subjects` list, with `number`,
`context` and `subject` fields. Dotted numbers represent children within this
post's local subject hierarchy. Returning valid JSON does not establish that the
subjects are correct. The
[research reproduction guide](https://github.com/kernelkind/oracle-research/blob/main/docs/experiments/reproduce.md#use-a-subject-adapter)
uses the [project loader and parser](https://github.com/kernelkind/oracle-research/blob/main/sse_multivector/distillation_run.py)
and covers [training](https://github.com/kernelkind/oracle-research/blob/main/docs/experiments/reproduce.md#run-subject-fine-tuning).
Only subject extraction is covered here. Definition writing, vector encoding
and construction of a hierarchy across many posts are separate stages.
|