| # Extract subjects from one post |
|
|
| Choose a model-family folder below. Download one checkpoint directory and |
| `model.json` from that folder, plus the shared `ASv6.100.md` from the repository |
| root. Read that family's `checkpoints.json` to identify the checkpoint and verify its |
| weight/configuration hashes. The numbers in checkpoint names count training-note |
| uses; they are not optimizer steps. |
|
|
| The original runtime pins Python 3.12, PyTorch 2.12.1+cu130, Transformers 5.17.0, |
| PEFT 0.21.0, Accelerate 1.15.0 and safetensors 0.8.0. Qwen CUDA execution also |
| uses fla-core 0.5.2. Use a CUDA installation compatible with your hardware; these |
| pins describe the research runtime rather than a promise of compatibility with |
| all GPU hosts. The code below follows its text-backbone loading and decoding |
| path. A fresh GPU execution of this standalone example has not been tested. |
|
|
| The published adapters have FP32 weights. Evaluation loaded an unquantized BF16 |
| base, applied the adapter and merged it into the in-memory evaluation copy. |
| The original experiments used 96 GB GPUs. Smaller-memory and quantized inference |
| have not been validated. The adapter download size excludes the base weights, |
| host memory during loading and generation cache. |
|
|
| Put the files in a local directory with this layout, and save one post as UTF-8 |
| in `post.txt`: |
|
|
| ```text |
| subject-model/ |
| ASv6.100.md |
| post.txt |
| qwen3.8-27b/ # or gemma-4-31b-it / gemma-4-26b-a4b-it |
| model.json |
| checkpoint-1000/ # or another checkpoint from that family's checkpoints.json |
| adapter_config.json |
| adapter_model.safetensors |
| ``` |
|
|
| Save the following as `extract_subjects.py` inside `subject-model/`. Set |
| `family` and `checkpoint` to the downloaded model-family/checkpoint folders. The example reads local |
| adapter files, downloads only the pinned upstream base/tokenizer if needed, and |
| makes no model API calls. |
|
|
| ```python |
| from pathlib import Path |
| import hashlib |
| import json |
| import torch |
| from peft import PeftModel |
| from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer |
| from torch.nn.attention import SDPBackend, sdpa_kernel |
| |
| root = Path(__file__).resolve().parent |
| family = root / "qwen3.8-27b" |
| settings = json.loads((family / "model.json").read_text()) |
| checkpoint = family / "checkpoint-1000" |
| base, revision = settings["base_model"], settings["base_revision"] |
| instruction = (root / "ASv6.100.md").read_text(encoding="utf-8").rstrip() |
| if hashlib.sha256(instruction.encode()).hexdigest() != settings["instruction_text_sha256"]: |
| raise ValueError("ASv6.100 instruction does not match training") |
| post = (root / "post.txt").read_text(encoding="utf-8") |
| messages = [{"role": "user", "content": instruction + "\n\n<post>\n" + post + "\n</post>"}] |
| |
| tokenizer = AutoTokenizer.from_pretrained(base, revision=revision, trust_remote_code=False) |
| if tokenizer.pad_token_id is None: |
| tokenizer.pad_token = tokenizer.eos_token |
| architecture = AutoConfig.from_pretrained(base, revision=revision, trust_remote_code=False) |
| options = dict(revision=revision, trust_remote_code=False, dtype=torch.bfloat16, |
| attn_implementation="sdpa", config=architecture.get_text_config()) |
| if architecture.model_type in {"gemma4", "gemma4_unified"}: |
| options["key_mapping"] = {r"^model\.language_model\.": "model."} |
| if architecture.model_type in {"qwen3_5", "qwen3_5_text"}: |
| from fla.ops.gated_delta_rule import chunk_gated_delta_rule, fused_recurrent_gated_delta_rule |
| model, loading = AutoModelForCausalLM.from_pretrained(base, output_loading_info=True, **options) |
| if loading["missing_keys"] or loading["mismatched_keys"] or loading.get("error_msgs"): |
| raise ValueError("Pinned base weights did not load completely") |
| model = model.to("cuda") |
| model = PeftModel.from_pretrained(model, checkpoint, is_trainable=False) |
| model = model.merge_and_unload(safe_merge=True).eval() |
| |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, |
| add_generation_prompt=True, enable_thinking=False) |
| tokens = tokenizer.encode(prompt, add_special_tokens=False) |
| allowance = settings["generation"]["max_new_tokens"] |
| if len(tokens) + allowance > settings["generation"]["max_length"]: |
| raise ValueError("Post and output allowance exceed the recorded context limit") |
| inputs = torch.tensor([tokens], device="cuda") |
| eos = model.generation_config.eos_token_id |
| if eos is None: |
| eos = tokenizer.eos_token_id |
| stop_ids = [eos] if isinstance(eos, int) else eos |
| with torch.inference_mode(), sdpa_kernel([SDPBackend.FLASH_ATTENTION, |
| SDPBackend.EFFICIENT_ATTENTION]): |
| output = model.generate(input_ids=inputs, attention_mask=torch.ones_like(inputs), |
| do_sample=False, max_new_tokens=allowance, use_cache=True, |
| pad_token_id=tokenizer.pad_token_id, eos_token_id=eos) |
| answer = output[0, len(tokens):].tolist() |
| stop = next((i for i, token in enumerate(answer) if token in stop_ids), None) |
| answer = answer if stop is None else answer[:stop + 1] |
| raw = tokenizer.decode(answer, skip_special_tokens=True) |
| print(raw, flush=True) |
| if stop is None: |
| raise ValueError("Output reached its limit without an end-of-sequence token") |
| ``` |
|
|
| From that directory: |
|
|
| ```bash |
| python extract_subjects.py |
| ``` |
|
|
| The recorded decoding uses thinking off, greedy generation, a 20,480-token |
| context and up to 10,240 new tokens. Do not silently truncate a post or the |
| instruction to fit. Output is JSON containing a `subjects` list, with `number`, |
| `context` and `subject` fields. Dotted numbers represent children within this |
| post's local subject hierarchy. Returning valid JSON does not establish that the |
| subjects are correct. The |
| [research reproduction guide](https://github.com/kernelkind/oracle-research/blob/main/docs/experiments/reproduce.md#use-a-subject-adapter) |
| uses the [project loader and parser](https://github.com/kernelkind/oracle-research/blob/main/sse_multivector/distillation_run.py) |
| and covers [training](https://github.com/kernelkind/oracle-research/blob/main/docs/experiments/reproduce.md#run-subject-fine-tuning). |
|
|
| Only subject extraction is covered here. Definition writing, vector encoding |
| and construction of a hierarchy across many posts are separate stages. |
|
|
