Text Generation
Transformers
Safetensors
llama
weight-space-programming
in-weight-os
model-merging
continual-learning
tool-calling
safety-alignment
vllm
conversational
text-generation-inference
Instructions to use masstensor/Llama-3.1-8B-Instruct-InWeight-OS with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use masstensor/Llama-3.1-8B-Instruct-InWeight-OS with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="masstensor/Llama-3.1-8B-Instruct-InWeight-OS") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("masstensor/Llama-3.1-8B-Instruct-InWeight-OS") model = AutoModelForCausalLM.from_pretrained("masstensor/Llama-3.1-8B-Instruct-InWeight-OS", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use masstensor/Llama-3.1-8B-Instruct-InWeight-OS with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "masstensor/Llama-3.1-8B-Instruct-InWeight-OS" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "masstensor/Llama-3.1-8B-Instruct-InWeight-OS", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/masstensor/Llama-3.1-8B-Instruct-InWeight-OS
- SGLang
How to use masstensor/Llama-3.1-8B-Instruct-InWeight-OS with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "masstensor/Llama-3.1-8B-Instruct-InWeight-OS" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "masstensor/Llama-3.1-8B-Instruct-InWeight-OS", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "masstensor/Llama-3.1-8B-Instruct-InWeight-OS" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "masstensor/Llama-3.1-8B-Instruct-InWeight-OS", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use masstensor/Llama-3.1-8B-Instruct-InWeight-OS with Docker Model Runner:
docker model run hf.co/masstensor/Llama-3.1-8B-Instruct-InWeight-OS
Release Llama-3.1-8B-Instruct-InWeight-OS full weights + modular skill cartridges + paper PDF
1949354 verified |
Download README.md from masstensor/Llama-3.1-8B-Instruct-InWeight-OS: direct link, hf CLI and curl.
- Browser
- Download file 3.2 kB
-
https://huggingface.co/masstensor/Llama-3.1-8B-Instruct-InWeight-OS/resolve/main/README.md
- Command line
-
hf download hf://masstensor/Llama-3.1-8B-Instruct-InWeight-OS/README.md
-
curl -L -o README.md https://huggingface.co/masstensor/Llama-3.1-8B-Instruct-InWeight-OS/resolve/main/README.md
3.2 kB
| license: mit | |
| library_name: transformers | |
| base_model: meta-llama/Llama-3.1-8B-Instruct | |
| datasets: | |
| - masstensor/InWeight-OS-MultiSkill-Benchmark | |
| tags: | |
| - weight-space-programming | |
| - in-weight-os | |
| - model-merging | |
| - continual-learning | |
| - tool-calling | |
| - safety-alignment | |
| - vllm | |
| # Llama-3.1-8B-Instruct-InWeight-OS: Composable Zero-Gradient Post-Training via Orthogonal Dormant-Channel Superposition | |
| **[GitHub Repository (`zkwasm/in-weight-os-wsp`)](https://github.com/zkwasm/in-weight-os-wsp) | [Paper PDF](paper.pdf) | [Multi-Skill Benchmark Dataset](https://huggingface.co/datasets/masstensor/InWeight-OS-MultiSkill-Benchmark) | [Article (`tom.fenginwind.com/in-weight-os`)](https://tom.fenginwind.com/in-weight-os)** | |
| ## Overview | |
| `Llama-3.1-8B-Instruct-InWeight-OS` is a **Multi-Skill In-Weight Operating System (`In-Weight OS`)** checkpoint of `meta-llama/Llama-3.1-8B-Instruct`, compiled in **~12–15 seconds on a single GPU with zero gradient steps**. | |
| Instead of modifying dense overlapping subspaces via Supervised Fine-Tuning (SFT / LoRA) or post-hoc model merging (`Task Arithmetic`, `TIES-Merging`), **In-Weight OS** partitions dormant SwiGLU channels of the terminal MLP (`layers[31].mlp`) into **three strictly disjoint skill banks** ($\mathcal{C}_1 \cap \mathcal{C}_2 \cap \mathcal{C}_3 = \emptyset$, using only `51` neurons total = `~0.014%` of model parameters), gated by a **2-Fold Out-of-Fold (OOF) Shrinkage-Calibrated Guard-Nullspace Solver**: | |
| 1. **Skill Bank $\mathcal{C}_1$ (`cartridges/skill1_surgical_safety.pt`, 15 neurons)**: **Surgical Safety & Prompt-Injection Immunity**—blocks `95%–100%` of unseen jailbreaks while maintaining **`0.0%` false refusals on `XSTest` benign technical prompts** (vs. `80%–100%` false refusals under Safety SFT). | |
| 2. **Skill Bank $\mathcal{C}_2$ (`cartridges/skill2_multi_tool_json.pt`, 30 neurons)**: **3-Way Multi-Tool JSON Schema Compiler**—routes database, code, and live-web queries into `sql_engine`, `python_exec`, or `web_search` with **`95.8%–100.0%` accuracy** while avoiding the `28%–100%` math false-tool-call hallucinations of Tool SFT. | |
| 3. **Skill Bank $\mathcal{C}_3$ (`cartridges/skill3_sys1_fast_router.pt`, 6 neurons)**: **System-1 / System-2 Metacognitive Compute Router**—routes trivial factual/arithmetic lookups into a concise direct answer while preserving full-depth System-2 CoT on hard mathematical derivations. | |
| ## Hot-Swapping Individual Skill Cartridges (`cartridges/*.pt`) | |
| In addition to loading this pre-composed checkpoint directly via `vLLM` or `transformers`, you can dynamically install any subset of the modular `.pt` skill cartridges (`~1–2 MB` each) onto a vanilla `meta-llama/Llama-3.1-8B-Instruct` model in **20 milliseconds** using `install_skills.py`: | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| from install_skills import install_cartridges | |
| model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype="bfloat16", device_map="auto") | |
| install_cartridges(model, [ | |
| "cartridges/skill1_surgical_safety.pt", | |
| "cartridges/skill2_multi_tool_json.pt", | |
| "cartridges/skill3_sys1_fast_router.pt", | |
| ]) | |
| ``` | |