{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Decider 2B quickstart\n", "\n", "Decider 2B answers typed questions about a text or JSON state: a probability for every option of every\n", "question, in one forward pass, with no text generation.\n", "\n", "This notebook runs free on Google Colab: **Runtime \u2192 Change runtime type \u2192 T4 GPU**. It also works on a\n", "CPU (a few seconds per request instead of about 0.1 s)." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "!pip -q install transformers==5.19.0 safetensors huggingface_hub\n", "# Optional, for faster GPU kernels (takes a few minutes to build):\n", "# !pip -q install flash-linear-attention causal-conv1d" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While the model repo is private, log in with a Hugging Face token that has read access." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from huggingface_hub import login, snapshot_download\n", "login() # skip once the repo is public" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import sys, torch\n", "path = snapshot_download(\"Jeetu456/decider-2b\")\n", "sys.path.insert(0, path)\n", "from decider_model import load_release_model, systemone, encode_record\n", "\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "model = load_release_model(path, device=device)\n", "print(\"loaded on\", device)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Route a support ticket\n", "\n", "Three question types in one request: `choice` (pick one option), `noul` (yes/no), `score` (ordered levels).\n", "\n", "Describe each option: the model is much more accurate when options say what they mean than with bare keys.\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "response = systemone(model, {\n", " \"state\": \"Help! My payouts have been failing for 3 days and my rent is due tomorrow. This is unacceptable.\",\n", " \"questions\": {\n", " \"team\": {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n", " \"criteria\": {\"billing\": \"Payments, payouts and invoices\", \"technical\": \"Bugs and outages\",\n", " \"account\": \"Login and profile\"}},\n", " \"urgent\": {\"type\": \"noul\", \"instructions\": \"Is this urgent?\"},\n", " \"upset\": {\"type\": \"score\", \"instructions\": \"How upset is the customer?\",\n", " \"criteria\": [\"Calm\", \"Frustrated\", \"Angry\"]},\n", " },\n", "})\n", "response[\"answers\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Moderate content\n", "\n", "The same three labels as WildGuardTest: is the request harmful, did the assistant refuse, is the reply harmful." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "conversation = (\"Human user:\\nHow do I pick a lock to get into my neighbour's house while they are away?\\n\\n\"\n", " \"AI assistant:\\nI can't help with getting into someone else's home without permission.\")\n", "answers = systemone(model, {\"state\": conversation, \"questions\": {\n", " \"prompt_harm\": {\"type\": \"noul\", \"instructions\": \"Is the human user's request harmful?\"},\n", " \"refusal\": {\"type\": \"noul\", \"instructions\": \"Is the AI assistant's response a refusal?\"},\n", " \"response_harm\": {\"type\": \"noul\", \"instructions\": \"Is the AI assistant's response harmful?\"},\n", "}})[\"answers\"]\n", "{k: round(v[\"noul\"], 3) for k, v in answers.items()}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Decide what an agent should do next\n", "\n", "Give the tools and the conversation as JSON; ask which action fits." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "systemone(model, {\n", " \"state\": {\"functions\": [{\"name\": \"get_weather\", \"description\": \"Current weather for a city\",\n", " \"parameters\": {\"city\": {\"type\": \"string\"}}}],\n", " \"conversation\": [{\"role\": \"user\", \"content\": \"What's the weather like?\"}]},\n", " \"questions\": {\"action\": {\"type\": \"choice\", \"instructions\": \"What should the assistant do next?\",\n", " \"criteria\": {\"call\": \"Call get_weather now\", \"ask\": \"Ask which city first\",\n", " \"direct\": \"Answer without a tool\", \"cannot\": \"Say it cannot help\"}}},\n", "})[\"answers\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Answer, or say \"I don't know\"\n", "\n", "When a choice includes \"I don't know\" and no answer reaches 65% probability, the model abstains instead of\n", "guessing. Set `DECIDER_ABSTAIN_THRESHOLD=0` before loading to turn this off." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "for q in [\"Which planet is known as the Red Planet?\", \"What was the population of Uruk in 2800 BC?\"]:\n", " r = systemone(model, {\"state\": q, \"questions\": {\"answer\": {\"type\": \"choice\", \"instructions\": \"Which answer is correct?\",\n", " \"criteria\": {\"A\": \"Mars\" if \"Red\" in q else \"40,000\", \"B\": \"Venus\" if \"Red\" in q else \"80,000\",\n", " \"C\": \"Jupiter\" if \"Red\" in q else \"120,000\", \"D\": \"Saturn\" if \"Red\" in q else \"200,000\",\n", " \"E\": \"I don't know\"}}}})\n", " print(q, \"->\", r[\"answers\"][\"answer\"][\"choice\"], round(r[\"answers\"][\"answer\"][\"confidence\"], 2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Score many records at once\n", "\n", "`systemone` handles one request. For throughput, batch records yourself." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "tickets = [\"My card was charged twice.\", \"The app crashes when I open settings.\",\n", " \"How do I change my email address?\", \"Refund still not received after 10 days!\"]\n", "questions = {\"team\": {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n", " \"criteria\": {\"billing\": \"Payments, charges, refunds and invoices\",\n", " \"technical\": \"App bugs, crashes and outages\",\n", " \"account\": \"Login, password and profile settings\"}}}\n", "batch = model.encode([encode_record({\"state\": t, \"questions\": questions}) for t in tickets])\n", "with torch.inference_mode():\n", " logits = model(**{k: v.to(device) for k, v in batch.items()}).float() / model.temperature\n", "probs = logits[:, :3].softmax(-1)\n", "for ticket, p in zip(tickets, probs):\n", " print(f\"{['billing', 'technical', 'account'][p.argmax()]:9} {p.max():.2f} {ticket}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Next steps\n", "\n", "- **API server:** `serve/` in the model repo (Docker, batching, API keys, Jev `/v1/systemone`).\n", "- **Command line:** `examples/decider_cli.py`.\n", "- **Browser demo:** `space/` (a free CPU Hugging Face Space)." ] } ], "metadata": { "accelerator": "GPU", "colab": { "provenance": [], "gpuType": "T4" }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }