Text Generation
Transformers
Safetensors
English
clarification
ambiguity-detection
question-answering
qwen2.5
grpo
reinforcement-learning
conversational
Instructions to use chrisjcc/ask-before-answer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chrisjcc/ask-before-answer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="chrisjcc/ask-before-answer") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("chrisjcc/ask-before-answer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use chrisjcc/ask-before-answer with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "chrisjcc/ask-before-answer" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "chrisjcc/ask-before-answer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/chrisjcc/ask-before-answer
- SGLang
How to use chrisjcc/ask-before-answer with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "chrisjcc/ask-before-answer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "chrisjcc/ask-before-answer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "chrisjcc/ask-before-answer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "chrisjcc/ask-before-answer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use chrisjcc/ask-before-answer with Docker Model Runner:
docker model run hf.co/chrisjcc/ask-before-answer
Upload folder using huggingface_hub
Browse files- README.md +15 -15
- adapter_config.json +9 -7
- adapter_model.safetensors +1 -1
- training_args.bin +1 -1
README.md
CHANGED
|
@@ -25,7 +25,7 @@ The **AskBeforeAnswer** model exhibits "clarification-seeking" behavior. When pr
|
|
| 25 |
## Pipeline
|
| 26 |
- **Base Model:** Qwen/Qwen2.5-7B-Instruct
|
| 27 |
- **Ablation Winner:** The model variant promoted to Production via W&B Registry is:
|
| 28 |
-
`
|
| 29 |
|
| 30 |
## LLM-as-a-Judge Evaluation Leaderboard
|
| 31 |
|
|
@@ -34,20 +34,20 @@ Gemini-based judge scorer on a randomly selected
|
|
| 34 |
**50-sample** subset
|
| 35 |
of the `sewon/ambig_qa` (validation split).
|
| 36 |
|
| 37 |
-
| Metric | base | dpo_only | sft | sft_dpo | clarifier_lora |
|
| 38 |
-
|:----------------------|----------:|-----------:|---------:|----------:|-----------------:|
|
| 39 |
-
| ambiguity_detection | 0.
|
| 40 |
-
| clarification_quality | 0.
|
| 41 |
-
| usefulness | 0.
|
| 42 |
-
| model_accuracy | 0.62 |
|
| 43 |
-
| clarify_precision | 0.617021 |
|
| 44 |
-
| clarify_recall | 0.966667 |
|
| 45 |
-
| clarify_f1 | 0.753247 |
|
| 46 |
-
|
|
| 47 |
-
| macro_f1 | 0.46358 |
|
| 48 |
-
| answer_accuracy | 0.05 |
|
| 49 |
-
| facet_generation_rate | 0.
|
| 50 |
-
| clarify_ratio | 1.56667 |
|
| 51 |
|
| 52 |
|
| 53 |
**GitHub Release:** [v0.0.5](https://github.com/chrisjcc/ask-before-answer/releases/tag/v0.0.5)
|
|
|
|
| 25 |
## Pipeline
|
| 26 |
- **Base Model:** Qwen/Qwen2.5-7B-Instruct
|
| 27 |
- **Ablation Winner:** The model variant promoted to Production via W&B Registry is:
|
| 28 |
+
`dpo`.
|
| 29 |
|
| 30 |
## LLM-as-a-Judge Evaluation Leaderboard
|
| 31 |
|
|
|
|
| 34 |
**50-sample** subset
|
| 35 |
of the `sewon/ambig_qa` (validation split).
|
| 36 |
|
| 37 |
+
| Metric | base | dpo_only | sft | sft_dpo | clarifier_lora | orpo | grpo |
|
| 38 |
+
|:----------------------|----------:|-----------:|---------:|----------:|-----------------:|---------:|---------:|
|
| 39 |
+
| ambiguity_detection | 0.966 | 0.944 | 0.97 | 0.972 | 0.994 | 0.972 | 0.948 |
|
| 40 |
+
| clarification_quality | 0.784 | 0.784 | 0.796 | 0.796 | 0.796 | 0.796 | 0.798 |
|
| 41 |
+
| usefulness | 0.88 | 0.88 | 0.896 | 0.898 | 0.896 | 0.896 | 0.898 |
|
| 42 |
+
| model_accuracy | 0.62 | 0.6 | 0.64 | 0.62 | 0.6 | 0.64 | 0.64 |
|
| 43 |
+
| clarify_precision | 0.617021 | 0.604167 | 0.657895 | 0.657143 | 0.6 | 0.636364 | 0.642857 |
|
| 44 |
+
| clarify_recall | 0.966667 | 0.966667 | 0.833333 | 0.766667 | 1 | 0.933333 | 0.9 |
|
| 45 |
+
| clarify_f1 | 0.753247 | 0.74359 | 0.735294 | 0.707692 | 0.75 | 0.756757 | 0.75 |
|
| 46 |
+
| action_f1_answer | 0.173913 | 0.0909091 | 0.4375 | 0.457143 | 0 | 0.307692 | 0.357143 |
|
| 47 |
+
| macro_f1 | 0.46358 | 0.417249 | 0.586397 | 0.582418 | 0.375 | 0.532225 | 0.553571 |
|
| 48 |
+
| answer_accuracy | 0.05 | 0.05 | 0.05 | 0 | 0 | 0.1 | 0.1 |
|
| 49 |
+
| facet_generation_rate | 0.0851064 | 0.0833333 | 1 | 1 | 0 | 0.931818 | 1 |
|
| 50 |
+
| clarify_ratio | 1.56667 | 1.6 | 1.26667 | 1.16667 | 1.66667 | 1.46667 | 1.4 |
|
| 51 |
|
| 52 |
|
| 53 |
**GitHub Release:** [v0.0.5](https://github.com/chrisjcc/ask-before-answer/releases/tag/v0.0.5)
|
adapter_config.json
CHANGED
|
@@ -27,20 +27,21 @@
|
|
| 27 |
"megatron_config": null,
|
| 28 |
"megatron_core": "megatron.core",
|
| 29 |
"modules_to_save": null,
|
|
|
|
| 30 |
"peft_type": "LORA",
|
| 31 |
-
"peft_version": "0.
|
| 32 |
"qalora_group_size": 16,
|
| 33 |
"r": 16,
|
| 34 |
"rank_pattern": {},
|
| 35 |
"revision": null,
|
| 36 |
"target_modules": [
|
| 37 |
-
"
|
| 38 |
"up_proj",
|
| 39 |
-
"k_proj",
|
| 40 |
-
"q_proj",
|
| 41 |
"o_proj",
|
| 42 |
-
"
|
| 43 |
-
"
|
|
|
|
|
|
|
| 44 |
],
|
| 45 |
"target_parameters": null,
|
| 46 |
"task_type": "CAUSAL_LM",
|
|
@@ -48,5 +49,6 @@
|
|
| 48 |
"use_bdlora": null,
|
| 49 |
"use_dora": false,
|
| 50 |
"use_qalora": false,
|
| 51 |
-
"use_rslora": false
|
|
|
|
| 52 |
}
|
|
|
|
| 27 |
"megatron_config": null,
|
| 28 |
"megatron_core": "megatron.core",
|
| 29 |
"modules_to_save": null,
|
| 30 |
+
"monteclora_config": null,
|
| 31 |
"peft_type": "LORA",
|
| 32 |
+
"peft_version": "0.20.0",
|
| 33 |
"qalora_group_size": 16,
|
| 34 |
"r": 16,
|
| 35 |
"rank_pattern": {},
|
| 36 |
"revision": null,
|
| 37 |
"target_modules": [
|
| 38 |
+
"gate_proj",
|
| 39 |
"up_proj",
|
|
|
|
|
|
|
| 40 |
"o_proj",
|
| 41 |
+
"k_proj",
|
| 42 |
+
"v_proj",
|
| 43 |
+
"down_proj",
|
| 44 |
+
"q_proj"
|
| 45 |
],
|
| 46 |
"target_parameters": null,
|
| 47 |
"task_type": "CAUSAL_LM",
|
|
|
|
| 49 |
"use_bdlora": null,
|
| 50 |
"use_dora": false,
|
| 51 |
"use_qalora": false,
|
| 52 |
+
"use_rslora": false,
|
| 53 |
+
"velora_config": null
|
| 54 |
}
|
adapter_model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 161533192
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8730e5a37e3df071b5da452c0ffaf1863062aec888f38e01adf1afd099ea3721
|
| 3 |
size 161533192
|
training_args.bin
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 6584
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f80303518705dcb84090e851e58a8ab093d94eaa41dc357e4f553c5130633140
|
| 3 |
size 6584
|