File size: 3,214 Bytes
238f5ab | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | ---
license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
base_model: google/gemma-4-31B-it
pipeline_tag: text-generation
tags:
- gemma4
- compressed-tensors
- lora
- peft
- vllm
- jevbench
---
# MetaDecision-31B-Rain
Decision service for JevBench. Base model:
[`google/gemma-4-31B-it`](https://huggingface.co/google/gemma-4-31B-it), used through Google's
official quantization-aware-training (QAT) releases of it. The model has two parts that are loaded
together and **not merged**; both are the files of
[`MeTask-Rain/metask-jev-rain-31B-trained`](https://huggingface.co/MeTask-Rain/metask-jev-rain-31B-trained/tree/2feff076f07711fe2fefe7a6fdcedfe4132b773e),
unchanged and pinned by SHA-256 in `MODEL_MANIFEST.json` of the service code. This entry uses the same weights as that
entry and changes only the serving rules.
| Part | Location in that model repository | What it is |
|---|---|---|
| Base | repository root | The files of [`google/gemma-4-31B-it-qat-w4a16-ct`](https://huggingface.co/google/gemma-4-31B-it-qat-w4a16-ct) at revision `52f3f65bc7a02d555763bc923bd1d9094898219d`, byte for byte: Google's official compressed-tensors W4A16 checkpoint (4-bit weights, group size 32, symmetric). |
| Adapter | `adapter/` | Our LoRA (PEFT format): rank 16, alpha 32, on the language model's q/k/v/o/gate/up/down projections, trained by us in BF16 on [`google/gemma-4-31B-it-qat-q4_0-unquantized`](https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized) at revision `1e4d8beecacb8b7590c1d8bedd7335f687bf311f`. |
`adapter/adapter_model.safetensors` contains the trained checkpoint tensors unchanged plus, for the 10 full-attention layers (5, 11, …, 59) that compute values with the key projection, a v_proj LoRA that is a byte copy of the trained k_proj LoRA (20 tensors), so that vLLM applies the update to keys and values as in training.
## Serving
The service code (`service/` in the Hugging Face repository `MeTask-Rain/MetaDecision-31B-Rain`, mirrored at https://github.com/chenyuwang166-oss/MetaDecision-31B-Rain) starts vLLM 0.31.0 on a folder holding the files of that weight
repository, with `enable_lora`, `max_lora_rank=16`, and the `adapter/` folder as the single LoRA,
applied to every request. Each decision is read from the next-token log-probabilities of the answer
forms in one prompt pass; up to 49% of decisions then take a second, short drafting pass
(at most 192 tokens) under a 9 s deadline, a choice decision answered in one pass although its length allowed the drafting pass is read a second time with the options in reverse order when its first-pass margin is below 4, and the reported probabilities get a per-type post-calibration (DISCLOSURE.md).
Equivalent vLLM usage of the model:
```python
from vllm import LLM
from vllm.lora.request import LoRARequest
llm = LLM(model=MODEL_DIR, dtype="bfloat16", enable_lora=True, max_lora_rank=16)
adapter = LoRARequest("adapter", 1, MODEL_DIR + "/adapter")
# llm.generate(prompts, sampling_params, lora_request=adapter)
```
Facts on data and the serving rules are in `DISCLOSURE.md` of the service code (`service/DISCLOSURE.md` in the
Hugging Face repository).
Contact: chenyuwang166@gmail.com
|