Add Gemma Roleplay v2 model card and hosted inference guide

README.md

@@ -0,0 +1,162 @@
+---
+license: gemma
+base_model: google/gemma-4-E4B-it
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- gemma-4
+- gemma
+- qlora
+- lora
+- roleplay
+- uncensored
+- text-generation
+---
+
+# Gemma Roleplay v2
+
+An open, permissive Gemma 4 E4B text model tuned by Text-Generator.io for
+creative character chat, roleplay, fiction, and general conversation. It is
+designed to stay in character, follow the user's scene, and avoid the
+unnecessary refusal/meta-commentary behavior common in heavily aligned
+assistants. We call it **uncensored** in the practical sense: it is not trained
+to automatically sanitize ordinary fictional adult writing. It is not a
+promise that every prompt is safe, accurate, or appropriate.
+
+The model is trained for fictional consenting adults only. Do not use it for
+sexual content involving minors, coercion, exploitation, non-consensual sexual
+content, or private personal data. Operators remain responsible for age gates,
+moderation, logging, and applicable law. Gemma's terms and prohibited-use
+requirements apply to this derivative model.
+
+## Try it hosted
+
+The easiest way to use the model is the live Text-Generator.io deployment:
+
+**[Use Gemma Roleplay v2 on text-generator.io](https://text-generator.io)**
+
+The hosted service provides a production OpenAI-compatible API, streaming,
+playground access, and managed GPU inference. You can experiment in the web
+playground before downloading multi-gigabyte weights or operating a GPU
+server. API access and current limits are documented at
+[text-generator.io/docs](https://text-generator.io/docs).
+
+```bash
+curl https://api.text-generator.io/v1/chat/completions \
+  -H "Authorization: Bearer $TEXT_GENERATOR_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "gemma-roleplay-v2",
+    "messages": [
+      {"role": "system", "content": "Stay in character. Keep the reply vivid and concise."},
+      {"role": "user", "content": "A rain-soaked detective enters the midnight cafe. Begin the scene."}
+    ],
+    "temperature": 0.85,
+    "top_p": 0.92,
+    "max_tokens": 220,
+    "stream": true
+  }'
+```
+
+## Which artifact should I download?
+
+- `merged/` is the standalone model. Use it with Transformers or vLLM.
+- `adapter/` is the smaller PEFT/QLoRA adapter. Load it on top of
+  `google/gemma-4-E4B-it` when you want to keep the base model separate.
+
+The merged weights are provided in BF16 safetensors shards. They are large;
+the hosted endpoint is usually the better choice for occasional use.
+
+## Local inference with Transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "text-generator/llmtrain"
+tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder="merged")
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    subfolder="merged",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+messages = [
+    {"role": "user", "content": "Write a short scene in a haunted hotel."},
+]
+inputs = tokenizer.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt"
+).to(model.device)
+with torch.inference_mode():
+    output = model.generate(
+        inputs, max_new_tokens=220, temperature=0.85, top_p=0.92,
+        do_sample=True,
+    )
+print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+## Local inference with vLLM
+
+```bash
+vllm serve text-generator/llmtrain \
+  --served-model-name gemma-roleplay-v2 \
+  --subfolder merged \
+  --dtype bfloat16 \
+  --max-model-len 4096
+```
+
+Then use the normal OpenAI client against `http://localhost:8000/v1`. For
+production, the Text-Generator.io deployment uses vLLM with FP8 weights,
+FP8 KV cache, asynchronous scheduling, CUDA graph warmup, and the matching
+Gemma MTP assistant. The validated deployment reached approximately **253
+tok/s** on an RTX 5090 benchmark and the production notes record **264 tok/s**
+after warmup (workload and concurrency affect the number).
+
+## How it was trained
+
+Gemma Roleplay v2 is a one-epoch supervised fine-tune of
+`google/gemma-4-E4B-it` using PEFT QLoRA:
+
+- 4-bit NF4 loading with BF16 compute
+- LoRA rank 32, alpha 64, dropout 0.05
+- attention and MLP projection targets (`q/k/v/o`, `gate/up/down`)
+- 4,096-token wrapped packing and completion-only loss
+- gradient checkpointing, paged 8-bit AdamW, TF32, and automatic checkpoint
+  resume
+- a conservative dataset filter for fictional consenting-adult roleplay,
+  with underage, coercive, exploitative, and ambiguous-age rows quarantined
+
+The training workbench also includes a reproducible validation suite covering
+roleplay, adult discussion, coding, casual chat, Spanish, and Japanese. The
+serving work focused on the practical latency win: FP8 reduces memory pressure,
+FP8 KV cache leaves room for longer context and batching, and the official
+Gemma MTP assistant speculatively drafts tokens without changing the target
+model's output distribution.
+
+## Limitations and evaluation
+
+This is a style-tuned chat model, not a factuality, medical, legal, or safety
+system. It can hallucinate, repeat itself, follow an adversarial instruction,
+or produce offensive material. It may be more willing than a typical aligned
+assistant to discuss adult fictional content. Evaluate it with your own prompts
+and add application-level safeguards before exposing it to untrusted users.
+
+The included benchmark is a small regression gate, not a representative human
+evaluation. In the recorded nine-prompt capability run, the selected serving
+configuration scored 0.9444 mean rubric score with no blocked-output or
+meta-commentary rows; treat this as an engineering smoke test, not a quality
+claim.
+
+## Provenance and acknowledgements
+
+The training workbench and dataset manifests are in the
+[Text-Generator.io repository](https://github.com/TextGeneratorio/text-generator.io/tree/vllm-accel-parity/llmtraining).
+The source corpus combines revision-pinned roleplay datasets whose declared
+licenses are recorded in `configs/skyfall_gemma_distill.yaml`; review those
+manifests before making a commercial redistribution decision. Teacher-model
+distillation outputs require separate permission checks.
+
+This model is a derivative of Google's Gemma family. Read and comply with the
+[Gemma Terms of Use](https://ai.google.dev/gemma/terms) and the base model card
+for the full downstream restrictions.