Text Generation
Transformers
Safetensors
English
nemotron_h
ai-text-detection
prose-provenance
conversational
Instructions to use rishanthrajendhran/ProseLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use rishanthrajendhran/ProseLens with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="rishanthrajendhran/ProseLens") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("rishanthrajendhran/ProseLens") model = AutoModelForCausalLM.from_pretrained("rishanthrajendhran/ProseLens", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use rishanthrajendhran/ProseLens with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "rishanthrajendhran/ProseLens" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rishanthrajendhran/ProseLens", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/rishanthrajendhran/ProseLens
- SGLang
How to use rishanthrajendhran/ProseLens with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "rishanthrajendhran/ProseLens" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rishanthrajendhran/ProseLens", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "rishanthrajendhran/ProseLens" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rishanthrajendhran/ProseLens", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use rishanthrajendhran/ProseLens with Docker Model Runner:
docker model run hf.co/rishanthrajendhran/ProseLens
Release: merged weights, adapter/ (Tinker LoRA), load_adapter.py, thresholds, card, LICENSE
Browse files- .gitattributes +1 -0
- LICENSE +51 -0
- README.md +183 -13
- adapter_config.json → adapter/adapter_config.json +0 -0
- adapter_model.safetensors → adapter/adapter_model.safetensors +0 -0
- chat_template.jinja +190 -0
- config.json +121 -0
- load_adapter.py +88 -0
- model-00001-of-00014.safetensors +3 -0
- model-00002-of-00014.safetensors +3 -0
- model-00003-of-00014.safetensors +3 -0
- model-00004-of-00014.safetensors +3 -0
- model-00005-of-00014.safetensors +3 -0
- model-00006-of-00014.safetensors +3 -0
- model-00007-of-00014.safetensors +3 -0
- model-00008-of-00014.safetensors +3 -0
- model-00009-of-00014.safetensors +3 -0
- model-00010-of-00014.safetensors +3 -0
- model-00011-of-00014.safetensors +3 -0
- model-00012-of-00014.safetensors +3 -0
- model-00013-of-00014.safetensors +3 -0
- model-00014-of-00014.safetensors +3 -0
- model.safetensors.index.json +0 -0
- special_tokens_map.json +30 -0
- thresholds.json +83 -0
- tokenizer.json +3 -0
- tokenizer_config.json +0 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
LICENSE
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
| 2 |
+
|
| 3 |
+
OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)
|
| 4 |
+
|
| 5 |
+
By exercising rights granted to you under this agreement, you accept and agree
|
| 6 |
+
to its terms.
|
| 7 |
+
|
| 8 |
+
As used in this agreement, "Model Materials" means the materials provided to
|
| 9 |
+
you under this agreement, consisting of: (1) one or more machine learning
|
| 10 |
+
models (including architecture and parameters); and (2) all related artifacts
|
| 11 |
+
(including associated data, documentation and software) that are provided to
|
| 12 |
+
you hereunder.
|
| 13 |
+
|
| 14 |
+
Subject to your compliance with this agreement, permission is hereby granted,
|
| 15 |
+
free of charge, to deal in the Model Materials without restriction, including
|
| 16 |
+
under all copyright, patent, database, and trade secret rights included or
|
| 17 |
+
embodied therein.
|
| 18 |
+
|
| 19 |
+
If you distribute any portion of the Model Materials, you shall retain in your
|
| 20 |
+
distribution (1) a copy of this agreement, and (2) all copyright notices and
|
| 21 |
+
other notices of origin included in the Model Materials that are applicable to
|
| 22 |
+
your distribution.
|
| 23 |
+
|
| 24 |
+
If you file, maintain, or voluntarily participate in a lawsuit against any
|
| 25 |
+
person or entity asserting that the Model Materials directly or indirectly
|
| 26 |
+
infringe any patent or copyright, then all rights and grants made to you
|
| 27 |
+
hereunder are terminated, unless that lawsuit was in response to a
|
| 28 |
+
corresponding lawsuit first brought against you.
|
| 29 |
+
|
| 30 |
+
This agreement does not impose any restrictions or obligations with respect to
|
| 31 |
+
any use, modification, or sharing of any outputs generated by using the Model
|
| 32 |
+
Materials.
|
| 33 |
+
|
| 34 |
+
THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
|
| 35 |
+
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 36 |
+
FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
|
| 37 |
+
ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
|
| 38 |
+
TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
|
| 39 |
+
|
| 40 |
+
YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
|
| 41 |
+
APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
|
| 42 |
+
ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
|
| 43 |
+
MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
|
| 44 |
+
REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
|
| 45 |
+
DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
|
| 46 |
+
ANYTHING INCORPORATED OR EMBODIED THEREIN.
|
| 47 |
+
|
| 48 |
+
IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
|
| 49 |
+
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
|
| 50 |
+
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
|
| 51 |
+
USE THEREOF OR OTHER DEALINGS THEREIN.
|
README.md
CHANGED
|
@@ -1,24 +1,194 @@
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
license_name: openmdw-1.1
|
|
|
|
| 4 |
base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
|
| 5 |
-
library_name:
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
extra_gated_prompt: "Access is granted individually. Please say who you are and what you intend to use the weights for."
|
| 8 |
---
|
| 9 |
|
| 10 |
-
#
|
| 11 |
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
|
| 16 |
-
-
|
| 17 |
-
|
| 18 |
|
| 19 |
-
|
| 20 |
-
normalised, giving P(human). The detector fires when that falls below a calibrated threshold.
|
| 21 |
-
Thresholds are quantiles over held-out human documents and are **not** included here: a cut
|
| 22 |
-
from one model or input form is meaningless against another's scores.
|
| 23 |
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
license_name: openmdw-1.1
|
| 4 |
+
license_link: LICENSE
|
| 5 |
base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
|
| 6 |
+
library_name: transformers
|
| 7 |
+
language:
|
| 8 |
+
- en
|
| 9 |
+
tags:
|
| 10 |
+
- ai-text-detection
|
| 11 |
+
- prose-provenance
|
| 12 |
+
datasets:
|
| 13 |
+
- rishanthrajendhran/IdeaLens-1M
|
| 14 |
extra_gated_prompt: "Access is granted individually. Please say who you are and what you intend to use the weights for."
|
| 15 |
---
|
| 16 |
|
| 17 |
+
# ProseLens
|
| 18 |
|
| 19 |
+
ProseLens detects **who wrote the words** of a document. It is IdeaLens's counterpart in the paper: the same
|
| 20 |
+
backbone, training documents and labels, but it reads the raw document text instead of an outline, so it learns
|
| 21 |
+
word-level provenance. It returns P(human), the probability that the document was written by a person.
|
| 22 |
|
| 23 |
+
ProseLens is `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16` fine-tuned with LoRA (rank 64) on 1M English web
|
| 24 |
+
documents ([IdeaLens-1M](https://huggingface.co/datasets/rishanthrajendhran/IdeaLens-1M)).
|
| 25 |
|
| 26 |
+
## Results
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
+
From the paper: ProseLens is accurate when a document's ideas and words come from the same source (99.1%), but
|
| 29 |
+
when they come from different sources it tracks the words and reaches 25.4% on idea provenance. IdeaLens, trained
|
| 30 |
+
identically on outlines, reaches 95.3% and 81.3%; Pangram 4 reaches 98.5% and 25.9%.
|
| 31 |
+
|
| 32 |
+
## Usage
|
| 33 |
+
|
| 34 |
+
Pass the document text as is.
|
| 35 |
+
|
| 36 |
+
### Load the merged model (66 GB download)
|
| 37 |
+
|
| 38 |
+
```python
|
| 39 |
+
import torch
|
| 40 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 41 |
+
|
| 42 |
+
tok = AutoTokenizer.from_pretrained("rishanthrajendhran/ProseLens")
|
| 43 |
+
model = AutoModelForCausalLM.from_pretrained("rishanthrajendhran/ProseLens", dtype=torch.bfloat16, device_map="auto").eval()
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
The weights take 59 GiB of GPU memory, and each input adds more; see *Hardware requirements*.
|
| 47 |
+
|
| 48 |
+
### Or apply the adapter to the base model (3 GB download)
|
| 49 |
+
|
| 50 |
+
`adapter/` holds the LoRA adapter as trained, in the layout of the Tinker training service. If you already have the
|
| 51 |
+
base model, `load_adapter.py` merges the adapter into it in memory. The resulting weights are bit-identical to the
|
| 52 |
+
merged model's:
|
| 53 |
+
|
| 54 |
+
```python
|
| 55 |
+
import importlib.util
|
| 56 |
+
from huggingface_hub import hf_hub_download
|
| 57 |
+
|
| 58 |
+
path = hf_hub_download("rishanthrajendhran/ProseLens", "load_adapter.py")
|
| 59 |
+
spec = importlib.util.spec_from_file_location("load_adapter", path)
|
| 60 |
+
la = importlib.util.module_from_spec(spec); spec.loader.exec_module(la)
|
| 61 |
+
model, tok = la.load_model() # base model + adapter/, then la.p_human(model, tok, text)
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
Do not load `adapter/` with `peft.PeftModel`. In transformers, Nemotron fuses the Mamba gate and x projections into
|
| 65 |
+
one `in_proj` and stores each layer's 128 routed experts as a single 3D tensor, so PEFT has nowhere to attach most of
|
| 66 |
+
the adapter and skips it without a warning; the model then scores close to the base model. `tinker-cookbook`'s
|
| 67 |
+
`weights.build_hf_model` can also merge the adapter into full weights.
|
| 68 |
+
|
| 69 |
+
### Score a document
|
| 70 |
+
|
| 71 |
+
ProseLens compares the next-token probabilities of `human` and `ai`:
|
| 72 |
+
|
| 73 |
+
```python
|
| 74 |
+
SYSTEM = "Given a document, answer with one word: human if the document was human-written, ai if it was AI-generated."
|
| 75 |
+
SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
|
| 76 |
+
HUMAN, AI = 50755, 2464 # token ids of "human" and "ai"
|
| 77 |
+
|
| 78 |
+
@torch.no_grad()
|
| 79 |
+
def p_human(text):
|
| 80 |
+
ids = tok.encode(f"<|im_start|>system\n{SYSTEM}<|im_end|>\n<|im_start|>user\n{text}{SUFFIX}",
|
| 81 |
+
add_special_tokens=False)
|
| 82 |
+
logits = model(torch.tensor([ids], device=model.device)).logits[0, -1].float()
|
| 83 |
+
return torch.softmax(logits[[HUMAN, AI]], -1)[0].item()
|
| 84 |
+
|
| 85 |
+
text = open("document.txt").read()
|
| 86 |
+
print(p_human(text))
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
Build the prompt string exactly as above rather than through the chat template.
|
| 90 |
+
|
| 91 |
+
### Thresholds
|
| 92 |
+
|
| 93 |
+
ProseLens flags a document as AI-written when P(human) is at or below a cut. Each cut is set so
|
| 94 |
+
that a given share of human documents is flagged (the false-positive rate, FPR), measured on the 80,000 human
|
| 95 |
+
documents in IdeaLens-1M's `calibration` split (10,000 per format). The paper's operating point is the global
|
| 96 |
+
cut at 1% FPR.
|
| 97 |
+
|
| 98 |
+
| FPR | 0.1% | 0.5% | 1% | 2% | 5% |
|
| 99 |
+
|---|---:|---:|---:|---:|---:|
|
| 100 |
+
| Global cut | 0.07082 | 0.30469 | 0.60539 | 0.92502 | 0.99889 |
|
| 101 |
+
|
| 102 |
+
Per-format cuts give each format its own operating point. They need the document's format, which the paper
|
| 103 |
+
assigns with WebOrganizer's FormatClassifier (FormatClassifier-NoURL when there is no URL). Each is the
|
| 104 |
+
format's own quantile, shrunk toward the global cut with weight n / (n + 2500); at 0.1% FPR 10,000 documents
|
| 105 |
+
per format are too few, so there is no per-format cut. A document outside these eight formats has no
|
| 106 |
+
per-format cut; do not fall back to the global cut for it.
|
| 107 |
+
|
| 108 |
+
| Format | 0.5% | 1% | 2% | 5% |
|
| 109 |
+
|---|---:|---:|---:|---:|
|
| 110 |
+
| Nonfiction Writing | 0.14387 | 0.27050 | 0.49499 | 0.93935 |
|
| 111 |
+
| Knowledge Article | 0.24236 | 0.38050 | 0.64044 | 0.95295 |
|
| 112 |
+
| Personal Blog | 0.68275 | 0.87993 | 0.98163 | 0.99966 |
|
| 113 |
+
| News Article | 0.29554 | 0.55535 | 0.90380 | 0.99788 |
|
| 114 |
+
| Academic Writing | 0.77788 | 0.89967 | 0.98279 | 0.99969 |
|
| 115 |
+
| User Reviews | 0.62135 | 0.85161 | 0.98035 | 0.99964 |
|
| 116 |
+
| Personal About Page | 0.41699 | 0.78714 | 0.97730 | 0.99966 |
|
| 117 |
+
| Creative Writing | 0.84923 | 0.91749 | 0.98411 | 0.99958 |
|
| 118 |
+
|
| 119 |
+
`thresholds.json` holds every cut at full precision.
|
| 120 |
+
|
| 121 |
+
These rates hold for English web documents like the training data. For another domain, fit the cut on
|
| 122 |
+
human documents from that domain.
|
| 123 |
+
|
| 124 |
+
ProseLens scores 97% of human calibration documents above 0.99, so its cuts at higher FPRs sit close
|
| 125 |
+
to 1 and small shifts in score move the realised FPR a long way.
|
| 126 |
+
|
| 127 |
+
## Hardware requirements
|
| 128 |
+
|
| 129 |
+
Measured with transformers 5.15 in bf16 on NVIDIA H100 80GB GPUs (our other runs used A100 80GB), with transformers'
|
| 130 |
+
PyTorch implementation of the Mamba layers (no fused Mamba kernels installed). We have not tried CPU-only inference.
|
| 131 |
+
|
| 132 |
+
| | Merged model | Adapter route (`load_adapter.py`) |
|
| 133 |
+
|---|---|---|
|
| 134 |
+
| Download | 65.8 GB | 65.8 GB base model + 3.1 GB adapter |
|
| 135 |
+
| Peak CPU RAM while loading | 60 GiB | 60 GiB |
|
| 136 |
+
| GPU memory once loaded | 58.8 GiB | 58.8 GiB (66 GiB during the ~10 s it takes to apply the adapter) |
|
| 137 |
+
|
| 138 |
+
GPU memory then grows with the length of the input, by about 4.2 MiB per token at typical lengths, scoring one
|
| 139 |
+
input at a time:
|
| 140 |
+
|
| 141 |
+
| Input tokens | 500 | 1,000 | 2,000 | 4,000 | 8,000 |
|
| 142 |
+
|---|---:|---:|---:|---:|---:|
|
| 143 |
+
| Peak GPU memory, one 80 GB GPU | 61.0 GiB | 63.1 GiB | 67.3 GiB | 75.7 GiB | does not fit |
|
| 144 |
+
| Peak memory per GPU, two 80 GB GPUs (`device_map="auto"`) | | | | 47.4 GiB | 63.7 GiB |
|
| 145 |
+
| Seconds per input, H100 | 0.18 | 0.34 | 0.66 | 1.32 | 2.70 |
|
| 146 |
+
|
| 147 |
+
Inputs of 6,000 tokens do not fit on one 80 GB GPU and 12,000 do not fit on two; lowering the Mamba chunk size from
|
| 148 |
+
128 to 64 did not change either limit.
|
| 149 |
+
|
| 150 |
+
ProseLens reads whole documents, so their length decides the hardware. One 80 GB GPU handles documents up to about
|
| 151 |
+
4,000 tokens (about 3,000 words); 85% of IdeaLens-1M's test documents are that short. Two 80 GB GPUs handle up to
|
| 152 |
+
8,000 tokens (about 6,000 words), which covers all but about 1 in 1,000 test documents. We have not measured longer
|
| 153 |
+
inputs or more GPUs.
|
| 154 |
+
|
| 155 |
+
## Intended use and limitations
|
| 156 |
+
|
| 157 |
+
- ProseLens estimates who wrote a document's words. It should not be the sole basis for decisions about a person's work.
|
| 158 |
+
- It was trained on English web documents of at least 500 words in eight long-form formats (Nonfiction Writing, Knowledge Article, Personal Blog, News Article, Academic Writing, User Reviews, Personal About Page, Creative Writing).
|
| 159 |
+
- Its training labels come from the Pangram prose detector, applied to whole documents.
|
| 160 |
+
|
| 161 |
+
## Related models
|
| 162 |
+
|
| 163 |
+
| Model | Backbone | Reads |
|
| 164 |
+
|---|---|---|
|
| 165 |
+
| [IdeaLens](https://huggingface.co/rishanthrajendhran/IdeaLens) | Nemotron-3.5-Lightning-30B-A3B, LoRA | outline |
|
| 166 |
+
| [ProseLens](https://huggingface.co/rishanthrajendhran/ProseLens) (this model) | Nemotron-3.5-Lightning-30B-A3B, LoRA | document text |
|
| 167 |
+
| [IdeaLens-NoParaphrase](https://huggingface.co/rishanthrajendhran/IdeaLens-NoParaphrase) | Nemotron-3.5-Lightning-30B-A3B, LoRA | outline, trained without paraphrasing |
|
| 168 |
+
| [IdeaLens-Qwen3.5-9B](https://huggingface.co/rishanthrajendhran/IdeaLens-Qwen3.5-9B) | Qwen3.5-9B, classification head | outline |
|
| 169 |
+
| [IdeaLens-ModernBERT-L](https://huggingface.co/rishanthrajendhran/IdeaLens-ModernBERT-L) | ModernBERT-large | outline |
|
| 170 |
+
| [ProseLens-ModernBERT-L](https://huggingface.co/rishanthrajendhran/ProseLens-ModernBERT-L) | ModernBERT-large | document text |
|
| 171 |
+
| [IdeaLens-LogisticClassifier](https://huggingface.co/rishanthrajendhran/IdeaLens-LogisticClassifier) | logistic regression over text-embedding-3-large | outline |
|
| 172 |
+
| [IdeaLens-ModernBERT-L-NoParaphrase](https://huggingface.co/rishanthrajendhran/IdeaLens-ModernBERT-L-NoParaphrase) | ModernBERT-large | outline, trained without paraphrasing |
|
| 173 |
+
| [IdeaLens-ModernBERT-L-RolesOnly](https://huggingface.co/rishanthrajendhran/IdeaLens-ModernBERT-L-RolesOnly) | ModernBERT-large | role labels only |
|
| 174 |
+
| [IdeaLens-Qwen3.5-9B-PerItem](https://huggingface.co/rishanthrajendhran/IdeaLens-Qwen3.5-9B-PerItem) | Qwen3.5-9B, classification head | single outline items, pooled |
|
| 175 |
+
| [IdeaLens-ModernBERT-L-PerItem](https://huggingface.co/rishanthrajendhran/IdeaLens-ModernBERT-L-PerItem) | ModernBERT-large | single outline items, pooled |
|
| 176 |
+
| [IdeaLens-LogisticClassifier-PerItem](https://huggingface.co/rishanthrajendhran/IdeaLens-LogisticClassifier-PerItem) | logistic regression over text-embedding-3-large | single outline items, pooled |
|
| 177 |
+
|
| 178 |
+
Training data: [IdeaLens-1M](https://huggingface.co/datasets/rishanthrajendhran/IdeaLens-1M).
|
| 179 |
+
|
| 180 |
+
## License
|
| 181 |
+
|
| 182 |
+
OpenMDW-1.1, the license of the base model (see `LICENSE`).
|
| 183 |
+
|
| 184 |
+
## Citation
|
| 185 |
+
|
| 186 |
+
```bibtex
|
| 187 |
+
@article{idealens2026,
|
| 188 |
+
title = {IdeaLens: Detecting AI Ideas in Long-form Writing},
|
| 189 |
+
author = {Anonymous},
|
| 190 |
+
journal = {arXiv preprint arXiv:TBD},
|
| 191 |
+
year = {2026},
|
| 192 |
+
url = {https://arxiv.org/abs/TBD}
|
| 193 |
+
}
|
| 194 |
+
```
|
adapter_config.json → adapter/adapter_config.json
RENAMED
|
File without changes
|
adapter_model.safetensors → adapter/adapter_model.safetensors
RENAMED
|
File without changes
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{% macro render_extra_keys(json_dict, handled_keys) %}
|
| 2 |
+
{%- if json_dict is mapping %}
|
| 3 |
+
{%- for json_key in json_dict if json_key not in handled_keys %}
|
| 4 |
+
{%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
|
| 5 |
+
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
|
| 6 |
+
{%- else %}
|
| 7 |
+
{{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
|
| 8 |
+
{%- endif %}
|
| 9 |
+
{%- endfor %}
|
| 10 |
+
{%- endif %}
|
| 11 |
+
{% endmacro %}
|
| 12 |
+
{%- set enable_thinking = enable_thinking if enable_thinking is defined else True %}
|
| 13 |
+
{%- set truncate_history_thinking = truncate_history_thinking if truncate_history_thinking is defined else True %}
|
| 14 |
+
{%- set ns = namespace(last_user_idx = -1) %}
|
| 15 |
+
{%- set loop_messages = messages %}
|
| 16 |
+
{%- for m in loop_messages %}
|
| 17 |
+
{%- if m["role"] == "user" %}
|
| 18 |
+
{%- set ns.last_user_idx = loop.index0 %}
|
| 19 |
+
{%- endif %}
|
| 20 |
+
{%- endfor %}
|
| 21 |
+
{%- if messages[0]["role"] == "system" %}
|
| 22 |
+
{%- set system_message = messages[0]["content"] %}
|
| 23 |
+
{%- set loop_messages = messages[1:] %}
|
| 24 |
+
{%- else %}
|
| 25 |
+
{%- set system_message = "" %}
|
| 26 |
+
{%- set loop_messages = messages %}
|
| 27 |
+
{%- endif %}
|
| 28 |
+
{%- if not tools is defined %}
|
| 29 |
+
{%- set tools = [] %}
|
| 30 |
+
{%- endif %}
|
| 31 |
+
{%- set ns = namespace(last_user_idx = -1) %}
|
| 32 |
+
{%- for m in loop_messages %}
|
| 33 |
+
{%- if m["role"] == "user" %}
|
| 34 |
+
{%- set ns.last_user_idx = loop.index0 %}
|
| 35 |
+
{%- endif %}
|
| 36 |
+
{%- endfor %}
|
| 37 |
+
{%- if system_message is defined %}
|
| 38 |
+
{{- "<|im_start|>system\n" + system_message }}
|
| 39 |
+
{%- else %}
|
| 40 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 41 |
+
{{- "<|im_start|>system\n" }}
|
| 42 |
+
{%- endif %}
|
| 43 |
+
{%- endif %}
|
| 44 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 45 |
+
{%- if system_message is defined and system_message | length > 0 %}
|
| 46 |
+
{{- "\n\n" }}
|
| 47 |
+
{%- endif %}
|
| 48 |
+
{{- "# Tools\n\nYou have access to the following functions:\n\n" }}
|
| 49 |
+
{{- "<tools>" }}
|
| 50 |
+
{%- for tool in tools %}
|
| 51 |
+
{%- if tool.function is defined %}
|
| 52 |
+
{%- set tool = tool.function %}
|
| 53 |
+
{%- endif %}
|
| 54 |
+
{{- "\n<function>\n<name>" ~ tool.name ~ "</name>" }}
|
| 55 |
+
{%- if tool.description is defined %}
|
| 56 |
+
{{- '\n<description>' ~ (tool.description | trim) ~ '</description>' }}
|
| 57 |
+
{%- endif %}
|
| 58 |
+
{{- '\n<parameters>' }}
|
| 59 |
+
{%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
|
| 60 |
+
{%- for param_name, param_fields in tool.parameters.properties|items %}
|
| 61 |
+
{{- '\n<parameter>' }}
|
| 62 |
+
{{- '\n<name>' ~ param_name ~ '</name>' }}
|
| 63 |
+
{%- if param_fields.type is defined %}
|
| 64 |
+
{{- '\n<type>' ~ (param_fields.type | string) ~ '</type>' }}
|
| 65 |
+
{%- endif %}
|
| 66 |
+
{%- if param_fields.description is defined %}
|
| 67 |
+
{{- '\n<description>' ~ (param_fields.description | trim) ~ '</description>' }}
|
| 68 |
+
{%- endif %}
|
| 69 |
+
{%- if param_fields.enum is defined %}
|
| 70 |
+
{{- '\n<enum>' ~ (param_fields.enum | tojson | safe) ~ '</enum>' }}
|
| 71 |
+
{%- endif %}
|
| 72 |
+
{%- set handled_keys = ['name', 'type', 'description', 'enum'] %}
|
| 73 |
+
{{- render_extra_keys(param_fields, handled_keys) }}
|
| 74 |
+
{{- '\n</parameter>' }}
|
| 75 |
+
{%- endfor %}
|
| 76 |
+
{%- endif %}
|
| 77 |
+
{% set handled_keys = ['type', 'properties', 'required'] %}
|
| 78 |
+
{{- render_extra_keys(tool.parameters, handled_keys) }}
|
| 79 |
+
{%- if tool.parameters is defined and tool.parameters.required is defined %}
|
| 80 |
+
{{- '\n<required>' ~ (tool.parameters.required | tojson | safe) ~ '</required>' }}
|
| 81 |
+
{%- endif %}
|
| 82 |
+
{{- '\n</parameters>' }}
|
| 83 |
+
{%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
|
| 84 |
+
{{- render_extra_keys(tool, handled_keys) }}
|
| 85 |
+
{{- '\n</function>' }}
|
| 86 |
+
{%- endfor %}
|
| 87 |
+
{{- "\n</tools>" }}
|
| 88 |
+
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 89 |
+
{%- endif %}
|
| 90 |
+
{%- if system_message is defined %}
|
| 91 |
+
{{- '<|im_end|>\n' }}
|
| 92 |
+
{%- else %}
|
| 93 |
+
{%- if tools is iterable and tools | length > 0 %}
|
| 94 |
+
{{- '<|im_end|>\n' }}
|
| 95 |
+
{%- endif %}
|
| 96 |
+
{%- endif %}
|
| 97 |
+
{%- for message in loop_messages %}
|
| 98 |
+
{%- if message.role == "assistant" %}
|
| 99 |
+
{%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
|
| 100 |
+
{%- set content = "<think>\n" ~ message.reasoning_content ~ "</think>" ~ (message.content | default('', true)) %}
|
| 101 |
+
{%- else %}
|
| 102 |
+
{%- set content = message.content | default('', true) %}
|
| 103 |
+
{%- if content is string -%}
|
| 104 |
+
{%- if '<think>' not in content and '</think>' not in content -%}
|
| 105 |
+
{%- set content = "<think></think>" ~ content -%}
|
| 106 |
+
{%- endif -%}
|
| 107 |
+
{%- else -%}
|
| 108 |
+
{%- set content = content -%}
|
| 109 |
+
{%- endif -%}
|
| 110 |
+
{%- endif %}
|
| 111 |
+
{%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
|
| 112 |
+
{{- '<|im_start|>assistant\n' }}
|
| 113 |
+
{%- set include_content = not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
|
| 114 |
+
{%- if content is string and content | trim | length > 0 %}
|
| 115 |
+
{%- if include_content %}
|
| 116 |
+
{{- (content | trim) ~ '\n' -}}
|
| 117 |
+
{%- else %}
|
| 118 |
+
{%- set c = (content | string) %}
|
| 119 |
+
{%- if '</think>' in c %}
|
| 120 |
+
{%- set c = c.split('</think>')[-1] %}
|
| 121 |
+
{%- elif '<think>' in c %}
|
| 122 |
+
{%- set c = c.split('<think>')[0] %}
|
| 123 |
+
{%- endif %}
|
| 124 |
+
{%- set c = "<think></think>" ~ c %}
|
| 125 |
+
{%- if c | length > 0 %}
|
| 126 |
+
{{- c ~ '\n' -}}
|
| 127 |
+
{%- endif %}
|
| 128 |
+
{%- endif %}
|
| 129 |
+
{%- else %}
|
| 130 |
+
{{- "<think></think>" -}}
|
| 131 |
+
{%- endif %}
|
| 132 |
+
{%- for tool_call in message.tool_calls %}
|
| 133 |
+
{%- if tool_call.function is defined %}
|
| 134 |
+
{%- set tool_call = tool_call.function %}
|
| 135 |
+
{%- endif %}
|
| 136 |
+
{{- '<tool_call>\n<function=' ~ tool_call.name ~ '>\n' -}}
|
| 137 |
+
{%- if tool_call.arguments is defined %}
|
| 138 |
+
{%- for args_name, args_value in tool_call.arguments|items %}
|
| 139 |
+
{{- '<parameter=' ~ args_name ~ '>\n' -}}
|
| 140 |
+
{%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
|
| 141 |
+
{{- args_value ~ '\n</parameter>\n' -}}
|
| 142 |
+
{%- endfor %}
|
| 143 |
+
{%- endif %}
|
| 144 |
+
{{- '</function>\n</tool_call>\n' -}}
|
| 145 |
+
{%- endfor %}
|
| 146 |
+
{{- '<|im_end|>\n' }}
|
| 147 |
+
{%- else %}
|
| 148 |
+
{%- if not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
|
| 149 |
+
{{- '<|im_start|>assistant\n' ~ (content | default('', true) | string | trim) ~ '<|im_end|>\n' }}
|
| 150 |
+
{%- else %}
|
| 151 |
+
{%- set c = (content | default('', true) | string) %}
|
| 152 |
+
{%- if '<think>' in c and '</think>' in c %}
|
| 153 |
+
{%- set c = "<think></think>" ~ c.split('</think>')[-1] %}
|
| 154 |
+
{%- endif %}
|
| 155 |
+
{%- set c = c | trim %}
|
| 156 |
+
{%- if c | length > 0 %}
|
| 157 |
+
{{- '<|im_start|>assistant\n' ~ c ~ '<|im_end|>\n' }}
|
| 158 |
+
{%- else %}
|
| 159 |
+
{{- '<|im_start|>assistant\n<|im_end|>\n' }}
|
| 160 |
+
{%- endif %}
|
| 161 |
+
{%- endif %}
|
| 162 |
+
{%- endif %}
|
| 163 |
+
{%- elif message.role == "user" or message.role == "system" %}
|
| 164 |
+
{{- '<|im_start|>' + message.role + '\n' }}
|
| 165 |
+
{%- set content = message.content | string %}
|
| 166 |
+
{{- content }}
|
| 167 |
+
{{- '<|im_end|>\n' }}
|
| 168 |
+
{%- elif message.role == "tool" %}
|
| 169 |
+
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
| 170 |
+
{{- '<|im_start|>user\n' }}
|
| 171 |
+
{%- endif %}
|
| 172 |
+
{{- '<tool_response>\n' }}
|
| 173 |
+
{{- message.content }}
|
| 174 |
+
{{- '\n</tool_response>\n' }}
|
| 175 |
+
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
| 176 |
+
{{- '<|im_end|>\n' }}
|
| 177 |
+
{%- elif loop.last %}
|
| 178 |
+
{{- '<|im_end|>\n' }}
|
| 179 |
+
{%- endif %}
|
| 180 |
+
{%- else %}
|
| 181 |
+
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
|
| 182 |
+
{%- endif %}
|
| 183 |
+
{%- endfor %}
|
| 184 |
+
{%- if add_generation_prompt %}
|
| 185 |
+
{%- if enable_thinking %}
|
| 186 |
+
{{- '<|im_start|>assistant\n<think>\n' }}
|
| 187 |
+
{%- else %}
|
| 188 |
+
{{- '<|im_start|>assistant\n<think></think>' }}
|
| 189 |
+
{%- endif %}
|
| 190 |
+
{%- endif %}
|
config.json
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"NemotronHForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"bos_token_id": 1,
|
| 8 |
+
"chunk_size": 128,
|
| 9 |
+
"conv_kernel": 4,
|
| 10 |
+
"dtype": "bfloat16",
|
| 11 |
+
"eos_token_id": 2,
|
| 12 |
+
"expand": 2,
|
| 13 |
+
"head_dim": 128,
|
| 14 |
+
"hidden_dropout": 0.0,
|
| 15 |
+
"hidden_size": 2688,
|
| 16 |
+
"initializer_range": 0.02,
|
| 17 |
+
"intermediate_size": 1856,
|
| 18 |
+
"layer_norm_epsilon": 1e-05,
|
| 19 |
+
"mamba_head_dim": 64,
|
| 20 |
+
"mamba_hidden_act": "silu",
|
| 21 |
+
"mamba_num_heads": 64,
|
| 22 |
+
"mamba_proj_bias": false,
|
| 23 |
+
"mamba_ssm_cache_dtype": "float32",
|
| 24 |
+
"max_position_embeddings": 262144,
|
| 25 |
+
"mlp_bias": false,
|
| 26 |
+
"mlp_hidden_act": "relu2",
|
| 27 |
+
"model_type": "nemotron_h",
|
| 28 |
+
"moe_intermediate_size": 1856,
|
| 29 |
+
"moe_latent_size": null,
|
| 30 |
+
"moe_shared_expert_intermediate_size": 3712,
|
| 31 |
+
"moe_shared_expert_overlap": true,
|
| 32 |
+
"n_group": 1,
|
| 33 |
+
"n_groups": 8,
|
| 34 |
+
"n_routed_experts": 128,
|
| 35 |
+
"n_shared_experts": 1,
|
| 36 |
+
"norm_eps": 1e-05,
|
| 37 |
+
"norm_topk_prob": true,
|
| 38 |
+
"num_attention_heads": 32,
|
| 39 |
+
"num_experts_per_tok": 6,
|
| 40 |
+
"num_hidden_layers": 52,
|
| 41 |
+
"num_key_value_heads": 2,
|
| 42 |
+
"num_logits_to_keep": 1,
|
| 43 |
+
"num_nextn_predict_layers": 1,
|
| 44 |
+
"pad_token_id": 0,
|
| 45 |
+
"partial_rotary_factor": 1.0,
|
| 46 |
+
"rescale_prenorm_residual": true,
|
| 47 |
+
"residual_in_fp32": false,
|
| 48 |
+
"rope_theta": 10000,
|
| 49 |
+
"routed_scaling_factor": 2.5,
|
| 50 |
+
"sliding_window": null,
|
| 51 |
+
"ssm_state_size": 128,
|
| 52 |
+
"tie_word_embeddings": false,
|
| 53 |
+
"time_step_floor": 0.0001,
|
| 54 |
+
"time_step_max": 0.1,
|
| 55 |
+
"time_step_min": 0.001,
|
| 56 |
+
"topk_group": 1,
|
| 57 |
+
"transformers_version": "4.57.6",
|
| 58 |
+
"use_bias": false,
|
| 59 |
+
"use_cache": true,
|
| 60 |
+
"use_conv_bias": true,
|
| 61 |
+
"use_mamba_kernels": true,
|
| 62 |
+
"vocab_size": 131072,
|
| 63 |
+
"layers_block_type": [
|
| 64 |
+
"mamba",
|
| 65 |
+
"moe",
|
| 66 |
+
"mamba",
|
| 67 |
+
"moe",
|
| 68 |
+
"mamba",
|
| 69 |
+
"attention",
|
| 70 |
+
"moe",
|
| 71 |
+
"mamba",
|
| 72 |
+
"moe",
|
| 73 |
+
"mamba",
|
| 74 |
+
"moe",
|
| 75 |
+
"mamba",
|
| 76 |
+
"attention",
|
| 77 |
+
"moe",
|
| 78 |
+
"mamba",
|
| 79 |
+
"moe",
|
| 80 |
+
"mamba",
|
| 81 |
+
"moe",
|
| 82 |
+
"mamba",
|
| 83 |
+
"attention",
|
| 84 |
+
"moe",
|
| 85 |
+
"mamba",
|
| 86 |
+
"moe",
|
| 87 |
+
"mamba",
|
| 88 |
+
"moe",
|
| 89 |
+
"mamba",
|
| 90 |
+
"attention",
|
| 91 |
+
"moe",
|
| 92 |
+
"mamba",
|
| 93 |
+
"moe",
|
| 94 |
+
"mamba",
|
| 95 |
+
"moe",
|
| 96 |
+
"mamba",
|
| 97 |
+
"attention",
|
| 98 |
+
"moe",
|
| 99 |
+
"mamba",
|
| 100 |
+
"moe",
|
| 101 |
+
"mamba",
|
| 102 |
+
"moe",
|
| 103 |
+
"mamba",
|
| 104 |
+
"moe",
|
| 105 |
+
"mamba",
|
| 106 |
+
"attention",
|
| 107 |
+
"moe",
|
| 108 |
+
"mamba",
|
| 109 |
+
"moe",
|
| 110 |
+
"mamba",
|
| 111 |
+
"moe",
|
| 112 |
+
"mamba",
|
| 113 |
+
"moe",
|
| 114 |
+
"mamba",
|
| 115 |
+
"moe"
|
| 116 |
+
],
|
| 117 |
+
"mtp_layers_block_type": [
|
| 118 |
+
"attention",
|
| 119 |
+
"moe"
|
| 120 |
+
]
|
| 121 |
+
}
|
load_adapter.py
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Apply the LoRA adapter in this repo's adapter/ folder to the base model, then score role-labelled outlines.
|
| 2 |
+
|
| 3 |
+
adapter/ holds the adapter as trained, in the layout of the Tinker training service. Do not load it with
|
| 4 |
+
peft.PeftModel. In transformers, Nemotron fuses the Mamba gate and x projections into one `in_proj`, and stores each
|
| 5 |
+
MoE layer's 128 routed experts as a single 3D tensor. PEFT has no module to attach those LoRA weights to and skips them
|
| 6 |
+
without a warning. This script instead merges every LoRA delta, W += (alpha / r) * B @ A, into the base weights in
|
| 7 |
+
place, following the same rules as the tinker-cookbook merge (tinker_cookbook.weights.build_hf_model). The resulting
|
| 8 |
+
weights are bit-identical to the merged model in this repo.
|
| 9 |
+
|
| 10 |
+
Needs transformers >= 5.15, torch, safetensors and huggingface_hub. The weights take 59 GiB of GPU memory and each
|
| 11 |
+
input adds about 4.2 MiB per token, so one 80 GB GPU handles inputs up to about 4,000 tokens; see the model card.
|
| 12 |
+
"""
|
| 13 |
+
import json
|
| 14 |
+
|
| 15 |
+
import torch
|
| 16 |
+
from huggingface_hub import snapshot_download
|
| 17 |
+
from safetensors.torch import load_file
|
| 18 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 19 |
+
|
| 20 |
+
REPO = "rishanthrajendhran/ProseLens"
|
| 21 |
+
BASE = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
|
| 22 |
+
SYSTEM = 'Given a role-labelled outline of a document, answer with one word: human if the source document was human-written, ai if it was AI-generated.'
|
| 23 |
+
SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
|
| 24 |
+
HUMAN, AI = 50755, 2464
|
| 25 |
+
CUT = 0.60539 # P(human) at or below this is flagged: the global cut at 1% FPR (thresholds.json)
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def apply_tinker_lora(model, adapter_dir):
|
| 29 |
+
"""Merge a Tinker LoRA archive for Nemotron-3.5 into `model` in place. Returns the parameter names changed."""
|
| 30 |
+
cfg = json.load(open(f"{adapter_dir}/adapter_config.json"))
|
| 31 |
+
scale = cfg["lora_alpha"] / cfg["r"]
|
| 32 |
+
lora = load_file(f"{adapter_dir}/adapter_model.safetensors")
|
| 33 |
+
params = dict(model.named_parameters())
|
| 34 |
+
touched = []
|
| 35 |
+
for key_a in sorted(k for k in lora if k.endswith(".lora_A.weight")):
|
| 36 |
+
A, B = lora[key_a], lora[key_a.replace(".lora_A.", ".lora_B.")]
|
| 37 |
+
if A.numel() == 0: # the experts have no gate projection; Tinker keeps an empty w3 placeholder
|
| 38 |
+
continue
|
| 39 |
+
name = key_a.removeprefix("base_model.model.").removesuffix(".lora_A.weight")
|
| 40 |
+
rows = None
|
| 41 |
+
if name == "model.lm_head": # Tinker nests the LM head under model.
|
| 42 |
+
target = "lm_head.weight"
|
| 43 |
+
elif name.endswith((".gate_proj", ".x_proj")): # Mamba: in_proj rows are [gate | x | B | C | dt]
|
| 44 |
+
layer, proj = name.rsplit(".", 1)
|
| 45 |
+
target = f"{layer}.in_proj.weight"
|
| 46 |
+
start = 0 if proj == "gate_proj" else lora[f"base_model.model.{layer}.gate_proj.lora_B.weight"].shape[0]
|
| 47 |
+
rows = slice(start, start + B.shape[0])
|
| 48 |
+
elif name.endswith(".experts.w1"): # routed experts: w1 = up_proj, one (expert, out, in) tensor per layer
|
| 49 |
+
target = name.removesuffix("w1") + "up_proj"
|
| 50 |
+
elif name.endswith(".experts.w2"): # w2 = down_proj
|
| 51 |
+
target = name.removesuffix("w2") + "down_proj"
|
| 52 |
+
else: # attention, Mamba out_proj, shared experts
|
| 53 |
+
target = name + ".weight"
|
| 54 |
+
W = params[target] # KeyError here means the adapter does not match this model
|
| 55 |
+
# For experts one side is shared (leading dim 1) and broadcasts across the 128 experts.
|
| 56 |
+
delta = scale * torch.matmul(B.to(W.device, torch.float32), A.to(W.device, torch.float32))
|
| 57 |
+
with torch.no_grad():
|
| 58 |
+
view = W.data if rows is None else W.data[rows]
|
| 59 |
+
assert view.shape == delta.shape, (target, tuple(view.shape), tuple(delta.shape))
|
| 60 |
+
view.copy_((view.float() + delta).to(W.dtype))
|
| 61 |
+
touched.append(target)
|
| 62 |
+
return touched
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def load_model(repo=REPO, device_map="auto"):
|
| 66 |
+
tok = AutoTokenizer.from_pretrained(BASE)
|
| 67 |
+
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map=device_map).eval()
|
| 68 |
+
adapter_dir = snapshot_download(repo, allow_patterns=["adapter/*"]) + "/adapter"
|
| 69 |
+
apply_tinker_lora(model, adapter_dir)
|
| 70 |
+
return model, tok
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
@torch.no_grad()
|
| 74 |
+
def p_human(model, tok, outline):
|
| 75 |
+
"""P(human) for one document's text."""
|
| 76 |
+
ids = tok.encode(f"<|im_start|>system\n{SYSTEM}<|im_end|>\n<|im_start|>user\n{outline}{SUFFIX}",
|
| 77 |
+
add_special_tokens=False)
|
| 78 |
+
logits = model(torch.tensor([ids], device=model.device)).logits[0, -1].float()
|
| 79 |
+
return torch.softmax(logits[[HUMAN, AI]], -1)[0].item()
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
if __name__ == "__main__":
|
| 83 |
+
model, tok = load_model()
|
| 84 |
+
outline = ("[Central Development] A town's water supply fails after a drought, and residents organise to share wells.\n"
|
| 85 |
+
"[Background Context] The reservoir has been shrinking for three summers.\n"
|
| 86 |
+
"[Open Question] Whether the council will fund a new pipeline remains undecided.")
|
| 87 |
+
p = p_human(model, tok, outline)
|
| 88 |
+
print(f"P(human) = {p:.4f} -> {'flagged as AI-written' if p <= CUT else 'not flagged'}")
|
model-00001-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cd128bfd93420bd0f0e5d036cd173f4ee09ae0ccbcc0de0fdcb375270b3ed39c
|
| 3 |
+
size 4991210024
|
model-00002-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:85969f9b02e6dcb1d11177a511ad77556fa1d383d2a602fd14e2fe540785f06e
|
| 3 |
+
size 4992601016
|
model-00003-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3c1cb332c55e654810fd6af3559f9d517ae44a36fe720fcbc305b98950e86e94
|
| 3 |
+
size 4992601368
|
model-00004-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:63098215bc454e76b3384f11ba92b9a812d30d533c2a6904626a86c8e3231b50
|
| 3 |
+
size 4995692800
|
model-00005-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:db7d1f3278bed9d1751495d5424cd37aa7cd90e4da236d9c52035b12fc75ea61
|
| 3 |
+
size 4980553832
|
model-00006-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e567c95f1fc5839210ae20d57d5c799c07bfbe1b561491118572f9cc24f03c52
|
| 3 |
+
size 4989423536
|
model-00007-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a9382a9eb90da4e011b36095bff2cac4be8337631eb919584d43120b4cf2d0ce
|
| 3 |
+
size 4992601488
|
model-00008-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0c26027509bb92721249a00f4f69231e6f8739a7604289d7f7943815908a0f3a
|
| 3 |
+
size 4992601520
|
model-00009-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c3cc352129ba528bcbb3764f922a718a6612b21e346fc8552c1341d9c548ad8a
|
| 3 |
+
size 4995692800
|
model-00010-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5723b717fbbe0a106d4aa75607ebc984118d2a028b014338fa9d5aa55515fc18
|
| 3 |
+
size 4992601520
|
model-00011-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:79067ee3d8428ad6480ef507794183f0b424ce431dd29882db2a34d12b3fd711
|
| 3 |
+
size 4995692800
|
model-00012-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e68b02cf6cce08dc4ee1f7adc72241282819026026d5fdb965d759410e6ca82c
|
| 3 |
+
size 4995692816
|
model-00013-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a0ec01201aa1a054a3d691bcf9495405b7f22ee0d3c10fbf06f4ac396ee78cd6
|
| 3 |
+
size 3249723504
|
model-00014-of-00014.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:64577b275ca4e7e5266eae0903674f7f46ec2a8cbf4f4f1a3207f80d503cd1d0
|
| 3 |
+
size 2670685240
|
model.safetensors.index.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": {
|
| 3 |
+
"content": "<s>",
|
| 4 |
+
"lstrip": false,
|
| 5 |
+
"normalized": false,
|
| 6 |
+
"rstrip": false,
|
| 7 |
+
"single_word": false
|
| 8 |
+
},
|
| 9 |
+
"eos_token": {
|
| 10 |
+
"content": "<|im_end|>",
|
| 11 |
+
"lstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"rstrip": false,
|
| 14 |
+
"single_word": false
|
| 15 |
+
},
|
| 16 |
+
"pad_token": {
|
| 17 |
+
"content": "<|im_end|>",
|
| 18 |
+
"lstrip": false,
|
| 19 |
+
"normalized": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"single_word": false
|
| 22 |
+
},
|
| 23 |
+
"unk_token": {
|
| 24 |
+
"content": "<unk>",
|
| 25 |
+
"lstrip": false,
|
| 26 |
+
"normalized": false,
|
| 27 |
+
"rstrip": false,
|
| 28 |
+
"single_word": false
|
| 29 |
+
}
|
| 30 |
+
}
|
thresholds.json
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "rishanthrajendhran/ProseLens",
|
| 3 |
+
"input": "document",
|
| 4 |
+
"flag_rule": "flag the document as AI when P(human) <= cut",
|
| 5 |
+
"calibration": {
|
| 6 |
+
"data": "IdeaLens-1M, calibration split (human documents only)",
|
| 7 |
+
"scores_from": "the training checkpoint, scored on Tinker",
|
| 8 |
+
"n_humans": 80000,
|
| 9 |
+
"n_per_format": {
|
| 10 |
+
"Academic Writing": 10000,
|
| 11 |
+
"Creative Writing": 10000,
|
| 12 |
+
"Knowledge Article": 10000,
|
| 13 |
+
"News Article": 10000,
|
| 14 |
+
"Nonfiction Writing": 10000,
|
| 15 |
+
"Personal About Page": 10000,
|
| 16 |
+
"Personal Blog": 10000,
|
| 17 |
+
"User Reviews": 10000
|
| 18 |
+
}
|
| 19 |
+
},
|
| 20 |
+
"method": {
|
| 21 |
+
"global": "the target-FPR quantile of P(human) over all calibration humans",
|
| 22 |
+
"per_group": "the group's own quantile, shrunk toward the global cut with weight n / (n + 2500)",
|
| 23 |
+
"min_group_humans": 200,
|
| 24 |
+
"estimability": "a cut at FPR q is given only when q * n >= 25 calibration humans",
|
| 25 |
+
"missing_cut": "no cut for a group means none could be estimated; do not substitute the global cut"
|
| 26 |
+
},
|
| 27 |
+
"fpr_targets": [
|
| 28 |
+
0.001,
|
| 29 |
+
0.005,
|
| 30 |
+
0.01,
|
| 31 |
+
0.02,
|
| 32 |
+
0.05
|
| 33 |
+
],
|
| 34 |
+
"global": {
|
| 35 |
+
"0.001": 0.07081838771272196,
|
| 36 |
+
"0.005": 0.30468918373143306,
|
| 37 |
+
"0.01": 0.6053894506737609,
|
| 38 |
+
"0.02": 0.925022270529183,
|
| 39 |
+
"0.05": 0.998888189099037
|
| 40 |
+
},
|
| 41 |
+
"per_format": {
|
| 42 |
+
"0.005": {
|
| 43 |
+
"Academic Writing": 0.7778763017339165,
|
| 44 |
+
"Creative Writing": 0.8492284331725566,
|
| 45 |
+
"Knowledge Article": 0.24235805929603155,
|
| 46 |
+
"News Article": 0.2955419059997487,
|
| 47 |
+
"Nonfiction Writing": 0.14387439727844545,
|
| 48 |
+
"Personal About Page": 0.4169940435489798,
|
| 49 |
+
"Personal Blog": 0.6827533426351194,
|
| 50 |
+
"User Reviews": 0.6213508153913166
|
| 51 |
+
},
|
| 52 |
+
"0.01": {
|
| 53 |
+
"Academic Writing": 0.8996738533714292,
|
| 54 |
+
"Creative Writing": 0.9174860344190486,
|
| 55 |
+
"Knowledge Article": 0.3804974222436696,
|
| 56 |
+
"News Article": 0.555354294854766,
|
| 57 |
+
"Nonfiction Writing": 0.27050169337460295,
|
| 58 |
+
"Personal About Page": 0.7871415559941629,
|
| 59 |
+
"Personal Blog": 0.8799318831045015,
|
| 60 |
+
"User Reviews": 0.8516071157574004
|
| 61 |
+
},
|
| 62 |
+
"0.02": {
|
| 63 |
+
"Academic Writing": 0.9827934935442895,
|
| 64 |
+
"Creative Writing": 0.9841078402345083,
|
| 65 |
+
"Knowledge Article": 0.6404405355320586,
|
| 66 |
+
"News Article": 0.9037969402135294,
|
| 67 |
+
"Nonfiction Writing": 0.494985268248519,
|
| 68 |
+
"Personal About Page": 0.9772992347199888,
|
| 69 |
+
"Personal Blog": 0.9816307917983351,
|
| 70 |
+
"User Reviews": 0.9803463940922453
|
| 71 |
+
},
|
| 72 |
+
"0.05": {
|
| 73 |
+
"Academic Writing": 0.9996898375219624,
|
| 74 |
+
"Creative Writing": 0.9995782080959261,
|
| 75 |
+
"Knowledge Article": 0.952949484412262,
|
| 76 |
+
"News Article": 0.997881027494693,
|
| 77 |
+
"Nonfiction Writing": 0.9393488591865028,
|
| 78 |
+
"Personal About Page": 0.9996622303035176,
|
| 79 |
+
"Personal Blog": 0.999661233340514,
|
| 80 |
+
"User Reviews": 0.9996383495931975
|
| 81 |
+
}
|
| 82 |
+
}
|
| 83 |
+
}
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:623c34567aebb18582765289fbe23d901c62704d6518d71866e0e58db892b5b7
|
| 3 |
+
size 17077484
|
tokenizer_config.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|