timosarkar commited on
Commit
aefea79
·
verified ·
1 Parent(s): b480919

added readme

Browse files
Files changed (1) hide show
  1. README.md +223 -0
README.md CHANGED
@@ -1,3 +1,226 @@
1
  ---
2
  license: bsd-3-clause
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: bsd-3-clause
3
+ language:
4
+ - en
5
+ library_name: transformers
6
+ pipeline_tag: text-classification
7
+ tags:
8
+ - bert
9
+ - bert-mini
10
+ - classification
11
+ - decision-model
12
+ - dynamic-classification
13
+ - calibrated-classification
14
+ - option-scoring
15
+ - oryn
16
+ datasets:
17
+ - zefang-liu/phishing-email-dataset
18
+ - fancyzhx/ag_news
19
+ - google/boolq
20
+ - SetFit/sst5
21
+ metrics:
22
+ - accuracy
23
+ - brier_score
24
+ - expected_calibration_error
25
+ base_model:
26
+ - boltuix/bert-mini
27
  ---
28
+
29
+ # Oryn
30
+
31
+ Oryn is a compact, low-latency decision model built on
32
+ [`boltuix/bert-mini`](https://huggingface.co/boltuix/bert-mini).
33
+
34
+ Unlike a conventional classifier with a fixed output layer, Oryn receives the
35
+ classification question and candidate answers at inference time. This allows
36
+ the same model to handle binary decisions, multiclass classification, ordinal
37
+ scoring, and other option-based tasks.
38
+
39
+ Oryn is inspired by the publicly described architecture and training ideas
40
+ behind Laya.
41
+
42
+ ## Highlights
43
+
44
+ - Approximately 21 MB of trained weights
45
+ - Approximately 3 ms warmed-up single-request latency in the author's local test
46
+ - Dynamically supplied classification options
47
+ - Supports binary, categorical, and ordinal questions
48
+ - Uses one `[MASK]` marker for each candidate answer
49
+ - Scores all supplied options relative to one another
50
+ - Trained on phishing detection, BoolQ, AG News, and SST-5
51
+ - Designed for local and CPU-friendly inference
52
+
53
+ Latency depends on the device, runtime, input length, and measurement method.
54
+ The reported 3 ms result should not be treated as a universal benchmark.
55
+
56
+ ## Architecture
57
+
58
+ Oryn consists of:
59
+
60
+ 1. A pretrained BERT Mini encoder
61
+ 2. Two additional Transformer encoder layers
62
+ 3. One `[MASK]` marker for each candidate answer
63
+ 4. A learned linear option scorer
64
+ 5. A softmax over the options provided with the request
65
+
66
+ A request is represented approximately as:
67
+
68
+ ```text
69
+ question: Which category best describes this text?
70
+ state: The company reported record quarterly revenue.
71
+ options: business [MASK], sports [MASK], technology [MASK], politics [MASK]
72
+ ```
73
+
74
+ The contextual representation at each option's `[MASK]` token is passed through
75
+ the shared scoring layer. The resulting logits are normalized across the
76
+ candidate options.
77
+
78
+ Because the answer space is supplied at request time, it is not permanently
79
+ encoded into a fixed classification head.
80
+
81
+ ## Evaluation
82
+
83
+ Oryn was evaluated on a random sample of 1,000 held-out examples from its
84
+ multitask validation set.
85
+
86
+ | Metric | Result |
87
+ |---|---:|
88
+ | Overall accuracy | 71.7% |
89
+ | Brier score | 0.3468 |
90
+ | Negative log-likelihood | 0.6228 |
91
+ | Expected calibration error | 0.0381 |
92
+ | Average confidence | 0.7518 |
93
+ | Batched throughput | 129.9 examples/second |
94
+
95
+ ### Results by question type
96
+
97
+ | Question type | Examples | Accuracy | Brier score |
98
+ |---|---:|---:|---:|
99
+ | Binary / `noul` | 591 | 85.62% | 0.2083 |
100
+ | Categorical / `choice` | 196 | 79.59% | 0.2793 |
101
+ | Ordinal / `score` | 213 | 25.82% | 0.7933 |
102
+
103
+ The ordinal sentiment task is currently the model's main weakness. The model is
104
+ substantially stronger on binary and categorical decisions.
105
+
106
+ The benchmark was performed with batched GPU inference. Batched latency is not
107
+ equivalent to end-to-end single-request latency.
108
+
109
+ ## Training data
110
+
111
+ Oryn was trained on a mixture of:
112
+
113
+ - `zefang-liu/phishing-email-dataset`
114
+ - `fancyzhx/ag_news`
115
+ - `google/boolq`
116
+ - `SetFit/sst5`
117
+
118
+ These datasets expose the model to:
119
+
120
+ - Phishing and fraud detection
121
+ - Binary reading comprehension
122
+ - News topic classification
123
+ - Five-level ordinal sentiment classification
124
+
125
+ ## Training method
126
+
127
+ Training used two phases:
128
+
129
+ 1. Supervised cross-entropy warmup
130
+ 2. Calibration-oriented optimization using differentiable proper-scoring
131
+ objectives
132
+
133
+ For nominal decisions, the calibration objective combines cross-entropy with a
134
+ spherical scoring term. For ordinal decisions, cross-entropy is combined with
135
+ ranked probability score.
136
+
137
+ This training procedure is Laya-inspired, but it does not reproduce the
138
+ original RLCD/REINFORCE procedure exactly.
139
+
140
+ ## Intended uses
141
+
142
+ Oryn is suitable for experimentation with:
143
+
144
+ - Message triage
145
+ - Phishing detection
146
+ - Topic classification
147
+ - Sentiment classification
148
+ - Support-request routing
149
+ - Urgency classification
150
+ - Small local decision systems
151
+ - Dynamic classification research
152
+ - Low-latency edge or local inference
153
+
154
+ ## General classification
155
+
156
+ Oryn can technically receive any question and any set of options:
157
+
158
+ ```python
159
+ result = predict(
160
+ text="The company reported record quarterly revenue.",
161
+ question="Which category best describes this text?",
162
+ options=[
163
+ "business",
164
+ "sports",
165
+ "technology",
166
+ "politics",
167
+ ],
168
+ )
169
+ ```
170
+
171
+ However, architectural support for arbitrary options does not guarantee
172
+ reliable knowledge of arbitrary domains.
173
+
174
+ Oryn is most reliable on tasks similar to its training mixture. New domains
175
+ should be evaluated and usually fine-tuned before production use.
176
+
177
+ ## Example output
178
+
179
+ ```python
180
+ {
181
+ "prediction": "business",
182
+ "confidence": 0.8421,
183
+ "probabilities": {
184
+ "business": 0.8421,
185
+ "sports": 0.0314,
186
+ "technology": 0.0987,
187
+ "politics": 0.0278
188
+ }
189
+ }
190
+ ```
191
+
192
+ ## Files
193
+
194
+
195
+ The custom `inference.py` implementation is required because Oryn uses a custom
196
+ option-scoring head rather than a standard Transformers sequence-classification
197
+ head.
198
+
199
+ ## Installation
200
+
201
+ ```bash
202
+ pip install torch transformers
203
+ ```
204
+
205
+ ## Inference
206
+
207
+ ```bash
208
+ python inference.py \
209
+ --text "The company reported record quarterly revenue." \
210
+ --question "Which category best describes this text?" \
211
+ --options business sports technology politics
212
+ ```
213
+
214
+ ## Citation
215
+
216
+ If you use Oryn, cite this model repository:
217
+
218
+ ```bibtex
219
+ @misc{oryn_bert_mini,
220
+ title = {Oryn: A compact low-latency BERT Mini option-scoring model},
221
+ author = {Timo Sarkar},
222
+ year = {2026},
223
+ howpublished = {Hugging Face},
224
+ note = {A compact low-latency BERT Mini option-scoring model}
225
+ }
226
+ ```