Tern-Decide-0.1
Tern-Decide-0.1 is the first, early version (0.1) of the Tern-Decide architecture from OkeyMeta Ltd: a small text decision and routing model that runs on a CPU in a few milliseconds per query.
- Trained from scratch by OkeyMeta. It is not a fine-tune. No pretrained model, no third-party embeddings and no external tokenizer were used. The tokenizer (Oji), the word and token embeddings, the pair encoder and every head were built by OkeyMeta.
- No GPU and no backpropagation. Every learned component was fitted on a 2-core CPU with closed-form or other gradient-free statistical methods. There is no gradient descent anywhere in training.
- CPU-only inference with
numpyandsafetensors. No deep-learning runtime. About 108 MB of weights. - Inference only. This repository contains inference code and weights. Training code is not released.
It is an early research release. Its intent heads are strong for their size and speed. Its zero-shot workflow decider is experimental and, on the one workflow benchmark we have, below a trivial baseline (details below).
What it does
| Call | What it does | Status |
|---|---|---|
classify(text, head="banking77") |
BANKING77 intent routing (77 banking intents) | Production-quality for that label set |
classify(text, head="clinc150") |
CLINC150 intent routing (150 intents + oos) |
Production-quality for that label set |
nli(premise, hypothesis) |
Entailment / neutral / contradiction | Research quality (MNLI 63.9) |
decide(state, question, options, kind) |
Zero-shot yes/no (Noul), Choice and Score questions over text or JSON state | Experimental: read the limitations first |
Quick start
git clone https://huggingface.co/OkeyMetaLtd/Tern-Decide-0.1 && cd Tern-Decide-0.1
pip install . # installs the tern-decide package (deps: numpy, safetensors)
python smoke_test.py # checks the install
from tern_decide import TernDecide
td = TernDecide.from_pretrained(".") # repo folder; components load lazily and thread-safely
r = td.classify("I still have not received my new card", head="banking77")
r.label, r.prob, r.top[:3] # 'card_arrival', ...
td.classify("what will the weather be tomorrow", head="clinc150", oos="threshold").label
n = td.nli("A man is playing a guitar on stage.", "A man is performing music.")
n.label, n.probabilities # 'entailment', {...}
state = {"messages": [{"from": "customer", "text": "My card was charged twice, please refund the duplicate."}]}
td.decide(state, "Is the customer asking for a refund?") # Noul -> Decision(label='true'|'false', ...)
td.decide(state, "Which team should handle this?", options={"billing": "billing team", "tech": "technical support"})
td.decide(state, "How urgent is this?", options=["low", "medium", "high"], kind="score").expected
# batch APIs
td.classify_batch(["top up failed", "card declined"], head="banking77")
td.nli_batch([("p1", "h1"), ("p2", "h2")])
td.decide_batch(["Is this a refund request?", {"question": "How urgent?", "options": ["low", "high"], "kind": "score"}], state=state)
Every call returns a typed, immutable result with .label, .probabilities (sums to 1), .kind and .prob.
Decision adds .expected for Score questions; Classification adds .head and .top. Invalid input raises
TernDecideError with a clear message. Outputs are deterministic. TernDecide.from_pretrained("OkeyMetaLtd/Tern-Decide-0.1")
also works if huggingface_hub is installed (pip install ".[hub]").
Run the tests with pytest tests or python tests/test_inference.py, and the latency benchmark with
OPENBLAS_NUM_THREADS=1 python benchmark_latency.py.
Benchmarks
Setup. aarch64 CPU, 2 cores, about 8 GB RAM. Inference is single-threaded (OPENBLAS_NUM_THREADS=1), batch size 1,
raw text in and result out. Every Tern number below was re-measured with this package through its public API and matches
the research predictions exactly. [matched] means same splits and same machine. Published baselines come from the cited
papers; their hardware differs, so cross-paper latency is indicative only. Hyperparameters were chosen on dev splits and
each test set was scored once.
Intent routing (supervised, full training data)
| Model | BANKING77 acc | CLINC150 in-scope acc | CLINC OOS recall | p50 / p99 latency, 1 query |
|---|---|---|---|---|
| Tern-Decide-0.1 | 92.01 | 93.27 | 21.9 (oos="train"); 55.3 at 92.98 in-scope (oos="threshold") |
BANKING 2.3 / 4.8 ms; CLINC 3.1 / 4.5 ms |
| TF-IDF + logistic regression [matched] | 91.53 | 92.42 | 24.9 | 1.0 / 1.4 ms |
| FastText / SVM (Larson et al. 2019) | – | 89.0 / 91.0 | 9.7 / 14.5 | n/p |
| USE / ConveRT encoders + MLP (Casanueva et al. 2020) | 92.81 / 93.01 | 93.5 (USE + MLP, Larson et al.) | 47.4 | ≈17–19 ms per sentence (encoder, 2-core CPU) |
| BERT-large / RoBERTa-base fine-tuned | 93.66 / 94.1 | 96.9 (BERT, Larson et al.) | 40.3 | GPU fine-tuning |
- Tern-Decide beats the matched TF-IDF + LR baseline on accuracy (+0.48 BANKING77, +0.85 CLINC in-scope) but is slower per query, and its OOS recall in the default mode is lower.
- It trails fine-tuned pretrained encoders by about 1.5–4 points.
- Calibration (expected calibration error, temperature fitted on dev): BANKING77 0.035, CLINC150 0.031.
- The CLINC threshold row (92.98 / 55.3) was measured with the research code; the package exposes the same rule.
Natural-language inference
| Model | MNLI-m dev | MNLI-mm dev | SNLI test | p50 / p99 per pair |
|---|---|---|---|---|
| majority class | 35.4 | – | – | – |
Tern-Decide-0.1 nli (trained on SNLI + MNLI + other NLI/QA, no gradients) |
63.94 | 64.40 | 75.51 | 1.6 / 4.5 ms |
| CBOW / BiLSTM / ESIM trained on MNLI only (Williams et al. 2018) | 64.8 / 66.9 / 72.3 | 64.5 / 66.9 / 72.1 | 51.5 / 50.8 / 60.7 | – |
| CBOW / BiLSTM / ESIM trained on MNLI + SNLI (Williams et al. 2018) | 65.2 / 67.5 / 72.4 | 64.6 / 67.1 / 71.9 | 74.7 / 74.0 / 79.7 | – |
- On MNLI, Tern is below all three published backprop-trained baselines (about 1 point below CBOW, 3.5 below BiLSTM, 8.5 below ESIM).
- On SNLI test it is above CBOW and BiLSTM trained on MNLI + SNLI and below ESIM. Its training mix differs from theirs (it also includes ANLI, QNLI, SciTail, RTE, CB, BoolQ, MultiRC, RACE and SciQ), so this is not a fully matched comparison.
- It is at chance on adversarial NLI (ANLI round 3: 33.9).
nli()probabilities use a temperature fitted on MNLI-mismatched dev.
Zero-shot routing with the decider (no in-domain labels)
| Method | CLINC150 in-scope | BANKING77 | AG News | SST-2 (val) |
|---|---|---|---|---|
| chance | 0.7 | 1.3 | 25.0 | 50.0 |
Tern-Decide-0.1 decide (Choice), full test sets |
47.7 | 39.6 | 60.2 | 56.3 |
| 41 open LLMs ≤ 9B, zero-shot, first 500 test items: mean / best (Ganesh et al. 2026) | 46.8 / 83.6 | 38.8 / 70.8 | – | – |
| GPT-3 175B zero-shot / with contextual calibration (Zhao et al. 2021) | – | – | 43.9 / 73.9 | 71.6 / 75.8 |
These comparisons are approximate (different test subsets and prompt formats). None of these datasets was used to train or select the decider. SST-2 is near chance.
Zero-shot workflow decisions vs TypeSafe Jev (honest comparison)
TypeSafe publishes 20 of the 711 cases in its workflow benchmark (customer service, invoice processing, security incidents, agent traces), with every question, the reference probabilities and Jev's answers. We built a matched per-question harness on those 354 public questions. The metric is agreement with the argmax of the reference probabilities. Every Tern row was scored once; nothing was fitted or selected on these questions.
| Method | all (354) | Noul (221) | Choice (106) | Score (27) | Noul Brier | Latency |
|---|---|---|---|---|---|---|
| TypeSafe Jev (published answers) | 88.1 | 92.8 | 82.1 | 74.1 | 0.028 | ≈0.44 s mean per case (network API) |
| trivial constant: Noul = false, Choice = first option, Score = lowest | 70.1 | 82.4 | 56.6 | 22.2 | 0.149 | – |
| uniform random (expected) | 39.1 | 50.0 | 20.6 | 23.2 | – | – |
| Tern-Decide-0.1 decider | 64.7 | 80.5 | 43.4 | 18.5 | 0.116 | 1.2 ms p50 per question; 13.5 ms p50 per case (local CPU) |
On accuracy, Jev leads clearly: 88.1% vs 64.7%. Tern-Decide is also below the 70.1% trivial constant baseline on this harness. Its Noul (yes/no) answers are close to an "almost always false" prior, Choice is below the first-option constant, and Score is below uniform. Per workflow, Tern scores 64.1 (customer service), 72.8 (invoice), 46.2 (security) and 46.2 (agent trace). This is 20 of 711 cases, so it is a small sample.
Where Tern-Decide beats Jev
Only these claims, and only as stated:
- Latency. About 1.2 ms per question (p50) and 13.5 ms per whole case (p50) on one local CPU thread, versus Jev's published ≈0.44 s mean per case over its API (0.18–1.02 s). One is local inference and the other is a network API, so this is an approximate comparison, not a like-for-like benchmark of the two models.
- Deployment. Runs offline and on-device, with no API cost and no data leaving the machine.
- Footprint. About 108 MB of weights,
numpy+safetensorsonly, no GPU.
Tern-Decide does not beat Jev on accuracy or calibration.
Files and sizes
| File | Size | Contents |
|---|---|---|
weights/intent_clinc150.safetensors |
45.8 MB | CLINC150 kernel head: hashed training rows, dual coefficients, sentence vectors |
weights/embeddings.safetensors |
37.6 MB | Tern word vectors (24k × 300), Oji token vectors (8,192 × 256), OOV map |
weights/intent_banking77.safetensors |
20.6 MB | BANKING77 kernel head |
weights/nli_encoder.safetensors |
1.9 MB | pair encoder: whitening, ternary layer (2-bit packed), NLI head, composed units |
weights/oji_tokenizer.safetensors |
1.2 MB | Oji v1 tokenizer (8,192 vocabulary) |
weights/decider.safetensors |
0.16 MB | decider weights (39 features × 3 kinds) and valence lexicon |
weights/word_vocab.json, weights/valence_vocab.json |
0.5 MB | vocabularies |
tern_decide/ |
≈0.07 MB | inference code (pure numpy) |
tests/, smoke_test.py, benchmark_latency.py |
small | inference tests, smoke test, latency benchmark |
TECHNICAL_REPORT.pdf / .md |
technical report |
The intent heads are kernel models and store their hashed training rows, which is why they are the largest files. Post-hoc int8 / ternary quantization of their coefficients lost too much accuracy (CLINC int8: 70.2%), so they ship in fp32.
Intended use
- Fast, private, offline intent routing for the BANKING77 and CLINC150 label sets.
- A cheap CPU NLI signal.
- Research on gradient-free, CPU-only language models.
Out-of-scope uses
- Commercial use (see the licence).
- Automated yes/no judgements about people, money, security, compliance, health, employment or legal matters. The decider is not reliable enough for these.
- Languages other than English. The tokenizer handles many scripts, but every head was trained on English data.
- New intent label sets. The intent heads are fixed; refitting is not part of this release.
Limitations
- Decider: below a trivial constant baseline on the only workflow benchmark we have; Noul behaves like a calibrated "probably false" prior; on held-out public flag tasks its evidence features rank true above false at about chance (AUROC ≈ 0.4–0.6); Score is near uniform. It reads topical similarity, not predicates (who did what, amounts, tense, cross-field consistency).
- Intent heads: fixed label sets; low OOS recall in the default CLINC mode (21.9).
- NLI: below CBOW, BiLSTM and ESIM on MNLI; chance on ANLI; no gain over majority on BoolQ-style passage QA.
- Latencies were measured on one 2-core aarch64 machine; yours will differ.
- Training data comes from public datasets and the web and carries their biases.
Licence
The weights, inference code and documentation are released by OkeyMeta Ltd under CC BY-NC 4.0 (attribution, non-commercial). Training code and training methods are not released.
Training and evaluation data
Tern-Decide was trained on public datasets. Each keeps its own licence; using this model does not grant any right to the
datasets. Several sources are non-commercial or research-only (ANLI, RACE, SciQ, DREAM, customer-support-tickets,
dair-ai/emotion), and several are share-alike (SNLI, BoolQ, QNLI/SQuAD, ARC, DROP, DBpedia-14, MultiNLI parts,
Bitext). The non-commercial licence of this model is consistent with the most restrictive of these. See
TECHNICAL_REPORT.md (Data licences) for the full table, including sources whose terms are restrictive or unclear.
| Component | Datasets (licence) |
|---|---|
| Intent heads | BANKING77 (CC BY 4.0), CLINC150 / oos-eval (CC BY 3.0) |
| Tokenizer and embeddings | FineWeb sample (ODC-By 1.0; Common Crawl terms of use) |
| NLI encoder | SNLI (CC BY-SA 4.0), MultiNLI (OANC / CC BY 3.0 / CC BY-SA 3.0 / MIT / other), ANLI (CC BY-NC 4.0), QNLI (from SQuAD, CC BY-SA 4.0), RTE, CB and MultiRC (no explicit licence; distributed for research via GLUE / SuperGLUE), SciTail (AI2), BoolQ (CC BY-SA 3.0), RACE (non-commercial research only), SciQ (CC BY-NC 3.0) |
| Decider | see the report: 30+ public tasks including Yelp reviews (Yelp Dataset Terms of Use), civil_comments (CC0), GoEmotions (Apache 2.0), TweetEval, Yahoo Answers Topics, DBpedia-14, ARC, CommonsenseQA, OpenBookQA, customer-support-tickets, HWU64, SNIPS, ATIS, Bitext, TabFact, DREAM, DROP, SMS Spam, STS-B, dair-ai/emotion |
| Evaluation only | TypeSafe public example cases (evals.typesafe.ai), AG News, SST-2 |
Acknowledgements & dataset credits
Tern-Decide-0.1 exists because of the researchers, companies and annotators who built and shared the datasets below. We thank all of them. Every dataset used to train or evaluate any component is listed here with its authors or provider, the licence or terms it is published under (as checked on 16 October 2026), and a source link. Each dataset keeps its own licence; using this model does not grant any right to the datasets themselves.
Please note. Some components were trained on data released under share-alike, non-commercial / research-only
or provider-specific terms (for example the Yelp Dataset Terms of Use, Twitter terms, and the LDC user agreement).
The decider component's training included Yelp review data (Yelp Open Dataset, used under the Yelp Dataset Terms of
Use): the decider's Score task and its valence lexicon (weights/decider.safetensors, weights/valence_vocab.json) are
derived from Yelp reviews, and the decider's Noul task used the TweetEval hate subset (HatEval). Users of this model are
responsible for complying with the terms of these datasets, in addition to this model's CC BY-NC 4.0 licence.
Training data
| Dataset | Used for | Authors / provider (citation) | Licence / terms | Source |
|---|---|---|---|---|
| BANKING77 | intent head | Casanueva et al., 2020, PolyAI | CC BY 4.0 | https://huggingface.co/datasets/PolyAI/banking77 |
| CLINC150 (oos-eval) | intent head | Larson et al., 2019, Clinc | CC BY 3.0 | https://github.com/clinc/oos-eval |
| FineWeb (sample) | tokenizer, embeddings | Penedo et al., 2024, Hugging Face | ODC-By 1.0; Common Crawl terms of use | https://huggingface.co/datasets/HuggingFaceFW/fineweb |
| SNLI | NLI, decider | Bowman et al., 2015, Stanford NLP | CC BY-SA 4.0 | https://huggingface.co/datasets/stanfordnlp/snli |
| MultiNLI | NLI, decider | Williams, Nangia & Bowman, 2018, NYU | OANC / CC BY 3.0 / CC BY-SA 3.0 / MIT / other (per genre) | https://cims.nyu.edu/~sbowman/multinli/ |
| ANLI | NLI, decider | Nie et al., 2020, Meta AI / UNC | CC BY-NC 4.0 | https://huggingface.co/datasets/facebook/anli |
| QNLI (from SQuAD) | NLI | Wang et al., 2019 (GLUE); Rajpurkar et al., 2016 (SQuAD) | CC BY-SA 4.0 | https://huggingface.co/datasets/nyu-mll/glue |
| RTE | NLI | Dagan et al., 2006 and later RTE challenges; via GLUE / SuperGLUE (Wang et al., 2019) | no explicit licence; distributed for research via GLUE / SuperGLUE | https://huggingface.co/datasets/aps/super_glue |
| CommitmentBank (CB) | NLI | de Marneffe, Simons & Tonhauser, 2019 | no explicit licence; research use via SuperGLUE | https://github.com/mcdm/CommitmentBank |
| MultiRC | NLI | Khashabi et al., 2018, CogComp | no explicit licence; research use via SuperGLUE | https://cogcomp.seas.upenn.edu/multirc/ |
| SciTail | NLI | Khot, Sabharwal & Clark, 2018, AI2 | not stated on the Hugging Face card | https://huggingface.co/datasets/allenai/scitail |
| BoolQ | NLI, decider | Clark et al., 2019, Google | CC BY-SA 3.0 | https://huggingface.co/datasets/google/boolq |
| RACE | NLI, decider | Lai et al., 2017, CMU | non-commercial research only (CMU terms) | https://huggingface.co/datasets/ehovy/race |
| SciQ | NLI | Welbl, Liu & Gardner, 2017, AI2 | CC BY-NC 3.0 | https://huggingface.co/datasets/allenai/sciq |
| Yelp Open Dataset (Yelp review stars) | decider (Score task, valence lexicon) | Yelp Inc.; review-full task by Zhang, Zhao & LeCun, 2015 | Yelp Dataset Terms of Use (version of 21 Feb 2020); used under those terms | https://www.yelp.com/dataset ; https://huggingface.co/datasets/Yelp/yelp_review_full |
| Civil Comments | decider | Borkan et al., 2019, Jigsaw / Civil Comments | CC0 1.0 | https://huggingface.co/datasets/google/civil_comments |
| STS-B | decider | Cer et al., 2017 | mixed; GLUE refers to the original licences | https://huggingface.co/datasets/sentence-transformers/stsb |
| DBpedia-14 | decider | Zhang, Zhao & LeCun, 2015; Lehmann et al., 2015 (DBpedia) | CC BY-SA 3.0 | https://huggingface.co/datasets/fancyzhx/dbpedia_14 |
| Yahoo Answers Topics | decider | Zhang, Zhao & LeCun, 2015; from Yahoo! Answers (Yahoo! Webscope) | "unknown" on the card; Yahoo! Webscope terms | https://huggingface.co/datasets/community-datasets/yahoo_answers_topics |
| Emotion (dair-ai) | decider | Saravia et al., 2018 | educational and research purposes only | https://huggingface.co/datasets/dair-ai/emotion |
| TweetEval (emotion, irony) | decider | Barbieri et al., 2020; Mohammad et al., 2018 (SemEval-2018 Task 1); Van Hee, Lefever & Hoste, 2018 (SemEval-2018 Task 3) | Twitter terms of service; licence undefined | https://huggingface.co/datasets/cardiffnlp/tweet_eval |
| TweetEval hate (HatEval) | decider (Noul task) | Basile et al., 2019 (SemEval-2019 Task 5, HatEval); Barbieri et al., 2020 (TweetEval) | "need permission" on the TweetEval card; Twitter terms of service | https://huggingface.co/datasets/cardiffnlp/tweet_eval ; https://github.com/cardiffnlp/tweeteval |
| GoEmotions | decider | Demszky et al., 2020, Google Research | Apache 2.0 | https://huggingface.co/datasets/google-research-datasets/go_emotions |
| SMS Spam Collection | decider | Almeida, Gómez Hidalgo & Yamakami, 2011 | CC BY 4.0 | https://archive.ics.uci.edu/dataset/228/sms+spam+collection |
| ARC | decider | Clark et al., 2018, AI2 | CC BY-SA 4.0 | https://huggingface.co/datasets/allenai/ai2_arc |
| CommonsenseQA | decider | Talmor et al., 2019 | MIT | https://huggingface.co/datasets/tau/commonsense_qa |
| OpenBookQA | decider | Mihaylov et al., 2018, AI2 | Apache 2.0 (AI2 repository) | https://github.com/allenai/OpenBookQA |
| customer-support-tickets | decider | Tobias Bueck (Hugging Face user Tobi-Bueck) | CC BY-NC 4.0 | https://huggingface.co/datasets/Tobi-Bueck/customer-support-tickets |
| HWU64 | decider | Liu et al., 2019 | CC BY 4.0 | https://github.com/xliuhw/NLU-Evaluation-Data |
| SNIPS | decider | Coucke et al., 2018, Snips | Apache 2.0 (Hugging Face copy) | https://huggingface.co/datasets/benayas/snips |
| ATIS | decider | Hemphill, Godfrey & Doddington, 1990; Linguistic Data Consortium (LDC93S5) | LDC User Agreement for Non-Members; the Hugging Face copy states no licence | https://catalog.ldc.upenn.edu/LDC93S5 ; https://huggingface.co/datasets/tuetschek/atis |
| Bitext customer support | decider | Bitext Innovations | CDLA-Sharing 1.0 | https://huggingface.co/bitext |
| TabFact | decider | Chen et al., 2020 | CC BY 4.0 | https://huggingface.co/datasets/wenhu/tab_fact |
| DREAM | decider | Sun et al., 2019 | non-commercial research purposes only | https://github.com/nlpdata/dream |
| DROP | decider | Dua et al., 2019, AI2 | CC BY-SA 4.0 | https://huggingface.co/datasets/ucinlp/drop |
Evaluation-only data (not used for training)
| Dataset | Authors / provider (citation) | Source |
|---|---|---|
| AG News | Zhang, Zhao & LeCun, 2015 | https://huggingface.co/datasets/fancyzhx/ag_news |
| SST-2 | Socher et al., 2013; via GLUE (Wang et al., 2019) | https://huggingface.co/datasets/nyu-mll/glue |
| TypeSafe public workflow cases (Jev harness) | TypeSafe | https://evals.typesafe.ai/ |
Dataset citations
- Almeida, T. A., Gómez Hidalgo, J. M., Yamakami, A. (2011). Contributions to the study of SMS spam filtering. DocEng.
- Barbieri, F., Camacho-Collados, J., Espinosa Anke, L., Neves, L. (2020). TweetEval: Unified benchmark and comparative evaluation for tweet classification. Findings of EMNLP.
- Basile, V., et al. (2019). SemEval-2019 Task 5: Multilingual detection of hate speech against immigrants and women in Twitter (HatEval). SemEval.
- Borkan, D., et al. (2019). Nuanced metrics for measuring unintended bias with real data for text classification. WWW Companion.
- Bowman, S. R., et al. (2015). A large annotated corpus for learning natural language inference. EMNLP.
- Casanueva, I., et al. (2020). Efficient intent detection with dual sentence encoders. NLP4ConvAI.
- Cer, D., et al. (2017). SemEval-2017 Task 1: Semantic textual similarity. SemEval.
- Chen, W., et al. (2020). TabFact: A large-scale dataset for table-based fact verification. ICLR.
- Clark, C., et al. (2019). BoolQ: Exploring the surprising difficulty of natural yes/no questions. NAACL.
- Clark, P., et al. (2018). Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457.
- Coucke, A., et al. (2018). Snips Voice Platform: an embedded spoken language understanding system. arXiv:1805.10190.
- Dagan, I., Glickman, O., Magnini, B. (2006). The PASCAL recognising textual entailment challenge. MLCW.
- de Marneffe, M.-C., Simons, M., Tonhauser, J. (2019). The CommitmentBank. Sinn und Bedeutung 23.
- Demszky, D., et al. (2020). GoEmotions: A dataset of fine-grained emotions. ACL.
- Dua, D., et al. (2019). DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. NAACL.
- Hemphill, C. T., Godfrey, J. J., Doddington, G. R. (1990). The ATIS spoken language systems pilot corpus. DARPA Speech and Natural Language Workshop.
- Khashabi, D., et al. (2018). Looking beyond the surface: A challenge set for reading comprehension over multiple sentences (MultiRC). NAACL.
- Khot, T., Sabharwal, A., Clark, P. (2018). SciTail: A textual entailment dataset from science question answering. AAAI.
- Lai, G., et al. (2017). RACE: Large-scale reading comprehension dataset from examinations. EMNLP.
- Larson, S., et al. (2019). An evaluation dataset for intent classification and out-of-scope prediction. EMNLP.
- Lehmann, J., et al. (2015). DBpedia: A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web Journal.
- Liu, X., Eshghi, A., Swietojanski, P., Rieser, V. (2019). Benchmarking natural language understanding services for building conversational agents. IWSDS.
- Mihaylov, T., et al. (2018). Can a suit of armor conduct electricity? A new dataset for open book question answering. EMNLP.
- Mohammad, S., et al. (2018). SemEval-2018 Task 1: Affect in tweets. SemEval.
- Nie, Y., et al. (2020). Adversarial NLI: A new benchmark for natural language understanding. ACL.
- Penedo, G., et al. (2024). The FineWeb datasets: Decanting the web for the finest text data at scale. NeurIPS Datasets and Benchmarks.
- Rajpurkar, P., et al. (2016). SQuAD: 100,000+ questions for machine comprehension of text. EMNLP.
- Saravia, E., et al. (2018). CARER: Contextualized affect representations for emotion recognition. EMNLP.
- Socher, R., et al. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP.
- Sun, K., et al. (2019). DREAM: A challenge dataset and models for dialogue-based reading comprehension. TACL.
- Talmor, A., et al. (2019). CommonsenseQA: A question answering challenge targeting commonsense knowledge. NAACL.
- Van Hee, C., Lefever, E., Hoste, V. (2018). SemEval-2018 Task 3: Irony detection in English tweets. SemEval.
- Wang, A., et al. (2019). GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR.
- Wang, A., et al. (2019). SuperGLUE: A stickier benchmark for general-purpose language understanding systems. NeurIPS.
- Welbl, J., Liu, N. F., Gardner, M. (2017). Crowdsourcing multiple choice science questions (SciQ). W-NUT.
- Williams, A., Nangia, N., Bowman, S. R. (2018). A broad-coverage challenge corpus for sentence understanding through inference. NAACL.
- Yelp Inc. Yelp Open Dataset. https://www.yelp.com/dataset (used under the Yelp Dataset Terms of Use).
- Zhang, X., Zhao, J., LeCun, Y. (2015). Character-level convolutional networks for text classification. NeurIPS.
Citation
@misc{okeymeta2026terndecide,
title = {Tern-Decide-0.1: a CPU-only, gradient-free decision and routing model trained from scratch},
author = {{OkeyMeta Ltd}},
year = {2026},
howpublished = {\url{https://huggingface.co/OkeyMetaLtd/Tern-Decide-0.1}},
note = {Version 0.1.0}
}
References: Larson et al. 2019 (CLINC150); Casanueva et al. 2020 (BANKING77); Williams et al. 2018 (MultiNLI); Bowman et al. 2015 (SNLI); Nie et al. 2020 (ANLI); Parikh et al. 2016 and Chen et al. 2017 (attention-based NLI); Arora et al. 2017 (SIF); Zhao et al. 2021 (calibration); TypeSafe evals (evals.typesafe.ai).
- Downloads last month
- 10
Datasets used to train OkeyMetaLtd/Tern-Decide-0.1
stanfordnlp/snli
nyu-mll/multi_nli
Papers for OkeyMetaLtd/Tern-Decide-0.1
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Evaluation results
- accuracy on BANKING77test set self-reported92.010
- accuracy on CLINC150 (in-scope)test set self-reported93.270
- accuracy on MNLI matchedself-reported63.940
- accuracy on SNLItest set self-reported75.510