Spaces:
Running on Zero
Running on Zero
| title: 'JevAlt: English, Turkish and German decision models' | |
| colorFrom: blue | |
| colorTo: gray | |
| sdk: gradio | |
| sdk_version: 6.29.0 | |
| app_file: app.py | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: Try small LLMs for text classification and local decisions | |
| models: | |
| - internlm/Intern-Decision-4B | |
| - mertkayacs/Deem-4B | |
| - mertkayacs/Karar-4B | |
| - mertkayacs/Wahler-4B | |
| datasets: | |
| - mertkayacs/jevalt-bench | |
| tags: | |
| - text-classification | |
| - small-language-model | |
| - local-llm | |
| - decision-model | |
| - calibration | |
| - turkish-llm | |
| - german-llm | |
| - jev | |
| language: | |
| - en | |
| - tr | |
| - de | |
| # JevAlt: try small English, Turkish and German decision models | |
| Try small language models for text classification and decisions with probabilities: Deem-4B in English, Karar-4B in Turkish and Wähler-4B in German. **Karar-4B scores 96.8% against Kev-4B's 87.1%** on held-out Turkish decisions from the training data pipeline. This result measures that decision task. | |
| ## Try a decision | |
| Choose a language and an example, then press **Decide**. Or enter a situation, a question and the possible answers. Each option returns a probability. Select **Allow “unknown”** when the deciding fact may be missing. | |
| [Open the decision app](https://mertkayacs-jevalt.hf.space). Use synthetic examples; the public Space receives the text you submit. Run locally for private data. | |
| ## Call from Python | |
| ```python | |
| import json | |
| from gradio_client import Client | |
| request = { | |
| "state": "I was charged twice for my subscription.", | |
| "questions": { | |
| "team": { | |
| "type": "choice", | |
| "instructions": "Which team should handle this ticket?", | |
| "criteria": {"billing": "payments, refunds", "technical": "bugs, outages"}, | |
| } | |
| }, | |
| "reasoning": "off", | |
| "abstain": False, | |
| } | |
| print(Client("mertkayacs/JevAlt").predict(json.dumps(request), "en", api_name="/decide")) | |
| ``` | |
| Use `tr` or `de` for Turkish or German, with the request written in that language. Install `gradio_client` for this example. | |
| ## Models and limits | |
| | Model | Language | Held-out accuracy | Kev-4B accuracy | | |
| |---|---|---|---| | |
| | [Deem-4B](https://huggingface.co/mertkayacs/Deem-4B) | English | 94.7% | 84.7% | | |
| | [Karar-4B](https://huggingface.co/mertkayacs/Karar-4B) | Turkish | 96.8% | 87.1% | | |
| | [Wähler-4B](https://huggingface.co/mertkayacs/Wahler-4B) | German | 92.0% | 81.1% | | |
| The tests share the training data pipeline. Planted instructions, long irrelevant text and date arithmetic remain failure cases. Refit calibration before using probabilities as thresholds, and review consequential decisions. [Recorded examples and test results](https://huggingface.co/datasets/mertkayacs/jevalt-bench). | |
| [Run on your computer](https://github.com/mertkayacs/jevalt/blob/main/docs/local.md) with Q4_K_M in about 3 GB RAM at a 4k context. [Play Emberwick](https://emberwick.mertkayacs.com) to see the models choose villagers' actions, or [watch the clips](https://huggingface.co/datasets/mertkayacs/emberwick-videos). | |
| Apache-2.0. [Code](https://github.com/mertkayacs/jevalt) | [Project](https://jevalt.mertkayacs.com). | |
| An [Eschatia Labs](https://eschatialabs.com) project. [Mert Kaya](https://mertkayacs.com). | |