Text Generation
GGUF
Japanese
japanese
instruction-tuning
little-language-model
tiny-language-model
edge-ai
embedded-ai
ex-word
llama-cpp
lm-studio
custom-code
conversational
Instructions to use ToTo-40417/EXLLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ToTo-40417/EXLLM with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: llama cli -hf ToTo-40417/EXLLM:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: llama cli -hf ToTo-40417/EXLLM:F16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: ./llama-cli -hf ToTo-40417/EXLLM:F16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf ToTo-40417/EXLLM:F16
Use Docker
docker model run hf.co/ToTo-40417/EXLLM:F16
- LM Studio
- Jan
- vLLM
How to use ToTo-40417/EXLLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ToTo-40417/EXLLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ToTo-40417/EXLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ToTo-40417/EXLLM:F16
- Ollama
How to use ToTo-40417/EXLLM with Ollama:
ollama run hf.co/ToTo-40417/EXLLM:F16
- Unsloth Desktop
- Docker Model Runner
How to use ToTo-40417/EXLLM with Docker Model Runner:
docker model run hf.co/ToTo-40417/EXLLM:F16
- Lemonade
How to use ToTo-40417/EXLLM with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ToTo-40417/EXLLM:F16
Run and chat with the model
lemonade run user.EXLLM-F16
List all available models
lemonade list
- Atomic Chat
File size: 2,360 Bytes
80300e5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | import re
from .tokenizer import normalize_text
from .infer import generate, bad_text
FALLBACK='ใใพใ็ญใใใใพใใใงใใใ่ณชๅใ็ญใ่จใๆใใฆใใ ใใใ'
TOO_LONG='่ณชๅใ้ทใใใพใใ็ญใๅใใฆๅ
ฅๅใใฆใใ ใใใ'
# Small deterministic tool path. This is algorithmic, not a table of fixed answers.
_CALC_PATTERNS=[
(re.compile(r'^\s*([+-]?\d{1,5})\s*([+\-*ร])\s*([+-]?\d{1,5})\s*(?:ใฏ|=)?\s*[?๏ผ]?\s*$'), None),
(re.compile(r'^\s*([+-]?\d{1,5})\s*(ใใ|ใฒใ|ใใใ)\s*([+-]?\d{1,5})\s*(?:ใจ|ใฏ)?\s*[?๏ผ]?\s*$'), None),
]
def try_calculate(prompt:str):
s=normalize_text(prompt)
for pat,_ in _CALC_PATTERNS:
m=pat.fullmatch(s)
if not m: continue
a=int(m.group(1)); op=m.group(2); b=int(m.group(3))
if op in ('+','ใใ'): r=a+b
elif op in ('-','ใฒใ'): r=a-b
elif op in ('*','ร','ใใใ'): r=a*b
else: return None
# keep device-side integer formatting simple and bounded
if not (-2147483648 <= r <= 2147483647): return '่จ็ฎ็ตๆใๆฑใใ็ฏๅฒใ่ถ
ใใฆใใพใใ'
return f'{r}ใงใใ'
return None
def _clean_output(text:str):
text=normalize_text(text)
if bad_text(text): return FALLBACK
try:
text.encode('utf-8','strict').decode('utf-8','strict')
except Exception:
return FALLBACK
# If a generation ran long and contains a complete Japanese sentence, keep the complete prefix.
if len(text)>72 and 'ใ' in text:
text=text[:text.rfind('ใ')+1]
# Long non-terminated fragments are safer as a fallback than as broken prose.
if len(text)>24 and text[-1] not in 'ใ๏ผ๏ผ?!':
cut=max(text.rfind('ใ'),text.rfind('๏ผ'),text.rfind('๏ผ'))
if cut>=6: text=text[:cut+1]
else: return FALLBACK
return text or FALLBACK
def answer(model,tok,prompt,max_new=64,temperature=0.0,top_k=8):
s=normalize_text(prompt)
if not s: return '่ณชๅใๅ
ฅๅใใฆใใ ใใใ'
# Unicode character count limit; byte-fallback may use more tokens internally.
if len(s)>160: return TOO_LONG
calc=try_calculate(s)
if calc is not None: return calc
text,_=generate(model,tok,s,max_new=max_new,temperature=temperature,top_k=top_k,confidence_fallback=True)
return _clean_output(text)
|