Text Generation
GGUF
Japanese
japanese
instruction-tuning
little-language-model
tiny-language-model
edge-ai
embedded-ai
ex-word
llama-cpp
lm-studio
custom-code
conversational
Instructions to use ToTo-40417/EXLLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ToTo-40417/EXLLM with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: llama cli -hf ToTo-40417/EXLLM:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: llama cli -hf ToTo-40417/EXLLM:F16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: ./llama-cli -hf ToTo-40417/EXLLM:F16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf ToTo-40417/EXLLM:F16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf ToTo-40417/EXLLM:F16
Use Docker
docker model run hf.co/ToTo-40417/EXLLM:F16
- LM Studio
- Jan
- vLLM
How to use ToTo-40417/EXLLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ToTo-40417/EXLLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ToTo-40417/EXLLM", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ToTo-40417/EXLLM:F16
- Ollama
How to use ToTo-40417/EXLLM with Ollama:
ollama run hf.co/ToTo-40417/EXLLM:F16
- Unsloth Desktop
- Docker Model Runner
How to use ToTo-40417/EXLLM with Docker Model Runner:
docker model run hf.co/ToTo-40417/EXLLM:F16
- Lemonade
How to use ToTo-40417/EXLLM with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ToTo-40417/EXLLM:F16
Run and chat with the model
lemonade run user.EXLLM-F16
List all available models
lemonade list
- Atomic Chat
| import re | |
| from .tokenizer import normalize_text | |
| from .infer import generate, bad_text | |
| FALLBACK='ใใพใ็ญใใใใพใใใงใใใ่ณชๅใ็ญใ่จใๆใใฆใใ ใใใ' | |
| TOO_LONG='่ณชๅใ้ทใใใพใใ็ญใๅใใฆๅ ฅๅใใฆใใ ใใใ' | |
| # Small deterministic tool path. This is algorithmic, not a table of fixed answers. | |
| _CALC_PATTERNS=[ | |
| (re.compile(r'^\s*([+-]?\d{1,5})\s*([+\-*ร])\s*([+-]?\d{1,5})\s*(?:ใฏ|=)?\s*[?๏ผ]?\s*$'), None), | |
| (re.compile(r'^\s*([+-]?\d{1,5})\s*(ใใ|ใฒใ|ใใใ)\s*([+-]?\d{1,5})\s*(?:ใจ|ใฏ)?\s*[?๏ผ]?\s*$'), None), | |
| ] | |
| def try_calculate(prompt:str): | |
| s=normalize_text(prompt) | |
| for pat,_ in _CALC_PATTERNS: | |
| m=pat.fullmatch(s) | |
| if not m: continue | |
| a=int(m.group(1)); op=m.group(2); b=int(m.group(3)) | |
| if op in ('+','ใใ'): r=a+b | |
| elif op in ('-','ใฒใ'): r=a-b | |
| elif op in ('*','ร','ใใใ'): r=a*b | |
| else: return None | |
| # keep device-side integer formatting simple and bounded | |
| if not (-2147483648 <= r <= 2147483647): return '่จ็ฎ็ตๆใๆฑใใ็ฏๅฒใ่ถ ใใฆใใพใใ' | |
| return f'{r}ใงใใ' | |
| return None | |
| def _clean_output(text:str): | |
| text=normalize_text(text) | |
| if bad_text(text): return FALLBACK | |
| try: | |
| text.encode('utf-8','strict').decode('utf-8','strict') | |
| except Exception: | |
| return FALLBACK | |
| # If a generation ran long and contains a complete Japanese sentence, keep the complete prefix. | |
| if len(text)>72 and 'ใ' in text: | |
| text=text[:text.rfind('ใ')+1] | |
| # Long non-terminated fragments are safer as a fallback than as broken prose. | |
| if len(text)>24 and text[-1] not in 'ใ๏ผ๏ผ?!': | |
| cut=max(text.rfind('ใ'),text.rfind('๏ผ'),text.rfind('๏ผ')) | |
| if cut>=6: text=text[:cut+1] | |
| else: return FALLBACK | |
| return text or FALLBACK | |
| def answer(model,tok,prompt,max_new=64,temperature=0.0,top_k=8): | |
| s=normalize_text(prompt) | |
| if not s: return '่ณชๅใๅ ฅๅใใฆใใ ใใใ' | |
| # Unicode character count limit; byte-fallback may use more tokens internally. | |
| if len(s)>160: return TOO_LONG | |
| calc=try_calculate(s) | |
| if calc is not None: return calc | |
| text,_=generate(model,tok,s,max_new=max_new,temperature=temperature,top_k=top_k,confidence_fallback=True) | |
| return _clean_output(text) | |