Instructions to use AviSoori1x/Mistral-Nemo-Think with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AviSoori1x/Mistral-Nemo-Think with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="AviSoori1x/Mistral-Nemo-Think")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("AviSoori1x/Mistral-Nemo-Think") model = AutoModelForCausalLM.from_pretrained("AviSoori1x/Mistral-Nemo-Think", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AviSoori1x/Mistral-Nemo-Think with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AviSoori1x/Mistral-Nemo-Think" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AviSoori1x/Mistral-Nemo-Think", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/AviSoori1x/Mistral-Nemo-Think
- SGLang
How to use AviSoori1x/Mistral-Nemo-Think with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AviSoori1x/Mistral-Nemo-Think" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AviSoori1x/Mistral-Nemo-Think", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AviSoori1x/Mistral-Nemo-Think" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AviSoori1x/Mistral-Nemo-Think", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use AviSoori1x/Mistral-Nemo-Think with Docker Model Runner:
docker model run hf.co/AviSoori1x/Mistral-Nemo-Think
Download params.json from AviSoori1x/Mistral-Nemo-Think: direct link, hf CLI and curl.
- Browser
- Download file 863 Bytes
-
https://huggingface.co/AviSoori1x/Mistral-Nemo-Think/resolve/main/params.json
- Command line
-
hf download hf://AviSoori1x/Mistral-Nemo-Think/params.json
-
curl -L -o params.json https://huggingface.co/AviSoori1x/Mistral-Nemo-Think/resolve/main/params.json
863 Bytes
| {"dim": 5120, "n_layers": 40, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "use_biases": false, "causal": true, "rope_theta": 1000000.0, "norm_eps": 1e-05, "init": "DEFAULT", "dropout": 0.0, "vocab_size": 131072, "model_parallel": 1, "is_sequence_parallel": false, "context_parallel": 1, "model_pipelining": 1, "efficient_attn": true, "fused_rms_norm": true, "ragged_attention": null, "checkpoint": true, "use_cache": false, "max_concurrent_tokens": 65536, "rms_norm": "PRE", "cust_bwd": false, "recompute_w1_every": 0, "recompute_w3_every": 0, "recompute_attn_every": 0, "freeze_nonembedding": false, "zero2": false, "cutlass": false, "attn_tanh_gating": null, "softmax_tanh_gating": null, "deterministic_flash_attn": false, "moe": null, "mamba": null, "multimodal": null, "quantization": null, "layer_drop": null, "max_seq_len": 131072} |