Instructions to use Lucasgitguy-co/wiki-model-tiny-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Lucasgitguy-co/wiki-model-tiny-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Lucasgitguy-co/wiki-model-tiny-GGUF:F16 # Run inference directly in the terminal: llama cli -hf Lucasgitguy-co/wiki-model-tiny-GGUF:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Lucasgitguy-co/wiki-model-tiny-GGUF:F16 # Run inference directly in the terminal: llama cli -hf Lucasgitguy-co/wiki-model-tiny-GGUF:F16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Lucasgitguy-co/wiki-model-tiny-GGUF:F16 # Run inference directly in the terminal: ./build/bin/llama cli -hf Lucasgitguy-co/wiki-model-tiny-GGUF:F16
Use Docker
docker model run hf.co/Lucasgitguy-co/wiki-model-tiny-GGUF:F16
- Llama new
- LM Studio
- Jan
- vLLM
How to use Lucasgitguy-co/wiki-model-tiny-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Lucasgitguy-co/wiki-model-tiny-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Lucasgitguy-co/wiki-model-tiny-GGUF", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Lucasgitguy-co/wiki-model-tiny-GGUF:F16
- Ollama
How to use Lucasgitguy-co/wiki-model-tiny-GGUF with Ollama:
ollama run hf.co/Lucasgitguy-co/wiki-model-tiny-GGUF:F16
- Unsloth Desktop
- Docker Model Runner
How to use Lucasgitguy-co/wiki-model-tiny-GGUF with Docker Model Runner:
docker model run hf.co/Lucasgitguy-co/wiki-model-tiny-GGUF:F16
- Lemonade
How to use Lucasgitguy-co/wiki-model-tiny-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Lucasgitguy-co/wiki-model-tiny-GGUF:F16
Run and chat with the model
lemonade run user.wiki-model-tiny-GGUF-F16
List all available models
lemonade list
- Atomic Chat
Download convert_tiny.py from Lucasgitguy-co/wiki-model-tiny-GGUF: direct link, hf CLI and curl.
- Browser
- Download file 3.47 kB
-
https://huggingface.co/Lucasgitguy-co/wiki-model-tiny-GGUF/resolve/main/convert_tiny.py
- Command line
-
hf download hf://Lucasgitguy-co/wiki-model-tiny-GGUF/convert_tiny.py
-
curl -L -o convert_tiny.py https://huggingface.co/Lucasgitguy-co/wiki-model-tiny-GGUF/resolve/main/convert_tiny.py
3.47 kB
| """Write wiki-model-tiny (TinyGPT) as a llama-arch GGUF directly with gguf-py. | |
| TinyGPT already uses interleaved (GPT-J style) RoPE pairs, which is llama.cpp's LLAMA rope, so q/k are NOT permuted. | |
| usage: python convert_tiny.py MODEL_DIR OUT.gguf [f32|f16]""" | |
| import json, sys | |
| import numpy as np | |
| from safetensors.numpy import load_file | |
| sys.path.insert(0, sys.argv[4] if len(sys.argv) > 4 else "/Users/lucasvandeweerd/.cache/spot-snake/llama.cpp/gguf-py") | |
| import gguf | |
| mdir, out = sys.argv[1], sys.argv[2] | |
| ft = sys.argv[3] if len(sys.argv) > 3 else "f16" | |
| c = json.load(open(f"{mdir}/config.json")) | |
| sd = load_file(f"{mdir}/model.safetensors") | |
| tj = json.load(open(f"{mdir}/tokenizer.json")) | |
| D, H, L, F = c["dim"], c["heads"], c["layers"], c["hidden"] | |
| w = gguf.GGUFWriter(out, "llama") | |
| w.add_name("wiki-model-tiny") | |
| w.add_context_length(c["ctx"]); w.add_embedding_length(D); w.add_block_count(L) | |
| w.add_feed_forward_length(F); w.add_head_count(H); w.add_head_count_kv(H) | |
| w.add_layer_norm_rms_eps(1e-5); w.add_rope_freq_base(c["rope_base"]); w.add_rope_dimension_count(D // H) | |
| w.add_vocab_size(c["vocab"]) | |
| w.add_file_type(gguf.LlamaFileType.MOSTLY_F16 if ft == "f16" else gguf.LlamaFileType.ALL_F32) | |
| # tokenizer: byte-level BPE with the GPT-2 regex == llama.cpp "gpt2" model + "gpt-2" pre-tokenizer | |
| vocab = tj["model"]["vocab"] | |
| added = {a["id"]: a for a in tj["added_tokens"]} | |
| tokens = [None] * c["vocab"] | |
| for t, i in vocab.items(): tokens[i] = t | |
| for i, a in added.items(): tokens[i] = a["content"] | |
| assert all(t is not None for t in tokens), "vocab has holes" | |
| types = [] | |
| for i in range(len(tokens)): | |
| if i in added: | |
| types.append(gguf.TokenType.CONTROL if added[i]["content"] in ("<pad>", "<eos>") else gguf.TokenType.USER_DEFINED) | |
| else: | |
| types.append(gguf.TokenType.NORMAL) | |
| merges = [m if isinstance(m, str) else " ".join(m) for m in tj["model"]["merges"]] | |
| eos = tj["model"]["vocab"].get("<eos>", next(i for i, a in added.items() if a["content"] == "<eos>")) | |
| pad = next(i for i, a in added.items() if a["content"] == "<pad>") | |
| w.add_tokenizer_model("gpt2"); w.add_tokenizer_pre("gpt-2") | |
| w.add_token_list(tokens); w.add_token_types(types); w.add_token_merges(merges) | |
| w.add_bos_token_id(eos); w.add_eos_token_id(eos); w.add_pad_token_id(pad) | |
| w.add_add_bos_token(False); w.add_add_eos_token(False) | |
| def put(name, a, norm=False): | |
| a = np.ascontiguousarray(a, dtype=np.float32 if (norm or ft == "f32") else np.float16) | |
| w.add_tensor(name, a) | |
| put("token_embd.weight", sd["emb.weight"]) | |
| for i in range(L): | |
| p = f"blocks.{i}." | |
| q, k, v = np.split(sd[p + "qkv.weight"], 3, axis=0) | |
| g, u = np.split(sd[p + "gate_up.weight"], 2, axis=0) # chunk(2): gate first, then up | |
| put(f"blk.{i}.attn_norm.weight", sd[p + "n1.w"], norm=True) | |
| put(f"blk.{i}.attn_q.weight", q); put(f"blk.{i}.attn_k.weight", k); put(f"blk.{i}.attn_v.weight", v) | |
| put(f"blk.{i}.attn_output.weight", sd[p + "o.weight"]) | |
| put(f"blk.{i}.ffn_norm.weight", sd[p + "n2.w"], norm=True) | |
| put(f"blk.{i}.ffn_gate.weight", g); put(f"blk.{i}.ffn_up.weight", u) | |
| put(f"blk.{i}.ffn_down.weight", sd[p + "down.weight"]) | |
| put("output_norm.weight", sd["norm.w"], norm=True) | |
| used = {"emb.weight", "norm.w"} | {f"blocks.{i}.{n}" for i in range(L) for n in ("qkv.weight", "gate_up.weight", "n1.w", "n2.w", "o.weight", "down.weight")} | |
| assert used == set(sd), set(sd) ^ used | |
| w.write_header_to_file(); w.write_kv_data_to_file(); w.write_tensors_to_file(); w.close() | |
| print("wrote", out) | |