Text Generation
Transformers
Safetensors
GGUF
English
qwen3
code
100m
conversational
text-generation-inference
Dmitriy-Zemskov commited on
Commit
a52ecf3
·
verified ·
1 Parent(s): 9d6a7a5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -21
README.md CHANGED
@@ -89,27 +89,6 @@ print(tok.decode(out[0, inputs["input_ids"].shape[1]:]).replace("<|im_end|>", ""
89
  Sampling defaults live in `generation_config.json` (temperature 0.7, top-k 40). Keep prompt + answer under
90
  1024 bytes; the model never saw longer sequences.
91
 
92
- ### GGUF / llama.cpp
93
-
94
- The tokenizer has no BPE merges and a pre-tokenizer unknown to the stock converter, so
95
- `convert_hf_to_gguf.py` fails on it. The `make_gguf.py` wrapper from the code repository patches this
96
- at conversion time without modifying llama.cpp (details in its header):
97
-
98
- ```bash
99
- git clone https://github.com/ggml-org/llama.cpp
100
- python make_gguf.py llama.cpp CalmaCatCoder-Next-96M calmacat-96m-f16.gguf f16 # or q8_0
101
- ```
102
-
103
- Because `<|im_end|>` is plain text, llama.cpp does not stop on it by itself. Save the ChatML prompt to a file
104
- and use a reverse prompt:
105
-
106
- ```bash
107
- llama-completion -m calmacat-96m-f16.gguf -no-cnv --no-escape -f prompt.txt -r "<|im_end|>" -n 300 --temp 0.7 --top-k 40
108
- ```
109
-
110
- A warning `special_eos_id is not in special_eog_ids` is expected: the model has no EOS token.
111
- Prefer f16/bf16 or q8_0; this model is so small that aggressive quantization will likely hurt it.
112
-
113
  ## How it was trained
114
 
115
  All of it ran on a single RTX 5060 Ti (16 GB) under Windows with a PyTorch nightly build.
 
89
  Sampling defaults live in `generation_config.json` (temperature 0.7, top-k 40). Keep prompt + answer under
90
  1024 bytes; the model never saw longer sequences.
91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
  ## How it was trained
93
 
94
  All of it ran on a single RTX 5060 Ti (16 GB) under Windows with a PyTorch nightly build.