Safetensors
GGUF
qwen-bot / llama.cpp /tests /peg-parser /simple-tokenize.h
anilsuthar2004's picture
Upload folder using huggingface_hub (part 11)
645e264 verified
Raw History Blame Contribute Delete
114 Bytes
#pragma once
#include <string>
#include <vector>
std::vector<std::string> simple_tokenize(const std::string &);