parakon-runtime / tests /peg-parser /simple-tokenize.h
kat0012's picture
Parakon runtime: llama.cpp fork with fused 1-bit CUDA/Metal/CPU kernels (part 7)
d2816aa verified
Raw History Blame Contribute Delete
114 Bytes
#pragma once
#include <string>
#include <vector>
std::vector<std::string> simple_tokenize(const std::string &);