I tried many small models, including thinking models, but unfortunately, their answers were full of hallucinations. Thinking models were particularly problematic because they often failed to reach a final answer at all.
I also tested a 12B Qwen model through the online API integrated into our app, and it produced reasonable answers. That made me optimistic that I could get a local generative model working offline as well, but unfortunately, it didn’t work out.
No, multilingual support doesn’t work particularly well, but that’s expected from a 300M-parameter model. When searching within a specific language, it works reasonably well and almost always retrieves relevant information, so the model is functional.
I originally tried to make the system run 100% offline, but I couldn’t find a small local generative model that could produce coherent answers. EmbeddingGemma 300M does its job—the issue was with the small generative models.
