LFM2.5-1.2B keeps learning new facts in small add-ons (in-car assistant test)

#26
by fintanlabs - opened

Hi Liquid team, I'm Troy, I run a small lab in Spain working on models that keep learning into their own weights after release. I tested it on LFM2.5-1.2B-Instruct, set up like an in-car assistant, and wanted to share the results.

Setup: one add-on for a carmaker's monthly update (release notes, owner's manual changes, a service campaign, charging prices) and one for a single driver's details, trained one after the other. Each takes about 2 minutes on one A40 and is about 10 MB. A small router switches an add-on on only for questions about its topics, so everything else goes to the unchanged base model.

Results (base model vs with add-ons):

  • Update facts, wordings never seen in training: 3.5% vs 85%
  • Driver facts, wordings never seen in training: 1.2% vs 87%
  • Driver facts, direct questions: 0% vs 90%
  • GSM8K / MMLU maths / MMLU other: 0.54 / 0.41 / 0.48 vs 0.55 / 0.41 / 0.49 (unchanged)

When the model's own confidence decides whether to answer, it was wrong on 2 of 76 confident answers and said it wasn't sure on the rest. A third add-on learnt a week of real news for one car model, town and job (88.5% on unseen wordings) without touching the other two. The add-ons also run in llama.cpp as standard LoRA files, with no change to the base model.

All car and driver data is fictional and written by me, and this is one run, so please treat it as early. I'd be glad to run the same thing on documents you choose, or share more detail on the numbers. Would this be useful for your on-device work, for example the in-car use?

Troy Clifford, Garnet Taurus Ltd

Sign up or log in to comment