Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
javadtaghia 
posted an update 29 days ago
Post
143
It is not always positive results that worth noting.
I ran a long run of training to improve AtMem (agent memory) on storage path using Laya as a decision model. However, I found it did not give me any better result than original AtMem.
Indeed we can use laya on the retrieval path, we have used than by pluging Jev. So we kinda proved something that is still a gap in memory systems (improving storage) is not as easy as it sounds. Check details https://huggingface.co/blog/javadtaghia/we-fine-tuned-a-memory-classifier-for-our-agent-he
dipankarsarkar

The sharpest row is missing from your replay table: the constant policy.

V3 put DO_NOT_STORE on all 100 and scored 20/100.
Original and fine-tuned Laya both scored 16/100.
On the same fixture, the learned arms land below storing nothing.

The 55-item bias against the baseline can be removed, not just caveated.
Deterministic is only scoreable on the 45 STORE / DO_NOT_STORE items, so its 30 all come from there: 30/45.
Fine-tuned V2 rejected all 25 STOREs, so on those 45 it can score at most 20. A tie with the constant, at best.

So "a safety metric that rewards abstention can be maxed out" holds for plain accuracy too, once the prior is 91%.

Should every write-path replay carry the all-DO_NOT_STORE row as its floor, next to the 30 the rules got?

·
javadtaghia

thinking about your comment, it's fair.

dipankarsarkar

Glad it holds up.

The floor is cheap to keep honest because it is just the label mix of the fixture.
Change the mix and the all-DO_NOT_STORE score moves with it, so it belongs in every table, not only this one.

Will the next replay reuse these 100 items, or draw a fresh set?

·
javadtaghia

updated the document. Thanks
In this post