Megh

Thanks Ander.

Datapoint from me - At work we've been using second llm with same idea (claims, etc.) but we provide basic ontology graph structure for composition, and we've found that it works reasonably well. Although our domain is mostly on research/analytics on news/finance/insurance with low-med volume. We basically are seeing that as model improves yr/yr, we are paying less and less attention on our attribution part of the pipeline.

Cheers,
Megh

I had a brief read on the paper, but I could not find any comparison with cheaper llm agent for source aware judge. Like this is my reading of the paper;

1. An LLM calls multiple tools and generates an answer using their outputs, which may describe many claims
2. A claim can be supported by (one or more) tool’s output while being incorrectly attributed to another.
3. Basically, split the answer into claims and uses embeddings to find a likely supporting source for each.
4. Using NLI & random forest, check if the selected source supports the claim;
5. Also check in response, claim's is attributed by correct source.

Like, it seems (2 - 5) can be done via basic prompts in lifecycle, if you own the executing agent. If not, this can also be done via hooks in harness as plugin. Do you have any comparison on this. Basically, how much of random forest, and all of these eng/plumbing is buying, vs second llm call over traces+ans?