vikp/instruct_rater

This model judges if a given output is sufficient to recreate a given instruction.

It's useful for filtering data to train a reverse instruct model. It could also have applications around determining if an output/instruction pair is linked, or around quality filtering data (data where the instruction can be recreated from the output might be higher quality).

The model is a binary classifier trained on top of Python 410m with 100k examples for 1 epoch. The final validation loss is .35. You can see an example of a dataset filtered with this model here.

To use it, pass in this prompt format:

Output

{output}

Instruction

{instruction}

Output should be the output from a model, and instruction should be the instruction that generated the output. The model will return a 0-1 score indicating how effectively the instruction can be recreated.