langchain.client.runner_utils.run_on_dataset¶
langchain.client.runner_utils.run_on_dataset(dataset_name: str, llm_or_chain_factory: Union[Callable[[], Chain], BaseLanguageModel], *, num_repetitions: int = 1, project_name: Optional[str] = None, verbose: bool = False, client: Optional[LangChainPlusClient] = None, tags: Optional[List[str]] = None, run_evaluators: Optional[Sequence[RunEvaluator]] = None, input_mapper: Optional[Callable[[Dict], Any]] = None) → Dict[str, Any][source]¶
Run the Chain or language model on a dataset and store traces
to the specified project name.
Parameters
dataset_name – Name of the dataset to run the chain on.
llm_or_chain_factory – Language model or Chain constructor to run
over the dataset. The Chain constructor is used to permit
independent calls on each example without carrying over state.
concurrency_level – Number of async workers to run in parallel.
num_repetitions – Number of times to run the model on each example.
This is useful when testing success rates or generating confidence
intervals.
project_name – Name of the project to store the traces in.
Defaults to {dataset_name}-{chain class name}-{datetime}.
verbose – Whether to print progress.
client – Client to use to access the dataset. If None, a new client
will be created using the credentials in the environment.
tags – Tags to add to each run in the project.
run_evaluators – Evaluators to run on the results of the chain.
input_mapper – A function to map to the inputs dictionary from an Example
to the format expected by the model to be evaluated. This is useful if
your model needs to deserialize more complex schema or if your dataset
has inputs with keys that differ from what is expected by your chain
or agent.


has inputs with keys that differ from what is expected by your chain
or agent.
Returns
A dictionary containing the run’s project name and the resulting model outputs.


langchain.client.runner_utils.InputFormatError¶
class langchain.client.runner_utils.InputFormatError[source]¶
Bases: Exception
Raised when the input format is invalid.
add_note()¶
Exception.add_note(note) –
add a note to the exception
with_traceback()¶
Exception.with_traceback(tb) –
set self.__traceback__ to tb and return self.
args¶


langchain.client.runner_utils.run_on_examples¶
langchain.client.runner_utils.run_on_examples(examples: Iterator[Example], llm_or_chain_factory: Union[Callable[[], Chain], BaseLanguageModel], *, num_repetitions: int = 1, project_name: Optional[str] = None, verbose: bool = False, client: Optional[LangChainPlusClient] = None, tags: Optional[List[str]] = None, run_evaluators: Optional[Sequence[RunEvaluator]] = None, input_mapper: Optional[Callable[[Dict], Any]] = None) → Dict[str, Any][source]¶
Run the Chain or language model on examples and store
traces to the specified project name.
Parameters
examples – Examples to run the model or chain over.
llm_or_chain_factory – Language model or Chain constructor to run
over the dataset. The Chain constructor is used to permit
independent calls on each example without carrying over state.
num_repetitions – Number of times to run the model on each example.
This is useful when testing success rates or generating confidence
intervals.
project_name – Name of the project to store the traces in.
Defaults to {dataset_name}-{chain class name}-{datetime}.
verbose – Whether to print progress.
client – Client to use to access the dataset. If None, a new client
will be created using the credentials in the environment.
tags – Tags to add to each run in the project.
run_evaluators – Evaluators to run on the results of the chain.
input_mapper – A function to map to the inputs dictionary from an Example
to the format expected by the model to be evaluated. This is useful if
your model needs to deserialize more complex schema or if your dataset
has inputs with keys that differ from what is expected by your chain
or agent.
Returns
A dictionary mapping example ids to the model outputs.


langchain.client.runner_utils.run_llm_or_chain¶
langchain.client.runner_utils.run_llm_or_chain(example: Example, llm_or_chain_factory: Union[Callable[[], Chain], BaseLanguageModel], n_repetitions: int, *, tags: Optional[List[str]] = None, callbacks: Optional[List[BaseCallbackHandler]] = None, input_mapper: Optional[Callable[[Dict], Any]] = None) → Union[List[dict], List[str], List[LLMResult], List[ChatResult]][source]¶
Run the Chain or language model synchronously.
Parameters
example – The example to run.
llm_or_chain_factory – The Chain or language model constructor to run.
n_repetitions – The number of times to run the model on each example.
tags – Optional tags to add to the run.
callbacks – Optional callbacks to use during the run.
Returns
A list of outputs.


langchain.evaluation.qa.generate_chain.QAGenerateChain¶
class langchain.evaluation.qa.generate_chain.QAGenerateChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseLLMOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None)[source]¶
Bases: LLMChain
LLM Chain specifically for generating examples for question answering.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.


There are many different types of memory - please see memory docs
for the full catalog.
param output_key: str = 'text'¶
param output_parser: BaseLLMOutputParser [Optional]¶
Output parser to use.
Defaults to one that takes the most likely string but does not change it
otherwise.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.


chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶


Generate LLM result from inputs.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
apply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async apredict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
async apredict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, str]]¶
Call apredict and then parse the results.
async aprep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
create_outputs(llm_result: LLMResult) → List[Dict[str, Any]]¶
Create outputs from response.


Create outputs from response.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
classmethod from_llm(llm: BaseLanguageModel, **kwargs: Any) → QAGenerateChain[source]¶
Load QA Generate Chain from LLM.
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
predict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
predict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, Any]]¶
Call predict and then parse the results.
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prep outputs.
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
validator raise_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.


Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
save(file_path: Union[Path, str]) → None¶
Save the chain.
Parameters
file_path – Path to file to save the chain to.
Example:
.. code-block:: python
chain.save(file_path=”path/chain.yaml”)
validator set_verbose  »  verbose¶
If verbose is None, set it.
This allows users to pass in None as verbose to access the global setting.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶
extra = 'forbid'¶


langchain.evaluation.qa.eval_chain.QAEvalChain¶
class langchain.evaluation.qa.eval_chain.QAEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseLLMOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None)[source]¶
Bases: LLMChain
LLM Chain specifically for evaluating question answering.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.


chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.


include_run_info – Whether to include run info in the response. Defaults
to False.
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → dict[source]¶
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
apply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async apredict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
async apredict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, str]]¶
Call apredict and then parse the results.


Call apredict and then parse the results.
async aprep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
create_outputs(llm_result: LLMResult) → List[Dict[str, Any]]¶
Create outputs from response.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
evaluate(examples: Sequence[dict], predictions: Sequence[dict], question_key: str = 'query', answer_key: str = 'answer', prediction_key: str = 'result', *, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[dict][source]¶
Evaluate question answering examples and predictions.
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → dict[source]¶
Evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – the LLM or chain prediction to evaluate.
reference (Optional[str], optional) – the reference label
to evaluate against.
input (Optional[str], optional) – the input to consider during evaluation
callbacks (Callbacks, optional) – the callbacks to use for tracing.
**kwargs – additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type


Returns
The evaluation results containing the score or value.
Return type
dict
classmethod from_llm(llm: BaseLanguageModel, prompt: PromptTemplate = PromptTemplate(input_variables=['query', 'result', 'answer'], output_parser=None, partial_variables={}, template="You are a teacher grading a quiz.\nYou are given a question, the student's answer, and the true answer, and are asked to score the student answer as either CORRECT or INCORRECT.\n\nExample Format:\nQUESTION: question here\nSTUDENT ANSWER: student's answer here\nTRUE ANSWER: true answer here\nGRADE: CORRECT or INCORRECT here\n\nGrade the student answers based ONLY on their factual accuracy. Ignore differences in punctuation and phrasing between the student answer and true answer. It is OK if the student answer contains more information than the true answer, as long as it does not contain any conflicting statements. Begin! \n\nQUESTION: {query}\nSTUDENT ANSWER: {result}\nTRUE ANSWER: {answer}\nGRADE:", template_format='f-string', validate_template=True), **kwargs: Any) → QAEvalChain[source]¶
Load QA Eval Chain from LLM.
Parameters
llm (BaseLanguageModel) – the base language model to use.
prompt ('answer' and 'result' that will be used as the) – A prompt template containing the input_variables:
'input' –
prompt –
evaluation. (for) –
PROMPT. (Defaults to) –
**kwargs – additional keyword arguments.
Returns
the loaded QA eval chain.
Return type
QAEvalChain
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.


Create LLMChain from LLM and template.
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
predict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
predict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, Any]]¶
Call predict and then parse the results.
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prep outputs.
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
validator raise_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
save(file_path: Union[Path, str]) → None¶
Save the chain.
Parameters


Save the chain.
Parameters
file_path – Path to file to save the chain to.
Example:
.. code-block:: python
chain.save(file_path=”path/chain.yaml”)
validator set_verbose  »  verbose¶
If verbose is None, set it.
This allows users to pass in None as verbose to access the global setting.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶
extra = 'forbid'¶


langchain.evaluation.run_evaluators.base.RunEvaluatorOutputParser¶
class langchain.evaluation.run_evaluators.base.RunEvaluatorOutputParser(*, eval_chain_output_key: str = 'text')[source]¶
Bases: BaseOutputParser[EvaluationResult]
Parse the output of a run.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param eval_chain_output_key: str = 'text'¶
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
abstract parse(text: str) → T¶
Parse the output of an LLM call.
A method which takes in a string (assumed output of a language model )
and parses it into some structure.
Parameters
text – output of language model
Returns
structured output
parse_chain_output(output: Dict[str, Any]) → EvaluationResult[source]¶
Parse the output of a run.
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.


serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.run_evaluators.implementations.StringRunEvaluatorInputMapper¶
class langchain.evaluation.run_evaluators.implementations.StringRunEvaluatorInputMapper(*, prediction_map: Dict[str, str], input_map: Dict[str, str], answer_map: Optional[Dict[str, str]] = None)[source]¶
Bases: RunEvaluatorInputMapper, BaseModel
Maps the Run and Optional[Example] to a dictionary.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param answer_map: Optional[Dict[str, str]] = None¶
Map from example outputs to the evaluation inputs.
param input_map: Dict[str, str] [Required]¶
Map from run inputs to the evaluation inputs.
param prediction_map: Dict[str, str] [Required]¶
Map from run outputs to the evaluation inputs.
map(run: Run, example: Optional[Example] = None) → Dict[str, Any][source]¶
Maps the Run and Optional[Example] to a dictionary


langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain¶
class langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, agent_tools: Optional[List[BaseTool]] = None, eval_chain: LLMChain, output_parser: TrajectoryOutputParser = None, return_reasoning: bool = False)[source]¶
Bases: Chain
A chain for evaluating ReAct style agents.
This chain is used to evaluate ReAct style agents by reasoning about
the sequence of actions taken and their outcomes.
Example
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param agent_tools: Optional[List[langchain.tools.base.BaseTool]] = None¶
A list of tools available to the agent.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param eval_chain: langchain.chains.llm.LLMChain [Required]¶
The language model chain used for evaluation.
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start


Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param output_parser: langchain.evaluation.agents.trajectory_eval_chain.TrajectoryOutputParser [Optional]¶
The output parser used to parse the output.
param return_reasoning: bool = False¶
Whether to return the reasoning along with the score.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any][source]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.


chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aevaluate_agent_trajectory(*, prediction: str, input: str, agent_trajectory: Union[str, List[Tuple[AgentAction, str]]], reference: Optional[str] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → dict[source]¶
Asynchronously evaluate a trajectory.
Parameters
input (str) – The input question.
agent_trajectory (Union[str, List[Tuple[AgentAction, str]]]) – The intermediate steps forming the agent trajectory.
prediction (str) – The expected prediction.
reference (Optional[str]) – The reference answer.
Returns
The evaluation result.
Return type
dict


Returns
The evaluation result.
Return type
dict
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Call the chain on all inputs in the list.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
evaluate_agent_trajectory(*, prediction: str, input: str, agent_trajectory: Union[str, List[Tuple[AgentAction, str]]], reference: Optional[str] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → dict[source]¶
Evaluate a trajectory.
Parameters
input (str) – The input question.
agent_trajectory (Union[str, List[Tuple[AgentAction, str]]]) – The intermediate steps forming the agent trajectory.
prediction (str) – The expected prediction.
reference (Optional[str]) – The reference answer.
Returns
The evaluation result.
Return type
dict
classmethod from_llm(llm: BaseChatModel, agent_tools: Optional[Sequence[BaseTool]] = None, output_parser: Optional[TrajectoryOutputParser] = None, return_reasoning: bool = False) → TrajectoryEvalChain[source]¶
Create a TrajectoryEvalChain object from a language model chain.
Parameters
llm (BaseChatModel) – The language model chain.
agent_tools (Optional[Sequence[BaseTool]]) – A list of tools
available tothe agent.


available tothe agent.
output_parser (Optional[TrajectoryOutputParser]) – The output parser
used to parse the chain output into a score.
return_reasoning (bool) – Whether to return the
reasoning along with the score.
Returns
The TrajectoryEvalChain object.
Return type
TrajectoryEvalChain
static get_agent_trajectory(steps: Union[str, List[Tuple[AgentAction, str]]]) → str[source]¶
Get the agent trajectory as a formatted string.
Parameters
steps (Union[str, List[Tuple[AgentAction, str]]]) – The agent trajectory.
Returns
The formatted agent trajectory.
Return type
str
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prep outputs.
validator raise_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
save(file_path: Union[Path, str]) → None¶
Save the chain.
Parameters
file_path – Path to file to save the chain to.
Example:
.. code-block:: python
chain.save(file_path=”path/chain.yaml”)
validator set_verbose  »  verbose¶
If verbose is None, set it.
This allows users to pass in None as verbose to access the global setting.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶


to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property input_keys: List[str]¶
Get the input keys for the chain.
Returns
The input keys.
Return type
List[str]
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
property output_keys: List[str]¶
Get the output keys for the chain.
Returns
The output keys.
Return type
List[str]
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶


langchain.evaluation.criteria.eval_chain.CriteriaEvalChain¶
class langchain.evaluation.criteria.eval_chain.CriteriaEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None, requires_reference: bool = False)[source]¶
Bases: LLMChain
LLM Chain for evaluating runs against criteria.
Parameters
llm (BaseLanguageModel) – The language model to use for evaluation.
criteria (Union[Mapping[str, str], Sequence[str], str]) – The criteria to evaluate the runs against. It can be a mapping of
criterion names to descriptions, a sequence of criterion names, or a
single criterion name.
prompt (Optional[BasePromptTemplate], default=None) – The prompt template to use for generating prompts. If not provided, a
default prompt template will be used based on the value of
requires_reference.
requires_reference (bool, default=False) – Whether the evaluation requires a reference text. If True, the
PROMPT_WITH_REFERENCES template will be used, which includes the
reference labels in the prompt. Otherwise, the PROMPT template will be
used, which is a reference-free prompt.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain constructor.
Returns
An instance of the CriteriaEvalChain class.
Return type
CriteriaEvalChain
Examples
>>> from langchain.chat_models import ChatAnthropic
>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = ChatAnthropic()


>>> llm = ChatAnthropic()
>>> criteria = {"my-custom-criterion": "Is the submission the most amazing ever?"}
>>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param output_parser: BaseOutputParser [Optional]¶
The parser to use to map the output to a structured result.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param requires_reference: bool = False¶
Whether the evaluation template expects a reference text.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.


If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.


Utilize the LLM generate method for speed gains.
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶
Asynchronously evaluate a prediction against the criteria.
Parameters
prediction (str) – The predicted text to evaluate.
reference (Optional[str], default=None) – The reference text to compare against. This is required if
requires_reference is True.
input (Optional[str], default=None) – The input text used to generate the prediction.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain acall
method.


method.
Returns
The evaluation results.
Return type
dict
Examples
>>> from langchain.llms import OpenAI
>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = OpenAI()
>>> criteria = "conciseness"
>>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
>>> await chain.aevaluate_strings(
        prediction="The answer is 42.",
        reference="42",
        input="What is the answer to life, the universe, and everything?",
    )
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
apply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async apredict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")


Completion from LLM.
Example
completion = llm.predict(adjective="funny")
async apredict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, str]]¶
Call apredict and then parse the results.
async aprep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
create_outputs(llm_result: LLMResult) → List[Dict[str, Any]]¶
Create outputs from response.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶
Evaluate a prediction against the criteria.
Parameters
prediction (str) – The predicted text to evaluate.
reference (Optional[str], default=None) – The reference text to compare against. This is required if
requires_reference is True.
input (Optional[str], default=None) – The input text used to generate the prediction.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain __call__
method.
Returns
The evaluation results.
Return type
dict
Examples
>>> from langchain.llms import OpenAI
>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = OpenAI()


>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = OpenAI()
>>> criteria = "conciseness"
>>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
>>> chain.evaluate_strings(
        prediction="The answer is 42.",
        reference="42",
        input="What is the answer to life, the universe, and everything?",
    )
classmethod from_llm(llm: BaseLanguageModel, criteria: Union[Mapping[str, str], Sequence[str], Sequence[ConstitutionalPrinciple], str, ConstitutionalPrinciple], *, prompt: Optional[BasePromptTemplate] = None, requires_reference: bool = False, **kwargs: Any) → CriteriaEvalChain[source]¶
Create a CriteriaEvalChain instance from an llm and criteria.
Parameters
llm (BaseLanguageModel) – The language model to use for evaluation.
criteria (CRITERIA_TYPE) –
The criteria to evaluate the runs against. It can be:
a mapping of criterion names to descriptions
a sequence of criterion names
a single criterion name present in one of the default criteria
a sequence of ConstitutionalPrinciple instances
a single ConstitutionalPrinciple instance
prompt (Optional[BasePromptTemplate], default=None) – The prompt template to use for generating prompts. If not provided,
a default prompt template will be used based on the value of
requires_reference.
requires_reference (bool, default=False) – Whether the evaluation requires a reference text. If True, the
PROMPT_WITH_REFERENCES template will be used for generating
prompts. If False, the PROMPT template will be used.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain
constructor.
Returns
An instance of the CriteriaEvalChain class.
Return type
CriteriaEvalChain
Examples


An instance of the CriteriaEvalChain class.
Return type
CriteriaEvalChain
Examples
>>> from langchain.llms import OpenAI
>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = OpenAI()
>>> criteria = {
        "hallucination": (
            "Does this submission contain information"
            " not present in the input or reference?"
        ),
    }
>>> chain = CriteriaEvalChain.from_llm(
        llm=llm,
        criteria=criteria,
        requires_reference=True,
    )
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
static get_supported_default_criteria() → List[str][source]¶
Get the list of supported default criteria.
Returns
The list of supported default criteria.
Return type
List[str]
Examples
>>> CriteriaEvalChain.supported_default_criteria()
['conciseness', 'relevance', 'coherence', 'harmfulness',
    'maliciousness', 'helpfulness',
    'controversiality', 'mysogyny', 'criminality', 'insensitive']
predict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")


Completion from LLM.
Example
completion = llm.predict(adjective="funny")
predict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, Any]]¶
Call predict and then parse the results.
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prep outputs.
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
validator raise_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
classmethod resolve_criteria(criteria: Union[Mapping[str, str], Sequence[str], Sequence[ConstitutionalPrinciple], str, ConstitutionalPrinciple]) → Dict[str, str][source]¶
Resolve the criteria to evaluate.
Parameters
criteria (CRITERIA_TYPE) –
The criteria to evaluate the runs against. It can be:
a mapping of criterion names to descriptions
a sequence of criterion names
a single criterion name present in one of the default criteria
a sequence of ConstitutionalPrinciple instances
a single ConstitutionalPrinciple instance
Returns
A dictionary mapping criterion names to descriptions.
Return type
Dict[str, str]
Examples
>>> criteria = ["relevance", "coherence"]
>>> CriteriaEvalChain.resolve_criteria(criteria)
{'relevance': 'Is the submission referring to a real quote from the text?',
 'coherence': 'Is the submission coherent, well-structured, and organized?'}


'coherence': 'Is the submission coherent, well-structured, and organized?'}
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
save(file_path: Union[Path, str]) → None¶
Save the chain.
Parameters
file_path – Path to file to save the chain to.
Example:
.. code-block:: python
chain.save(file_path=”path/chain.yaml”)
validator set_verbose  »  verbose¶
If verbose is None, set it.
This allows users to pass in None as verbose to access the global setting.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶
extra = 'forbid'¶


langchain.evaluation.run_evaluators.implementations.get_trajectory_evaluator¶


langchain.evaluation.run_evaluators.implementations.get_trajectory_evaluator(llm: BaseChatModel, agent_tools: Union[Sequence[str], Sequence[BaseTool]], *, input_key: str = 'input', prediction_key: str = 'output', tool_input_key: str = 'input', tool_output_key: str = 'output', prompt: BasePromptTemplate = ChatPromptTemplate(input_variables=['tool_descriptions', 'reference', 'agent_trajectory', 'answer', 'question'], output_parser=None, partial_variables={}, messages=[SystemMessage(content='You are a helpful assistant that evaluates language models.', additional_kwargs={}), HumanMessage(content="An AI language model has been given acces to the following set of tools to help answer a user's question.\n\nThe tools given to the AI model are:\n[TOOL_DESCRIPTIONS]\nTool 1:\nName: Search\nDescription: useful for when you need to ask with search\n\nTool 2:\nName: Lookup\nDescription: useful for when you need to ask with lookup\n\nTool 3:\nName: Calculator\nDescription: useful for doing calculations\n\nTool 4:\nName: Search the Web (SerpAPI)\nDescription: useful for when you need to answer questions about current events\n[END_TOOL_DESCRIPTIONS]\n\nThe question the human asked the AI model was: If laid the Statue of Liberty end to end, how many times would it stretch across the United States?\n\nThe AI language model decided to use the following set of tools to answer the question:\n[AGENT_TRAJECTORY]\nStep 1:\nTool used: Search the Web (SerpAPI)\nTool input: If laid the Statue of Liberty end to end, how many times would it stretch across the United States?\nTool output: The Statue of Liberty was given to the United States by France, as a symbol of the two countries' friendship. It was erected


to the United States by France, as a symbol of the two countries' friendship. It was erected atop an American-designed ...\n[END_AGENT_TRAJECTORY]\n\n[RESPONSE]\nThe AI language model's final answer to the question was: There are different ways to measure the length of the United States, but if we use the distance between the Statue of Liberty and the westernmost point of the contiguous United States (Cape Alava, Washington), which is approximately 2,857 miles (4,596 km), and assume that the Statue of Liberty is 305 feet (93 meters) tall, then the statue would stretch across the United States approximately 17.5 times if laid end to end.\n[END_RESPONSE]\n\nLet's to do a detailed evaluation of the AI language model's answer step by step.\n\nWe consider the following criteria before giving a score from 1 to 5:\n\ni. Is the final answer helpful?\nii. Does the AI language use a logical sequence of tools to answer the question?\niii. Does the AI language model use the tools in a helpful way?\niv. Does the AI language model use too many steps to answer the question?\nv. Are the appropriate tools used to answer the question?", additional_kwargs={}, example=False), AIMessage(content="First, let's evaluate the final answer. The final uses good reasoning but is wrong. 2,857 divided by 305 is not 17.5.The model should have used the calculator to figure this out. Second does the model use a logical sequence of tools to answer the question?The way model uses the search is not helpful. The model should have used the search tool to figure the width of the US or the height of the statue.The model didn't use the calculator tool and gave an incorrect answer. The search API should be used for current events or specific questions.The tools were not used in a helpful way. The model did


used for current events or specific questions.The tools were not used in a helpful way. The model did not use too many steps to answer the question.The model did not use the appropriate tools to answer the question.    \nJudgment: Given the good reasoning in the final answer but otherwise poor performance, we give the model a score of 2.\n\nScore: 2", additional_kwargs={}, example=False), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['agent_trajectory', 'answer', 'question', 'reference', 'tool_descriptions'], output_parser=None, partial_variables={}, template="An AI language model has been given access to the following set of tools to help answer a user's question.\n\nThe tools given to the AI model are:\n[TOOL_DESCRIPTIONS]\n{tool_descriptions}\n[END_TOOL_DESCRIPTIONS]\n\nThe question the human asked the AI model was:\n[QUESTION]\n{question}\n[END_QUESTION]{reference}\n\nThe AI language model decided to use the following set of tools to answer the question:\n[AGENT_TRAJECTORY]\n{agent_trajectory}\n[END_AGENT_TRAJECTORY]\n\nThe AI language model's final answer to the question was:\n[RESPONSE]\n{answer}\n[END_RESPONSE]\n\nLet's to do a detailed evaluation of the AI language model's answer step by step.\n\nWe consider the following criteria before giving a score from 1 to 5:\n\ni. Is the final answer helpful?\nii. Does the AI language use a logical sequence of tools to answer the question?\niii. Does the AI language model use the tools in a helpful way?\niv. Does the AI language model use too many steps to answer the question?\nv. Are the appropriate tools used to answer the question?", template_format='f-string', validate_template=True), additional_kwargs={})]), evaluation_name: str =


template_format='f-string', validate_template=True), additional_kwargs={})]), evaluation_name: str = 'Agent Trajectory', **kwargs: Any) → RunEvaluatorChain[source]¶


Get an eval chain for grading a model’s response against a map of criteria.


langchain.evaluation.run_evaluators.implementations.TrajectoryEvalOutputParser¶
class langchain.evaluation.run_evaluators.implementations.TrajectoryEvalOutputParser(*, eval_chain_output_key: str = 'text', evaluation_name: str = 'Agent Trajectory', evaluator_info: dict = None)[source]¶
Bases: RunEvaluatorOutputParser
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param eval_chain_output_key: str = 'text'¶
param evaluation_name: str = 'Agent Trajectory'¶
The name assigned to the evaluation feedback.
param evaluator_info: dict [Optional]¶
Additional information to log as feedback metadata.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
parse(text: str) → EvaluationResult[source]¶
Parse the output of an LLM call.
A method which takes in a string (assumed output of a language model )
and parses it into some structure.
Parameters
text – output of language model
Returns
structured output
parse_chain_output(output: Dict[str, Any]) → EvaluationResult¶
Parse the output of a run.
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output


Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.loading.load_dataset¶
langchain.evaluation.loading.load_dataset(uri: str) → List[Dict][source]¶


langchain.evaluation.schema.PairwiseStringEvaluator¶
class langchain.evaluation.schema.PairwiseStringEvaluator(*args, **kwargs)[source]¶
Bases: Protocol
A protocol for comparing the output of two models.
Methods
__init__(*args, **kwargs)
aevaluate_string_pairs(prediction, prediction_b)
Evaluate the output string pairs.
evaluate_string_pairs(*, prediction, ...[, ...])
Evaluate the output string pairs.
async aevaluate_string_pairs(prediction: str, prediction_b: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶
Evaluate the output string pairs.
Parameters
prediction (str) – The output string from the first model.
prediction_b (str) – The output string from the second model.
reference (str, optional) – The expected output / reference
string. Defaults to None.
input (str, optional) – The input string. Defaults to None.
**kwargs (Any) – Additional keyword arguments, such
as callbacks and optional reference strings.
Returns
A dictionary containing the preference, scores, and/orother information.
Return type
dict
abstract evaluate_string_pairs(*, prediction: str, prediction_b: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶
Evaluate the output string pairs.
Parameters
prediction (str) – The output string from the first model.
prediction_b (str) – The output string from the second model.
reference (str, optional) – The expected output / reference
string. Defaults to None.
input (str, optional) – The input string. Defaults to None.
**kwargs (Any) – Additional keyword arguments, such
as callbacks and optional reference strings.
Returns


as callbacks and optional reference strings.
Returns
A dictionary containing the preference, scores, and/orother information.
Return type
dict


langchain.evaluation.comparison.eval_chain.PairwiseStringResultOutputParser¶
class langchain.evaluation.comparison.eval_chain.PairwiseStringResultOutputParser[source]¶
Bases: BaseOutputParser[dict]
A parser for the output of the PairwiseStringEvalChain.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
parse(text: str) → Any[source]¶
Parse the output text.
Parameters
text (str) – The output text to parse.
Returns
The parsed output.
Return type
Any
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.


Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.schema.StringEvaluator¶
class langchain.evaluation.schema.StringEvaluator(*args, **kwargs)[source]¶
Bases: Protocol
Protocol for evaluating strings.
Methods
__init__(*args, **kwargs)
aevaluate_strings(*, prediction[, ...])
Asynchronously evaluate Chain or LLM output, based on optional
evaluate_strings(*, prediction[, reference, ...])
Evaluate Chain or LLM output, based on optional input and label.
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶
Asynchronously evaluate Chain or LLM output, based on optionalinput and label.
Parameters
prediction (str) – the LLM or chain prediction to evaluate.
reference (Optional[str], optional) – the reference label
to evaluate against.
input (Optional[str], optional) – the input to consider during evaluation
**kwargs – additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
abstract evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶
Evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – the LLM or chain prediction to evaluate.
reference (Optional[str], optional) – the reference label
to evaluate against.
input (Optional[str], optional) – the input to consider during evaluation
**kwargs – additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict


langchain.evaluation.agents.trajectory_eval_chain.TrajectoryOutputParser¶
class langchain.evaluation.agents.trajectory_eval_chain.TrajectoryOutputParser[source]¶
Bases: BaseOutputParser
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
parse(text: str) → TrajectoryEval[source]¶
Parse the output text and extract the score and reasoning.
Parameters
text (str) – The output text to parse.
Returns
A named tuple containing the score and reasoning.
Return type
TrajectoryEval
Raises
OutputParserException – If the score is not found in the output text or
    if the score is not a digit in the range 1-5.
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.


property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain¶
class langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None)[source]¶
Bases: LLMChain
A chain for comparing the output of two models.
Example:
>>> from langchain.chat_models import ChatOpenAI
>>> from langchain.evaluation.comparison import PairwiseStringEvalChain
>>> llm = ChatOpenAI(temperature=0)
>>> chain = PairwiseStringEvalChain.from_llm(llm=llm)
>>> result = chain.evaluate_string_pairs(
…     input = “What is the chemical formula for water?”,
…     prediction = “H2O”,
…     prediction_b = (
…        “The chemical formula for water is H2O, which means”
…        ” there are two hydrogen atoms and one oxygen atom.”
…     referenc = “The chemical formula for water is H2O.”,
… )
>>> print(result[“text”])
# {
#    “value”: “B”,
#    “comment”: “Both responses accurately state”
#       ” that the chemical formula for water is H2O.”
#       ” However, Response B provides additional information”
# .     ” by explaining what the formula means.
[[B]]”# }


# .     ” by explaining what the formula means.
[[B]]”# }
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param output_parser: BaseOutputParser [Optional]¶
Output parser to use.
Defaults to one that takes the most likely string but does not change it
otherwise.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,


These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.


Call apply and then parse the results.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aevaluate_string_pairs(*, prediction: str, prediction_b: str, input: str, reference: Optional[str] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → dict[source]¶
Asynchronously evaluate whether output A is preferred to output B.
Parameters
prediction (str) – The output string from the first model.
prediction_b (str) – The output string from the second model.
input (str) – The input or task string.
callbacks (Callbacks, optional) – The callbacks to use.
reference (str, optional) – The reference string, if any.
**kwargs (Any) – Additional keyword arguments.
Returns
A dictionary containing:
reasoning: The reasoning for the preference.
value: The preference value, which is either ‘A’, ‘B’, or Nonefor no preference.


score: The preference score, which is 1 for ‘A’, 0 for ‘B’,and 0.5 for None.
Return type
dict
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
apply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async apredict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
async apredict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, str]]¶
Call apredict and then parse the results.
async aprep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.


Prepare prompts from inputs.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
create_outputs(llm_result: LLMResult) → List[Dict[str, Any]]¶
Create outputs from response.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
evaluate_string_pairs(*, prediction: str, prediction_b: str, input: str, reference: Optional[str] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → dict[source]¶
Evaluate whether output A is preferred to output B.
Parameters
prediction (str) – The output string from the first model.
prediction_b (str) – The output string from the second model.
input (str) – The input or task string.
callbacks (Callbacks, optional) – The callbacks to use.
reference (str, optional) – The reference string, if any.
**kwargs (Any) – Additional keyword arguments.
Returns
A dictionary containing:
reasoning: The reasoning for the preference.
value: The preference value, which is either ‘A’, ‘B’, or Nonefor no preference.
score: The preference score, which is 1 for ‘A’, 0 for ‘B’,and 0.5 for None.
Return type
dict
classmethod from_llm(*, llm: BaseLanguageModel, prompt: Optional[PromptTemplate] = None, require_reference: bool = False, **kwargs: Any) → PairwiseStringEvalChain[source]¶
Initialize the PairwiseStringEvalChain from an LLM.
Parameters


Initialize the PairwiseStringEvalChain from an LLM.
Parameters
llm (BaseLanguageModel) – The LLM to use.
prompt (PromptTemplate, optional) – The prompt to use.
require_reference (bool, optional) – Whether to require a reference
string. Defaults to False.
**kwargs (Any) – Additional keyword arguments.
Returns
The initialized PairwiseStringEvalChain.
Return type
PairwiseStringEvalChain
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
predict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
predict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, Any]]¶
Call predict and then parse the results.
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prep outputs.


Validate and prep outputs.
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
validator raise_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
save(file_path: Union[Path, str]) → None¶
Save the chain.
Parameters
file_path – Path to file to save the chain to.
Example:
.. code-block:: python
chain.save(file_path=”path/chain.yaml”)
validator set_verbose  »  verbose¶
If verbose is None, set it.
This allows users to pass in None as verbose to access the global setting.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
Configuration for this pydantic object.


model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶
extra = 'forbid'¶


langchain.evaluation.run_evaluators.implementations.CriteriaOutputParser¶
class langchain.evaluation.run_evaluators.implementations.CriteriaOutputParser(*, eval_chain_output_key: str = 'text', evaluation_name: str)[source]¶
Bases: RunEvaluatorOutputParser
Parse a criteria results into an evaluation result.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param eval_chain_output_key: str = 'text'¶
param evaluation_name: str [Required]¶
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
parse(parsed_output: Union[str, dict]) → EvaluationResult[source]¶
Parse the last line of the text and return an evaluation result.
parse_chain_output(output: Dict[str, Any]) → EvaluationResult¶
Parse the output of a run.
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.


langchain.evaluation.run_evaluators.implementations.ChoicesOutputParser¶
class langchain.evaluation.run_evaluators.implementations.ChoicesOutputParser(*, eval_chain_output_key: str = 'text', evaluation_name: str, choices_map: Optional[Dict[str, int]] = None)[source]¶
Bases: RunEvaluatorOutputParser
Parse a feedback run with optional choices.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param choices_map: Optional[Dict[str, int]] = None¶
param eval_chain_output_key: str = 'text'¶
param evaluation_name: str [Required]¶
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
parse(text: str) → EvaluationResult[source]¶
Parse the last line of the text and return an evaluation result.
parse_chain_output(output: Dict[str, Any]) → EvaluationResult¶
Parse the output of a run.
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the


property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEval¶
class langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEval(score, reasoning)[source]¶
Bases: NamedTuple
Create new instance of TrajectoryEval(score, reasoning)
Methods
__init__()
count(value, /)
Return number of occurrences of value.
index(value[, start, stop])
Return first index of value.
Attributes
reasoning
Alias for field number 1
score
Alias for field number 0
count(value, /)¶
Return number of occurrences of value.
index(value, start=0, stop=9223372036854775807, /)¶
Return first index of value.
Raises ValueError if the value is not present.
reasoning: str¶
Alias for field number 1
score: int¶
Alias for field number 0


langchain.evaluation.run_evaluators.implementations.TrajectoryInputMapper¶
class langchain.evaluation.run_evaluators.implementations.TrajectoryInputMapper(*, tool_descriptions: List[str], agent_input_key: str = 'input', agent_output_key: str = 'output', tool_input_key: str = 'input', tool_output_key: str = 'output')[source]¶
Bases: RunEvaluatorInputMapper, BaseModel
Maps the Run and Optional[Example] to a dictionary.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param agent_input_key: str = 'input'¶
The key to load from the agent executor’s run input dictionary.
param agent_output_key: str = 'output'¶
The key to load from the agent executor’s run output dictionary.
param tool_descriptions: List[str] [Required]¶
The descriptions for each of the tools available to the agent.
param tool_input_key: str = 'input'¶
The key to load from the tool executor’s run input dictionary.
param tool_output_key: str = 'output'¶
The key to load from the tool executor’s run output dictionary.
map(run: Run, example: Optional[Example] = None) → Dict[str, str][source]¶
Maps the Run and Optional[Example] to a dictionary


langchain.evaluation.criteria.eval_chain.CriteriaResultOutputParser¶
class langchain.evaluation.criteria.eval_chain.CriteriaResultOutputParser[source]¶
Bases: BaseOutputParser[dict]
A parser for the output of the CriteriaEvalChain.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
parse(text: str) → Any[source]¶
Parse the output text.
Parameters
text (str) – The output text to parse.
Returns
The parsed output.
Return type
Any
parse_result(result: List[Generation]) → T¶
Parse LLM Result.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Optional method to parse the output of an LLM call with a prompt.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – output of language model
prompt – prompt value
Returns
structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.


Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.qa.eval_chain.CotQAEvalChain¶
class langchain.evaluation.qa.eval_chain.CotQAEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseLLMOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None)[source]¶
Bases: ContextQAEvalChain
LLM Chain specifically for evaluating QA using chain of thought reasoning.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs


There are many different types of memory - please see memory docs
for the full catalog.
param output_key: str = 'text'¶
param output_parser: BaseLLMOutputParser [Optional]¶
Output parser to use.
Defaults to one that takes the most likely string but does not change it
otherwise.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.


chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Run the logic of this chain and add to output if desired.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param.
return_only_outputs – boolean for whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. If not provided, will
use the callbacks provided to the chain.
include_run_info – Whether to include run info in the response. Defaults
to False.


include_run_info – Whether to include run info in the response. Defaults
to False.
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
apply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async apredict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
async apredict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, str]]¶
Call apredict and then parse the results.
async aprep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.


Prepare prompts from inputs.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶
Run the chain as text in, text out or multiple variables, text out.
create_outputs(llm_result: LLMResult) → List[Dict[str, Any]]¶
Create outputs from response.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
evaluate(examples: List[dict], predictions: List[dict], question_key: str = 'query', context_key: str = 'context', prediction_key: str = 'result', *, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[dict]¶
Evaluate question answering examples and predictions.
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶


classmethod from_llm(llm: BaseLanguageModel, prompt: PromptTemplate = PromptTemplate(input_variables=['query', 'context', 'result'], output_parser=None, partial_variables={}, template="You are a teacher grading a quiz.\nYou are given a question, the context the question is about, and the student's answer. You are asked to score the student's answer as either CORRECT or INCORRECT, based on the context.\nWrite out in a step by step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset.\n\nExample Format:\nQUESTION: question here\nCONTEXT: context the question is about here\nSTUDENT ANSWER: student's answer here\nEXPLANATION: step by step reasoning here\nGRADE: CORRECT or INCORRECT here\n\nGrade the student answers based ONLY on their factual accuracy. Ignore differences in punctuation and phrasing between the student answer and true answer. It is OK if the student answer contains more information than the true answer, as long as it does not contain any conflicting statements. Begin! \n\nQUESTION: {query}\nCONTEXT: {context}\nSTUDENT ANSWER: {result}\nEXPLANATION:", template_format='f-string', validate_template=True), **kwargs: Any) → CotQAEvalChain[source]¶
Load QA Eval Chain from LLM.
Parameters
llm (BaseLanguageModel) – the base language model to use.
prompt ('context' and 'result' that will be used as the) – A prompt template containing the input_variables:
'query' –
prompt –
evaluation. (for) –
PROMPT. (Defaults to) –
**kwargs – additional keyword arguments.
Returns
the loaded QA eval chain.
Return type
ContextQAEvalChain


Returns
the loaded QA eval chain.
Return type
ContextQAEvalChain
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
predict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
predict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, Any]]¶
Call predict and then parse the results.
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prep outputs.
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
validator raise_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, **kwargs: Any) → str¶