NovaEval by Noveum.ai
Evaluate AI models with NovaEval
Tracing, evaluation and scenario-based testing for production chat, voice and workflow agents.
Tracing, evaluation and testing for production AI agents.
Noveum AI builds infrastructure for teams developing chat, voice and workflow agents. This is the official Noveum organization profile. It brings together our current product information, open-source work and founder background.
Shashank Agarwal is the founder and CEO of Noveum AI and the creator of API.market. His background includes the original AWS SageMaker launch team, AWS SDK for PHP work, machine-learning personalization at Amazon Prime Video, and engineering leadership at Activeloop and Levity. Earlier ventures include Hopdata and PipFeed.
Shashank holds an MS in Computer Science from The Ohio State University and is the author of Intelligence at Scale. His work connects production machine learning, developer infrastructure and company building.
Connect with Shashank on LinkedIn for his professional history and current work.
A response from an agent is only one part of the evidence needed to evaluate a task. Teams also need to understand the instructions, tool interactions and intermediate steps that produced it. A change to a prompt, model or tool can improve one behavior while making another less reliable.
Noveum connects tracing, datasets, evaluation and scenario-based testing so engineers can investigate behavior and validate candidate changes. The practical workflow is:
| Stage | Engineering question | Useful evidence |
|---|---|---|
| Inspect a run | What actually happened? | Instrumented traces, inputs, outputs and tool interactions |
| Build a dataset | Which examples should recur in testing? | Representative successes, failures and edge cases |
| Select checks | What does success mean for this task? | Task-specific criteria and checks compatible with the available data |
| Compare changes | Did the proposed change improve the intended behavior? | Results on the same examples under a defined configuration |
| Review and verify | What should the team change next? | Investigated failures, reviewed recommendations and follow-up validation |
Evaluation criteria should match the application. A factual-answer check, a tool-selection check and a voice-interaction scenario measure different properties. A catalog of available scorers does not imply that every scorer is appropriate for every run. Engineers remain responsible for reviewing recommendations and verifying outcomes before shipping changes.
NovaSynth supports pre-production testing of configured voice and chat agents using personas and scenarios. Scenario-based testing complements production tracing: one explores configured interactions before release, while the other helps teams inspect instrumented behavior from actual runs.
A useful scenario has an explicit objective, relevant context and observable acceptance criteria. After changing the agent, compare the same scenario and investigate the reasons behind any changed result. Synthetic scenarios provide controlled test inputs; they should not be presented as proof of real-world customer performance.
See the official Noveum website for current product capabilities, setup requirements and deployment information.
The Noveum managed platform and the NovaEval open-source repository are distinct projects. The managed platform is closed source; Noveum also maintains open-source tooling, including its tracing SDK.
This organization card previously reproduced a lengthy NovaEval README. For NovaEval installation instructions, APIs, examples, release status and contribution guidance, use the repository and its documentation. Check the version and project status before depending on an example. The repository's license applies to that project's code, rather than to the separate managed platform.
For project-specific problems, use NovaEval issues. For ideas and questions about the open-source project, see NovaEval discussions.
Updated 30 October 2026 by the Noveum team.