| INTRODUCTION_TEXT = """ |
| A **Coding Agent** is more than just a model - it's the combination of a **Model** and a **Harness** (the tool/framework driving the model). |
| This leaderboard tracks how these components work together, because the same model can perform very differently depending on the harness it's paired with. |
| """ |
|
|
| LLM_BENCHMARKS_TEXT = """ |
| ## What is a Coding Agent? |
| |
| A coding agent is a system that autonomously solves software engineering tasks - reading code, reasoning about bugs, and writing patches. Its performance depends on two components: |
| |
| - **Model** - The underlying language model (e.g. Claude Opus 4.7, Qwen3.6-35B) |
| - **Harness** - The framework or tool that orchestrates the model's actions (e.g. Claude Code, OpenCode, Pi) |
| |
| ## How to Read the Table |
| |
| | Column | Description | |
| |--------|-------------| |
| | **Benchmark** | The benchmark used for evaluation (e.g. SWE-bench Verified - 500 real GitHub issues) | |
| | **Harness** | The agent framework driving the model. | |
| | **Model** | The language model being evaluated | |
| | **Skills** | The set of instructions guiding the agent's behavior | |
| | **Score** | Outcome of the benchmark, often the fraction of tasks solved correctly (higher is better) | |
| | **Precision** | Model weight format (e.g. bf16, fp4) - affects speed, memory footprint, and quality | |
| |
| ## Key Concepts |
| |
| - **FOSS vs Proprietary** - Filters let you compare fully open-source agents against proprietary ones. A FOSS model with a FOSS harness means anyone can reproduce the result |
| - **Skills** - Some harnesses augment the model with extra capabilities (tools, retrieval, etc.). Listed in the "skills" column when present |
| - **Internal results (`*`)** - Benchmarks run by the model provider where the harness and environment were not made public. These are useful reference points but are not independently reproducible |
| |
| ## Learn More |
| |
| Visit the [GitHub repo](https://github.com/redhat-et/coding_agent_bench) for details about the project, methodology, and how to submit your own results. |
| """ |
|
|
| HOW_TO_USE_TEXT = """ |
| --- |
| ## How to interpret these results |
| |
| In the absence of enterprise-specific datasets, public benchmarks provide a means of comparing the performance of coding agents across a wide range of tasks. |
| Better performance on these benchmarks generally translates to better performance on real-world tasks. |
| All benchmarks are run using Harbor, a sandboxed environment for evaluating coding agents. |
| |
| Each benchmark measures the performance of the coding agent on different tasks: |
| |
| * **SWE-Bench Verified**: Measures performance on solving GitHub issues in popular Python repositories. |
| * **SWE-Bench Pro -- Ansible**: Measures performance on solving GitHub issues in the [ansible/ansible](https://github.com/ansible/ansible) repository. |
| Demonstrates how benchmarking can be used to evaluate coding agents on enterprise-specific tasks. |
| |
| Higher scores indicate better performance on the benchmarks. |
| If an agent scores better on a given benchmark than another, it can be generally considered to be better at those kinds of tasks. |
| Compare agents within each benchmark rather than relying on a cross-benchmark average. Each benchmark emphasizes different task types, so per-benchmark scores are the clearest way to judge performance. |
| """ |
|
|
