Update README.md
Browse files
README.md
CHANGED
|
@@ -7,4 +7,31 @@ sdk: static
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# SimReal
|
| 11 |
+
|
| 12 |
+
**Make the world's work checkable.**
|
| 13 |
+
|
| 14 |
+
AI agents are getting good at passing tests. We build the places where they have to do the work instead: realistic environments where an agent acts, the world responds, and real outcomes decide the score.
|
| 15 |
+
|
| 16 |
+
Every benchmark we release follows the same rules. The verifier is a program, not a judge model. The scored data is never shown to the agent. And wherever possible, the bar is a real human result, not an estimate.
|
| 17 |
+
|
| 18 |
+
## Benchmarks
|
| 19 |
+
|
| 20 |
+
| | What the agent does | How it's scored |
|
| 21 |
+
|---|---|---|
|
| 22 |
+
| **[Xitadel-QuantBench](https://huggingface.co/spaces/SimrealAI/xitadel-quantbench-leaderboard)** · [GitHub](https://github.com/Simreal-AI/Xitadel-QuantBench) | Researches historical order books, writes a Python trading strategy and ships it | Replayed on an unseen market day against the best human competition strategy that day: P&L, drawdown, Sharpe |
|
| 23 |
+
| **[Hard Puzzle Benchmark](https://huggingface.co/datasets/SimrealAI/hard-puzzle-benchmark)** · [GitHub](https://github.com/Simreal-AI/hard-puzzle-benchmark) | Solves 749 hard, human-written puzzles across 15 topics, many with images | Exact answers on a 707-item core, plus a separate proof-certificate track |
|
| 24 |
+
| **MathmoBench** · [GitHub](https://github.com/Simreal-AI/MathmoBench) | Solves families of near-identical combinatorics problems where one small edit flips the answer | Machine-checkable certificates: a witness if feasible, an impossibility proof if not |
|
| 25 |
+
| **ML Benchmark** · [GitHub](https://github.com/Simreal-AI/Simreal-MLBench) | Runs a full ML research loop across 60 tasks, from tabular data to vision and audio | Test labels stay with the competition platform, so the scorer is not us |
|
| 26 |
+
|
| 27 |
+
## Latest result
|
| 28 |
+
|
| 29 |
+
On Xitadel-QuantBench, **no model has cleared the human bar overall**. The best model scores 77.28 against a human reference of 80, under a shortened pilot research budget. [See the leaderboard →](https://huggingface.co/spaces/SimrealAI/xitadel-quantbench-leaderboard)
|
| 30 |
+
|
| 31 |
+
## Work with us
|
| 32 |
+
|
| 33 |
+
- **Evaluate your agent.** Research teams can request an official evaluation. Results go to you first and are published only with your consent. → [business@simreal.co](mailto:business@simreal.co?subject=Evaluate%20my%20agent)
|
| 34 |
+
- **Partner with us.** Labs and teams can get full access to our environments. → [business@simreal.co](mailto:business@simreal.co)
|
| 35 |
+
- **Join us.** We work with researchers, engineers and domain experts. → [simreal.co/experts](https://simreal.co/experts)
|
| 36 |
+
|
| 37 |
+
[Website](https://simreal.co) · [GitHub](https://github.com/Simreal-AI) · [X](https://x.com/simrealhq) · [LinkedIn](https://www.linkedin.com/company/simreal-ai)
|