Model card and records for the 0.1.0-preview release

APP_AND_PACK_LICENSES.md

@@ -1,4 +1,4 @@
-# Licences: Timegravity Tamil app, models and knowledge packs (generated 2026-09-14 10:12 UTC by build_licenses_md.py)
+# Licences: Timegravity Tamil app, models and knowledge packs (generated 2026-09-14 12:04 UTC by build_licenses_md.py)
 
 This is an independent research project; no legal review has been performed on data licensing.
 
@@ -10,7 +10,7 @@ The app is not open source. The weights and the recipe are open; the app is prop
 - Kept private: the Android app source (timegravity/tamil-lm-android), its UI and on-device serving stack, and the plain-text lexicon and full rule list. The earlier line keeping the KB packs proprietary is superseded: pack contents keep their own licensing (CC BY-SA sources published under CC BY-SA in Timegravity/tamil-lm-2b-gguf; anything not redistributable stays out).
 - Rationale: nothing in the stack obliges source release, and the app is the product while the weights are the research contribution. The plain-text lexicon and rule list stay private because publishing the exact blocklist would be a map around it, and the list contains material that is not published.
 
-- On the device (ruling 2026-09-14): the lexicon (assets/safety/lexicon.hashed) and the request rule lists (assets/safety/rules.hashed.json) ship as salted SHA-256 hashes with exact matching preserved; the plain-text files never ship. The guard classifier (assets/safety/guard_v2.json) ships in plain text: it holds short character n-grams (2 to 5 characters) and their weights, hashing cannot hide strings that short because every one of them can be enumerated, and the weights reveal patterns rather than a word list. The sensitive material is the lexicon and the rule lists, which are hashed.
+- On the device: the lexicon (assets/safety/lexicon.hashed) and the request rule lists (assets/safety/rules.hashed.json) ship as salted SHA-256 hashes with exact matching preserved; the plain-text files never ship. The guard classifier (assets/safety/guard_v2.json) ships in plain text: it holds short character n-grams (2 to 5 characters) and their weights, hashing cannot hide strings that short because every one of them can be enumerated, and the weights reveal patterns rather than a word list. The sensitive material is the lexicon and the rule lists, which are hashed.
 ## Weights licence
 
 tamil-lm-2b weights: Apache License 2.0, matching the base model Qwen/Qwen3.5-2B-Base. Prohibited uses are stated in the model card as expectations, not licence terms. Enforceable use restrictions live in the app's Terms and govern people using the app. Any GGUF conversion Timegravity hosts inherits the licence of its source weights.
@@ -28,7 +28,7 @@ tamil-lm-2b weights: Apache License 2.0, matching the base model Qwen/Qwen3.5-2B
 Automated check: android/app/app/licence-audit.gradle.kts (task :app:licenceAudit, run before every assemble and bundle task and in CI by .github/workflows/licence-audit.yml).
 
 1. GPL and AGPL. Shipped runtime modules checked: 114 (both editions, Maven POM licences with parent POMs followed); build tooling modules checked: 144 (the Android Gradle plugin and the Kotlin Gradle plugin, transitively); native code compiled into the APK scanned for GPL headers. GPL or AGPL components found that fail the build: 0.
-   - Build tooling only, dual-licensed with a non-GPL option, not packaged into the APK: 2. The non-GPL licence (CDDL) is elected for these. They are recorded, not a build failure, because they run only on the build machine and nothing from them is linked into or packaged in the app; only a GPL or AGPL component in the shipped runtime graph or in the native code compiled into the APK fails the build (confirmed by Vignesh 2026-09-14).
+   - Build tooling only, dual-licensed with a non-GPL option, not packaged into the APK: 2. The non-GPL licence (CDDL) is elected for these. They are recorded, not a build failure, because they run only on the build machine and nothing from them is linked into or packaged in the app; only a GPL or AGPL component in the shipped runtime graph or in the native code compiled into the APK fails the build.
      - com.sun.activation:javax.activation:1.2.0 [build tooling] licence: CDDL/GPLv2+CE https://github.com/javaee/activation/blob/master/LICENSE.txt
      - javax.annotation:javax.annotation-api:1.3.2 [build tooling] licence: CDDL + GPLv2 with classpath exception https://github.com/javaee/javax.annotation/blob/master/LICENSE (with Classpath Exception)
    - LGPL (not GPL) in build tooling, offered with Apache 2.0: 2.
@@ -207,7 +207,7 @@ The only network client in the app is android/app/app/src/main/java/ai/timegravi
 
 | pin | status |
 |---|---|
-| No GPL or AGPL in the dependency graph, checked in CI | automated; 0 failing hits; build-time-only dual-licensed tooling recorded, not failed (confirmed by Vignesh 2026-09-14) |
+| No GPL or AGPL in the dependency graph, checked in CI | automated; 0 failing hits; build-time-only dual-licensed tooling recorded, not failed |
 | Licences screen renders every component with full text | the build fails if a shipped library has no mapped full text |
 | First-launch acceptance recorded locally and not shown again | Prefs.eulaAccepted |
 | APK SHA-256 in the release notes matches the hosted file | scripts/release_info.py computes it from the APK; to check at release |

DATACARD.md

@@ -3,7 +3,7 @@
 Releasing entity: Timegravity Labs Private Limited (Coimbatore, India), Hugging Face organisation "Timegravity".
 Dataset agreements recorded as accepted by Timegravity Labs Private Limited. No legal review has been performed on data licensing.
 
-Generated by datacard.py from data/manifest.json (profile `main3b_ext_e1`) on 2026-09-12; license rows come from data/LICENSES.md.
+Generated by datacard.py from data/manifest.json (profile `main3b_ext_e1`) on 2026-09-14; license rows come from data/LICENSES.md.
 
 | bucket | source_id | documents | tokens | share |
 |---|---|---|---|---|
@@ -269,13 +269,12 @@ Project Madurai's moolam edition was used instead), and all 20th-century
 commentaries bundled in Project Madurai urai editions.
 
 
-## Decisions by Vignesh (2026-08-25)
+## Source decisions
 - ai4bharat/samanantar (CC BY-NC 4.0): EXCLUDED from training. The bucket built from it was set aside (data/clean/EXCLUDED_samanantar_parallel.jsonl, gitignored, not used). Parallel bucket rebuilt from verified-permissive sources only: BPCC subsets individually verified as CC BY 4.0 or more permissive, plus OPUS corpora with open terms.
 - ai4bharat/IndicSentiment (no stated license): EVAL-ONLY, never redistributed, not in training.
 - Divyanshu/indicxnli (cc0 tag vs CC BY-NC text): EVAL-ONLY, never redistributed, not in training.
 - Tamil Virtual Academy: NOT USED. No permission request sent.
 - FLORES (gsarti/flores_101, CC BY-SA 4.0): eval-only.
-- HF_TOKEN: provided via environment only, never written to any file. Gated sources (uonlp/CulturaX, ai4bharat/MILU, bigcode/starcoderdata) become accessible once it is set; confirmation recorded in STATUS.md.
 
 ## English-Tamil parallel corpus sources (2026-08-25)
 
@@ -441,14 +440,14 @@ actually loaded (after the noted drops).
 ## Access update (2026-08-26)
 With an HF token present (stored outside the repo), the previously gated sources are accessible: ai4bharat/MILU (CC BY 4.0, eval-only), ai4bharat/IN22-Gen (CC BY 4.0, eval-only), uonlp/CulturaX (mC4/OSCAR terms, include-no-redistribute), bigcode/starcoderdata (The Stack ToU, include-no-redistribute), ai4bharat/BPCC (per-subset: CC0/CC-BY-4.0 subsets only; Samanantar-derived subsets excluded). Training inclusion of the last three is a separate decision, not part of the run-2 literature fix.
 
-## Recency and retrieval sources (added 2026-08-26, ruling B/A)
+## Recency and retrieval sources
 
 | # | Source | URL | License (verified 2026-08-26 via the MediaWiki siteinfo rightsinfo API) | Use |
 |---|---|---|---|---|
-| R1 | Tamil Wikipedia dump tawiki-20260801-pages-articles.xml.bz2 (275 MB, dump dated 2026-08-04) | https://dumps.wikimedia.org/tawiki/20260801/ | CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/deed.ta) | Retrieval index for serve.py (Phase 5b); candidate recency top-up CPT bucket (ruling A, improvement-loop only) |
+| R1 | Tamil Wikipedia dump tawiki-20260801-pages-articles.xml.bz2 (275 MB, dump dated 2026-08-04) | https://dumps.wikimedia.org/tawiki/20260801/ | CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/deed.ta) | Retrieval index for serve.py (not used in the training data) |
 | R2 | Tamil Wikinews (ta.wikinews.org) via API | https://ta.wikinews.org | CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) | Candidate recency top-up bucket only |
 
-No copyrighted news sites are used for recency (ruling A). Attribution for CC BY / CC BY-SA content is carried in data/manifest.json and the data card.
+No copyrighted news sites are used for recency. Attribution for CC BY / CC BY-SA content is carried in data/manifest.json and the data card.
 
 ## Bundled tools (added 2026-09-07)
 
@@ -457,9 +456,23 @@ No copyrighted news sites are used for recency (ruling A). Attribution for CC BY
 | Tamil phonetic transliterator (browser, dependency-free) | ui/transliterate.js, ui/README.md | MIT, Timegravity Labs Private Limited 2026 (header in file) | written for this project; no third-party code |
 | Reference serving stack (guard, routing, retrieval, chat UI) | serve.py, guard.py, guard_small.py, retrieval/, chat_server.py | same license as the repository code | this project |
 
+## AG-TU: Tamil University encyclopedias on Tamil Wikisource (added 2026-09-14)
+
+| # | source | url | licence (as stated, quoted) | verified at | decision | what was taken |
+|---|---|---|---|---|---|---|
+| AG-TU | Tamil University, Thanjavur: அறிவியல் களஞ்சியம் (science encyclopedia, 19 volumes) and வாழ்வியற் களஞ்சியம் (social science encyclopedia, 15 volumes), proofreading pages on Tamil Wikisource | https://ta.wikisource.org/wiki/அட்டவணை:அறிவியல்_களஞ்சியம்_14.pdf (one index per volume) | Wikimedia Commons file page of each volume (checked for volumes 5 and 14 of the science encyclopedia and volume 1 of the social science encyclopedia): "The Tamil university released all its publications under CC-BY-SA. refer the below document."; licence template {{cc-by-sa-1.0+}}; Permission field: [[File:GoTN Tamil Development Departments order on creative commons cc by sa.pdf]], whose Commons page reads "Government Order of Department of Tamil Development Government of Tamil Nadu declared books publications under CC-BY-SA" with date 2016-08-12 | https://commons.wikimedia.org/wiki/File:அறிவியல்_களஞ்சியம்_14.pdf, https://commons.wikimedia.org/wiki/File:அறிவியல்_களஞ்சியம்_5.pdf, https://commons.wikimedia.org/wiki/File:வாழ்வியற்_களஞ்சியம்_1.pdf and https://commons.wikimedia.org/wiki/File:GoTN_Tamil_Development_Departments_order_on_creative_commons_cc_by_sa.pdf, fetched 2026-09-14 | include (on the strength of the government order) | entry text on crops, pests and farming practice (build_pack_tamil_university.py, data/packs/agriculture/chunks_tamil_university.jsonl) |
+
+- Pending review, to revisit: every volume's Commons file page carries {{LicenseReview}}, meaning the Commons licence review of the
+  CC BY-SA claim has not been completed (checked 2026-09-14). The inclusion rests on the Tamil Nadu government order as cited on those
+  pages. If the review fails, or the order is found not to cover these volumes, remove chunks_tamil_university.jsonl from the pack
+  (PACK_EXTRA in build_app_packs.py) and rebuild.
+- Attribution: every chunk carries the entry title, the series, volume and printed page (section), the Wikisource page URL, and the
+  licence line; the published pack keeps them.
+- Share-alike: the chunks are redistributed under CC BY-SA 4.0 (the {{cc-by-sa-1.0+}} tag permits later versions) with that attribution.
+
 
 ## Knowledge cutoff
-Tamil Wikipedia content is from the 2023-11 dump (to be updated to 2026-08 only if the recency top-up experiment is kept); web crawl cutoff 2025-10 (fineweb-2 release); IndicCorpV2 is a 2022-2023 crawl. The model has no knowledge of events after that, and as a 2B model it makes factual errors outside the literature knowledge base.
+Tamil Wikipedia content in the training data is from the 2023-11 dump (the serving stack's retrieval index uses the 2026-08 dump); web crawl cutoff 2025-10 (fineweb-2 release); IndicCorpV2 is a 2022-2023 crawl. The model has no knowledge of events after that, and as a 2B model it makes factual errors outside the literature knowledge base.
 
 ## Wikipedia-derived instruction rows (round 3 onwards)
 Some supervised fine-tuning rows are grounded on Tamil Wikipedia and Tamil Wikibooks passages (CC BY-SA 4.0). Each such row carries the source title and URL; served answers that draw on Wikipedia carry an attribution line; these rows are Wikipedia-derived under CC BY-SA and are never redistributed under any other license.

LICENSES.md

@@ -185,13 +185,12 @@ Project Madurai's moolam edition was used instead), and all 20th-century
 commentaries bundled in Project Madurai urai editions.
 
 
-## Decisions by Vignesh (2026-08-25)
+## Source decisions
 - ai4bharat/samanantar (CC BY-NC 4.0): EXCLUDED from training. The bucket built from it was set aside (data/clean/EXCLUDED_samanantar_parallel.jsonl, gitignored, not used). Parallel bucket rebuilt from verified-permissive sources only: BPCC subsets individually verified as CC BY 4.0 or more permissive, plus OPUS corpora with open terms.
 - ai4bharat/IndicSentiment (no stated license): EVAL-ONLY, never redistributed, not in training.
 - Divyanshu/indicxnli (cc0 tag vs CC BY-NC text): EVAL-ONLY, never redistributed, not in training.
 - Tamil Virtual Academy: NOT USED. No permission request sent.
 - FLORES (gsarti/flores_101, CC BY-SA 4.0): eval-only.
-- HF_TOKEN: provided via environment only, never written to any file. Gated sources (uonlp/CulturaX, ai4bharat/MILU, bigcode/starcoderdata) become accessible once it is set; confirmation recorded in STATUS.md.
 
 ## English-Tamil parallel corpus sources (2026-08-25)
 
@@ -357,14 +356,14 @@ actually loaded (after the noted drops).
 ## Access update (2026-08-26)
 With an HF token present (stored outside the repo), the previously gated sources are accessible: ai4bharat/MILU (CC BY 4.0, eval-only), ai4bharat/IN22-Gen (CC BY 4.0, eval-only), uonlp/CulturaX (mC4/OSCAR terms, include-no-redistribute), bigcode/starcoderdata (The Stack ToU, include-no-redistribute), ai4bharat/BPCC (per-subset: CC0/CC-BY-4.0 subsets only; Samanantar-derived subsets excluded). Training inclusion of the last three is a separate decision, not part of the run-2 literature fix.
 
-## Recency and retrieval sources (added 2026-08-26, ruling B/A)
+## Recency and retrieval sources
 
 | # | Source | URL | License (verified 2026-08-26 via the MediaWiki siteinfo rightsinfo API) | Use |
 |---|---|---|---|---|
-| R1 | Tamil Wikipedia dump tawiki-20260801-pages-articles.xml.bz2 (275 MB, dump dated 2026-08-04) | https://dumps.wikimedia.org/tawiki/20260801/ | CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/deed.ta) | Retrieval index for serve.py (Phase 5b); candidate recency top-up CPT bucket (ruling A, improvement-loop only) |
+| R1 | Tamil Wikipedia dump tawiki-20260801-pages-articles.xml.bz2 (275 MB, dump dated 2026-08-04) | https://dumps.wikimedia.org/tawiki/20260801/ | CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/deed.ta) | Retrieval index for serve.py (not used in the training data) |
 | R2 | Tamil Wikinews (ta.wikinews.org) via API | https://ta.wikinews.org | CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) | Candidate recency top-up bucket only |
 
-No copyrighted news sites are used for recency (ruling A). Attribution for CC BY / CC BY-SA content is carried in data/manifest.json and the data card.
+No copyrighted news sites are used for recency. Attribution for CC BY / CC BY-SA content is carried in data/manifest.json and the data card.
 
 ## Bundled tools (added 2026-09-07)
 
@@ -373,11 +372,11 @@ No copyrighted news sites are used for recency (ruling A). Attribution for CC BY
 | Tamil phonetic transliterator (browser, dependency-free) | ui/transliterate.js, ui/README.md | MIT, Timegravity Labs Private Limited 2026 (header in file) | written for this project; no third-party code |
 | Reference serving stack (guard, routing, retrieval, chat UI) | serve.py, guard.py, guard_small.py, retrieval/, chat_server.py | same license as the repository code | this project |
 
-## AG-TU: Tamil University encyclopedias on Tamil Wikisource (added 2026-09-14, ruling item 15)
+## AG-TU: Tamil University encyclopedias on Tamil Wikisource (added 2026-09-14)
 
 | # | source | url | licence (as stated, quoted) | verified at | decision | what was taken |
 |---|---|---|---|---|---|---|
-| AG-TU | Tamil University, Thanjavur: அறிவியல் களஞ்சியம் (science encyclopedia, 19 volumes) and வாழ்வியற் களஞ்சியம் (social science encyclopedia, 15 volumes), proofreading pages on Tamil Wikisource | https://ta.wikisource.org/wiki/அட்டவணை:அறிவியல்_களஞ்சியம்_14.pdf (one index per volume) | Wikimedia Commons file page of each volume (checked for volumes 5 and 14 of the science encyclopedia and volume 1 of the social science encyclopedia): "The Tamil university released all its publications under CC-BY-SA. refer the below document."; licence template {{cc-by-sa-1.0+}}; Permission field: [[File:GoTN Tamil Development Departments order on creative commons cc by sa.pdf]], whose Commons page reads "Government Order of Department of Tamil Development Government of Tamil Nadu declared books publications under CC-BY-SA" with date 2016-08-12 | https://commons.wikimedia.org/wiki/File:அறிவியல்_களஞ்சியம்_14.pdf, https://commons.wikimedia.org/wiki/File:அறிவியல்_களஞ்சியம்_5.pdf, https://commons.wikimedia.org/wiki/File:வாழ்வியற்_களஞ்சியம்_1.pdf and https://commons.wikimedia.org/wiki/File:GoTN_Tamil_Development_Departments_order_on_creative_commons_cc_by_sa.pdf, fetched 2026-09-14 | include (Vignesh, 2026-09-14: on the strength of the government order) | entry text on crops, pests and farming practice (build_pack_tamil_university.py, data/packs/agriculture/chunks_tamil_university.jsonl) |
+| AG-TU | Tamil University, Thanjavur: அறிவியல் களஞ்சியம் (science encyclopedia, 19 volumes) and வாழ்வியற் களஞ்சியம் (social science encyclopedia, 15 volumes), proofreading pages on Tamil Wikisource | https://ta.wikisource.org/wiki/அட்டவணை:அறிவியல்_களஞ்சியம்_14.pdf (one index per volume) | Wikimedia Commons file page of each volume (checked for volumes 5 and 14 of the science encyclopedia and volume 1 of the social science encyclopedia): "The Tamil university released all its publications under CC-BY-SA. refer the below document."; licence template {{cc-by-sa-1.0+}}; Permission field: [[File:GoTN Tamil Development Departments order on creative commons cc by sa.pdf]], whose Commons page reads "Government Order of Department of Tamil Development Government of Tamil Nadu declared books publications under CC-BY-SA" with date 2016-08-12 | https://commons.wikimedia.org/wiki/File:அறிவியல்_களஞ்சியம்_14.pdf, https://commons.wikimedia.org/wiki/File:அறிவியல்_களஞ்சியம்_5.pdf, https://commons.wikimedia.org/wiki/File:வாழ்வியற்_களஞ்சியம்_1.pdf and https://commons.wikimedia.org/wiki/File:GoTN_Tamil_Development_Departments_order_on_creative_commons_cc_by_sa.pdf, fetched 2026-09-14 | include (on the strength of the government order) | entry text on crops, pests and farming practice (build_pack_tamil_university.py, data/packs/agriculture/chunks_tamil_university.jsonl) |
 
 - Pending review, to revisit: every volume's Commons file page carries {{LicenseReview}}, meaning the Commons licence review of the
   CC BY-SA claim has not been completed (checked 2026-09-14). The inclusion rests on the Tamil Nadu government order as cited on those

eval_results/cmp_Sarvam-1_chat_test.json

@@ -0,0 +1,283 @@
+[
+ {
+  "benchmark": "milu_ta",
+  "split": "test",
+  "n": 6372,
+  "metric": "acc",
+  "score": 0.384,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:06:25",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3"
+ },
+ {
+  "benchmark": "mmlu_en",
+  "split": "test",
+  "n": 400,
+  "metric": "acc",
+  "score": 0.435,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:06:53",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3"
+ },
+ {
+  "benchmark": "gsm8k_en",
+  "split": "test",
+  "n": 160,
+  "metric": "acc",
+  "score": 0.125,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:16:42",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 512
+ },
+ {
+  "benchmark": "indicqa_ta",
+  "split": "test",
+  "n": 1022,
+  "metric": "f1",
+  "score": 0.213,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:18:09",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 48
+ },
+ {
+  "benchmark": "indicqa_ta",
+  "split": "test",
+  "n": 1022,
+  "metric": "contains",
+  "score": 0.3982,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:18:09",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 48
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++",
+  "score": 31.8652,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:19:10",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu",
+  "score": 4.3535,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:19:10",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++",
+  "score": 14.1283,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:20:01",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu",
+  "score": 5.0863,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:20:01",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++",
+  "score": 30.4741,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:20:52",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 192
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu",
+  "score": 3.7903,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:20:52",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 192
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++",
+  "score": 14.8956,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:21:33",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu",
+  "score": 6.151,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:21:33",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "tanglish_heldout",
+  "split": "test",
+  "n": 2740,
+  "metric": "bpc",
+  "score": 3.2914,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:22:18",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3"
+ },
+ {
+  "benchmark": "tamil_heldout",
+  "split": "test",
+  "n": 1012,
+  "metric": "bpc",
+  "score": 1.2949,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T11:22:37",
+  "stage": "cmp_Sarvam-1_chat",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "chat",
+  "rules": "v3"
+ }
+]

eval_results/cmp_Sarvam-1_test.json

@@ -0,0 +1,283 @@
+[
+ {
+  "benchmark": "milu_ta",
+  "split": "test",
+  "n": 6372,
+  "metric": "acc",
+  "score": 0.3382,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:38:50",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3"
+ },
+ {
+  "benchmark": "mmlu_en",
+  "split": "test",
+  "n": 400,
+  "metric": "acc",
+  "score": 0.425,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:39:19",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3"
+ },
+ {
+  "benchmark": "gsm8k_en",
+  "split": "test",
+  "n": 160,
+  "metric": "acc",
+  "score": 0.0813,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:53:26",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 512
+ },
+ {
+  "benchmark": "indicqa_ta",
+  "split": "test",
+  "n": 1022,
+  "metric": "f1",
+  "score": 0.2576,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:54:57",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 48
+ },
+ {
+  "benchmark": "indicqa_ta",
+  "split": "test",
+  "n": 1022,
+  "metric": "contains",
+  "score": 0.4432,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:54:57",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 48
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++",
+  "score": 19.77,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:56:08",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu",
+  "score": 2.3405,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:56:08",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++",
+  "score": 36.1294,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:56:49",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu",
+  "score": 15.1873,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:56:49",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++",
+  "score": 18.2346,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:57:54",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 192
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu",
+  "score": 1.3324,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:57:54",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 192
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++",
+  "score": 36.0348,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:58:34",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu",
+  "score": 14.3238,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:58:34",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3",
+  "gen_cap": 160
+ },
+ {
+  "benchmark": "tanglish_heldout",
+  "split": "test",
+  "n": 2740,
+  "metric": "bpc",
+  "score": 3.2914,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:59:18",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3"
+ },
+ {
+  "benchmark": "tamil_heldout",
+  "split": "test",
+  "n": 1012,
+  "metric": "bpc",
+  "score": 1.2949,
+  "script": "eval/suite.py",
+  "commit": "6bf9373",
+  "harness": "ce037bd7c0:batched-eager",
+  "gen_mode": "batched-eager",
+  "gen_batch": 32,
+  "timestamp": "2026-09-14T10:59:35",
+  "stage": "cmp_Sarvam-1",
+  "model": "sarvamai/sarvam-1",
+  "adapter": null,
+  "prompt_mode": "raw",
+  "rules": "v3"
+ }
+]

eval_results/extract_cmp_Sarvam-1_chat_test.json

@@ -0,0 +1,226 @@
+[
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_firstline",
+  "score": 31.8652,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 7,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:06"
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_extracted",
+  "score": 31.9365,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 7,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:06"
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_firstline",
+  "score": 4.3535,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 7,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:06"
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_extracted",
+  "score": 4.3525,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 7,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:06"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_firstline",
+  "score": 14.1283,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 3,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:09"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_extracted",
+  "score": 14.1803,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 3,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:09"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_firstline",
+  "score": 5.0863,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 3,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:09"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_extracted",
+  "score": 5.1628,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 3,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:09"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_firstline",
+  "score": 30.4741,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:10"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_extracted",
+  "score": 30.5594,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:10"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_firstline",
+  "score": 3.7903,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:10"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_extracted",
+  "score": 3.8063,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:10"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_firstline",
+  "score": 14.8956,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 4,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:11"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_extracted",
+  "score": 14.9061,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 4,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:11"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_firstline",
+  "score": 6.151,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 4,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:11"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_extracted",
+  "score": 6.1524,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1_chat",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 4,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:27:11"
+ }
+]

eval_results/extract_cmp_Sarvam-1_test.json

@@ -0,0 +1,226 @@
+[
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_firstline",
+  "score": 19.77,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 17,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:55"
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_extracted",
+  "score": 19.8356,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 17,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:55"
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_firstline",
+  "score": 2.3405,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 17,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:55"
+ },
+ {
+  "benchmark": "flores_en_ta",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_extracted",
+  "score": 2.3176,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 17,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:55"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_firstline",
+  "score": 36.1294,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 14,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:57"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "chrf++_extracted",
+  "score": 36.3411,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 14,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:57"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_firstline",
+  "score": 15.1873,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 14,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:57"
+ },
+ {
+  "benchmark": "flores_ta_en",
+  "split": "test",
+  "n": 1012,
+  "metric": "bleu_extracted",
+  "score": 15.2213,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 14,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:57"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_firstline",
+  "score": 18.2346,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:58"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_extracted",
+  "score": 18.3384,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:58"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_firstline",
+  "score": 1.3324,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:58"
+ },
+ {
+  "benchmark": "in22gen_en_ta",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_extracted",
+  "score": 1.3265,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 10,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:58"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_firstline",
+  "score": 36.0348,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 25,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:59"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "chrf++_extracted",
+  "score": 36.8761,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 25,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:59"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_firstline",
+  "score": 14.3238,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 25,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:59"
+ },
+ {
+  "benchmark": "in22gen_ta_en",
+  "split": "test",
+  "n": 820,
+  "metric": "bleu_extracted",
+  "score": 14.814,
+  "harness": "extract-v1",
+  "source": "cmp_Sarvam-1",
+  "source_harness": "ce037bd7c0:batched-eager",
+  "changed_by_rule": 25,
+  "script": "eval/extract_score.py",
+  "commit": "5ce2759",
+  "timestamp": "2026-09-14T11:26:59"
+ }
+]

eval_results/probe_cmp_Sarvam-1.json

@@ -0,0 +1,30 @@
+{
+ "model": "sarvamai/sarvam-1",
+ "adapter": null,
+ "n_items": 763,
+ "per_type_acc": {
+  "which_kural": 0.0,
+  "quote_kural": 0.0,
+  "identify_source": 0.4526,
+  "meaning_mcq": 0.3842
+ },
+ "probe_acc": 0.2092,
+ "quote_chrf": 10.42,
+ "per_type_n": {
+  "which_kural": 190,
+  "quote_kural": 178,
+  "identify_source": 190,
+  "meaning_mcq": 190
+ },
+ "per_type_acc_option_text": {
+  "identify_source": 0.2474,
+  "meaning_mcq": 0.3
+ },
+ "gate_identify_source_option_text": 0.2474,
+ "shuffle_seed": 20260828,
+ "harness": "ce037bd7c0:batched-eager",
+ "prompt_rule": "bos-v1",
+ "quote_cap": 64,
+ "letter_acc_choice_types": 0.4184,
+ "time_s": 237
+}