Publish the trained metadata-only router checkpoint with its fitted feature builders, verified label map, and a runnable example

README.md

@@ -1,10 +1,10 @@
 ---
 license: mit
-# NOTE: library_name is deliberately NOT set. Declaring `sklearn` makes Hugging
-# Face render a "Use this model" widget with a joblib.load(...) snippet pointing
-# at a checkpoint file. This repository ships no checkpoint, by design (see
-# "Why no checkpoint is published" below), so that widget would send readers to
-# a file that does not exist.
+library_name: sklearn
+# A real checkpoint ships here: router_metadata_only/model.joblib, with its
+# fitted feature builders, an explicit label map, and a runnable example.
+# `inference: false` because this is a local scikit-learn artifact -- there is
+# no hosted inference endpoint for it.
 inference: false
 datasets:
   - AgentsSci/EMNLP_Cost-Aware-Protocol-Routing
@@ -43,41 +43,96 @@ paper evaluates two third-party solver families — `openai/gpt-oss-120b` and
 `google/gemma-4-31B-it` — which we did **not** train, do not own, and do not
 redistribute. Their licenses and terms are their own.
 
-**This is a research-artifact card, not a checkpoint release.** It documents the
-paper's lightweight *routers*: small scikit-learn classifiers that pick which
-collaboration protocol to run. It ships their training code, configurations,
-selected hyperparameters, and per-problem held-out predictions — but **no
-serialized model file**. The reason is stated honestly below.
-
-### Why no checkpoint is published
-
-> **If you see a `joblib.load(...)` snippet on this page, ignore it.** Hugging
-> Face auto-generates a "Use this model" widget for scikit-learn repositories,
-> and it points at a `sklearn_model.joblib` that **does not exist here**. That
-> is not a broken upload: no checkpoint is published, deliberately, for the
-> reasons below.
-
-A trained metadata-only router exists in our internal results. We chose not to
-publish it:
-
-1. Its serialized estimator stores integer classes `[0, 1, 2, 3, 4]` with **no
-   embedded label mapping**. The mapping from index to protocol is recoverable
-   only by assuming sorted label-name order. That is a convention, not a stored
-   fact, and we could not verify it from the checkpoint alone. Publishing it
-   would ask users to guess which action index `3` means — and a wrong guess
-   produces confident, silently incorrect routing.
-2. It was pickled under a different scikit-learn version and already emits
-   `InconsistentVersionWarning` on load here. Pickled estimators are
-   version-fragile and are not a durable research artifact.
-3. Nothing is lost. Every number in the paper can be reproduced without it: the
-   **per-problem held-out predictions are released with string labels**, and the
-   **training code is released and tested**. A user can re-fit the router in
-   minutes on CPU, or score our exact predictions directly.
-
-If you need a checkpoint for a downstream system, train one with the released
-code and record your own label mapping.
+**What this is:** the paper's lightweight *router* — a small scikit-learn
+classifier that picks which collaboration protocol to run for a given problem.
+A trained checkpoint ships here, along with its fitted feature builders, an
+explicit label map, held-out predictions, and a runnable example.
 
----
+> **Using the "Use this model" snippet?** It loads `sklearn_model.joblib` from
+> the repository root, which is a byte-identical copy of the checkpoint provided
+> so that snippet works. It gives you the bare estimator and nothing else — **no
+> feature builders and no label map**, so its integer output is not
+> interpretable on its own and features built by hand will not match the fitted
+> pipeline. Use `router_metadata_only/predict_example.py` instead; it loads the
+> same model together with everything needed to use it correctly.
+
+### What ships here
+
+**A trained router checkpoint**, in `router_metadata_only/`:
+
+| File | What it is |
+|---|---|
+| `model.joblib` | The fitted multinomial logistic regression (220 features, 5 actions) |
+| `feature_builders.joblib` | The fitted `OneHotEncoder`, `MultiLabelBinarizer` and `StandardScaler` |
+| `label_mapping.json` | **Read this before decoding predictions.** See the warning below |
+| `predict_example.py` | A runnable, offline end-to-end example |
+| `model_metadata.json` | Feature order, split, hyperparameters, checksums, environment |
+| `metrics.csv`, `class_recalls.csv`, `test_predictions.csv` | Held-out results and per-problem predictions |
+| `selected_config.json` | The dev-only hyperparameter search and what it chose |
+
+A byte-identical copy of `model.joblib` also sits at the repository root as
+`sklearn_model.joblib`, purely so Hugging Face's auto-generated snippet resolves.
+
+```bash
+pip install scikit-learn pandas scipy joblib
+python router_metadata_only/predict_example.py
+```
+
+### Two things that will bite you
+
+**1. The class indices are not alphabetical.** The estimator stores integer
+classes `[0, 1, 2, 3, 4]` with no embedded label map. The order is the paper's
+**fixed cost order**, not `sorted()`:
+
+| index | 0 | 1 | 2 | 3 | 4 |
+|---|---|---|---|---|---|
+| **correct** | `baseline_llm` | `single_agent` | `per` | `broadcast` | `none` |
+| ~~alphabetical~~ | ~~baseline_llm~~ | ~~broadcast~~ | ~~none~~ | ~~per~~ | ~~single_agent~~ |
+
+This matters because the wrong decoding looks plausible. Decoding the released
+test set alphabetically agrees with the published predictions on **352 of 423**
+rows — it silently mislabels 71 and still produces a sensible-looking
+distribution. Use `label_mapping.json`, which we verified reproduces
+**423 of 423**.
+
+**2. The pickle is version-fragile.** It was fitted under scikit-learn 1.8.0 and
+emits `InconsistentVersionWarning` on other versions. In our test environment
+(1.6.1) `predict()` works but `predict_proba()` raises `AttributeError`, because
+an internal attribute moved between versions. `predict_example.py` catches this
+and falls back to a softmax over `decision_function`. If you need exact
+probabilities, either match 1.8.0 or retrain with the released code — training
+takes minutes on CPU.
+
+A checkpoint is a snapshot of a library version as much as of a model. The
+training code is the durable artifact; this file is a convenience.
+
+### What this checkpoint actually achieves
+
+Held-out results for **this** artifact, read from `router_metadata_only/metrics.csv`:
+
+| Split | n | Oracle-action accuracy | Macro-F1 | Realized solve rate | Avg tokens |
+|---|--:|--:|--:|--:|--:|
+| dev | 416 | 0.5697 | 0.2679 | 0.6274 | 51.7K |
+| test | 423 | 0.5721 | 0.2295 | 0.6076 | 44.3K |
+
+Read those numbers honestly. A macro-F1 of **0.23** on five actions means the
+router is good at the majority action and weak on the rare, expensive ones —
+which is exactly the paper's finding, not a defect in this file. On the same
+split, always-Baseline solves 56.8% and the retrospective oracle reaches 92.4%.
+This router lands at 60.8%: better than the fixed policy, far short of the
+ceiling, and it is the *metadata-only ablation*, deliberately the weakest of the
+three variants.
+
+Do not deploy it as-is. It is a reproducible baseline to beat.
+
+### Which router is this?
+
+The **metadata-only** variant, trained on the primary 423-problem split. It is
+the paper's ablation that isolates what metadata alone can do, and it is the one
+variant with serialized weights. The main **text+metadata** router and the
+six-setting evaluations have no saved estimator; their per-problem predictions
+and full hyperparameter searches are released under `artifacts/`, and the
+training code reproduces them.
 
 ## What the router predicts
 

router_metadata_only/class_recalls.csv

@@ -0,0 +1,11 @@
+model_name,split,label,recall
+metadata_only_logreg,dev,baseline_llm,0.8860759493670886
+metadata_only_logreg,dev,single_agent,0.1875
+metadata_only_logreg,dev,per,0.1388888888888889
+metadata_only_logreg,dev,broadcast,0.0
+metadata_only_logreg,dev,none,0.13333333333333333
+metadata_only_logreg,test,baseline_llm,0.9411764705882353
+metadata_only_logreg,test,single_agent,0.13402061855670103
+metadata_only_logreg,test,per,0.07894736842105263
+metadata_only_logreg,test,broadcast,0.0
+metadata_only_logreg,test,none,0.0625

router_metadata_only/feature_builders.joblib

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3252c274c6ae5a90afe3655823e2a16977c21796f7d76fbe3e6141454ca61e62
+size 23857

router_metadata_only/label_mapping.json

@@ -0,0 +1,30 @@
+{
+  "label_order": [
+    "baseline_llm",
+    "single_agent",
+    "per",
+    "broadcast",
+    "none"
+  ],
+  "index_to_label": {
+    "0": "baseline_llm",
+    "1": "single_agent",
+    "2": "per",
+    "3": "broadcast",
+    "4": "none"
+  },
+  "label_to_index": {
+    "baseline_llm": 0,
+    "single_agent": 1,
+    "per": 2,
+    "broadcast": 3,
+    "none": 4
+  },
+  "note": "The estimator stores integer classes [0..4] with no embedded label map. This file IS that map. It is the paper's fixed cost order, NOT alphabetical order: decoding with sorted(labels) agrees with the released predictions on only 352 of 423 test rows, while this mapping agrees on 423 of 423.",
+  "verified": {
+    "check": "model.predict on reconstructed test features, decoded with index_to_label, compared to test_predictions.csv",
+    "n_test": 423,
+    "agreement_label_order": "423/423",
+    "agreement_sorted_order": "352/423"
+  }
+}

router_metadata_only/metrics.csv

@@ -0,0 +1,3 @@
+model_name,split,n_examples,accuracy,macro_f1,realized_solve_rate,avg_total_tokens,avg_model_calls,avg_wall_time_seconds,avg_excess_cost_tokens_vs_oracle,missed_success_rate
+metadata_only_logreg,dev,416,0.569712,0.267917,0.627404,51731.351,15.312,89.666,19073.486,0.323834
+metadata_only_logreg,test,423,0.572104,0.229503,0.607565,44298.277,14.069,84.125,20451.118,0.342711

router_metadata_only/model.joblib

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63b5358423b430b39f2a586913d48bf84f2b6774cda78bf1bc383c76de7f0c3b
+size 9711

router_metadata_only/model_metadata.json

@@ -0,0 +1,92 @@
+{
+  "artifact": "metadata-only protocol router (multinomial logistic regression)",
+  "task": "predict the fixed-order-oracle action for a problem, before any protocol runs",
+  "actions": [
+    "baseline_llm",
+    "single_agent",
+    "per",
+    "broadcast",
+    "none"
+  ],
+  "n_features_in": 220,
+  "feature_blocks": [
+    {
+      "block": "source",
+      "type": "OneHotEncoder",
+      "note": "competition/source identifier"
+    },
+    {
+      "block": "domain",
+      "type": "MultiLabelBinarizer",
+      "note": "topic tags; unseen tags dropped"
+    },
+    {
+      "block": "numeric",
+      "type": "StandardScaler(with_mean=False)",
+      "columns": [
+        "difficulty",
+        "difficulty_tier"
+      ]
+    }
+  ],
+  "feature_order": "source one-hot, then domain multi-hot, then the two scaled numeric columns",
+  "split": {
+    "scheme": "stratified 80/10/10",
+    "seed": 42,
+    "train": 3342,
+    "dev": 416,
+    "test": 423
+  },
+  "selected_hyperparameters": {
+    "name": "plain_l2_c0.25",
+    "class_weight": null,
+    "C": 0.25
+  },
+  "selection": "macro-F1 then excess-token-cost on DEV only, with validation-based early stopping",
+  "leakage": "features are problem metadata only: no gold answers, no correctness, no protocol outcomes, no oracle label",
+  "pickled_with": {
+    "note": "recorded so a loader can explain a version warning",
+    "sklearn_at_fit": "1.8.0 (from the InconsistentVersionWarning emitted on load)"
+  },
+  "verified_in_this_environment": {
+    "sklearn": "1.6.1",
+    "numpy": "2.0.2",
+    "scipy": "1.13.1",
+    "predict": "works",
+    "predict_proba": "raises AttributeError under this version; predict_example.py falls back to the decision function"
+  },
+  "files": {
+    "class_recalls.csv": {
+      "bytes": 525,
+      "sha256": "a4cbb87ec4a07cb3e56543aef9dd54aaa929ab95933adfc106cf52845da7540b"
+    },
+    "feature_builders.joblib": {
+      "bytes": 23857,
+      "sha256": "3252c274c6ae5a90afe3655823e2a16977c21796f7d76fbe3e6141454ca61e62"
+    },
+    "label_mapping.json": {
+      "bytes": 906,
+      "sha256": "ac5c5cfef8e8d8a556351d078e3b2fd4a6415c19c3ff8d439f53c543cf6c590e"
+    },
+    "metrics.csv": {
+      "bytes": 376,
+      "sha256": "7e043f4946fc3219cb1c8ba5a7b259b3b991084c452b8ef388fd07d4f3356833"
+    },
+    "model.joblib": {
+      "bytes": 9711,
+      "sha256": "63b5358423b430b39f2a586913d48bf84f2b6774cda78bf1bc383c76de7f0c3b"
+    },
+    "predict_example.py": {
+      "bytes": 4724,
+      "sha256": "a61bdbf294349bc2c21a3543d4e197b2bb83ba0b4be824cbfd981496ad40a813"
+    },
+    "selected_config.json": {
+      "bytes": 1283,
+      "sha256": "44542b6696893bbd23e630b208a2061f67085c5dc56fbc04cb8b61c1464770ed"
+    },
+    "test_predictions.csv": {
+      "bytes": 17241,
+      "sha256": "5bcb554174f5cc34588d201938835473170c1c57154718dfede792341fcf3ee1"
+    }
+  }
+}

router_metadata_only/predict_example.py

@@ -0,0 +1,123 @@
+"""Load the released metadata-only router and route a few problems.
+
+Runs offline. No model endpoint, no download beyond this repository.
+
+    pip install scikit-learn pandas scipy joblib
+    python predict_example.py
+
+The router sees problem METADATA only -- difficulty, difficulty tier, source,
+and domain. It never sees gold answers, correctness, protocol outcomes, or the
+oracle label. That boundary is the point of the paper, so this example does not
+give you a way to pass those in.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import joblib
+import pandas as pd
+from scipy import sparse
+
+HERE = Path(__file__).resolve().parent
+
+# The estimator stores integer classes [0..4] and does NOT carry the label names.
+# Decode with this file, not with sorted(labels): the order is the paper's fixed
+# cost order, and alphabetical decoding disagrees with the released predictions
+# on 71 of 423 test rows while looking perfectly plausible.
+MAPPING = json.loads((HERE / "label_mapping.json").read_text())
+INDEX_TO_LABEL = {int(k): v for k, v in MAPPING["index_to_label"].items()}
+
+
+def load():
+    return (
+        joblib.load(HERE / "model.joblib"),
+        joblib.load(HERE / "feature_builders.joblib"),
+    )
+
+
+def build_features(df: pd.DataFrame, builders: dict):
+    """Reproduce the training-time feature pipeline exactly.
+
+    Column order matters: source one-hot, then domain multi-hot, then the two
+    scaled numeric columns. The estimator was fitted on 220 features in that
+    order and will silently mispredict if they are assembled differently.
+    """
+    x_source = builders["source_encoder"].transform(df[["source"]])
+
+    # Domains unseen at training time are dropped rather than erroring, which is
+    # what the training code does; an unknown domain simply contributes nothing.
+    known = set(builders["domain_binarizer"].classes_.tolist())
+    x_domain = builders["domain_binarizer"].transform(
+        [[d for d in row if d in known] for row in df["domain_list"]]
+    )
+
+    x_numeric = builders["scaler"].transform(
+        df[["difficulty", "difficulty_tier"]].to_numpy(dtype=float)
+    )
+    if not sparse.issparse(x_numeric):
+        x_numeric = sparse.csr_matrix(x_numeric)
+
+    return sparse.hstack([x_source, x_domain, x_numeric], format="csr")
+
+
+def main() -> None:
+    model, builders = load()
+
+    # Three made-up problems, in the schema the router expects.
+    problems = pd.DataFrame(
+        [
+            {"problem_id": "demo-easy", "source": "cayley", "difficulty": 1.5,
+             "difficulty_tier": 1,
+             "domain_list": ["Mathematics -> Algebra -> Prealgebra -> Simple Equations"]},
+            {"problem_id": "demo-mid", "source": "HMMT_2", "difficulty": 4.5,
+             "difficulty_tier": 5,
+             "domain_list": ["Mathematics -> Discrete Mathematics -> Combinatorics"]},
+            {"problem_id": "demo-hard", "source": "imo_shortlist", "difficulty": 8.0,
+             "difficulty_tier": 8,
+             "domain_list": ["Mathematics -> Number Theory -> Congruences"]},
+        ]
+    )
+
+    X = build_features(problems, builders)
+    assert X.shape[1] == model.n_features_in_, (
+        f"built {X.shape[1]} features but the model expects {model.n_features_in_}"
+    )
+
+    indices = model.predict(X)
+
+    # predict_proba reaches into attributes that moved between scikit-learn
+    # versions, so it can raise on an estimator pickled by a different version
+    # even though predict() works. Fall back to the decision function, which is
+    # stable, rather than failing the example.
+    try:
+        scores = model.predict_proba(X)
+        score_label = "confidence"
+    except Exception:
+        import numpy as np
+
+        margins = model.decision_function(X)
+        e = np.exp(margins - margins.max(axis=1, keepdims=True))
+        scores = e / e.sum(axis=1, keepdims=True)
+        score_label = "softmax(margin)"
+        print("note: predict_proba is unavailable under this scikit-learn "
+              "version; showing a softmax over the decision function instead.\n")
+
+    print(f"{'problem':12s} {'action':14s} {score_label:16s} runner-up")
+    for pid, idx, row in zip(problems["problem_id"], indices, scores):
+        order = row.argsort()[::-1]
+        top, second = int(order[0]), int(order[1])
+        print(
+            f"{pid:12s} {INDEX_TO_LABEL[int(idx)]:14s} {row[top]:.3f}"
+            f"            {INDEX_TO_LABEL[second]} ({row[second]:.3f})"
+        )
+
+    print(
+        "\n'none' means: do not escalate -- no protocol was expected to succeed,"
+        "\nso spending on collaboration would be wasted. It is a router action,"
+        "\nnot a fifth protocol."
+    )
+
+
+if __name__ == "__main__":
+    main()

router_metadata_only/selected_config.json

@@ -0,0 +1,55 @@
+{
+  "selected_config": {
+    "name": "plain_l2_c0.25",
+    "class_weight": null,
+    "C": 0.25
+  },
+  "selection_metric": "macro_f1 then avg_excess_cost_tokens_vs_oracle after validation-based early stopping",
+  "dev_metrics_before_refit": {
+    "n_examples": 416,
+    "accuracy": 0.569712,
+    "macro_f1": 0.267917,
+    "realized_solve_rate": 0.627404,
+    "avg_total_tokens": 51731.351,
+    "avg_model_calls": 15.312,
+    "avg_wall_time_seconds": 89.666,
+    "avg_excess_cost_tokens_vs_oracle": 19073.486,
+    "missed_success_rate": 0.323834
+  },
+  "loss_history_epochs": 30,
+  "early_stopping_metric": "dev_log_loss",
+  "selected_epoch": 1,
+  "best_dev_log_loss": 1.177944,
+  "feature_dimensions": {
+    "source_cardinality": 63,
+    "domain_cardinality": 155,
+    "total_feature_dim": 220
+  },
+  "label_counts_train": {
+    "baseline_llm": 1898,
+    "single_agent": 768,
+    "none": 244,
+    "per": 292,
+    "broadcast": 140
+  },
+  "label_counts_dev": {
+    "baseline_llm": 237,
+    "single_agent": 96,
+    "per": 36,
+    "none": 30,
+    "broadcast": 17
+  },
+  "label_counts_test": {
+    "baseline_llm": 238,
+    "none": 32,
+    "single_agent": 97,
+    "broadcast": 18,
+    "per": 38
+  },
+  "features": [
+    "source",
+    "domain",
+    "difficulty",
+    "difficulty_tier"
+  ]
+}

router_metadata_only/test_predictions.csv

@@ -0,0 +1,424 @@
+problem_id,split,gold_label,predicted_label
+tier01:10,test,baseline_llm,baseline_llm
+tier01:100,test,baseline_llm,baseline_llm
+tier01:101,test,baseline_llm,baseline_llm
+tier01:103,test,baseline_llm,baseline_llm
+tier01:109,test,baseline_llm,baseline_llm
+tier01:15,test,baseline_llm,baseline_llm
+tier01:21,test,baseline_llm,baseline_llm
+tier01:39,test,baseline_llm,baseline_llm
+tier01:65,test,baseline_llm,baseline_llm
+tier01:66,test,baseline_llm,baseline_llm
+tier01:71,test,baseline_llm,baseline_llm
+tier01:79,test,baseline_llm,baseline_llm
+tier01:92,test,baseline_llm,baseline_llm
+tier01:95,test,baseline_llm,baseline_llm
+tier01:96,test,baseline_llm,baseline_llm
+tier02:113,test,baseline_llm,baseline_llm
+tier02:121,test,baseline_llm,baseline_llm
+tier02:125,test,baseline_llm,baseline_llm
+tier02:136,test,none,baseline_llm
+tier02:170,test,baseline_llm,baseline_llm
+tier02:173,test,single_agent,baseline_llm
+tier02:177,test,baseline_llm,baseline_llm
+tier02:188,test,none,baseline_llm
+tier02:204,test,baseline_llm,baseline_llm
+tier02:21,test,baseline_llm,baseline_llm
+tier02:251,test,baseline_llm,baseline_llm
+tier02:253,test,baseline_llm,baseline_llm
+tier02:265,test,baseline_llm,baseline_llm
+tier02:278,test,baseline_llm,baseline_llm
+tier02:284,test,baseline_llm,baseline_llm
+tier02:29,test,baseline_llm,baseline_llm
+tier02:298,test,baseline_llm,baseline_llm
+tier02:316,test,baseline_llm,baseline_llm
+tier02:326,test,baseline_llm,baseline_llm
+tier02:330,test,none,baseline_llm
+tier02:336,test,baseline_llm,baseline_llm
+tier02:351,test,baseline_llm,baseline_llm
+tier02:353,test,baseline_llm,baseline_llm
+tier02:356,test,baseline_llm,baseline_llm
+tier02:363,test,baseline_llm,baseline_llm
+tier02:37,test,baseline_llm,baseline_llm
+tier02:372,test,baseline_llm,baseline_llm
+tier02:379,test,baseline_llm,baseline_llm
+tier02:405,test,baseline_llm,baseline_llm
+tier02:441,test,single_agent,baseline_llm
+tier02:445,test,baseline_llm,baseline_llm
+tier02:455,test,baseline_llm,baseline_llm
+tier02:462,test,baseline_llm,baseline_llm
+tier02:471,test,baseline_llm,baseline_llm
+tier02:473,test,baseline_llm,baseline_llm
+tier02:477,test,single_agent,baseline_llm
+tier02:478,test,baseline_llm,baseline_llm
+tier02:484,test,baseline_llm,baseline_llm
+tier02:488,test,baseline_llm,baseline_llm
+tier02:510,test,baseline_llm,baseline_llm
+tier02:71,test,baseline_llm,baseline_llm
+tier02:90,test,baseline_llm,baseline_llm
+tier03:11,test,baseline_llm,baseline_llm
+tier03:19,test,baseline_llm,baseline_llm
+tier04:1012,test,baseline_llm,baseline_llm
+tier04:1038,test,single_agent,none
+tier04:1041,test,single_agent,baseline_llm
+tier04:1049,test,none,per
+tier04:1057,test,baseline_llm,baseline_llm
+tier04:1068,test,single_agent,baseline_llm
+tier04:1074,test,single_agent,baseline_llm
+tier04:1076,test,broadcast,baseline_llm
+tier04:108,test,baseline_llm,baseline_llm
+tier04:1084,test,per,baseline_llm
+tier04:1093,test,per,baseline_llm
+tier04:1095,test,baseline_llm,baseline_llm
+tier04:1101,test,broadcast,baseline_llm
+tier04:1104,test,baseline_llm,baseline_llm
+tier04:1106,test,baseline_llm,baseline_llm
+tier04:111,test,baseline_llm,baseline_llm
+tier04:118,test,none,baseline_llm
+tier04:121,test,baseline_llm,baseline_llm
+tier04:129,test,single_agent,baseline_llm
+tier04:141,test,baseline_llm,baseline_llm
+tier04:147,test,baseline_llm,baseline_llm
+tier04:153,test,baseline_llm,baseline_llm
+tier04:157,test,baseline_llm,baseline_llm
+tier04:16,test,baseline_llm,baseline_llm
+tier04:165,test,single_agent,baseline_llm
+tier04:166,test,baseline_llm,baseline_llm
+tier04:180,test,none,baseline_llm
+tier04:196,test,baseline_llm,baseline_llm
+tier04:222,test,baseline_llm,baseline_llm
+tier04:224,test,single_agent,baseline_llm
+tier04:233,test,baseline_llm,baseline_llm
+tier04:238,test,baseline_llm,baseline_llm
+tier04:246,test,baseline_llm,baseline_llm
+tier04:247,test,none,baseline_llm
+tier04:252,test,baseline_llm,baseline_llm
+tier04:263,test,broadcast,baseline_llm
+tier04:277,test,single_agent,baseline_llm
+tier04:278,test,baseline_llm,baseline_llm
+tier04:293,test,baseline_llm,baseline_llm
+tier04:304,test,baseline_llm,baseline_llm
+tier04:350,test,baseline_llm,baseline_llm
+tier04:36,test,baseline_llm,baseline_llm
+tier04:378,test,baseline_llm,baseline_llm
+tier04:379,test,baseline_llm,baseline_llm
+tier04:384,test,baseline_llm,baseline_llm
+tier04:399,test,baseline_llm,baseline_llm
+tier04:416,test,baseline_llm,baseline_llm
+tier04:420,test,baseline_llm,baseline_llm
+tier04:422,test,single_agent,baseline_llm
+tier04:426,test,baseline_llm,baseline_llm
+tier04:438,test,baseline_llm,single_agent
+tier04:445,test,none,baseline_llm
+tier04:452,test,none,baseline_llm
+tier04:464,test,single_agent,baseline_llm
+tier04:468,test,single_agent,baseline_llm
+tier04:470,test,baseline_llm,baseline_llm
+tier04:476,test,baseline_llm,baseline_llm
+tier04:48,test,baseline_llm,baseline_llm
+tier04:484,test,baseline_llm,baseline_llm
+tier04:494,test,baseline_llm,baseline_llm
+tier04:504,test,baseline_llm,baseline_llm
+tier04:533,test,none,baseline_llm
+tier04:538,test,baseline_llm,baseline_llm
+tier04:540,test,single_agent,baseline_llm
+tier04:561,test,baseline_llm,baseline_llm
+tier04:570,test,baseline_llm,baseline_llm
+tier04:575,test,single_agent,baseline_llm
+tier04:578,test,baseline_llm,baseline_llm
+tier04:586,test,baseline_llm,baseline_llm
+tier04:591,test,baseline_llm,baseline_llm
+tier04:603,test,baseline_llm,baseline_llm
+tier04:615,test,baseline_llm,baseline_llm
+tier04:640,test,single_agent,baseline_llm
+tier04:650,test,single_agent,baseline_llm
+tier04:659,test,baseline_llm,baseline_llm
+tier04:672,test,baseline_llm,baseline_llm
+tier04:685,test,baseline_llm,baseline_llm
+tier04:686,test,baseline_llm,baseline_llm
+tier04:69,test,single_agent,baseline_llm
+tier04:692,test,baseline_llm,baseline_llm
+tier04:694,test,baseline_llm,baseline_llm
+tier04:696,test,baseline_llm,baseline_llm
+tier04:713,test,baseline_llm,baseline_llm
+tier04:714,test,baseline_llm,baseline_llm
+tier04:719,test,baseline_llm,baseline_llm
+tier04:72,test,baseline_llm,baseline_llm
+tier04:727,test,baseline_llm,baseline_llm
+tier04:742,test,baseline_llm,baseline_llm
+tier04:765,test,baseline_llm,baseline_llm
+tier04:771,test,baseline_llm,baseline_llm
+tier04:775,test,baseline_llm,baseline_llm
+tier04:78,test,baseline_llm,baseline_llm
+tier04:780,test,baseline_llm,baseline_llm
+tier04:787,test,baseline_llm,baseline_llm
+tier04:799,test,baseline_llm,baseline_llm
+tier04:800,test,single_agent,baseline_llm
+tier04:81,test,baseline_llm,baseline_llm
+tier04:811,test,single_agent,baseline_llm
+tier04:814,test,baseline_llm,baseline_llm
+tier04:816,test,baseline_llm,baseline_llm
+tier04:820,test,baseline_llm,baseline_llm
+tier04:84,test,baseline_llm,baseline_llm
+tier04:842,test,baseline_llm,baseline_llm
+tier04:847,test,baseline_llm,baseline_llm
+tier04:848,test,baseline_llm,baseline_llm
+tier04:85,test,baseline_llm,baseline_llm
+tier04:858,test,baseline_llm,baseline_llm
+tier04:86,test,single_agent,baseline_llm
+tier04:871,test,baseline_llm,baseline_llm
+tier04:878,test,per,baseline_llm
+tier04:89,test,baseline_llm,baseline_llm
+tier04:892,test,baseline_llm,baseline_llm
+tier04:896,test,baseline_llm,baseline_llm
+tier04:897,test,baseline_llm,baseline_llm
+tier04:907,test,baseline_llm,baseline_llm
+tier04:925,test,baseline_llm,baseline_llm
+tier04:932,test,baseline_llm,baseline_llm
+tier04:936,test,single_agent,baseline_llm
+tier04:947,test,single_agent,baseline_llm
+tier04:957,test,baseline_llm,baseline_llm
+tier04:963,test,baseline_llm,baseline_llm
+tier04:966,test,baseline_llm,baseline_llm
+tier04:972,test,baseline_llm,baseline_llm
+tier04:977,test,single_agent,baseline_llm
+tier04:996,test,baseline_llm,baseline_llm
+tier05:1025,test,single_agent,baseline_llm
+tier05:1039,test,baseline_llm,baseline_llm
+tier05:1058,test,baseline_llm,baseline_llm
+tier05:1067,test,per,none
+tier05:11,test,single_agent,baseline_llm
+tier05:139,test,baseline_llm,baseline_llm
+tier05:14,test,baseline_llm,baseline_llm
+tier05:152,test,single_agent,baseline_llm
+tier05:171,test,broadcast,baseline_llm
+tier05:175,test,baseline_llm,baseline_llm
+tier05:180,test,baseline_llm,baseline_llm
+tier05:19,test,baseline_llm,baseline_llm
+tier05:206,test,baseline_llm,single_agent
+tier05:215,test,per,baseline_llm
+tier05:219,test,single_agent,baseline_llm
+tier05:22,test,baseline_llm,baseline_llm
+tier05:236,test,broadcast,baseline_llm
+tier05:242,test,baseline_llm,baseline_llm
+tier05:250,test,baseline_llm,baseline_llm
+tier05:255,test,baseline_llm,baseline_llm
+tier05:26,test,baseline_llm,baseline_llm
+tier05:261,test,single_agent,baseline_llm
+tier05:267,test,single_agent,baseline_llm
+tier05:273,test,baseline_llm,baseline_llm
+tier05:280,test,baseline_llm,baseline_llm
+tier05:281,test,single_agent,baseline_llm
+tier05:283,test,per,none
+tier05:29,test,single_agent,baseline_llm
+tier05:292,test,baseline_llm,baseline_llm
+tier05:30,test,baseline_llm,baseline_llm
+tier05:326,test,baseline_llm,baseline_llm
+tier05:332,test,single_agent,single_agent
+tier05:334,test,baseline_llm,baseline_llm
+tier05:338,test,single_agent,baseline_llm
+tier05:350,test,none,baseline_llm
+tier05:351,test,baseline_llm,baseline_llm
+tier05:354,test,per,baseline_llm
+tier05:368,test,baseline_llm,baseline_llm
+tier05:397,test,baseline_llm,baseline_llm
+tier05:402,test,baseline_llm,baseline_llm
+tier05:403,test,baseline_llm,baseline_llm
+tier05:405,test,single_agent,baseline_llm
+tier05:411,test,baseline_llm,baseline_llm
+tier05:417,test,baseline_llm,baseline_llm
+tier05:441,test,baseline_llm,baseline_llm
+tier05:443,test,per,baseline_llm
+tier05:445,test,broadcast,baseline_llm
+tier05:452,test,baseline_llm,baseline_llm
+tier05:463,test,baseline_llm,baseline_llm
+tier05:469,test,baseline_llm,baseline_llm
+tier05:47,test,baseline_llm,baseline_llm
+tier05:481,test,baseline_llm,baseline_llm
+tier05:492,test,per,baseline_llm
+tier05:513,test,none,baseline_llm
+tier05:520,test,baseline_llm,baseline_llm
+tier05:524,test,single_agent,baseline_llm
+tier05:530,test,none,baseline_llm
+tier05:548,test,single_agent,baseline_llm
+tier05:55,test,baseline_llm,baseline_llm
+tier05:555,test,baseline_llm,baseline_llm
+tier05:583,test,per,baseline_llm
+tier05:590,test,single_agent,baseline_llm
+tier05:60,test,none,baseline_llm
+tier05:614,test,baseline_llm,baseline_llm
+tier05:619,test,baseline_llm,baseline_llm
+tier05:634,test,broadcast,baseline_llm
+tier05:635,test,none,baseline_llm
+tier05:636,test,single_agent,baseline_llm
+tier05:65,test,baseline_llm,baseline_llm
+tier05:650,test,baseline_llm,baseline_llm
+tier05:656,test,baseline_llm,baseline_llm
+tier05:663,test,broadcast,baseline_llm
+tier05:671,test,none,baseline_llm
+tier05:675,test,single_agent,baseline_llm
+tier05:680,test,baseline_llm,baseline_llm
+tier05:687,test,per,baseline_llm
+tier05:696,test,baseline_llm,baseline_llm
+tier05:704,test,baseline_llm,baseline_llm
+tier05:711,test,single_agent,baseline_llm
+tier05:714,test,baseline_llm,baseline_llm
+tier05:737,test,baseline_llm,baseline_llm
+tier05:739,test,baseline_llm,baseline_llm
+tier05:755,test,single_agent,broadcast
+tier05:760,test,none,baseline_llm
+tier05:774,test,baseline_llm,baseline_llm
+tier05:776,test,broadcast,baseline_llm
+tier05:782,test,single_agent,baseline_llm
+tier05:784,test,none,baseline_llm
+tier05:785,test,per,baseline_llm
+tier05:793,test,broadcast,baseline_llm
+tier05:798,test,none,baseline_llm
+tier05:80,test,baseline_llm,baseline_llm
+tier05:811,test,single_agent,baseline_llm
+tier05:823,test,baseline_llm,baseline_llm
+tier05:829,test,baseline_llm,baseline_llm
+tier05:843,test,broadcast,baseline_llm
+tier05:851,test,baseline_llm,none
+tier05:854,test,baseline_llm,baseline_llm
+tier05:86,test,baseline_llm,baseline_llm
+tier05:877,test,baseline_llm,baseline_llm
+tier05:892,test,single_agent,baseline_llm
+tier05:90,test,baseline_llm,baseline_llm
+tier05:918,test,baseline_llm,baseline_llm
+tier05:93,test,baseline_llm,baseline_llm
+tier05:931,test,baseline_llm,baseline_llm
+tier05:947,test,single_agent,baseline_llm
+tier05:999,test,baseline_llm,baseline_llm
+tier06:100,test,baseline_llm,baseline_llm
+tier06:130,test,baseline_llm,baseline_llm
+tier06:142,test,per,baseline_llm
+tier06:167,test,single_agent,baseline_llm
+tier06:173,test,baseline_llm,baseline_llm
+tier06:176,test,per,baseline_llm
+tier06:179,test,single_agent,baseline_llm
+tier06:18,test,per,none
+tier06:180,test,baseline_llm,single_agent
+tier06:192,test,none,per
+tier06:244,test,broadcast,baseline_llm
+tier06:257,test,per,baseline_llm
+tier06:262,test,per,baseline_llm
+tier06:264,test,single_agent,none
+tier06:266,test,single_agent,single_agent
+tier06:275,test,single_agent,baseline_llm
+tier06:28,test,none,none
+tier06:288,test,per,single_agent
+tier06:302,test,baseline_llm,baseline_llm
+tier06:305,test,baseline_llm,baseline_llm
+tier06:307,test,baseline_llm,baseline_llm
+tier06:324,test,single_agent,baseline_llm
+tier06:345,test,none,single_agent
+tier06:347,test,single_agent,baseline_llm
+tier06:361,test,baseline_llm,baseline_llm
+tier06:363,test,baseline_llm,baseline_llm
+tier06:370,test,baseline_llm,baseline_llm
+tier06:396,test,baseline_llm,baseline_llm
+tier06:407,test,baseline_llm,baseline_llm
+tier06:424,test,baseline_llm,single_agent
+tier06:426,test,per,baseline_llm
+tier06:427,test,single_agent,baseline_llm
+tier06:442,test,broadcast,none
+tier06:462,test,baseline_llm,baseline_llm
+tier06:473,test,single_agent,baseline_llm
+tier06:481,test,single_agent,single_agent
+tier06:484,test,single_agent,baseline_llm
+tier06:485,test,baseline_llm,baseline_llm
+tier06:63,test,per,baseline_llm
+tier06:66,test,per,baseline_llm
+tier06:73,test,baseline_llm,none
+tier06:79,test,baseline_llm,baseline_llm
+tier06:8,test,single_agent,none
+tier06:86,test,none,none
+tier06:89,test,single_agent,none
+tier07:105,test,none,baseline_llm
+tier07:114,test,single_agent,baseline_llm
+tier07:119,test,single_agent,baseline_llm
+tier07:134,test,single_agent,baseline_llm
+tier07:157,test,single_agent,single_agent
+tier07:163,test,per,baseline_llm
+tier07:174,test,single_agent,baseline_llm
+tier07:175,test,baseline_llm,single_agent
+tier07:184,test,per,single_agent
+tier07:208,test,broadcast,single_agent
+tier07:222,test,single_agent,single_agent
+tier07:227,test,single_agent,baseline_llm
+tier07:240,test,baseline_llm,baseline_llm
+tier07:263,test,per,none
+tier07:267,test,baseline_llm,baseline_llm
+tier07:311,test,baseline_llm,baseline_llm
+tier07:321,test,baseline_llm,baseline_llm
+tier07:322,test,single_agent,baseline_llm
+tier07:328,test,per,baseline_llm
+tier07:34,test,none,baseline_llm
+tier07:357,test,per,per
+tier07:38,test,per,baseline_llm
+tier07:44,test,per,single_agent
+tier07:57,test,per,none
+tier07:65,test,broadcast,none
+tier07:7,test,single_agent,single_agent
+tier07:74,test,single_agent,baseline_llm
+tier07:87,test,baseline_llm,baseline_llm
+tier07:88,test,none,single_agent
+tier08:103,test,single_agent,per
+tier08:107,test,single_agent,per
+tier08:112,test,per,per
+tier08:113,test,baseline_llm,per
+tier08:120,test,baseline_llm,single_agent
+tier08:135,test,baseline_llm,single_agent
+tier08:136,test,single_agent,baseline_llm
+tier08:147,test,none,baseline_llm
+tier08:149,test,per,single_agent
+tier08:168,test,single_agent,baseline_llm
+tier08:179,test,single_agent,none
+tier08:185,test,single_agent,baseline_llm
+tier08:200,test,single_agent,none
+tier08:201,test,per,single_agent
+tier08:216,test,single_agent,single_agent
+tier08:224,test,single_agent,baseline_llm
+tier08:225,test,per,single_agent
+tier08:226,test,single_agent,single_agent
+tier08:237,test,single_agent,baseline_llm
+tier08:259,test,single_agent,single_agent
+tier08:26,test,none,single_agent
+tier08:261,test,baseline_llm,baseline_llm
+tier08:264,test,none,baseline_llm
+tier08:265,test,baseline_llm,baseline_llm
+tier08:285,test,broadcast,baseline_llm
+tier08:286,test,per,single_agent
+tier08:298,test,baseline_llm,baseline_llm
+tier08:305,test,none,single_agent
+tier08:306,test,single_agent,baseline_llm
+tier08:320,test,broadcast,per
+tier08:328,test,single_agent,baseline_llm
+tier08:53,test,baseline_llm,baseline_llm
+tier08:6,test,single_agent,baseline_llm
+tier08:61,test,baseline_llm,baseline_llm
+tier08:67,test,single_agent,per
+tier08:70,test,single_agent,per
+tier08:76,test,single_agent,baseline_llm
+tier08:83,test,none,single_agent
+tier09:130,test,baseline_llm,single_agent
+tier09:132,test,per,single_agent
+tier09:18,test,broadcast,single_agent
+tier09:27,test,baseline_llm,single_agent
+tier09:3,test,single_agent,baseline_llm
+tier09:32,test,per,baseline_llm
+tier09:39,test,none,per
+tier09:42,test,single_agent,single_agent
+tier09:46,test,per,per
+tier09:58,test,baseline_llm,per
+tier09:61,test,baseline_llm,baseline_llm
+tier09:67,test,single_agent,baseline_llm
+tier09:81,test,single_agent,baseline_llm
+tier09:83,test,single_agent,single_agent
+tier09:86,test,single_agent,baseline_llm
+tier09:89,test,single_agent,single_agent
+tier09:91,test,per,single_agent
+tier10:15,test,single_agent,per
+tier10:5,test,baseline_llm,single_agent
+tier10:9,test,single_agent,single_agent

sklearn_model.joblib

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63b5358423b430b39f2a586913d48bf84f2b6774cda78bf1bc383c76de7f0c3b
+size 9711