HatemSaadallah/tiny-aya-global-strategy-agent

Model Card: Strategic Opportunity-Analysis Agent (Tiny Aya Fine-Tuned)

Team Two-eyed Ravens: Hatem Saadallah and Nour Jennane | Lexsi AI Hackathon Track 3 (Agentic Fine-Tuning) | November 2026

1. Model Details

• Model Name: Strategic Opportunity-Analysis Agent (Fine-Tuned Tiny Aya Global)
• Base Model: CohereLabs/tiny-aya-global (pinned commit af89d21)
• Base Architecture: Compact multilingual autoregressive transformer
• Base Model License: CC-BY-NC-4.0 (Non-Commercial)
• Post-Training Method: Supervised distillation via Parameter-Efficient Fine-Tuning (PEFT LoRA) through AgentTune
• Teacher Model: Command A+ (command-a-plus-05-2026-Q4_K_M, Apache 2.0)
• Evaluation Framework: AuditKIT episode replay and double-blind LLM rubric evaluation
• Training job: a university GPU cluster (Slurm job 744559), code revision 20d3610; adapter sha256 740c4a9b…, converted GGUF sha256 41794b57…

2. Intended Use

Primary Intended Use

The model is trained to act as an agentic strategy analyst for manufacturers exploring new consumer market opportunities. Following the scientific approach to strategic decision-making (Camuffo, Gambardella & Pignataro, 2024), the agent interacts with an environment of twelve guarded tools to:

1. Frame the Strategic Problem: Clarify company capabilities, boundary constraints, and alternative paths (including declining expansion).
2. Retrieve Grounded Evidence: Query and cite public regulatory standards, clinical studies, and corporate filings verbatim, avoiding ungrounded claims.
3. Construct Competing Causal Theories: Propose at least two distinct causal mechanisms explaining who benefits, why their behavior changes, and how the firm captures value.
4. Conduct Disconfirmation Searches: Actively seek evidence that could prove its favored theory wrong.
5. Design Discriminating Experiments: Specify affordable, feasible tests with measurable behavioral metrics and threshold-based decision rules (or explicitly decline a test if further information is not cost-justified).
6. Revise Beliefs Under Evidence Shocks: Update its confidence ratings rationally when exposed to new market signals.

Target Audience

Researchers, strategy analysts, and students studying AI agent workflows, scientific decision-making, and agentic fine-tuning.

3. Out-of-Scope Use & Limitations

• Fabricated Evidence: The model must not invent customer interviews, market size statistics, or test outcomes.
• Commercial Exploitation: The base model is governed by CC-BY-NC-4.0; commercial use, resale, or commercial deployment is prohibited by the base model license.
• Real-World Business Impact: Improved automated evaluation scores indicate better adherence to a scientific reasoning protocol; they do not prove that recommendations are commercially profitable or viable.

4. Training Procedure & Hyperparameters

Training is executed via src/strategy_agent/training/train.py using AgentTune's distillation trainer (agenttune.agentic.distill_trainer.create_distill_trainer).

LoRA Configuration

• Target Modules: all-linear (all linear projection layers in attention and MLP blocks)
• LoRA rank (r): 16
• LoRA alpha (α): 32
• LoRA Dropout: 0.05
• Task Type: CAUSAL_LM

Hyperparameters

• Precision: bfloat16
• Optimizer: AdamW
• Learning Rate: 1 × 10⁻⁴
• Learning Rate Schedule: Linear warmup over 5% of steps, cosine decay
• Per-Device Batch Size: 1
• Gradient Accumulation Steps: 4 (Effective batch size: 4)
• Epochs: 2.0 (1,971 optimizer steps on 3,942 rows: 3,891 teacher rows from 158 investigations on 57 training companies, plus 51 correction rows)
• Max Sequence Length: 8,192 tokens
• Loss Masking: Supervised loss is computed strictly on target reply tokens (labels_for() in src/strategy_agent/pipeline/rows.py). Context tokens are assigned label -100. Rows exceeding 8,192 tokens are refused and logged, never truncated.

Checkpoint Selection

Validation is conducted per company across six held-out validation companies (src/strategy_agent/pipeline/splits.py: Hellberg, Neumann, Jotun, Tanita, Deere, ACR), capped at 40 evenly spaced rows per company. The retained checkpoint is the one with the lowest validation loss; in the final run that was the last checkpoint (step 1,971, validation loss 0.7125, down from 1.045 at the start).

Pre-Training and Conversion Gates

1. Overfit Gate (--check 10 --require 8): Before launching the full training job, an overfit run on 10 demonstration rows must reproduce at least 8 target replies greedily; otherwise, the job aborts.
2. Adapter Conversion Gate v2 (src/strategy_agent/evaluation/conversion_gate.py): the converted GGUF adapter is replayed against its PEFT original on the same token inputs. On every replayed point the adapter's effect must have cosine at least 0.99 with the PEFT effect over the full vocabulary and a probability-weighted residual at most 0.15, and the weight-file audit must find every factor pair equal to its expected BF16 cast. The final adapter passed (11 points, lowest cosine 0.9990, largest residual 0.080), and the GGUF evaluated is byte-identical to the one gated.

5. Evaluation Design

The plan (docs/FINAL_EVAL_PLAN.md) and every choice (docs/DECISIONS.md) were fixed before the final test, which ran once.

Arms

• Base: untrained Tiny Aya Global under the same stage controller, tools, evidence and limits, with the prompt chosen on development data (staged, over compact).
• Trained: the same model with this adapter, chosen on development data over an earlier adapter.

Measures

1. Blind LLM judging by two judges that are neither the student nor the teacher (Claude claude-opus-5-5 and Codex gpt-6-astra), each qualified on 20 matched sound/flawed pairs, on nine dimensions scored 0 to 3: problem framing, causal mechanism, distinct theories, assumptions and predictions, evidence judgement, seeking challenge, experiment design, revision, recommendation. A dimension never reached counts 0. Packets carry no model name.
2. Mechanical checks and AuditKIT diagnostics: finished, the executable checks in src/strategy_agent/env/checks.py, refused and repeated calls, tool validity, redundant calls and loop detection.
3. What-if evidence: each held-out company has a case with no new evidence, one with irrelevant evidence, and a matched pair with a supporting and an undermining simulated result.

Held-out test (docs/final_test_freeze.json, digest dce26d4f…)

Four companies (Aquaporin A/S, Electrolux Professional Group, RATIONAL AG, Simonelli Group), 16 cases, 76 records, three attempts per case and arm: 96 runs. No case was used for prompt development, teacher data or training.

6. Evaluation Results

Final test (run once, 2026-11-08)

	Base	Trained	Trained minus base, paired, 95% case-cluster bootstrap
Claude	0.56	1.20	+0.64 [0.51, 0.77]
Codex	0.74	1.77	+1.03 [0.87, 1.18]
Mean of judges	0.65	1.48	+0.83 [0.69, 0.97]
Finished investigations	4 of 48	28 of 48
Passed every mechanical check	1 of 48	1 of 48

Higher at every company under both judges (mean of judges: Aquaporin +1.10, Simonelli +0.98, RATIONAL +0.66, Electrolux Professional +0.60) and for every kind of new evidence. AuditKIT replay mismatches: 0 in every run. Reports: reports/judge/final/.

Development (6 validation companies, 15 cases)

Base 0.14 (compact prompt) and 0.43 (staged); earlier adapter 1.09; this adapter 1.29, mean of judges. Finished: 0, 0, 6 and 7 of 15. Reports: reports/judge/dev/.

Known failures (docs/FINAL_TEST_DIAGNOSIS.md)

20 of 48 trained runs stop before a recommendation (rereading to the turn limit; finishes citing only simulated evidence, refused until the error limit). Finished runs miss checks mainly because the model never declines a test (no training example declines one), mishandles supporting and undermining what-if evidence (rare in training), and leans on company claims. On Aquaporin no run of either arm found the company's insolvency in its dossier, and every finished trained run recommends investing: treat its recommendations as practice output, not advice.

What these results do not show

• Broad generalisation. The final test covers four held-out companies (Aquaporin, Simonelli, RATIONAL, Electrolux Professional), all business-to-business manufacturers asked the same family of question: whether to build a proposition for individual buyers. Four companies support only a limited claim. Other industries, other kinds of strategic question, and companies outside the dossier's style are untested. The four have since informed the failure diagnosis, so they no longer count as unseen for any later claim.
• Independent replications. Each case was run three times, but at temperature 0 (0.4 only on a retry after a refused call), so the attempts are close to repeats. The effective sample is nearer 16 cases than 48 runs; the intervals resample whole cases for that reason.
• Human expert judgement. Quality was scored by two LLM judges (Claude and Codex), each qualified first on 20 matched sound and flawed investigations. They agree closely (Spearman 0.94), but they can share biases, for example towards structure or length, and no strategy expert has scored the runs.
• Good absolute quality. The fine-tuned agent averages 1.48 on a 0 to 3 rubric. 20 of its 48 final-test runs ended before a recommendation (11 after repeated refused calls, 9 at the turn limit, 8 of them rereading sources), and its weakest dimensions remain the recommendation, revision after new evidence, experiment design and judging evidence.
• A comparison with stronger models on quality. The comparison is with the same base model under its best prompt. The teacher (Command A+) and closed models were not run on the final test for quality; Claude Opus appears only in the cost comparison.
• Calibrated confidence. Beliefs are the agent's own five-step verbal scale within one run, not probabilities. The belief check tests direction only (undermining evidence should not raise the affected theory, supporting evidence should not lower it), and the Strategy Lab's confidence score summarises the run's own judgement and the sources it weighed.
• Real-world sources. Evaluation used a fixed, curated dossier (1,334 records with short excerpts) and simulated what-if results written by the case authors. The open web was never used in evaluation; with live web search the agent reads unchecked pages, and its behaviour may differ.
• Business outcomes. No recommendation was checked against what a company later did, and no proposed test was run. The results measure the quality of the investigation, not whether its advice would have paid off.
• Robustness of training. One training run, one data set and one seed. Variance across training runs is not measured. The training data has a known defect: constrained generation cut short text fields at 240 characters, so 132 of the 158 test-design targets have a decision rule that stops mid-sentence.
• Negative controls on the final test. The scripted always-keep and always-revise policies could not run on the held-out cases (they were written for the original earplug cases), so they are reported for development only.
• The Strategy Lab's settings. The app adds a reread limit, help after refused steps, more allowed steps, draft answers when a run stops early, memory, attachments and web search. None of these were part of the evaluated setup; the reported numbers apply to the evaluated settings (45 turns, 4 refusals in a row).
• Exact reproduction from the public data. The public dossier cuts every excerpt to one sentence of at most 25 words, so runs on the published data read shorter excerpts than the originals and can differ.

Files

file	sha256
adapter_model.safetensors	740c4a9b2f415c7b83162cb463d9eaec56a34feff090d9bf882680914eb3a112
tiny-aya-global-strategy-lora-bf16.gguf	41794b5756b9e0f87c58dd14e451756d15c897d595886dee9921181298fdb9d1