laya-erp-text2sql-rag-router

A fine-tune of convaiinnovations/laya (ModernBERT-large encoder) that routes ERP chatbot questions to one of two answering pipelines:

Label	Meaning	Example
TEXT_TO_SQL	Needs live ERP records: amounts, totals, counts, statuses, lists, reports	"What is my material receipt amount?"
RAG	Needs company documents: definitions, processes, how-tos, policies	"What is a material receipt?"

The model was trained to route by intent, not topic: every ERPNext doctype and business term in the training data is asked both ways.

Results

Eval set	Base convaiinnovations/laya	This model
Held-out templates, doctypes and terms (600)	62.5%	98.3%
Handwritten realistic questions (100)	64%	94%

Inference takes about 0.6 s per question on CPU. The training-time validation score (100%) is left out because that split comes from the same templates as training.

Usage

pip install laya==0.4.2

import laya

agent = laya.load("harshagale/laya-erp-text2sql-rag-router")

# Use this exact question. The model was fine-tuned on this wording.
QUESTIONS = {"route": {
    "type": "choice",
    "instructions": ("Route an ERPNext chatbot question by the source of information needed to answer it. "
                     "Focus on what the user wants, not on the business terms used."),
    "criteria": {
        "TEXT_TO_SQL": ("Needs live ERP records: the user's own documents, specific records or IDs, "
                        "amounts, totals, counts, balances, statuses, lists or reports."),
        "RAG": ("Needs company documents: what a term means, how a process or feature works, "
                "how to do something in ERPNext, policies, rules and troubleshooting."),
    },
}}

result = agent.predict({"text": "How many invoices are overdue?"}, QUESTIONS)["answers"]["route"]
print(result["choice"], result["confidence"])   # TEXT_TO_SQL 0.99...

The full chatbot, which routes with this model and answers with Gemini (text-to-SQL over SQLite, or RAG over policy docs), is in the project repository.

Training

• Data: about 3,000 synthetic, class-balanced questions built by crossing ERPNext doctypes (Sales Invoice, Material Receipt, BOM, …) and business terms with question templates. Some templates, doctypes and terms were held out entirely for the test set.
• Procedure: a full fine-tune (encoder not frozen) with laya.train_cli: --loss soft-ce --shuffle-options --epochs 4, encoder LR 2.5e-5, head LR 1e-4, effective batch size 64, bf16, on one T4 GPU (about 2 hours).

Limitations

• The training questions are synthetic and ERPNext-flavoured. Other ERPs, other languages, and phrasings far from the templates may route less accurately.
• Confidences are not calibrated. The calibration slice was answered perfectly, so temperature scaling had nothing to correct, and the model reports about 0.99 even on its mistakes. Don't use the confidence as an abstain threshold without recalibrating on your own data.
• A known failure mode: policy questions phrased as counts, e.g. "How many leave days can I carry forward?", tend to go to TEXT_TO_SQL.

License

Apache-2.0, the same as the base model convaiinnovations/laya and its encoder answerdotai/ModernBERT-large.