henhua21/foursight-intent-e5

FourSight intent classifier (multilingual-e5, with conversation context)

Routes a question asked to the FourSight logistics chat assistant to one of seven intents:

id	intent	meaning
0	shipment_information	where a shipment is right now (visibility)
1	realtime_query	live lookups: ETAs, statuses, reference numbers of specific loads
2	analytics	aggregated answers over historic data (counts, trends)
3	disruptions	weather, port, border or capacity disruptions
4	chitchat	small talk with the bot
5	knowledge_base	how a product or feature works
6	other	out of scope

Code, training and evaluation: https://github.com/genzwildsoul0824/foursight-intent-classifier Training runs: https://wandb.ai/henhua21-tiktok/foursight-intent

How to use

The model reads the current question together with the previous two questions of the conversation, as a text pair. calibration.json holds the post-processing that turns the logits into the final label and confidence: a per-class bias, a transition prior on the previous turn's predicted intent, and a temperature.

The simplest way is the repo's predict.py, which does all of this:

git clone https://github.com/genzwildsoul0824/foursight-intent-classifier
cd foursight-intent-classifier && pip install -r requirements.txt
# put test.csv in data/
cd src && python predict.py --model henhua21/foursight-intent-e5

Raw model only (no calibration):

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

name = "henhua21/foursight-intent-e5"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

question = "what about last month?"
context = "how many loads were delivered late in March"  # previous turn(s), most recent first, joined with " | "
with torch.no_grad():
    logits = model(**tok(question, context, truncation="longest_first", max_length=128, return_tensors="pt")).logits
print(model.config.id2label[int(logits.argmax())])

Without the calibration, the raw model over-predicts the rare classes (see below), so use predict.py for real decisions.

Training data

1,904 anonymised user questions from the FourSight assistant, each labelled with one intent. The data is not public. It is imbalanced: realtime_query 782, knowledge_base 509, analytics 257, shipment_information 256, other 55, disruptions 28, chitchat 17. It is multilingual, mostly English. Rows are in conversation order, which is where the context comes from.

Training

• Base model intfloat/multilingual-e5-base, with a new 7-class head. Word embeddings are frozen.
• Input is (question, previous two questions) as a text pair, max 128 tokens. Context dropout 0.5: during training, half the time the model sees the question alone.
• 8 epochs, batch 16, AdamW lr 3e-5, 10% warmup then linear decay, weight decay 0.01.
• Cross-entropy with balanced class weights and label smoothing 0.1. bf16 autocast, fp32 weights.
• Seed 42 with deterministic CUDA algorithms. This model was trained on all 1,904 rows. The numbers below come from 5-fold cross-validation of the same recipe.

Evaluation

5-fold cross-validation, grouped so that duplicate questions and neighbouring turns of a conversation never fall in both the training and the validation folds. The post-processing is tuned inside the cross-validation, so the scores are not optimistic. The score is the challenge metric, OVERALL = 0.6 macro-F1 + 0.4 accuracy.

	OVERALL	macro-F1	accuracy	ECE
raw model	59.3	52.9	68.8	0.210
with calibration.json	67.7	63.8	73.5	0.035

class	precision	recall	F1
shipment_information	44.6	30.5	36.2
realtime_query	74.1	83.5	78.5
analytics	68.8	57.6	62.7
disruptions	74.2	82.1	78.0
chitchat	75.0	88.2	81.1
knowledge_base	86.7	93.1	89.8
other	25.7	16.4	20.0

needs_review: flagging predictions with calibrated confidence below 0.5 flags 13% of questions. 60% of the flagged ones are wrong (against 26.5% overall), the flag catches 29% of all errors, and accuracy on the unflagged 87% is 78.5%.

Intended use and limitations

• Intended as the first routing step of the FourSight assistant, with low-confidence questions sent to a fallback or to human review. Not intended for other domains.
• shipment_information vs realtime_query is the weakest boundary. The training labels are inconsistent there too: the same load ID appears under both labels.
• "other" is a small catch-all class (55 examples) and is often missed.
• Context comes from the previous rows. Session boundaries were not available in the training data, so a conversation's first question sometimes gets context from another conversation. With real session IDs this goes away.
• Deterministic: the same input gives the same label (argmax, no sampling). CPU and GPU gave identical labels on the test set.