Galal123/rasd-laya-track-classifier

RASD Laya track classifier

Classifies a radar/sensor track described in plain English as drone, bird, aircraft or unknown (clutter), with a calibrated probability. It replaces the hand-written rule-based classifier in RASD (Qatar Airspace Guardian), a hackathon simulation of airspace monitoring around three Qatari sites. It only classifies: it has no actions, flight control or engagement outputs.

• Base model: convaiinnovations/laya (Laya, ModernBERT-large, Apache 2.0), fine-tuned with the laya library's own finetune (soft-ce loss, then temperature calibration on a held-out slice of the training rows).
• Selected run: run-D, chosen by macro-F1 on a class-balanced validation subset (2000 rows from validation runs; the test split was not used for selection): validation accuracy 0.960, macro-F1 0.960.
• Data: Galal123/rasd-tracks-synthetic, synthetic track descriptions from the simulation. 10814 class-balanced training rows, 2 epochs, 6.4 min on one NVIDIA A100-SXM4-80GB.
• Test: the dataset's held-out test split (23358 rows from runs never seen in training), in its natural class mix.

Results on the held-out test split (measured)

Classifier	Accuracy	Macro F1	ECE (15 bins)	Drones called bird	Birds called drone
RASD Laya (fine-tuned)	0.956	0.903	0.005	264	367
Laya base (zero-shot)	0.031	0.036	0.485	0	83
Rule-based (current)	0.386	0.477	0.033	65	31

The rule-based classifier answers unknown whenever its evidence is thin; that counts as wrong for a real object. Its confidence is a heuristic score, not a probability.

Fine-tuned model per class

Class	Precision	Recall	F1	Support
drone	0.946	0.935	0.940	7031
bird	0.980	0.962	0.971	13694
aircraft	0.986	0.981	0.984	2145
unknown	0.571	0.967	0.718	488

Confusion matrix (rows = truth, columns = prediction)

	drone	bird	aircraft	unknown
drone	6574	264	15	178
bird	367	13174	8	145
aircraft	8	1	2105	31
unknown	4	5	7	472

Hard cases (accuracy; columns: RASD Laya (fine-tuned) | Laya base (zero-shot) | Rule-based (current))

Tag	Rows	RASD Laya (fine-tuned)	Laya base (zero-shot)	Rule-based (current)
unregistered_drone	3817	0.963	0.011	0.343
registered_drone	2739	0.957	0.051	0.581
camera_disagrees	835	0.902	0.090	0.290
clutter	488	0.967	0.814	0.887
loose_match	867	0.570	0.026	0.038
bad_weather	9231	0.923	0.039	0.263

Full numbers: metrics.json. Batched inference: 3.28 ms per track on the NVIDIA A100-SXM4-80GB.

Use

import json, laya
from huggingface_hub import snapshot_download

path = snapshot_download("Galal123/rasd-laya-track-classifier")
agent = laya.Agent(path)
questions = json.load(open(f"{path}/question.json"))
text = "Site: Ras Laffan Industrial City. Conditions: night. Track confirmed, age 40 s, ..."
answer = agent.predict(text, questions)["answers"]["object_class"]
print(answer["choice"], answer["probabilities"])

Input text must follow the dataset's format (built by engine/track_text.py in the project).

Training sweep (selection on validation only)

Four runs were trained in parallel; the one with the best macro-F1 on the class-balanced validation subset was published here. Test numbers above are for the selected run only. Every run is kept on its own branch of this repo.

Run (branch)	Train rows	Epochs	Loss	Val accuracy	Val macro F1
run-A	10814	2	rlcd	0.948	0.948
run-D	10814	2	soft-ce	0.960	0.960	selected

Limitations

• Trained and tested only on synthetic data from the team's simulation. These numbers say nothing about real sensors, real sites or real airspace.
• The test split comes from the same simulator and generator as training (different seeds and scenarios), so it measures in-simulation generalisation only.
• Four classes; anything else (kites, balloons, gliders...) will be forced into one of them.
• English input in the dataset's fixed format.
• A classification aid for human operators. It must not be used to take any action against an aircraft; in RASD, people decide.