TensorGreed/brewslm-ntsb-accident-extractor

Qwen2.5-1.5B accident-report extractor (NTSB aviation narratives → JSON)

This is Qwen/Qwen2.5-1.5B-Instruct, fine-tuned with LoRA (merged into the weights) to turn a U.S. aviation accident narrative into one flat JSON record of 13 fields: aircraft, damage, injuries, operating rule, flight purpose, date, state and pilot hours. When the narrative doesn't state a field, it writes null instead of guessing.

For education only. This model is a demonstration of a training method. It is not an aviation-safety or accident-investigation expert, and it makes no claim to subject-matter expertise. It can be wrong, and it is not a source of NTSB findings. For any real purpose, use the official NTSB records at data.ntsb.gov. Don't use its output as evidence of what caused an accident (see Limitations).

Trained with BrewSLM, a platform for building small language models from your own data. Every step below (import, splits, training, evaluation, packaging) ran through it. How it was built, including where it still misses: Narrative in, JSON out: a 1.5B accident-report extractor that knows when to say null, and more on the BrewSLM blog. Non-commercial showcase; no warranty.

Update 2026-10-26: decoding fix

This repo first shipped the base model's generation_config.json, which sets repetition_penalty: 1.1. Greedy decoding (do_sample=False) still applies it. By the aircraft_model field the model has already written " several times, so that token is penalised, while null hasn't been written yet and wins close calls. The penalty, not the training, caused most of the "returns null for stated aircraft models" weakness. The repo now ships greedy decoding with no repetition penalty, and the snippet below passes repetition_penalty=1.0 explicitly. The weights are unchanged.

Same weights, all 1,256 validation narratives, BrewSLM's scorer:

	Old decoding (penalty 1.1)	Fixed decoding (no penalty)
Stated aircraft models returned as null	232 of 468	16 of 468
aircraft_model exactly right	0.761	0.891
Fields exactly right (13 fields)	0.962	0.977
Null precision	0.955	0.994
Null recall	0.982	0.972
All 13 fields right	0.639	0.785

The test numbers in the rest of this card were all measured with the old decoding and haven't been re-run. The pass rule allowed one test look per seed, and those looks are spent. These validation numbers are a check made afterwards, not a result the rule covered. A new test result will come with v2, under a rule written before it is measured. Null recall drops slightly with the fix, because the model now gives a value more often. For the aircraft model, every value it gives where the answer key says null (77 on validation) appears in the narrative; the key is what's wrong there (see Limitations).

Result

1,234 held-out test narratives, decoded with the old setting (repetition penalty 1.1; see the update above). Each output is parsed as JSON and compared field by field against the answer key. Comparison is typed: numbers as numbers, strings ignoring case and spacing, and null / missing / "" all count as null.

	Base model, same prompt	Base model, schema-aware prompt¹	This model
Output matches the schema (valid JSON, every field, right types and allowed values)	0.042	0.770	0.995
Fields exactly right (13 fields, all rows)	0.494	0.481	0.959
When it says null, the field really is unstated (null precision)	0.901	0.533	0.954
When a field is unstated, it says null (null recall)	0.545	0.371	0.976

¹ A 300-row subset of the test set. The prompt adds the chat template, the types and allowed values, and "use null when not stated". It fixes the format but makes the base model invent more, for example guessed dates and "Not specified" strings. Prompting alone doesn't give you this behaviour. (This one column was run by a separate script, scored with BrewSLM's own scorer; everything else ran inside BrewSLM.)

Per field (seed 42, test):

field	exact (all rows)	right when the text states it (rows)	null recall	base exact
aircraft_category	0.986	0.991 (1220)	0.571	0.120
aircraft_make	0.917	0.811 (508)	0.992	0.777
aircraft_model	0.752	0.388 (454)	0.964	0.702
damage	0.977	0.990 (1155)	0.785	0.490
event_date	0.994	0.996 (547)	0.991	0.796
flight_purpose	0.960	0.975 (563)	0.946	0.089
highest_injury	0.985	0.995 (567)	0.978	0.556
injuries_fatal	0.984	0.991 (567)	0.978	0.581
injuries_minor	0.977	0.982 (564)	0.973	0.507
injuries_serious	0.980	0.982 (567)	0.978	0.569
location_state	0.991	0.991 (555)	0.991	0.256
pilot_total_hours	0.970	0.765 (85)	0.985	0.574
regulation_part	0.991	0.993 (542)	0.990	0.406

How much to trust these numbers:

• The pass rule was written before any model was scored, and frozen with a checksum. It was amended twice before any evaluation ran: the paired gain is measured on per-row field F1, because per-row exact match isn't stored; and the published seed is the best on validation. Three seeds (13, 42, 1234) were trained on identical data, and each was evaluated on test once. All three had to pass every gate: schema-valid ≥ 0.95, overall field accuracy ≥ 0.85, null precision ≥ 0.95, null recall ≥ 0.90, per-field accuracy ≥ 0.70 and per-field null recall ≥ 0.80 on the 8 fields a narrative states directly, and a paired gain over the base model whose 95% CI excludes 0. The published seed is the one with the best validation score, chosen before any test result existed.

  Seed	Schema-valid	Field accuracy	Null precision	Null recall	Paired gain vs base, per-row field F1 (95% CI)	Val field accuracy
  13	0.994	0.958	0.955	0.976	+0.444 (+0.430 to +0.458)	0.9595
  42 (this model)	0.995	0.959	0.954	0.976	+0.444 (+0.430 to +0.458)	0.9614
  1234	0.997	0.957	0.952	0.977	+0.443 (+0.429 to +0.457)	0.9610

• What "right" means here. The answer key is the investigators' coded value, kept only when a fixed text check finds it stated in the narrative, and set to null otherwise. So the score measures "extracts what the text says". It doesn't measure "knows the true facts of the accident". A check can miss a value that is phrased unusually. The model is then marked wrong for extracting it, so the scores lean conservative.

• The 5 category fields aren't gated. aircraft_category, damage, highest_injury, regulation_part and flight_purpose map free wording onto fixed codes. They are reported above but weren't part of the per-field pass rule.

• No language model wrote any of the data, training or test. Every label comes from the NTSB's own coded fields and a deterministic text check.

How the data was made

• Source: the NTSB aviation accident database (bulk download from data.ntsb.gov/avdata, October 2026 snapshot).
• Population: accidents (not incidents) from 2015 on, with exactly one aircraft and a narrative that includes the probable cause, at most 14,000 characters: 12,421 events.
• Text: the published narrative sections joined and cleaned: the long narrative where there is one (70% of events), the summary narrative, and the probable-cause statement, with repeated paragraphs dropped.
• Answer key: for each field, the investigators' coded value, set to null unless a deterministic check finds it stated in the text. Examples: the make or model appears as written; the injury counts are stated; the date appears in the text (if the text's date is a day off the coded one, the text wins); total hours appear. Most narratives don't state most fields: only 7% state pilot total hours, and about 40% name the make.
• Splits: by a hash of the NTSB event id: 9,931 train, 1,256 validation, 1,234 test. No event is in two splits, and no narrative text is shared between splits.

Training

LoRA r 16, alpha 32, dropout 0.05, on every linear layer. bf16, AdamW, learning rate 2e-4 with cosine decay, warmup 3%, effective batch 16. Loss is on the JSON answer tokens only. Max sequence length is 4,096; the longest row is 3,366 tokens, so nothing is truncated. 1 epoch, 621 steps. Final train / eval loss for seed 13 is 0.010 / 0.008; for seed 42 (this model) 0.009 / 0.008. Trained on one NVIDIA GB10. The adapter was merged into the base weights.

Prompt format

The model was trained on a plain prompt with no chat template. Use it exactly; other layouts were never trained or tested.

Extract the following fields as JSON: aircraft_category, aircraft_make, aircraft_model, damage, event_date, flight_purpose, highest_injury, injuries_fatal, injuries_minor, injuries_serious, location_state, pilot_total_hours, regulation_part.
Reply with a single JSON object, nothing else.
Input: <narrative>
Output:

import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "TensorGreed/brewslm-ntsb-accident-extractor"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

FIELDS = ("aircraft_category, aircraft_make, aircraft_model, damage, event_date, flight_purpose, highest_injury, "
          "injuries_fatal, injuries_minor, injuries_serious, location_state, pilot_total_hours, regulation_part")
narrative = "..."  # one accident narrative
prompt = (f"Extract the following fields as JSON: {FIELDS}.\nReply with a single JSON object, nothing else.\n"
          f"Input: {narrative}\nOutput:")
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False, repetition_penalty=1.0)
record = json.loads(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Output schema (allowed values; null is always allowed):

field	type	allowed values
aircraft_category	string	airplane, helicopter, glider, balloon, gyroplane, weight_shift, powered_parachute, other
aircraft_make, aircraft_model	string	as written in the narrative
damage	string	destroyed, substantial, minor, none
highest_injury	string	fatal, serious, minor, none
injuries_fatal / _serious / _minor	integer
regulation_part	string	part_91, part_121, part_135, part_137, other
flight_purpose	string	personal, instructional, aerial_application, business, positioning, other
event_date	string	YYYY-MM-DD
location_state	string	two-letter U.S. state code
pilot_total_hours	integer	rounded

Limitations

• One source, one writing style. It was trained and tested only on NTSB narratives written since 2015, for accidents involving one aircraft. Other agencies' reports, older reports, incidents and multi-aircraft accidents weren't tested.
• It extracts; it doesn't judge. The answer key covers what the narrative states, not what is true. A value the narrative gets wrong will be extracted wrong.
• Not evidence. Under 49 U.S.C. § 1154(b), NTSB accident reports can't be admitted as evidence in a civil action for damages. A model trained on them is no better: don't use its output to argue cause or liability.
• Aircraft model: mostly fixed by the decoding update. With the old decoding, for 454 test narratives that state the model designator it returned null for 232 and a different designator for 46. With the fixed decoding, on validation it returns null for 16 of 468 and a different designator for 44. Most of those 44 are span boundaries: the answer key says 206 and the model writes "206 L-1", or the key says STOL CH 750 and the model writes "Zenith STOL CH 750".
• The answer key undercounts aircraft models. It keeps the coded designator only when the text contains it in full. When the text is more specific than the code (code 172, text "Cessna 172N"), the key says null. On validation with fixed decoding, all 77 aircraft-model values the key marks as invented appear in the narrative (ignoring case, spaces and hyphens). v2 will fix the key.
• Null precision was 0.954 on test with the old decoding, just over its 0.95 floor. With the fixed decoding it is 0.994 on validation.
• Whole records. With the old decoding, all 13 fields were right on 62% of test narratives (base model: 0%). With the fixed decoding, 78.5% on validation.
• Educational demonstration only, with no claim to aviation or investigation expertise. Use the official NTSB records for anything real.

Citation

Trained and evaluated with BrewSLM. The full story is in the build write-up on the BrewSLM blog. If you reuse the method, a link back is appreciated.

License

The model is Apache 2.0, the base model's license.

The training text is NTSB material: text prepared by U.S. Government employees as part of their official duties, which the NTSB states it "has placed … in the public domain" (NTSB policies; 17 U.S.C. § 105). Courtesy: National Transportation Safety Board. The answer key is a masked derivative of the NTSB's coded fields, not NTSB findings. The model is not endorsed by or affiliated with the NTSB.