LLMs Are Cool and All, But Let's Not Forget About Cost-Effective Machine Learning
Most of what a business wants out of AI does not require a language model to get there.
I spend most of my week inside large language models. Most of what I publish on this profile is QLoRA fine-tuning work on Qwen, and the agent systems I've shipped run on small, fine-tuned checkpoints. So read this as criticism from inside the tent: many of the business problems I get asked about don't need a language model at all, and the ones that do anyway tend to be quietly switched off eighteen months later.
The first question I usually get is some version of "well, the LLM can do it, so why wouldn't I use the LLM?" Because in most cases, it isn't doing what you think it is. Strip a business problem down to what actually has to come out the other end, and the language generation part is usually a thin layer on top, or it isn't there at all. Most of the targets I get handed are reachable with methods that were mature in 2016, and reaching them that way tends to produce a system that survives contact with production.
Start from the output, not the model
Here's the check I run before writing any code. Look at what leaves the system, not what goes into it.
If the output is a number, that's regression. A category, classification. A box around something in a video frame, detection. A ranked list, a ranker. A score that says "this one looks nothing like the other forty thousand," anomaly detection. Only when the thing leaving the system is language that a human reads and acts on do you actually have a language problem.
That distinction gets blurred because the input to all of these is often text, and text feels like LLM territory. It isn't. Classifying an incoming email into one of six queues is a classification problem that happens to have text as a feature. Embeddings plus logistic regression solved it years ago, it runs in milliseconds, and you can show a compliance auditor exactly which features drove the decision.
The systems version of this: keep the language model at the seams, where human language enters and leaves your architecture, and keep the middle deterministic. Built that way, the pipeline degrades gracefully. When the model at the edge is unavailable you still have structured output to fall back on. A pipeline with an LLM sitting in the middle just stops.
Where the boring models still win
Rows and columns. Most business data is tabular, and tabular is exactly where the classical stack refuses to die. Grinsztajn and colleagues asked in a NeurIPS 2022 benchmark paper why tree-based models still outperform deep learning on typical tabular data. McElfresh and colleagues followed up in 2023 with 19 algorithms across 176 datasets and found the neural-versus-boosted-trees debate is overstated: for a large share of datasets the performance gap is negligible, and light hyperparameter tuning on a GBDT matters more than which family you picked. Tabular foundation models are making real progress, and I follow that work closely, but as of the TabBench V2 results this summer, boosted trees still take the trophy on smaller datasets with many categorical features. Churn, credit risk, demand forecasting, claim triage, lead scoring, fraud. CatBoost gets you to production faster, and you can explain it afterward.
Pixels. Video review shows the layering clearly. YOLO handles detection, something like ByteTrack keeps an identity attached to the same object across frames, and a language model only earns its place at the very end if what you want out is a written summary. Finding people and vehicles across thousands of frames is a geometry problem, and geometry is cheap. Turning eleven tracked objects into a paragraph a security manager will actually read is a language problem, and that narrow slice is the part worth paying token prices for. Push every frame through a vision language model instead and you have paid a hundred times more for a less precise answer.
Deviation from normal. Isolation Forest, an autoencoder, a seasonal decomposition with a control limit. Fast, cheap to retrain, and the maintenance team can see why the alarm fired.
The pattern across all three: the LLM was never the thing producing the value. It was the interface on top of something else that was producing the value.
The failure statistics are not telling you the models are bad
The numbers on enterprise AI have gotten grim enough to take seriously. MIT's Project NANDA reported in July 2025 that roughly 95% of organizations deploying generative AI saw no measurable P&L impact, based on 300 public deployments and 150 leadership interviews. S&P Global found 42% of companies scrapped at least one AI initiative in 2025, up from 17% the year before. RAND put overall AI project failure above 80%, about double the rate of conventional IT work. Gartner has been forecasting that through 2026, 60% of AI projects get abandoned for want of AI-ready data and integration infrastructure.
Look at the root causes those studies land on. Data readiness. Workflow integration. No agreed definition of success before the build started. Executive sponsorship that evaporated when results came back modest. Not one of those is an algorithm problem. In my experience the model is almost never the constraint. The constraint is that the pipeline can only refresh overnight, or that "active customer" means three different things in three different systems, or that nobody decided who owns the thing after go-live.
So if the failure mode is upstream of the model, the model should be the simplest and cheapest thing that clears the accuracy bar. Every week you spend on inference sophistication is a week you didn't spend on the part that actually kills projects.
And then there's the bill
Cost isn't my main argument, but it's the one that makes this hard to ignore once the pilot goes to volume.
An LLM call is a variable cost that never stops. Every prediction burns tokens, and output tokens typically run four to ten times the price of input tokens, so a chatty model on a high-volume task compounds against you. Self-hosting doesn't make that go away; it just changes who owns the meter. Public B200 list rates in April 2026 ran roughly $3.49 to $14.24 per GPU hour depending on the cloud, and the break-even math on self-hosting only works if you keep utilization high, which most enterprise workloads with a daily traffic peak cannot.
Now price the boring version. A gradient-boosted tree trained on eighteen months of your own transaction history costs a few dollars of compute one time. After that, it serves predictions on a CPU you already bought, in single-digit milliseconds, at a marginal cost that rounds to zero. Same for a YOLO detector on a $250 edge box. The expensive part of that project is the data pipeline, and you were paying for it either way.
Paying per token for a repeated structured decision is like keeping a translator on retainer to read the same one page intake form ten thousand times a day. The translator is genuinely good at their job. You needed a checkbox.
The regulated environment argument
In a regulated industry, "explain why the model said that" is not a nice-to-have conversation, it's the whole conversation. A tree gives you feature importances, a fixed decision threshold, and a training run you can reproduce for an auditor. It also gives you determinism, which means you can write regression tests that fail loudly when something drifts.
Put a nondeterministic language model in that same slot and you've signed up for an evaluation harness, a hallucination detection strategy, and a human review loop before you've shipped any business value. I know the size of that bill because I built it. On my travel risk work I ended up running a three-judge LLM ensemble just for hallucination detection, on top of ROUGE-L scoring and a structured adherence harness. Worth it there, because the deliverable genuinely is prose. It would be an absurd amount of scaffolding to wrap around a yes or no decision.
When you do need language, size it to the task
None of this is an argument against language models. It's an argument for using one when language is the deliverable, and for reaching for the smallest one that clears the bar when it is.
My OSINT travel risk agent runs on fine-tuned Qwen2.5 checkpoints between 0.5B and 3B parameters, because the output really is a written brief and nothing smaller than a language model produces that. Fine-tuning a small model on domain data got the quality where it needed to be without a frontier API in the loop.
Nobody is going to invite you to keynote a conference for putting a logistic regression into production. The finance team will notice, though. That's usually the win.
Sources
- MIT Project NANDA, The GenAI Divide: State of AI in Business 2025 (July 2025), via Fortune coverage
- S&P Global Market Intelligence AI initiative abandonment data (2025), and RAND / Gartner failure and abandonment forecasts, summarized at Pertama Partners
- Grinsztajn, Oyallon and Varoquaux, Why do tree-based models still outperform deep learning on typical tabular data? (NeurIPS Datasets and Benchmarks, 2022)
- McElfresh et al., When Do Neural Nets Outperform Boosted Trees on Tabular Data? (NeurIPS 2023)
- When trees still beat tabular foundation models, TabBench V2 analysis (June 2026)
- GPU hour and per-token inference economics: Inworld, LLM Inference Cost at Scale (July 2026)