Smaller, cheaper AI can do most jobs. Do employers need LLMs?

Enterprise AI spending has climbed sharply over the past two years, with most companies betting on the largest, most capable models on the market. New research suggests much of that work could be handled by something smaller.

A Stanford University study found that small AI models running locally on a laptop or phone can now handle 88.7% of everyday chat and reasoning tasks, the kind most people use AI for day to day. The researchers also found these models have become far more efficient, delivering more than five times the performance per unit of energy compared to two years ago.

That 88.7% figure suggests many of the tasks employers are paying enterprise AI platforms to handle may not require them at all.

The problem, according to Stacey Harris, chief research officer and managing partner at Sapient Insights Group in Raleigh, North Carolina, is that most buyers don't know the alternative exists.

"It's not coming up in any of the conversations I'm having with most of these organizations," Harris said.

Buying enterprise AI on fear, not fit

David Nicholson, chief advisor at The Futurum Group in San Francisco and an instructor in Wharton's executive education programs on AI, said most senior leaders aren't really choosing between large language models, or LLMs, at all. They're choosing which existing vendor and governance setup to trust, and that choice is driven more by a fear of making a costly mistake than by a search for the best value.

"CIOs and CTOs are coming out of this fear of missing out era and they've entered into the fear of messing up era," Nicholson said.

Model decisions tend to fall inside whatever enterprise ecosystem a company already runs, whether that means Microsoft's Copilot tools or Google's Gemini, he said.

Running smaller models on local devices instead of in the cloud could cut the cost of generating AI output by roughly 80%, Nicholson said, but almost no large organization is set up to make that kind of granular, task-by-task decision yet.

Nicholson pointed to how AI providers already ask customers to make that kind of call, without giving them much to go on.

"Anthropic today gives you a drop down menu and sort of says, okay, well for most everyday tasks, select this one, for more complex tasks, select this one. And it's like, well, gosh, I don't know. I wanted to write a draft of a blog post. This one's going to cost you 4 cents and this one's going to cost you 39 cents. We don't have a good way of conceptualizing that yet," he said.

Most everyday tasks, he argued, don't call for the most powerful AI model on the market.

"If all I'm doing is walking down to the corner store, do I need to drive the minivan and burn all that gas? And the answer is no, you don't," Nicholson said.

That question hasn't hit most budgets yet because the market has been subsidizing it, he said.

"The true cost of those models has not yet been passed along to either consumers or business users at this point," he said. "As big companies start getting that sticker shock from how much they're paying for tokens, they will increasingly be asking the question, well, hold on a minute."

That sticker shock is already arriving. Brian Kropp, vice president of CPO insights and community engagement at Heidrick & Struggles in Washington, D.C., said the conversation inside companies has shifted sharply in the past several months, from driving adoption to demanding returns.

"CFOs are starting to ask the question, 'well, what is the value that we're getting out of them?'" Kropp said, noting that token costs have been rising steadily and are expected to climb further next year.

Kropp likened it to the early days of solar panels or flat panel TVs, when prices kept falling year after year until the technology matured and the improvements slowed down. AI hasn't reached that point of maturity yet, which makes long term vendor commitments a gamble.

"If we get really locked in with one vendor, then those prices could rise significantly in the future and then we're kind of stuck," Kropp said. "And then the cost of swapping out one vendor for another vendor, it's expensive and painful."

Cheaper isn't automatically safer

Harris cautioned that in HR, the case for small models is more complicated than the cost savings alone would suggest. She said Sapient Insights Group's research shows that roughly 45% of enterprise organizations used AI in a workforce process last year, a figure she expects to reach 60 to 70% this year, but employers remain deliberately careful with people data because the regulatory exposure is internal as much as external.

"Yes, the SLM might be smaller," Harris said, referring to the small language models, or SLMs, at the center of the Stanford research. "But the question is how was it developed and what are the guidelines around what's developed and who owns it."

What matters as much as the model, she argued, is the controls placed around it. Harris said embedded tools tend to come with more built-in guardrails and give employers more predictable control over token costs, which is part of why she is seeing them adopted faster than standalone tools. Separate industry research points to a similar trend of AI becoming embedded in day to day HR work.

Harris used the term "harnesses" to describe those controls, the rules that govern what a model can access and share.

"The harnesses tell me what I can do with that model," she said. "It's what gives it the context and gives it who it can talk to, what it can talk about, where it can put that data."

Her advice to buyers is to start with what the company already uses and look at what's already embedded in that technology before adding a new, separate tool.

She also argued that some of the burden belongs on the sell side. Vendors pitching AI to HR audiences should be modeling out realistic costs for companies of different sizes, and standing behind those estimates.

"If our buyers get burnt too quickly on the cost, they are going to go back to on-premise desktop solutions that are not connected, and those have a lot of risks with them," Harris said.

Whose job is this anyway?

For Kropp, the Stanford findings matter less as a technology verdict than as a governance test, one that many companies are failing because nobody owns the question. The trend lands directly on HR, as more organizations follow examples like Atlassian's chief people and AI enablement officer in merging people and AI leadership, and as CHRO pay packages continue to climb alongside their growing responsibilities.

"It tends to be one of these things where it's everybody's job to pay attention to that and nobody's job at the same time," Kropp said.

His firm's analysis found that the single biggest difference between high performing and low performing companies is that the CEOs of the top performers are the ones asking their head of HR to lead the AI transformation of work. That echoes a recent IBM Institute for Business Value study of 2,000 CEOs, in which 59% said they expect the CHRO's influence to grow in the coming years. That's why he believes organizations need a dedicated senior role, something like a VP of AI workforce transformation, sitting within the HR team.

"The reality is three months from now, six months from now, 12 months from now, there's going to be some other big, huge technology breakthrough. We just don't know what they're going to be," Kropp said.

He compared the scale of the shift to the last major disruption HR leaders lived through.

"I think the AI transformation of work will be an even bigger change and evolution and have more of an impact than everything that happened during COVID," Kropp said.