The End of AI Scaling
Why the next wave of innovation will be about applications, not raw power
There's a story in technology that we've seen play out many times: a breakthrough innovation leads to a gold rush of scaling up, followed by a plateau where the real value creation happens through applications rather than raw capabilities.
We saw it with microprocessors, with the internet, and with mobile.
Now we're seeing it with artificial intelligence.
The Physics of AI Progress
Imagine you're building the world's tallest skyscraper. Initially, each additional floor gives you dramatic new views of the city. But as you build higher, you face increasing engineering challenges, costs grow exponentially, and the marginal value of each new floor diminishes.
The laws of physics and economics eventually force you to stop building up and start thinking about how to best use the height you've achieved.
AI is approaching a similar inflection point. The industry has been riding a wave of what we call "scaling laws" – empirical observations about how AI models improve as we make them bigger. These laws have been as reliable as gravity: double the computing power, get a predictable improvement in capability.
It's been a clear guiding path so far: throw more computers at the problem, get better results.
But we're starting to see the limits of this approach, and it’s unclear what the next leap in performance will look like.
The Numbers Behind the Scaling Story
Let me give you a concrete example of what we're talking about. GPT-3, released in 2020, cost roughly $4-5 million to train and required about 3,640 petaflop-days of compute.
Its successor, GPT-4, reportedly cost over $100 million and used around 200,000 petaflop-days. That's a 50x increase in compute for what most observers would call a 2-3x improvement in capabilities.
This isn't just about money. OpenAI's Ilya Sutskever recently pointed out something even more fundamental: we're running out of training data. As he puts it, "we have but one internet." Unlike computing power, which continues to grow through better hardware and algorithms, the amount of high-quality human-generated text available for training these models is essentially fixed.
Think about this like farming: you can buy more tractors and hire more workers, but if you've already planted all the arable land, your yields won't increase much further.
The Real World Isn't About Raw Power
Here's where it gets interesting for businesses and technologists. While academics and AI labs debate the theoretical limits of model scaling, something more important is happening in the real world: the emergence of specialized applications that create actual business value.
Consider Bloomberg's experience. They created BloombergGPT, a specialized financial AI model trained on their vast troves of financial data. Despite their domain expertise and focused training data, they found that GPT-4, a general-purpose model,still performed better at financial tasks.
This tells us something crucial: raw scale has been winning over specialization.
But—and this is the key insight—we're approaching the point where throwing more compute at the problem won't yield proportional improvements. This is where the game changes.
The Next Wave: Agents and Applications
The future of AI innovation isn't going to come from building bigger models, it's going to come from building better applications and workflows around the models we already have. This is where the concept of AI agents becomes crucial.
Think of an AI agent like an employee rather than a calculator. A calculator gives you answers, but an employee understands context, takes initiative, learns from mistakes, and improves over time. The next wave of innovation will be about creating these kinds of AI systems that can:
Work autonomously toward defined goals
Learn from their interactions and improve over time
Understand context and adapt their behavior accordingly
Collaborate with humans and other AI systems effectively
Imagine an AI system that doesn't just answer customer service queries, but proactively manages customer relationships, identifies potential issues before they become problems, and orchestrates solutions across multiple departments.
This isn't about raw language processing power—it's about creating intelligent workflows and decision-making processes.
The Business Implications
For business leaders and technologists, this shift has profound implications. The key questions are no longer "How can we access the most powerful AI models?" but rather:
"How can we design workflows that combine AI capabilities with human expertise most effectively?"
"What specific business processes could be transformed by autonomous AI agents?"
"How do we create feedback loops that allow our AI systems to learn and improve from real-world interactions?"
The companies that will win in the next phase of AI won't be the ones with the biggest models or the most compute power—they'll be the ones that figure out how to apply AI capabilities to solve real business problems in novel ways.
Learning from History: The Database Parallel
This reminds me of the evolution of database technology. In the early days, the focus was on raw performance: how many transactions per second and how much data could be stored. But the real revolution came when people started building applications that used databases in creative ways to solve specific business problems.
The companies that won weren't the ones with the fastest databases—they were the ones that figured out how to use databases to transform business processes.
Amazon didn't win retail because they had better databases than Walmart; they won because they figured out how to use databases to create entirely new shopping experiences.
What This Means for You
If you're building AI applications or integrating AI into your business, this shift has practical implications:
Focus on workflow design over raw model capability. The key differentiator will be how well you can integrate AI into existing business processes and create new ones.
Invest in feedback loops. Create systems that can learn from their interactions and improve over time.
Think in terms of agents rather than models. Design systems that can take initiative and work autonomously toward business goals.
Don't wait for bigger models. The models we have today are already powerful enough for most applications—the limiting factor is our ability to apply them effectively.
The Road Ahead
There's a certain irony here: as AI models approach their scaling limits, they're becoming more interesting, not less. The constraints of scaling are pushing us toward more creative applications of AI technology, much like how the limitations of early computers pushed developers to create more efficient and innovative software.
The next few years will be less about breakthrough capabilities and more about breakthrough applications. We'll see fewer headlines about trillion-parameter models and more about AI systems that solve specific business problems in novel ways.
For those of us building in this space, it's an exciting time. The end of scaling means the beginning of something more interesting: the era where we figure out how to use these powerful tools to their full potential.
The future of AI isn't about bigger models—it's about smarter applications. And that's a future worth building for.