Is AI Progress Driven by Compute?

When we look at the explosive progress of artificial intelligence over the last decade—particularly the breakthroughs in large language models (LLMs) and multi-modal AI systems—a common question arises:

Is all of this simply due to more compute?

The short answer: Compute is necessary, but not sufficient. Let’s unpack this by looking at the three core forces behind AI’s progress.

1. Compute: The Engine Room of Modern AI

Compute is the infrastructure—the foundation that makes it possible to train increasingly complex and capable models.

• In 2012, AlexNet won the ImageNet competition and reignited the deep learning revolution. A big part of that success came from training on GPUs.
• From GPT-2 to GPT-4, we’ve seen a ~100x increase in training FLOPs (floating-point operations). GPT-4 reportedly consumed tens of exaflops over weeks—something unimaginable a few years ago.

Without massive compute, models like GPT-4, Claude 3.5, and Gemini would simply not exist.

2. Data: The Fuel of Machine Learning

Compute is meaningless without data. AI models need large-scale, high-quality data to learn patterns and reason about the world.

• GPT-style models are trained on huge swaths of the internet—code, books, articles, forums.
• Multi-modal models require diverse inputs: text, images, audio, video, and beyond.

As the saying goes, garbage in, garbage out. Better data = better models.

3. Algorithms: The Real Game-Changer

Data and compute are the raw materials. Algorithms are the design—they determine what’s possible with those resources.

• The introduction of the Transformer architecture in 2017 is what enabled all modern LLMs.
• Techniques like RLHF (Reinforcement Learning from Human Feedback), LoRA (Low-Rank Adaptation), and MoE (Mixture of Experts) have improved model quality and efficiency.
• Quantization, pruning, and distillation help models run faster and smaller—critical for deployment.

Often, a better algorithm can outperform raw compute.

So Where Does AI Progress Really Come From?

Let’s simplify it:

Compute is the foundation, data is the fuel, and algorithms are the engine. AI progress requires all three.

However, in the last 5 years, compute has been the key accelerator—giving us the ability to scale up everything else.

Looking Ahead

Here’s how the landscape is shifting:

While we still need huge compute to train frontier models, there’s also a wave of innovation focused on “small but smart” AI—running models efficiently, even on the edge.

Final Thoughts

If you’re a developer or indie hacker, here’s what matters:

• You don’t need a supercomputer to build valuable AI products.
• Focus on understanding users, leveraging open models, and solving niche problems.
• The real competitive edge lies in how you apply AI, not just how powerful your model is.