Tomasz Tunguz Blog · 2024-03-03 · 823d

AI Model Performance Convergence and the Commoditization of AI Infrastructure

As AI model performance converges across vendors and model sizes on benchmarks like MMLU, the competitive advantage in the model layer will collapse, forcing differentiation to shift toward applications and developer tooling. Similar to cloud infrastructure, the AI model layer will commoditize with a few dominant providers, making open-source alternatives increasingly competitive with proprietary solutions.

6 metrics· Cited 0× in the knowledge base ·Open source ↗

Metrics in this report

Cloud Market Concentration

65%

aggregate

Top 3 cloud providers (AWS, Azure, GCP)

GPT-2 MMLU Score

0.324score

point-in-time

Released 2/14/19, 1.5B parameters

GPT-3 MMLU Score

0.539score

point-in-time

Released 6/11/20, 175B parameters

GPT-3.5 MMLU Score

0.7score

point-in-time

Released 3/15/22, 175B parameters

GPT-4 MMLU Score

0.864score

point-in-time

Released 3/14/23, 1760B parameters

OpenAI Model Parameter Growth

1000+x multiplier

range

From GPT-2 to GPT-4