Tomasz Tunguz Blog · 2024-03-03
· 823d
AI Model Performance Convergence and the Commoditization of AI Infrastructure
As AI model performance converges across vendors and model sizes on benchmarks like MMLU, the competitive advantage in the model layer will collapse, forcing differentiation to shift toward applications and developer tooling. Similar to cloud infrastructure, the AI model layer will commoditize with a few dominant providers, making open-source alternatives increasingly competitive with proprietary solutions.
Metrics in this report
Cloud Market Concentration
65%
aggregate
Top 3 cloud providers (AWS, Azure, GCP)
GPT-2 MMLU Score
0.324score
point-in-time
Released 2/14/19, 1.5B parameters
GPT-3 MMLU Score
0.539score
point-in-time
Released 6/11/20, 175B parameters
GPT-3.5 MMLU Score
0.7score
point-in-time
Released 3/15/22, 175B parameters
GPT-4 MMLU Score
0.864score
point-in-time
Released 3/14/23, 1760B parameters
OpenAI Model Parameter Growth
1000+x multiplier
range
From GPT-2 to GPT-4