Last updated: May 21, 2026 | Reading time: 11 minutes

What makes a large language model truly intelligent? Not the polite manners it learns in fine-tuning, not the safety rules programmed in later. The answer is pre-training — the foundational phase where a model like Claude, ChatGPT, or Gemini consumes the vast majority of its knowledge and reasoning ability.

Understanding this phase is key to decoding the entire AI industry. Why is Anthropic betting its future on Andrej Karpathy leading a team focused on accelerating pre-training? Why does a single GPT-5 training run cost half a billion dollars in compute alone? And why, after all that, do models still make embarrassing mistakes?

This article explains what pre-training actually is, why it costs so much, and why it’s the single most critical — and most expensive — phase in building AI.

What Is Pre-Training? The Model’s “Childhood”

Imagine teaching a child language and basic reasoning without ever giving them specific homework. They absorb grammar from listening to conversations, learn that rain requires an umbrella from observing the world, and develop common sense without anyone explicitly “teaching” them facts. That is pre-training.

Technically, pre-training is the initial phase of training a large language model using vast amounts of unlabeled text data — the entire public internet, digitized books, academic papers, and code repositories. The model learns to predict the next word in a sequence, and in doing so, it internalizes:

• Grammar and syntax — how sentences are structured
• World knowledge — who wrote “Hamlet,” what the capital of France is, how photosynthesis works
• Reasoning patterns — basic logic, cause and effect
• Code and structured data — programming languages, JSON, tables

As one researcher put it, pre-training is what gives a model its “common sense” — the foundational understanding that you need an umbrella when it rains.

Everything that comes after — fine-tuning, RLHF (Reinforcement Learning from Human Feedback), safety alignment — merely adjusts the surface. The core intelligence, the “IQ ceiling” of the model, is set during pre-training. This is why every frontier AI lab places pre-training at the center of its strategy.

Compare it this way:

Phase	Purpose	Data Size	Cost
Pre‑training	Learn language, knowledge, reasoning from scratch	Trillions of tokens	50M–50M–1B+
Fine‑tuning	Adapt model to specific tasks or tone	Thousands to millions of tokens	10K–10K–1M
RLHF / Alignment	Teach safety, helpfulness, refusal	Millions of preference comparisons	1M–1M–10M

Pre-training is the mountain. Everything else is the flag on top. This is why, when Anthropic announced that Andrej Karpathy would build a team to use Claude itself to accelerate pre-training research, it signaled that the company is betting on recursion — teaching AI to help build its own successor — as the path to the next breakthrough.

The Astronomical Costs: From Millions to Billions

The cost of pre-training has exploded along a curve that makes Moore’s Law look modest.

• GPT‑3 (2020): 175 billion parameters, trained on 300 billion tokens, cost approximately $4.6 million in compute alone.
• GPT‑4 (2023): Rumored to have ~1.8 trillion parameters, trained on 13 trillion tokens, estimated cost $100–200 million.
• GPT‑5 (estimated 2026): A single six-month training run is estimated to cost over $500 million in compute costs alone.

And that is just the final training run. Labs typically run dozens of failed experiments — each costing millions — before they get a model that converges. As one report noted, “multiple runs have failed to meet researchers’ expectations”.

Where does the money go?

• GPU hardware: A single Nvidia H100 costs $25,000\text{–}$25,000–35,000. A cluster of 20,000 H100s costs $500–700 million just for the chips. Add networking, storage, cooling, and power delivery, and the bill multiplies.
• Electricity: A 20,000‑GPU cluster running for six months consumes approximately 40–50 megawatts of power continuously — enough to power a small city. At industrial electricity rates, that adds $10–15 million to the bill.
• Data preparation: Curating trillions of tokens from the internet, removing duplicates, filtering low-quality content, and managing licensing is a multi‑million‑dollar effort in itself.

The scale is almost incomprehensible. The five largest US cloud providers have committed $660–690 billion in 2026 capital expenditure, approximately 75% of which is AI‑specific. The vast majority of that spending targets pre‑training capacity.

The Hidden Resource Crisis: Energy, Water, and Carbon

The cost of pre-training is not just financial. Training frontier AI models consumes staggering amounts of electricity, water, and carbon budget — often in regions already facing scarcity.

• GPT‑4’s training energy: Estimated to have consumed 50 gigawatt‑hours of electricity — enough to power San Francisco for three days.
• Water consumption: Training a model like GPT‑3 requires over 700 kiloliters of water for cooling alone, enough to fill a quarter of an Olympic swimming pool. Mistral’s Large 2 model, over 18 months of development and use, consumed 281,000 cubic meters of water and generated 20 kilotons of CO₂ — equivalent to the carbon footprint of 150 European residents for an entire year.
• Gemini’s footprint: Google measured the environmental impact of its Gemini models directly and found that between May 2024 and May 2025, energy consumption fell by a factor of 33 and greenhouse gas emissions fell by a factor of 44, thanks to clean‑energy procurement and more efficient hardware. But the absolute numbers remain enormous.

As AI models grow larger, the environmental footprint grows with them. Every frontier pre‑training run is an infrastructure event — requiring dedicated power substations, water treatment facilities, and carbon offsets that cost millions.

The Economics of Scarcity: Training vs. Inference

Here is the counterintuitive twist: while pre‑training is astronomically expensive, it is a one‑time cost. Once a model is trained, the ongoing expense is inference — the cost of running the model every time a user asks a question. And inference costs are now overtaking training.

• In 2023, training accounted for roughly one‑third of AI compute.
• By 2026, inference will account for two‑thirds of all AI compute.
• The crossover happened in 2025, when global investment in inference infrastructure surpassed training infrastructure for the first time.
• OpenAI’s monthly inference spend now exceeds GPT‑4’s training costs every 24 days.

Why does this matter for understanding pre‑training? Because pre‑training determines how efficient the model will be during inference. A well‑designed pre‑training architecture can dramatically reduce inference costs — which, over the lifetime of a model serving millions of users, can save hundreds of millions of dollars.

This is why DeepSeek’s Mixture‑of‑Experts (MoE) architecture was such a breakthrough: by activating only a fraction of its parameters per query, it delivers 70B‑class performance with the inference cost of a much smaller model. Similarly, Google’s Gemini 3.5 Flash — the new default model announced at I/O 2026 — was explicitly designed for “speed and low cost” to make agentic AI affordable at scale.

Pre‑training decisions ripple through the entire economic life of an AI system. This is why the most important strategic choice a lab makes is not which features to add, but how to architect pre‑training for the inference era.

Why Andrej Karpathy Joined Anthropic: The Recursive Pre‑training Bet

This brings us back to the news that sparked this article. When OpenAI co‑founder Andrej Karpathy — a researcher who has worked at OpenAI twice, led Tesla’s AI team, and built a legendary online AI curriculum — joined Anthropic to lead a team focused on pre‑training, it was not a minor personnel move.

Karpathy’s mandate is to use Claude itself to accelerate pre‑training research. The idea is recursive: an AI model helps design and run the experiments that will build its own successor. This could break through the current scaling law bottlenecks by making pre‑training dramatically more efficient.

Why is Karpathy the right person for this? He is one of the few researchers who bridges the gap between deep learning theory, large‑scale engineering, and product reality. As one commentator put it, “Many AI researchers only know how to run experiments and publish papers, but not how to turn models into products. Karpathy understands AI, engineering, and product”.

His arrival signals that the frontier of AI competition has shifted. It is no longer just about who has the most GPUs; it is about who can accelerate pre‑training through smarter, AI‑assisted research.

The Bottom Line: Why Pre‑training Defines the AI Race

Pre‑training is not just a technical phase. It is the economic, strategic, and intellectual center of gravity for every frontier AI lab.

• Economically, pre‑training consumes the vast majority of capital expenditure — $500 million per run and rising.
• Strategically, the architecture and data choices made during pre‑training determine a model’s “IQ ceiling” and its long‑term inference costs.
• Intellectually, pre‑training is where the hardest research problems live — data curation, scaling laws, architecture design, and now, recursive self‑improvement.

The companies that master pre‑training will dominate the AI era. Those that do not will fall behind, no matter how polished their user interfaces or how extensive their fine‑tuning.

When you hear about a new AI model, ask yourself: What happened during its pre‑training? That is where the real intelligence — and the real expense — was built.