Part 1 · Model Factory

Understand what weights are, how training creates them, and how product choices become a deployable model artifact.

Predict, measure, update · 9 min

How training changes weights

Training repeats a loop: make a prediction, measure error, and adjust weights to reduce future error.

Essential path

Plain language, the core causal flow, and required checks.

How a model gets built

“Training” often hides four different stages. They use different data, produce different improvements, and differ enormously in cost.

  1. Pretraining reads a very large corpus and learns broad language and world patterns through next-token prediction.
  2. Supervised tuning uses prompt-and-answer examples to teach useful instruction-following behavior.
  3. Preference training uses comparisons or feedback to favor more helpful, safe, or honest responses.
  4. Evaluation and release tests the candidate, chooses a version, and packages a deployable artifact.

As an example, a customer-support model may begin with general language ability, learn how good support answers are structured, learn that admitting uncertainty is preferable to inventing a policy, and finally pass policy, safety, and quality gates before release.

Preference training is the broad stage; reinforcement learning is one possible method inside it. RLHF trains a reward model from human comparisons and then uses reinforcement learning to improve the answering model. DPO uses preference pairs more directly without a separate reward-model-and-RL loop.

Interactive explanation

Compare model-building stages

Core + Expert

How a model gets built

See where model-building compute really goes

interactive explanation

The word training hides stages that differ by orders of magnitude in cost and by the kind of value they create.

Selected stagePretraining
What entersVery large general corpus
What happensPredict tokens and update all weights
Indicative cost$10M–$100M+
Primary outcomeGeneral capability

Pretraining usually dominates compute, but post-training often determines much of the behavior users notice. That is why “train our own model” is not a useful plan until the intended stage is named.

One weight-update step

A batch enters the forward pass. The model predicts, loss measures the error, backpropagation computes gradients, and an optimizer applies a small update. Repeating this across many batches creates the learned weights.

Interactive explanation

Simulate a distributed training run

Core + Expert

Distributed training

Watch four workers learn one consistent set of weights

interactive explanation

Each worker sees different examples, computes local gradients, synchronizes them, and applies the same update. A checkpoint is a separate saved artifact.

Worker 1Receives a different data shardbatch A
Worker 2Receives a different data shardbatch B
Worker 3Receives a different data shardbatch C
Worker 4Receives a different data shardbatch D
Training work0.1 × 10²³ FLOPs
Wall-clock estimate8.5 days
Estimated GPU compute cost$32,719Assumes $2.50 per GPU-hour

The animation also shows distributed training: workers calculate local gradients, synchronize them, update consistently, and periodically write a checkpoint. Adding GPUs reduces elapsed time only while communication, input delivery, synchronization, and utilization remain efficient.

Teaching judgement

Supervised examples show what an answer looks like. Preference data compares alternatives and teaches what “better” means. Feedback is a proxy, not truth: raters may reward confidence, agreeableness, or style instead of correctness.

The human rater does not choose between “Model A” and “Model B.” The answering model produces two candidate responses to the same prompt. The rater uses an explicit rubric to record which candidate is preferable. Many such labelled pairs become the training data for a judge or for direct preference training.

Interactive explanation

Train a preference signal

Core + Expert

Preference training

Act as the human rater, then inspect what the judge learns

interactive explanation

Candidate answers come from the answering model. Your comparison becomes training data for a reward model—the judge—not an automatic statement of truth.

Your role: human rater

Use the rubric to choose the better candidate response. Prefer evidence, honesty about uncertainty, helpfulness, and safety—not confidence or praise by itself.

Answering model → two candidates → your preference

From checkpoint to release

Training ranks write consistent state. A publication job selects serving weights, converts and shards them, attaches the architecture and tokenizer, verifies integrity, and publishes an immutable version. Inference loads that version; it does not attach to the live training job.

After this chapter, you can:
  • Trace one forward and backward training step.
  • Explain the purpose of loss and an optimizer.
  • The forward pass produces a prediction.
  • Loss measures how far it is from the target.
  • Backpropagation assigns responsibility.
  • The optimizer updates the weights.

Core check

What provides the signal used to update weights?