Bias-Variance Tradeoff

Understanding the fundamental tradeoff between model simplicity and flexibility in machine learning.

The Dataset

We're exploring a simple relationship: how does the number of hours of sleep before an exam affect the number of mistakes made? The true relationship is U-shaped—both too little and too much sleep lead to more mistakes, with an optimal amount around 7 hours.

What are Bias and Variance?

Before diving into the tradeoff, let's understand these two fundamental sources of error in machine learning models.

Bias

Bias is the error from overly simplistic assumptions in the learning algorithm. A model with high bias pays little attention to the training data and oversimplifies the problem.

  • Think of it as the model being "stubborn"
  • It assumes the relationship is simpler than it really is
  • Example: Fitting a straight line to curved data

Variance

Variance is the error from sensitivity to small fluctuations in the training data. A model with high variance pays too much attention to training data, including noise.

  • Think of it as the model being "too flexible"
  • It memorizes noise instead of learning patterns
  • Example: A wiggly curve that passes through every point

The Tradeoff

Here's the key insight: you can't minimize both at once. As you decrease bias (make the model more complex), variance typically increases. As you decrease variance (make the model simpler), bias typically increases.

The goal is to find the sweet spot where total error (bias² + variance) is minimized. This is what we'll explore interactively below.

Interactive Exploration

Use the controls below to adjust the polynomial degree (model complexity) and noise level. Watch how the model fit and error curves change as you explore different settings.

Higher = more flexible
Random variation
Good Fit
Train
0.00
Valid
0.00
Gap
0.00
Presets

Data and Model Fit

Dashed = true relationship, Red = model prediction

Error vs Complexity

U-shaped validation curve

Current Errors

Train vs validation

Key Concepts

Underfitting

Model is too simple. It cannot capture the underlying pattern. Both training and validation errors are high.

Good Fit

Model complexity matches the data. Validation error is minimized. The gap between errors is small.

Overfitting

Model is too complex. It memorizes noise instead of learning patterns. Validation error increases.

Try These Experiments

Experiment 1: Finding the Sweet Spot

  • Start with degree = 1 and gradually increase
  • Watch the validation error (red line) in the Error vs Complexity chart
  • Find the degree where validation error is lowest

Experiment 2: The Effect of Noise

  • Set degree = 2, noise = 0.5 and note the errors
  • Increase noise to 2.5 — validation error increases
  • Now try degree = 10 with noise = 2.5 — observe severe overfitting

Experiment 3: Understanding the Gap

  • At degree = 2, the gap between training and validation error is small
  • At degree = 12, the gap becomes very large
  • A large gap is a sign of overfitting