Model Complexity: Model Complexity | Saylor University

Model Complexity

A model predicting house prices might perform exceptionally well on historical data but struggle to accurately predict new properties. Similarly, in fraud detection, a model trained on past fraudulent activities might fail to detect new, unknown forms of fraud. These examples illustrate how overfitting can hinder a model's ability to generalize to new, unseen data. In this section, you will explore strategies to mitigate overfitting, such as controlling model complexity, applying regularization techniques, and interpreting loss curves.

As you read, reflect on these questions: What does it mean if training loss decreases while validation loss increases? How can early stopping or tuning hyperparameters help mitigate overfitting? How do loss curves help distinguish between underfitting and overfitting?

The previous unit introduced the following model, which miscategorized a lot of trees in the test set:

Figure 16. The same image as Figure 13. This is a complex shape that
miscategorizes many trees. — **Figure 16.** The misbehaving complex model from the previous unit.

The preceding model contains a lot of complex shapes. Would a simpler model handle new data better? Suppose you replace the complex model with a ridiculously simple model--a straight line.

Figure 17. A straight line model that does an excellent job
separating the sick trees from the healthy trees. — **Figure 17.** A much simpler model.

The simple model generalizes better than the complex model on new data. That is, the simple model made better predictions on the test set than the complex model.

Simplicity has been beating complexity for a long time. In fact, the preference for simplicity dates back to ancient Greece. Centuries later, a fourteenth-century friar named William of Occam formalized the preference for simplicity in a philosophy known as Occam's razor. This philosophy remains an essential underlying principle of many sciences, including machine learning.

Exercises: Check your understanding

You are developing a physics equation. Which of the following formulas conform more closely to Occam's Razor?
1. A formula with twelve variables.
2. A formula with three variables.
  Answer: A formula with three variables.
  
  Three variables is more Occam-friendly than twelve variables.
You're on a brand-new machine learning project, about to select your first features. How many features should you pick?
1. Pick 1–3 features that seem to have strong predictive power.
2. Pick as many features as you can, so you can start observing which features have the strongest predictive power.
3. Pick 4–6 features that seem to have strong predictive power.
  Answer: Pick 1–3 features that seem to have strong predictive power.
  It's best for your data collection pipeline to start with only one or two features. This will help you confirm that the ML model works as intended. Also, when you build a baseline from a couple of features, you'll feel like you're making progress!

Regularization

Machine learning models must simultaneously meet two conflicting goals:

Fit data well.
Fit data as simply as possible.

One approach to keeping a model simple is to penalize complex models; that is, to force the model to become simpler during training. Penalizing complex models is one form of regularization.

Loss and complexity

So far, this course has suggested that the only goal when training was to minimize loss; that is:

\(\text{minimize(loss)}\)

As you've seen, models focused solely on minimizing loss tend to overfit. A better training optimization algorithm minimizes some combination of loss and complexity:

\(\text{minimize(loss + complexity)}\)

Unfortunately, loss and complexity are typically inversely related. As complexity increases, loss decreases. As complexity decreases, loss increases. You should find a reasonable middle ground where the model makes good predictions on both the training data and real-world data. That is, your model should find a reasonable compromise between loss and complexity.

What is complexity?

You've already seen a few different ways of quantifying loss. How would you quantify complexity? Start your exploration through the following exercise:

Exercise: Check your intuition

So far, we've been pretty vague about what complexity actually is. Which of the following ideas do you think would be reasonable complexity metrics?

Complexity is a function of the square of the model's weights.
Complexity is a function of the biases of all the features in the model.
Complexity is a function of the model's weights.
Answer: (1, 3)
- Yes, you can measure some models' complexity this way. This metric is called L₂ regularization.
- Yes, this is one way to measure some models' complexity. This metric is called L₁ regularization.

Source: Google for Developers, https://developers.google.com/machine-learning/crash-course/overfitting/model-complexity
This work is licensed under a Creative Commons Attribution 4.0 License.