Bias and Variance in Machine Learning

A brief introduction to the concept of Bias and variance and how they contribute to underfitting and overfitting in Machine Learning

Photo by Pritesh Sudra on Unsplash

Evaluation is the most important step in building a machine learning model. While evaluating a model, the error can be decomposed into two components: bias and variance. Understanding bias and variance is important to learn the concepts of overfitting and underfitting.

So what do we mean by bias and variance

Bias is the degree of error encountered on train data. If the model performs well on the train data and gives low error, it is said to have low bias. Whereas a model which is not trained well and gives high error on training data, will be having high bias.

Similarly variance is the degree of error calculated on test data. A model performing well on the test data with low error is said to have low variance. High error on test data indicates high variance.

Overfitting and Underfitting

In Machine Learning, our purpose is to create a generalized model. Such type of model should have following characteristics:

• Learns from the train data such that it finds the necessary patterns. At the same time does not over train.
• Gives good predictions on test data with low error.

If a model is trained so well on the train data that it fits each data point accurately, it will give highly accurate results on the train data but the same model will fail on test data. Thus we can say that low bias and high variance leads to overfitting of the model.

If the model does not fit well for both the train as well as the test data, the error rate will remain high for both train and test data. Such model will have high bias and high variance. This phenomenon is called underfitting.

Bias-Variance Trade-off

Since both overfitting and underfitting lead to poor performance of the model, an optimal model will be somewhere between an underfitting and overfitting model, having low bias and low variance. Overfitting is more common than underfitting.

Avoiding Overfitting

To avoid overfitting and making the model more generalized, optimizations are used in algorithms. Some of the examples are :

• In regression problems, degree of polynomial function is chosen such that the model works well for both train and test data.
• Using regularization methods such as Ridge and Lasso while dealing with complex regression problems.
• Decision tree algorithm provides option to set the depth and minimum leaf size of the tree to avoid fitting each and every point on train data.
• Using ensembling techniques such as bagging and boosting to combine results from multiple individual models to get a more generalized model.