3.201
Effects of Multicollinearity in Regression

---

Contents at a Glance

• Motivation
• Assumptions of Regression Methods

---

Motivation

Multicollinearity is an important issue that must be addressed when developing machine learning models, particularly regression models – and especially when using ordinary least squares (OLS) regression. Unfortunately, in practice many data analytics professionals ignore multicollinearity and consequently create models that perform poorly, i.e., resulting in predictions with high variance and poor accuracy. The result is that the predictive model is not useful.

The issue may be that many data analytics professionals do not have a mathematics, statistics, or computer science background and therefore are often not aware of some of the critical underpinnings and assumptions of many machine learning algorithms. They often falsely assume that if the algorithm produces a model, that the model is “good”. Nothing can be further from the truth; just because R or Python didn’t crash doesn’t mean that one has built a useful model that makes usable predictions.

This tutorial takes a closer look at a particularly pernicious problem most often encountered when building a regression model.

Assumptions of Regression Methods

There are several different ways to build a regression model: a linear equation relating the features and weights in an additive manner.

Regression makes the assumption that the features (the independent variables) and the target variable (the dependent variable) all are reasonably normally distributed. In other words, they follow a Gaussian or Gauss-Laplace Distribution. If a variable is not normally distributed, as established through visual inspection of the variable’s histogram or a statistical test such as the Shapiro-Wilk Normality Test or the Kolmogorov–Smirnov Test, then one or more transforms are applied which may change the distribution just enough to be meet the normality criterion.

Another important assumption is the assumption of equal variance, or heteroscedasticity. If the variance is uneven, then there are likely clusters and different regression models must be constructed for the different clusters. Applying a clustering algorithm such as k-means can often reveal the clusters computationally rather than through visual inspection.

A third assumption is that the variables are independent and are not correlated to one another but rather only to the target variable. This assumption is the topic of this tutorial.