Ridge Regression

Similar to Lasso Regression, Ridge Regression adds a penalty to basic regression procedure, with the penalty being calculated as equivalent to the square of the magnitude times lambda, causing all the coefficients to shrink toward zero.

Ridge Regression controls the magnitude of polynomial coefficients by introducing the parameter alpha(). Selecting alpha value is important because If alpha is too large, the coefficients will approach zero and underfit the data, and If alpha is zero, it'll overfit the data.


Notes:

  • Ridge regression is used when data suffers from Multicollinearity (independent variables are highly correlated).
  • Ridge regression penalizes the squares of regression coefficients but doesn’t allow the coefficients to reach zeros (It uses L2 regularization).
  • It is not good for feature reduction, because does not reduce the number of variables.