Boosting

Boosting is an Ensemble Model(meta-algorithm) used in supervised learning. It uses sequential learning method to train the Weak Learners by fixing model errors in each iteration of sequence.

Train Model
Input Data
Bootstrap Sample
Bootstrap Sample
...
Bootstrap Sample
Weak Learner
Update Model
Weak Learner
Update Model
Weak Learner
Update Model
Final Model

Goals:

  • Convert weak learners to strong learners.
  • reducing bias and variance.

Algorithms:


Notes:

  • Unlike Bagging, each model's prediction may be weighted based on its accuracy.
  • Boosting improves accuracy,
  • Boosting can reduce the risk of Overfitting by reweighting the inputs that are classified wrongly.
  • Boosting can make the models more interpretable as the smaller models are easier to interpret.
  • Boosting is generally sensitive to outliers.
  • Boosting is generally computationally expensive and not suitable for real-time applications.
  • LightGBM is a popular boosting framework.