Mutual Information (MI)

Mutual Information is a metric used to measures the relationship between two quantities. It's similar to Correlation, but it can detect any kind of relationship, while correlation only detects linear relationships.


Benefits:

  • Easy to use and interpret
  • Computationally efficient
  • Resistant to Overfitting
  • Able to detect any kind of relationship

Notes:

  • Mutual information describes relationships in terms of uncertainty and measures of the extent to which knowledge of one quantity reduces uncertainty about the other. "Uncertainty" here is measured using a quantity from information theory known as "entropy"
  • Mutual information can help describe the relative potential of a feature as a predictor of the target
  • Mutual information is a univariate metric and can't detect interactions between features.
  • Depending on the model, Feature Transformation may be needed to expose the associations as usefulness of a feature depends on the model used.