Loss Functions

A loss function measures how the predicted value deviates from the true value and shows the ability of a model to estimate the relationship between model’s output and actual value using a mathematical formula. Loss Function can be in form of squared difference or absolute difference.

Loss Function in Machine Learning: While training an Artificial Neural Networks (ANN), input is multiplied by assigned weight and Bias value is added to it, then one of Activation Functions is applied to it, and finally the output is passed to next layer and so on, until final output layer is reached. this is done in many iterations/epochs to optimize model by getting minimum deviation between training values and model’s output. the method to calculate this deviation is called a Loss Function.

Categorization and algorithms:

  • Classification: which is about predicting a label, by identifying which category an object belongs to based on different parameters.
    • Cross Entropy
    • Binary Cross-Entropy: used for binary classification.
      • Hinge Loss
      • Squared hinge loss: Squared Hinge Loss for numerical values to reduce the error function. It identifies the categorization border that establishes the largest possible difference between data points of different classes.
      • Kullback Leibler Divergence Loss (KL Loss)
    • Multi-class Classification
      • Multi-class Cross-Entropy: Calculates accuracy score by averaging the differences between actual and anticipated probability values. It often uses One-Hot Encoding to create numerical values from categorical data, however it’s not suitable for a high number of data points.
      • Multi-class Sparse Cross-Entropy: Similar to Multi-class Cross-Entropy, however it doesn’t use One-Hot Encoding and doesn’t have it’s problems.
    • Relative Entropy
    • Exponential Loss
    • Triplet loss: similar to Hinge Loss but with three inputs(anchor, positive, and negative). it’s used in Image classification and Word embedding. The objective is to Minimize the distance between the anchor and the positive image and maximize it between the anchor and the negative image.
    • Focal Loss: Application in Object Detection
  • Regression: which is about predicting a continuous output, by finding the correlations between dependent and independent variables.
  • GAN
    • Discriminator loss
    • Minmax GAN loss
  • AutoEncoder
    • KL Divergence: it measures how two different probabilities are different and is used in Variational Autoencoders.
  • Input Loss
    • Contrastive Loss: It is a distance-based Loss Function, which measures the distance or similarity between two inputs.