tags: [AI/EvaluationMetrics, ]
Evaluation metrics are used to measure the performance of the machine learning models.
In following equations:
Error: The difference between the real value and the predicted value is called the error.
Mean Squared Error(MSE): presents he square of the error over all samples.
Root Mean Squared Error(RMSE): RMSE is the most used regression metric and measures the differences between the predicted and actual values.
Root Mean Squared Log Error(RMSLE): Similar to RMSE while transforming the predicted and real dependent variable into a logarithmic value.
Mean Absolute Error (MAE): sensitive to outliers.
Mean Absolute Percentage Error
Mean Squared Logarithmic Error
Sum of Squared Errors (
Total Sum of Squares (
R-Squared(
Adjusted R-Squared(
Unlike regression that handles continuous dependent variable, classification problem handles dependent variables that are classes and is focused on estimating the probability of an observation belonging to each class. Dependent variables in classification problem are discrete and mutually exclusive groups or classes.
Approaches to solving classification problems:
Following metrics are used to determine model effectiveness. Variables used in their formula is extracted according to Confusion matrix.
AUC-ROC: Area under the ROC Curve is a measure of
Log Loss(Logarithmic Loss, or Cross-Entropy Loss): Used as a cost function for Logistic Regression and loss function for binary classification problems.
Silhouette Score: mean Silhouette Coefficient for all clusters
Calinski-Harabaz Index: measure the distinctiveness between groups by calculating between-cluster dispersion and within-cluster dispersion
Davies-Bouldin Index: average similarity of each cluster with its most similar cluster
Calculate intra-cluster dispersion
Calculate separation measure
Calculate similarity between clusters
Find most similar cluster for each cluster
Calculate Davies-Bouldin Index
Probabilistic Measures are metrics of model performance and complexity. Model complexity itself is the measure of the model’s ability to capture the variance in data.
Akaike Information Criterion (AIC)
Minimum Description Length (MDL)
Similarity metrics are used to compare and evaluate the level of similarity(or closeness) in different data points.
Euclidean Distance is used to calculate straight line distance between two points in an N-dimensional space
Manhattan Distance uses absolute differences of data point’s coordinates to calculate distance in each dimension and then sums them up.
Cosine Similarity uses the angle between two vectors to calculate their similarity.
Jaccard Similarity is measured by the size of intersection and union of two sets.
Pearson Correlation Coefficient is used to calculate linear correlation of variables.