{"id":3750,"date":"2022-03-28T05:50:13","date_gmt":"2022-03-28T05:50:13","guid":{"rendered":"https:\/\/cloudxlab.com\/blog\/?p=3750"},"modified":"2022-10-13T10:34:10","modified_gmt":"2022-10-13T10:34:10","slug":"classification-metrics","status":"publish","type":"post","link":"https:\/\/cloudxlab.com\/blog\/classification-metrics\/","title":{"rendered":"Classification metrics and their Use Cases"},"content":{"rendered":"\n


In this blog, we will discuss about commonly used classification metrics. We will be covering Accuracy Score<\/strong>, Confusion Matrix<\/strong>, Precision<\/strong>, Recall<\/strong>, F-Score<\/strong>, ROC-AUC<\/strong> and will then learn how to extend them to the multi-class classification<\/strong>. We will also discuss in which scenarios, which metric will be most suitable to use.<\/p>\n\n\n\n

First let\u2019s understand some important terms used throughout the blog-<\/p>\n\n\n\n

True Positive (TP): <\/strong>When you predict an observation belongs to a class and it actually does belong to that class.<\/p>\n\n\n\n

True Negative (TN): <\/strong>When you predict an observation does not belong to a class and it actually does not belong to that class.<\/p>\n\n\n\n

False Positive (FP)<\/strong>: When you predict an observation belongs to a class and it actually does not belong to that class.<\/p>\n\n\n\n

False Negative(FN): <\/strong>When you predict an observation does not belong to a class and it actually does belong to that class.<\/p>\n\n\n\n

All classification metrics work on these four terms. Let\u2019s start understanding classification metrics-<\/p>\n\n\n\n\n\n\n\n

Accuracy Score-<\/h2>\n\n\n\n

Classification Accuracy is what we usually mean, when we use the term accuracy<\/strong>. It is the ratio of number of correct predictions to the total number of input samples.<\/p>\n\n\n\n

\"Accuracy<\/figure>\n\n\n\n

For binary classification, we can calculate accuracy in terms of positives and negatives using the below formula:<\/p>\n\n\n\n

Accuracy=(TP+TN)\/(TP+TN+FP+FN)<\/pre>\n\n\n\n
It works well only if there are equal number of samples belonging to each class. For example, consider that there are 98% samples of class A and 2% samples of class B in our training set. Then our model can easily get 98% training accuracy<\/strong> by simply predicting every training sample belonging to class A. When the same model is tested on a test set with 60% samples of class A and 40% samples of class B, then the test accuracy would drop down to 60%. <\/strong>Classification Accuracy is great, but gives us the false sense of achieving high accuracy.<\/p>\n\n\n\n
So, you should use accuracy score only for class balanced data.<\/p><\/blockquote>\n\n\n\n
You can use it by-<\/p>\n\n\n\n
from<\/strong> sklearn.metrics<\/strong> import<\/strong> accuracy_score<\/pre>\n\n\n\n
In sklearn, there is also balanced_accuracy_score<\/a> which works for imbalanced class data. The balanced_accuracy_score<\/strong><\/a><\/code> function computes the balanced accuracy<\/a>, which avoids inflated performance estimates on imbalanced datasets. It is the macro-average of recall scores per class or, equivalently, raw accuracy where each sample is weighted according to the inverse prevalence of its true class. Thus for balanced datasets, the score is equal to accuracy.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Confusion matrix-<\/h2>\n\n\n\n
A confusion matrix is a table that is often used to describe the performance of a classification model<\/strong> on a set of test data for which the true values are known.<\/p>\n\n\n\n
\"Confusion<\/figure>\n\n\n\n
It is extremely useful for measuring Recall, Precision, Specificity, Accuracy and most importantly AUC-ROC Curve.<\/p>\n\n\n\n
from<\/strong> sklearn.metrics<\/strong> import<\/strong> confusion_matrix<\/pre>\n\n\n\n
Precision-<\/h2>\n\n\n\n
It is the ratio of the true positives and all the positives. It tells you that Out of all the positive classes we have predicted, how many are actually positive.<\/p>\n\n\n\n
\"Precision<\/figure>\n\n\n\n
from<\/strong> sklearn.metrics<\/strong> import<\/strong> precision_score<\/pre>\n\n\n\n
Recall(True Positive Rate)-<\/h2>\n\n\n\n
It tells you that out of all the positive classes, how many we predicted correctly.<\/p>\n\n\n\n
Recall should be as high as possible. Note that it is also called as sensitivity.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
from<\/strong> sklearn.metrics<\/strong> import<\/strong> recall_score<\/pre>\n\n\n\n
F1-Score-<\/h2>\n\n\n\n
It is difficult to compare two models with low precision and high recall or vice versa. If you try to increase precision, then it may decrease recall and vice versa. So it ends up in a lot of confusion.<\/p>\n\n\n\n
So to make them comparable, we use F1-Score . F1-score helps to measure Recall and Precision at the same time.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
It uses Harmonic Mean in place of Arithmetic Mean by punishing the extreme values more.<\/p>\n\n\n\n
from<\/strong> sklearn.metrics<\/strong> import<\/strong> f1_score<\/pre>\n\n\n\n
We use it when we have imbalanced class data. In most real-life classification problems, imbalanced class distribution exists and thus F1-score is a better metric to evaluate our model on than accuracy.<\/p>\n\n\n\n
But it is Less interpretable. Precision and recall are more interpretable than f1-score, since precision measures the type-1 error and recall measures the type-2 error. However, f1-score measures the trade-off between these two. So, instead of working with both and confusing ourselves, we use f1-score.<\/p>\n\n\n\n
Specificity(True Negative Rate): <\/strong>It tells you what fraction of all negative samples are correctly predicted as negative by the classifier.. To calculate specificity, use the following formula.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
False Positive Rate <\/strong>: FPR tells us what proportion of the negative class got incorrectly classified by the classifier.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
False Negative Rate: <\/strong>False Negative Rate (FNR) tells us what proportion of the positive class got incorrectly classified by the classifier.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
See, If you know clearly what task you have to accomplish, then it’s better to use precision and recall over f1-score. For example, suppose government launched a scheme for free cancer detection. Now, it’s costly to perform a single test. So, government assigned you a task to build a machine learning model to identify if a person is having cancer. It will be an initial screening test as government will take predictions from your model and will test those persons which your model predicted to have cancer, with real machines that whether they really are having cancer or not. That will reduce the cost of the scheme to much a extent.<\/p>
So in such case, it will be more important to identify all the persons having cancer because we can tolerate that a person not having cancer is detected to have cancer because after testing with real machines, the truth will prevail but we can’t tolerate a person having cancer been not detected cancer because that can cost a person his life. So, here you will use recall metric to check the performance of your model.<\/p>
But, if you are working on such a task where both precision and recall are important equally, then you may use f1-score over precision and recall.<\/p><\/blockquote>\n\n\n\n
ROC-AUC curve-<\/h2>\n\n\n\n
Not only numerical metrics, we also have plot metrics like ROC(Receiver Characteristic Operator) and AUC(Area Under the Curve) curve.<\/p>\n\n\n\n
AUC \u2014 ROC curve is a performance measurement for the classification problems at various threshold settings. This graph is plotted between true positive and false positive rates. The area under the curve (AUC) is the summary of this curve that tells about how good a model is when we talk about its ability to generalize.<\/p>\n\n\n\n
If any model captures more AUC than other models then it is considered to be a good model among all others. So, we can conclude that more the AUC the better the model will be on classifying actual positive and actual negative.<\/p>\n\n\n\n