Scribd Logo
Upload

Welcome to Scribd!

  • 31 views

    06 - ML - Classificaion Performance Evaluation Measures

    Uploaded by

    In Tech
    This document discusses various performance evaluation measures for classification algorithms, including accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). It defines key terms like true positives, false positives, and provides examples of computing measures from a confusion matrix. The document also addresses challenges in class imbalance and evaluating sensitivity and specificity.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    06 - ML - Classificaion Performance Evaluation Measures

    Uploaded by

    In Tech
    31 views19 pages
    This document discusses various performance evaluation measures for classification algorithms, including accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). It defines key terms like true positives, false positives, and provides examples of computing measures from a confusion matrix. The document also addresses challenges in class imbalance and evaluating sensitivity and specificity.

    Original Title

    06_ML_Classificaion Performance Evaluation Measures

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document discusses various performance evaluation measures for classification algorithms, including accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). It defines key terms like true positives, false positives, and provides examples of computing measures from a confusion matrix. The document also addresses challenges in class imbalance and evaluating sensitivity and specificity.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    31 views19 pages

    06 - ML - Classificaion Performance Evaluation Measures

    Uploaded by

    In Tech
    This document discusses various performance evaluation measures for classification algorithms, including accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). It defines key terms like true positives, false positives, and provides examples of computing measures from a confusion matrix. The document also addresses challenges in class imbalance and evaluating sensitivity and specificity.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 19

    Classification Algorithms

    Performance Evaluation Measures


    Dr Hikmat Ullah Khan
    Naïve Bayes revisited
     Have you computed any other example of Naïve Bayes?
     Online Source
     Synthetic Example
     Have you checked how to implement the Naïve Bayes?
     Have you checked how to run Naïve Bayes in
     WEKA
     C#
     R
     Python
     RapidMiner/anyother toll you are working on. ?
    Try it Again?

    3
    Evaluation
     HOW
     Comparison b/w Human and Method
     Method is simple
     Method is intuitive

    4 MHS (Dr Hikmat Ullah Khan)


    Confusion Matrix

    Actual class C1 ¬ C1
    \Predicted class
    C1 True Positives False Negatives
    (TP) (FN)
    ¬ C1 False Positives True Negatives
    (FP) (TN)

     Confusion matrix is the base


     All PEM emerge from it

    5
    5
    Confusion Matrix
    Confusion Matrix:
    Actual class\Predicted class C1 ¬ C1
    C1 True Positives (TP) False Negatives (FN)
    ¬ C1 False Positives (FP) True Negatives (TN)

    Example of Confusion Matrix:

    Actual class\Predicted buy_computer buy_computer Total


    class = yes = no
    buy_computer = yes 6954 46 7000
    buy_computer = no 412 2588 3000
    Total 7366 2634 10000

    6
    6
    Classifier Evaluation Metrics: Accuracy,
    A\P C ¬C
    C TP FN P
    ¬C FP TN N
    P’ N’ All

    • Classifier Accuracy, or recognition rate:


    • percentage of test set tuples that are correctly classified

    Accuracy = (TP + TN)/All


    • Error rate: 1 – accuracy, or
    Error rate = (FP + FN)/All

    7 MHS (Dr Hikmat Ullah Khan)


    7
    Compute Accuracy and Error

    Actual class\Predicted class C1 ¬ C1


    C1 True Positives (TP) False Negatives (FN)
    ¬ C1 False Positives (FP) True Negatives (TN)

    Example of Confusion Matrix:

    Actual class\Predicted buy_computer buy_computer Total


    class = yes = no
    buy_computer = yes 6954 46 7000
    buy_computer = no 412 2588 3000
    Total 7366 2634 10000

    8
    8
    Classifier Evaluation Metrics:
    Precision and Recall
     Precision: exactness – what % of tuples that the classifier
    labeled as positive are actually positive

     Recall: completeness – what % of positive tuples did the


    classifier label as positive?
     Perfect score is 1.0
     Inverse relationship between precision & recall

    9 MHS (Dr Hikmat Ullah Khan)


    9
    Classifier Evaluation Metrics:
    F-measures

     F measure (F1 or F-score):


     harmonic mean of precision and recall,

    10 MHS (Dr Hikmat Ullah Khan)


    10
    Precision vs Recall vs F-Measure (Example)

    Compute
    Precision, Recall , and F-Measure
    and
    then Compare the results
    Do we need another Performance Evaluation
    Measures?
    Actual Positive Acutal Negative
    Predicted Positive 10 25
    Predicted Negative 15 100

    Compute Accuracy
    Do we need another Performance Evaluation
    Measures?
    Actual Positive Acutal Negative
    Predicted Positive 10 25
    Predicted Negative 15 100

    Compute Accuracy

    Actual Positive Acutal Negative


    Predicted Positive 0 25
    Predicted Negative 0 125
    Classifier Evaluation Metrics:
    Sensitivity and Specificity
    A\P C ¬C  Class Imbalance Problem:
    C TP FN P
     One class may be rare,
    ¬C FP TN N
    P’ N’ All e.g. fraud, or HIV-positive
     Significant majority of the negative class

    and minority of the positive class


     Sensitivity: True Positive Recognition rate
    (and ?)
     Sensitivity = TP/P
     Specificity: True Negative recognition rate
     Specificity = TN/N

    14 MHS (Dr Hikmat Ullah Khan)


    14
    Receive operating characteristics curve
     It is commonly called the ROC curve.
     It is a plot of the true positive rate (TPR) against the false
    positive rate (FPR).
     True positive rate:
     Ratio of What is True Positive w.r..t Positive

     False positive rate:


     Ratio of Fasle Positive w.r.t Negative

    CS583, Bing Liu, UIC 15


    Example ROC curves

    CS583, Bing Liu, UIC 16


    Area under the curve (AUC)
     Which classifier is better, C1 or C2?
     It depends on which region you talk about.
     Can we have one measure?
     Yes, we compute the area under the curve (AUC)
     If AUC for Ci is greater than that of Cj, it is said that Ci is
    better than Cj.
     If a classifier is perfect, its AUC value is 1
     If a classifier makes all random guesses, its AUC value is 0.5.

    CS583, Bing Liu, UIC 17


    Want to learn more
     Learn Confusion Matric from Wikipedia
     https://en.wikipedia.org/wiki/Confusion_matrix

     Want to Compute Online

     There are many online Sources of Computation of these


    Measures
     http://www.marcovanetti.com/pages/cfmatrix/
     http://onlineconfusionmatrix.com/

    You might also like