UNIT 6 : Evaluation

AI Model Evaluation

1. Introduction to AI Model Evaluation

• Overview: In the AI project cycle, after going through the stages of problem scoping, data acquisition, exploration, and modeling, we reach the crucial stage of evaluation. This step is essential for determining how well a model performs by testing its ability to make accurate predictions on unseen data. The goal of evaluation is to select the best model that balances complexity with performance and will be able to handle new, unseen data effectively.
• Importance of Evaluation: Evaluation prevents overfitting, which occurs when a model learns the training data too well, including noise or minor fluctuations, leading to poor performance on new data. The evaluation step ensures that the model has generalized its learning and performs well across different scenarios.

2. What is Evaluation?

• Definition: Evaluation is the process of determining the reliability of an AI model by testing it with a test dataset and comparing its predictions to the actual outcomes. It helps to assess how the model will perform in the real world.
• Key Concept: It is important to use unseen data (test data) for evaluation, not the same data that was used for training the model. Using training data for evaluation would lead to overfitting, where the model becomes too specific to the training examples and does not generalize well to new inputs.

3. Key Evaluation Terminologies

In the context of evaluating an AI model, we need to understand certain key terms that represent the relationship between the model’s predictions and actual outcomes. These terms are crucial in understanding how effective the model is in making predictions.

• True Positive (TP): The model predicts an event correctly.
  • Example: A forest fire occurs, and the model correctly predicts it. This is a True Positive, as both the prediction and reality match.
• True Negative (TN): The model correctly predicts that an event will not happen.
  • Example: There is no fire, and the model correctly predicts that there is no fire. This is a True Negative.
• False Positive (FP): The model predicts an event incorrectly, stating that something has happened when it hasn’t.
  • Example: The model predicts that there is a forest fire, but in reality, there is no fire. This is a False Positive and represents an unnecessary alarm.
• False Negative (FN): The model fails to predict an actual event, stating that nothing happened when it did.
  • Example: A forest fire occurs, but the model fails to predict it. This is a False Negative, where the model incorrectly predicts no fire, despite one actually happening.

4. Confusion Matrix

Definition: The confusion matrix is a table used to describe the performance of a classification model. It helps visualize the performance of a model by showing how the predictions correspond to the actual outcomes.

• Structure of the Confusion Matrix:
  • True Positives (TP) and True Negatives (TN) represent correct predictions.
  • False Positives (FP) and False Negatives (FN) represent incorrect predictions.
• Example: Consider the forest fire prediction model. A confusion matrix for this model would look like:

• This matrix helps us understand not only how many times the model was right, but also how many mistakes it made and what types of mistakes (FP or FN) occurred.

5. Evaluation Metrics

To determine the performance of an AI model, several evaluation metrics are used. These metrics provide different perspectives on how well the model is working.

• Accuracy:
  • Definition: Accuracy is the proportion of correct predictions (both positive and negative) out of all predictions made.
  • Formula:

Limitations: While high accuracy seems ideal, it may not always indicate good performance. For instance, if forest fires are rare and occur in only 2% of cases, the model can predict “no fire” all the time and still achieve 98% accuracy without ever detecting an actual fire. This is why accuracy alone is not always reliable.

• Precision:
  • Definition: Precision focuses on how many of the predicted positive cases were actually positive. It tells us the accuracy of the positive predictions.
  • Formula:

​Importance: High precision means fewer false positives. In the forest fire scenario, low precision would lead to unnecessary fire alarms, potentially causing the firefighters to stop taking the alarms seriously.

• Recall:
  • Definition: Recall (or sensitivity) focuses on the model’s ability to identify actual positive cases. It answers the question: “Out of all the actual positive cases, how many did the model correctly predict?”
  • Importance: High recall ensures fewer false negatives. In critical situations like forest fires, a false negative (failing to predict a fire) could lead to catastrophic outcomes.

6. Choosing Between Precision and Recall

• Scenario-Based Selection:
  • In some cases, precision is more important than recall (e.g., in mining where false alarms can lead to wasted resources).
  • In other scenarios, recall is more important (e.g., in medical diagnoses or forest fire prediction, where missing a positive case could be very dangerous).

7. F1 Score

Definition: The F1 Score provides a balance between precision and recall. It is the harmonic mean of the two metrics, offering a single score that considers both.

• Formula:

• Importance: The F1 score is especially useful when there is an imbalance between precision and recall. A high F1 score indicates that both metrics are performing well.

8. Practical Examples

• Scenario 1: School Water Shortage Prediction: A model is designed to predict whether there will be a water shortage in a school. Evaluating this model using accuracy, precision, recall, and F1 score will help determine how well it predicts shortages.
• Scenario 2: Flood Prediction: In regions prone to floods, a model predicts whether floods are likely. High recall is crucial here, as missing a flood prediction (false negative) can result in significant damage and loss of life.
• Scenario 3: Rain Prediction: A model predicts whether it will rain, helping people avoid unexpected downpours. Precision might be important here, as false alarms can make people unnecessarily alter their plans.