Error Analysis + Explainability + Signal Detection

Question 1

How would we debug poor model performance?

Answer 1

1. Start by checking we are computing the metrics correctly, using the correct data, and the correct metric for the problem, in many cases, accuracy might not be the correct choice
2. Check for any data leakage if the training accuracy is high, but the testing is low
3. Do a quality check of the data being used, is there noise? Incorrect labels? Poor data quality can make models perform poorly
4. Observe Bias-Variance tradeoff behaviour
5. Evaluate model architecture and training process. Is everything expected and done within reason?

Question 2

What is LIME? How does it work?

Answer 2

Method to explain black-box model predictions.

Basic idea is to explain a prediction from a complex model by fitting a local surrogate model that is simpler, with predictions that are easy to explain.

1️⃣ Pick one input datapoint

2️⃣ Create small variations of that datapoint

This set of nearby points is called the neighborhood.

3️⃣ Ask the black-box model for predictions on these new points

4️⃣ Fit a simple model (usually linear) on this neighborhood

The simple model tries to mimic the black-box model just in this local region.

5️⃣ Use the simple model to explain

The simple model (like a linear model) is easy to interpret.

You look at its coefficients to understand which features matter for the prediction — for this particular datapoint.

Question 3

What are the limitations of LIME?

Answer 3

small changes in input can lead to different conflicting explanations. Local models can vary each time from random perturbations

High dimensionality - in datasets with a large number of features, it’s hard for surrogate models to accurately mimic the complex model.

Since local by design, may disregard global patterns

Question 4

What is SHAP?

Answer 4

Method to explain black-box model predictions

1️⃣ You want to explain one prediction — e.g., why did the model predict 85 for this input?

2️⃣ You consider all possible combinations of features being present or absent.

3️⃣ For each feature:

Measure how much it contributes to the prediction across all possible combinations.

This becomes the Shapley value for that feature.

4️⃣ You add up all feature contributions:
​

φ₀ is the baseline (prediction if no features are present).

φᵢ is how much feature i contributes.

For each feature i:

Compare model prediction with and without feature i, across all feature subsets.

Compute the average of these differences (weighted).

This way, every feature gets fair credit for its contribution to the final prediction.

Question 5

What are the limitations of SHAP?

Answer 5

Computationally expensive – needs to predict for large subset of features

Implementations depend on model – model agnostic in theory, but ideally needs custom implementations for speed/accuracy

Correlation, not causality

Visually complex results/plots

Question 6

What is feature confounding?

Answer 6

When two or more features are correlated, and model cannot distinguish between the 2.

Question 7

What are spurious correlations?

Answer 7

A spurious correlation is when two variables appear correlated, but the relationship is not causal or meaningful.

The model picks up on patterns that are accidental, coincidental, or dataset-specific.

The correlation may vanish or reverse when deployed in production or on new data.

Question 8

How do we mitigate spurious correlations and confounding features?

Answer 8

Collect better data (more diverse, more representative).

Use domain knowledge to select features carefully.

Apply causal inference methods if appropriate.

Test model generalization across multiple environments or populations (robustness testing).

Question 9

How do you know if there’s enough signal in your dataset?

Answer 9

Exploratory Data Analysis (EDA):

• Visualize feature distributions.
• Correlation heatmaps.
• Class separability plots (PCA, t-SNE).

Simple model sanity checks:

• Train simple models (logistic regression, shallow trees).
• If even simple models can’t beat random → possibly no signal.

Model learning curves:

• If model underperforms regardless of data size, may indicate weak signal.

Explainability tools:

• LIME/SHAP show which features the model is relying on.
• If models focus on irrelevant features → noise.