05 - Recommender Systems Evaluation Flashcards
What questions should you ask yourself is you develop a recommender system?
- Objective: What do you want to achieve with the model?
- How to measure: Evaluation methods and Evaluation metrics
- How good/relevant are the results?
What are the goals of the business world?
- A successful business
- Maximum profit, income, and user satisfaction
- Minimize costs
- Get as many users as possible
- To have the best product
What are possible costs, that may arise?
- Labour costs
- Server
- Legal/Licenses
- etc.
What is Goodhart’s Law?
When a measure becomes a target, it ceases to be a good measure (dt. wenn ein Messwert zu einem Ziel wird, ist es kein geeigneter Messwert)
What are the three main evaluation methods and metrics?
- Online Evaluations
- Offline Evaluations
- User Studies
What is part of Online Evaluations?
- Sales
- Profit
- Clicks
What is part of Offline Evaluations?
- Errors
- Accuracy
What is part of User Studies?
- User feedback
- User observations
How does an A/B Test work?
- Typical Online Test
- 50% of the users see Variante A
- 50% of the users see Variante B
How does Interleaving work?
- Randomize Rankings
- All kinds of variations (Random Mix, Top n Mix, Fixed amount Mix)
What is a typical metric for classification?
Accuracy
What is a typical metric for Regression?
Error Metrics
What is a typical metric for Ranking?
Ranking Metrics
Is Regression = Classification?
- Regression tasks can be interpreted as classification/ranking problem
- Define intervals and treat them as classes (and use a classification algorithm instead of regression algorithm)
What regression metrics do you know?
- Mean Absolute Error (MAE)
- (Root) Mean Square Error ((R)MSE)
