Machine Learning Systems Flashcards
(30 cards)
What are the three main components of a ML system that we need to decide on?
- Training algorithm
- Training data
- Features
What two methods are there to collect training data?
- Online: Users’ preexisting interactions with the system, e.g., impressions for a search engine
- Offline: Human annotators, which are expensive, must be QA’d, etc.
What kind of human annotation is there?
- MTurk/cheap
- Specialized/hired annotators
- Public datasets
What are the two most common ML system SLA metrics?
- Performance
2. Capacity
What is the ML system performance metric?
Time to get back results, usually measured in ms for 99% of queries
What is the ML system capacity metric?
Load the system can handle, usually measured in queries per second (QPS)
What are the three types of ML system complexity? (Think what decisions make it slow.)
- Training: time to train the model
- Evaluation/Inference: time to evaluate input at testing/deployment time
- Sampling: total number of samples required to learn the target
What bad thing happens when you switch from a popularity-based recommendation model to a user-specific one?
Bias in recommendations leads to user-specific recommendations being shown lower in the list, leading to a self-reinforcing cycle of only ever showing popularity-based recommendations.
What is a SERP?
Search engine results page
What are the names for the two types of pages in an A/B test?
- Control
2. Variation
What are the five (+optional sixth) steps in designing an A/B test?
- Come up with a hypothesis
- Design an experiment to measure the hypothesis using a NHST
- Use a power test to determine the number of samples you need to form a conclusion
- Run the experiment
- Check for a statistically significant result via NHST
- (optional) Consider backtesting
What is backtesting in the context of an A/B test?
After the A/B test is complete and the null hypothesis rejected, swapping the control and variations and confirming that a negative effect exists.
What is an embedding?
A low-dimensional version of a high-dimensional feature (speech, image, text) such that it captures the important semantic information.
How are embeddings used?
They allow us to train machine learning models on dense representations of data that captures the dimensions we care about.
What are two ways of generating text embeddings?
- word2vec
2. Context-based embeddings
What are two models for word2vec text embeddings?
- Continuous bag of words (CBOW)
2. Skipgrams
How does continuous bag of words (CBOW) work?
It’s an LSTM model where, given a sequence of words of time $t_(-2)$, $t_(-1)$, $t_1$, $t_2$, all one-hot encoded, it predicts $t$.
How do skipgrams work?
It’s an LSTM model where, given a one-hot encoded word at $t$, it predicts $t_(-2)$, $t_(-1)$, $t_1$, and $t_2$.
Why do we need context-based text embeddings?
Word2vec strips out context-based information. For example, the word “apple” can refer to either Apple the company or an apple the fruit.
What are two models for context-based text embeddings?
- Embeddings from Language Models (ELMo)
2. Bidirectional Encoders Representations from Transformers (BERT)
How does ELMo work?
Uses a bi-directional LSTM to capture words that appear before and after the current word.
How does BERT work?
Uses an attention mechanism to see words in context, and only uses the ones that help with predictions.
What are two models for generating visual embeddings?
- Auto-encoders
2. Visual supervised learning tasks
How do auto-encoders generate visual embeddings?
They generate low-dimensional representations of images with another path that re-creates the higher dimensional representation from that lower one. The model is optimized for recreating the original image as closely as possible. The path after the low-dimensional representation is then cut out and used as a feature vector.
