Theory Cards: Econometrics, Statistics, Causal Inference

Question 1

What is the purpose of A/B testing in experiments?

A/B Testing

Answer 1

A/B testing aims to determine if changes to a variable (e.g., a webpage design) lead to a statistically significant difference in a key metric by comparing control and treatment groups.

Question 2

What is a null hypothesis in A/B testing?

A/B Testing

Answer 2

The null hypothesis states that there is no effect or difference between the control and treatment groups.

Question 3

What is a p-value in hypothesis testing?

A/B Testing

Answer 3

A p-value represents the probability of observing results as extreme as those in the experiment if the null hypothesis is true. If p < 0.05, results are considered statistically significant.

Question 4

What is a Type I error in A/B testing?

A/B Testing

Answer 4

A Type I error (false positive) occurs when the null hypothesis is incorrectly rejected, suggesting an effect exists when it doesn’t.

Question 5

What is a Type II error in A/B testing?

A/B testing

Answer 5

A Type II error (false negative) happens when the null hypothesis is not rejected despite there being an actual effect, failing to detect a real difference.

Question 6

What is power analysis in the context of A/B testing?

A/B testing

Answer 6

Power analysis calculates the probability of correctly rejecting a false null hypothesis, reducing the chance of Type II errors and increasing confidence in detecting meaningful effects.

Question 7

Why is sample size important in A/B testing?

A/B testing

Answer 7

A sufficient sample size is needed to detect a statistically significant difference. It depends on the expected effect size, statistical power (usually 0.8), and significance level (often 0.05).

Question 8

What is a multi-variant test?

A/B testing

Answer 8

A multi-variant test compares more than two variations simultaneously, rather than just a control and treatment.

Question 9

Name a common pitfall in A/B testing and its solution.

A/B testing

Answer 9

Sample contamination (when participants in one group are influenced by another) can skew results. One solution is to strictly separate groups or adjust analysis methods.

Question 10

What statistical test would you use for comparing means in A/B testing?

A/B testing

Answer 10

A t-test is commonly used to compare the means of two samples, such as conversion rates or average order values, assuming normally distributed data and equal variances.

Question 11

When should T-Test be used?

A/B testing

Answer 11

For comparing the means for 2 samples (conversion rates or avg order values)
Assumes normally distributed data and equal variances in both groups
Example: comparing avg revenue per user between a control and a treatment group
Example: comparing avg time spent on a platform between two user groups: those who made a purchase and those that didnt

Question 12

When Should Chi-Square Test be used?

A/B testing

Answer 12

For categorical data (e.g. converted vs not converted) between 2 groups
Assumes that each observation is independent and expected frequencies in each cell are adequate
Example: checking if there is an associasion between a user gender and whether they purchased a product

Question 13

When should Z-Test be used?

A/B testing

Answer 13

Similar to t-test but specifically for large samples or when the population variance/mean is known
Assumes normally distributed data with large sample sizes
Example: comparing click-through rates when you have very large sample
Example: If historial avg # of listings per user is known, we can compare it to the avg # of listings per user from a new sample when a new feature is introduced, to assess whether the difference is statistically significant

Question 14

When should non-parametric tests be used?

A/B testing

Answer 14

When the data doesnt meet the assumptions of normality (e.g. revenue data often has outliers and is not normally distributed)
Example: For skewed data, like transaction amounts between 2 groups

Question 15

When should Bayesian Test be used?

A/B testing

Answer 15

To estimate the probability of one version being better than the other. Provides probability rather than a p-values
Example: estimate how likely the new version is to be better by a specific margin.

Question 16

What is the difference between correlation and causation?

Causal Inference

Answer 16

Correlation is when two variables move together but don’t imply one causes the other. Causation implies a direct effect of one variable on another.

Question 17

What is a confounding variable in causal inference?

Causal Inference

Answer 17

confounding variable is an external factor that influences both variables studied, potentially creating a false impression of causation.

Question 18

What are counterfactuals in causal inference?

Causal Inference

Answer 18

Counterfactuals represent “what could have happened” under a different scenario, such as considering if a recovery would still happen without taking medicine.

Question 19

What is the Difference-in-Differences method used for?

Causal Inference

Answer 19

It’s used in policy impact analysis where randomized trials are not feasible, comparing changes over time between a treatment and control group.

Question 20

What is an instrumental variable (IV) in causal inference?

Causal Inference

Answer 20

An IV is a variable affecting the treatment but not directly influencing the outcome except through the treatment, used to address unobserved confounding.

Example: Imagine studying the effect of education on income, but family background (unobserved) affects both. IF we use “distance to the nearest college” as an instrument, it affects the likelyhood of attending a college (treatment) but likely doesn’t directly influence income.

Quasi-Experimental Methods:

Question 21

How does Propensity Score Matching (PSM) work?

Causal Inference

Answer 21

PSM matches individuals in treatment and control groups based on observed characteristics to simulate randomization, though it only controls for observed variables.

Example: if we want to study the impact of a job training program, we could match participants(treatment) with non-participants(control) based on characteristics like age, education, and prior job experience.
Limitation: PSM can only control for observed variables.

Quasi-Experimental Methods:

Question 22

What is the use case of Quasi-Experimental Methods? What are some of these methods?

Causal Inference

Answer 22

These methods help us make causal inferences when randomized experiments arent feasible
e.g.

• Diffrence in difference
• Instrumental Variables
• Propensity score matching
• Regression Discontinuity Design (RDD)

Question 23

What is Regression Discontinuity Design (RDD)?

Causal Inference

Answer 23

Used when treatment is assigned based on a cutoff score (e.g. test scores, age)
Compares individuals just above and just below the cutoff, assuming they are otherwise similar.
Example: suppose scholarships are given only to students with GPA >3.0. RDD would compare students just above (eligible) and just below (ineligible) the cutoff to estimate the effect of scholarships on academic success.
Requires a clear cutoff.

Question 24

What are the key assumptions of regression analysis?

Econometrics

Answer 24

• Linearity: linear relationship between predictors and outcome variable
• Independence: observations are independent of each other
• Homoscedasticity: Constant variance of residuals across values of independent variables
• No multicollinearity: independent variables arent highly correlated

Question 25

What are the 2 types of Panel Data Models? ## Footnote Econometrics

Answer 25

* Fixed Effects Model * Random Effects Model

Question 26

What is the Fixed Effects model in panel data analysis? ## Footnote Econometrics

Answer 26

It controls for unobserved variables that are constant over time but vary between entities, useful when unique characteristics could bias results. Example: analyzing how wages change over time within a group of individuals where we assume each person has unique, unobserved traits that don’t change over time

Question 27

When is a Random Effects model suitable in panel data analysis? ## Footnote Econometrics

Answer 27

It’s used when variation across entities is assumed to be random and uncorrelated with predictors, ideal when entity-specific traits are randomly distributed. Example: If studying wage changes across different cities, assuming that individual city characteristics are randomly distributed and uncorrelated with factors like education or experience.

Question 28

What are the 2 types of models in Time Series Analysis? ## Footnote Time Series Analysis

Answer 28

* Autoregressive (AR) models * Moving Average (MA) Models

Question 29

What is an Autoregressive (AR) model? ## Footnote Time Series Analysis

Answer 29

An AR model uses past values of a variable to predict its future values, like predicting tomorrow’s temperature based on today’s.

Question 30

# ** What is a Moving Average (MA) model? ## Footnote Time Series Analysis

Answer 30

An MA model uses past forecast errors to predict future values, adjusting for consistent under- or overestimation in past forecasts.

Question 31

What is stationarity in time series? ## Footnote Time Series Analysis

Answer 31

A time series is stationary if its statistical properties, like mean and variance, remain constant over time, which is essential for accurate modeling.

Question 32

What are some of the common techniques in Time Series Forecasting? ## Footnote Time Series Analysis

Answer 32

Common techniques include** Exponential Smoothing** (which gives more weight to recent observations) and **ARIMA** (AutoRegressive Integrated Moving Average) for stationary data.

Question 33

What is Heteroskedasticity and how can you correct it? ## Footnote Time Series Analysis

Answer 33

**Heteroskedasticity**: Occurs when the variance of residuals (errors) isnt constant across observations. It can lead to inefficient estimates and affect the reliability of hypothesis tests. **Correction**: use techniques such as** robust standard errors **or **transform variables** (e.g. log transformation) to stabilize variance. **Example**: in a model predicting household income, high-income households might have more variance than low-income households, violating the homoscedasticity assumption.

Question 34

What is Autocorrelation and how can you correct it? ## Footnote Time Series Analysis

Answer 34

**Autocorrelation**: Occurs when **residuals are correlated across time** (common in time series data). This violates the independence assumption in OLS and can lead to biased estimates. **Correction**: Use models that account for autocorrelation, like ARIMA, or add lagged variables to the model **Example**: Monthly sales data might show autocorrelation, as sales in one month can be correlated with sales in the previous month.

Question 35

What is the Akaike Information Criterion (AIC)?

Answer 35

AIC is a measure used to compare models, balancing goodness of fit with model complexity; lower AIC indicates a better model.

Question 36

What is cross-validation?

Answer 36

Cross-validation evaluates model performance by splitting data into training and testing sets multiple times, often using K-fold cross-validation. Example: in a dataset with 1000 observations, we could use 10-fold cross-validation to train the model on the 90% of the data and test on 10%, rotating through all portions.

Question 37

What is the Central Limit Theorem? ## Footnote Statistical Modeling and Inference

Answer 37

The Central Limit Theorem states that the sample mean’s distribution approaches normal distribution as sample size increases, regardless of the population's original distribution.

Question 38

What is Maximum Likelihood Estimation (MLE)? ## Footnote Statistical Modeling and Inference

Answer 38

MLE estimates parameters by maximizing the likelihood that the observed data occurred under the model, used in logistic regression for probability estimation.

Question 39

What is Bayesian inference? ## Footnote Statistical Modeling and Inference

Answer 39

Bayesian inference uses Bayes’ theorem to update the probability of a hypothesis as new evidence emerges, combining prior beliefs with new data.

Question 40

What are the key types of distributions? ## Footnote Distributions

Answer 40

*** Normal (Gaussian) Distribution: **symmetrical, bell-shaped curve. Example: heights or test scores in a population *** Binomial Distribution:** Describes the number of successes in a fixed number of independent trials, each with the same probability of success. Example: Flipping a coin 10 times to get a certain number of heads *** Poisson Distribution: **represents the number of events occurring in a fixed interval of time or space, with a known constant mean rate and independent of previous events. Example: Number of customer arrivals at a store in an hour * **Exponential Distributon:** Continuous distribution, describes the time between independent events occurring at a constant average rate.

Question 41

What are the properties, Use Case, and Key Concepts of **Normal (Gaussian) distribution**? ## Footnote Distributions

Answer 41

* **Properties: **Symmetric, bell-shaped curve; defined by mean (μ) and standard deviation (σ); 68% of data within ±1σ, 95% within ±2σ, and 99.7% within ±3σ. * **Examples: **Heights of people, measurement errors, exam scores. * **Use Case:** Commonly used for naturally occurring data that clusters around an average. * **Key concepts: **68-95-99.7 rule (about 68% of data falls within 1 standad deviation, 95% within 2, and 99.7% within 3) The width of the curve is defined by standard deviation To draw you need to know: * The avg measurement (mean) * The standard deviation of the measurements

Question 42

What are the properties, Use Case, and Key Concepts of **Binomial Distribution**? ## Footnote Distributions

Answer 42

* **Properties:** Discrete distribution; defined by number of trials (n) and probability of success (p); mean = np, variance = np(1-p); symmetric for p ≈ 0.5, skewed otherwise. * **Examples: ** Number of heads in 10 coin flips, number of conversions in a marketing campaign, pass/fail outcomes in a series of tests. * **Use Case: ** Used when there’s a fixed number of independent trials, each with the same probability of success. **Laplace's Rule of Succession:** 48 out of 50 positive reviews: add one more positive and one more negative --> 49/52 ~ 94.2% chance of having a positive experience

Question 43

What are the properties, Use Case, and Key Concepts of **Poisson Distribution**? ## Footnote Distributions

Answer 43

* **Properties: **Models count of events in a fixed interval; mean and variance are equal (λ); skewed for small λ, more symmetric as λ increases. * **Examples: **Number of customer calls per hour, website visits per minute, number of defects per unit. * **Use Case:** Suitable for modeling rare events over time or space, where events occur independently. * **Key Concept: **Lambda (A)- the average rate of occurence, which is the mean and the variance of the distribution *

Question 44

What are the properties, Use Case, and Key Concepts of **Exponential Distribution**? ## Footnote Distributions

Answer 44

* **Properties: **Models time until the next event; defined by rate parameter (λ) with mean = 1/λ and variance = 1/λ²; memoryless property (probability of future events does not depend on past events). * **Examples: **Time between arrivals in a queue, time until a machine failure, time until customer churn. * **Use Case**: Ideal for modeling “time until” events in systems with a constant rate.

Question 45

What are the properties, Use Case, and Key Concepts of **Uniform Distribution**?

Answer 45

* **Properties: **All outcomes within a specified range are equally likely; mean = (a + b) / 2, variance = (b - a)² / 12 for continuous uniform; flat distribution. * **Examples**: Random assignment within a time range, selecting random test data, random numbers within a set range. * **Use Case**: Used when all values within a range are equally likely, either in continuous or discrete form.

Question 46

What are the properties, Use Case, and Key Concepts of **Gama and Beta Distributions**?

Answer 46

**Gamma Distribution** * **Properties**: Continuous, skewed distribution; defined by shape (k) and scale (θ) parameters; mean = kθ, variance = kθ²; as k increases, shape resembles a normal distribution. * **Examples**: Insurance claim amounts, waiting times in queuing systems, rainfall amounts. * **Use Case:** Suitable for modeling waiting times where multiple events need to happen, such as total claim costs or aggregated waiting times. **Beta Distribution** * **Properties**: Continuous distribution on [0,1]; defined by shape parameters α and β; mean = α / (α + β), variance = αβ / [(α + β)²(α + β + 1)]; flexible shape, can be symmetric or skewed. * **Examples**: Conversion rates, probability of success in Bayesian inference, proportions in surveys. * **Use Case:** Used for modeling probabilities, proportions, or percentages, especially in Bayesian statistics.

Question 47

What is a confidence interval?

Answer 47

A confidence interval is a **range of values that estimates where the true population parameter lies**, based on sample data. It shows the uncertainty around an estimate, with a specific confidence level (e.g., 95%) indicating how sure we are that the true value falls within the range. **Example**: If you are running an A/B test on Marktplaats and estimate that a new feature increases user engagement by 10%, you might calculate a 95% confidence interval for this increase as [7%, 13%]. This means you are 95% confident that the true increase in user engagement due to the feature lies between 7% and 13%.