experimental design

Question 1

A/B testing

Answer 1

• basic in statistics, and comes up eg in context of causal modeling
  
  • a general term for a method of comparing the outcomes of two different choices
  • there are two treatments and one acts as the control for the other
• also seen is the term “A/A” testing, which is just using the same choice on both the control and treatment group (ideally, I suppose, there should be no difference between the 2)

Question 2

split-half reliability

Answer 2

a method for assessing internal consistency (of a survey, eg)
split a test (eg an IQ test) randomly into parts A and B, give each member of a group both parts

check for positive correlation in results between parts A and B (eg per person)

Question 3

paired experiments

Answer 3

• a population is paired off, with both elements within any given pair having relevant traits in common;
  eg in a drug trial, people are paired by age-group similarity
• different variations of the experiment are run on the different elements in each pairing

Question 4

blocking

Answer 4

• a method in conducting experiments where attempts are made to block out unwanted sources of variation, what some describe as “nuisance factors”;
  eg we block on seasonal effects while studying differences in store sales among a group of stores
• blocking may be less necessary if the sample sizes are large enough
• paired experiments are a special case of blocking