Module 1: Study Design and Inference

Question 1

Statistics Definition (Summarised)

Answer 1

Science of learning data, and how to control and communicate uncertainty.

This involves study design, data collection, analysis, and interpretation; for the purpose of drawing conclusions and presenting uncertainty.

Question 2

Scope of Inference

Answer 2

Whether the results from the sample can be generalised to the broader population.

“What group items do we want our conclusion to be valid for?”

Question 3

Study Types

Answer 3

Survey, Experimental, Observations

Question 4

Goal of Surveys Studies

Answer 4

Sampling a population for the purpose of describing a populations characteristics.

Question 5

Survey Studies, the ‘Researchers Role’

Answer 5

The researcher has control over the collection method of the sample.

Question 6

Goal of Experimental Studies

Answer 6

Establish a causal relationship, by assigning treatments to experimental units.

Question 7

Experimental Studies, the ‘Researchers Role’

Answer 7

The experimenter has control over treatment assignment.

Question 8

Goal of Observational Studies

Answer 8

Investigate relationships (associations) between variables, as they occur in nature.

Question 9

Observational Studies, the ‘Researchers Role’

Answer 9

The researcher has control over what data to include, and how to investigate data.

Question 10

Ways to reduce Noise/”haphazard variation”:

Answer 10

• clever design that yields preciser results for the same cost/sample size.
• measuring covariates, to explain more variation.
• increase sample size.

Question 11

Ways to avoid bias:

Answer 11

• appropriate study design, analysis, and interpretation.
• Collecting a sample that’s representative of the population.

Question 12

Population of Inference is defined by…

Answer 12

• Scope of Inference
• Sampling Frame
• And all items within the PoI follow a Probabilistic Sampling Scheme.

Question 13

Probabilistic Sampling Scheme Definition

Answer 13

each sample unit in a population has a definable probability of being included in that sample.

Question 14

Design-based Inference vs. Model-based Inference

Answer 14

Design-based inference relies on randomly assigning some population to the sample. (the sample being representative of population).
While, Model-based Inference relies on the distributional assumptions. (normal vs non-normal).

Question 15

Sampling Types

Answer 15

• Simple Random Sampling
• Stratified Random Sampling
• Cluster Sampling (I vs II)
• Systematic Sampling
• Convenience / Ad-Hoc Sampling

Question 16

Simple-Random Sampling

Answer 16

All possible items (including subgroups) have the same chance of being selected.

Question 17

Benefits and Cons:
Simple-Random Sampling

Answer 17

This sampling method is cheap and easy to implement.

If the population is too large this sampling method may not be truly representative (mis-match), potentially leading to inaccuracies.

Question 18

Stratified-Random Sampling

Answer 18

In the sample, the size of subgroups are proportional to in population.

Question 19

Benefits and Cons:
Stratified-Random Sampling

Answer 19

The sample will be proportionally representative of the population. And can over-sample smaller strata to get strata specific estimates.

There is a potential for misclassification. And it may not be possible to identify every subgroup.

Question 20

Cluster Sampling

Answer 20

When the population may be comprised of similar and naturally occurring groups.
There are two types, single-stage and two-stage.

Question 21

Benefits and Cons:
Cluster Sampling

Answer 21

This sampling method is cheaper and can be easier to study clusters than individuals, e.g. a school’s avg vs student avg.

But less precise and can have multiple levels of variation.

Question 22

Single-Stage Cluster Sampling

Answer 22

Every unit within a cluster is sampled, e.g. everyone in the household.

Question 23

Two-Stage Cluster Sampling

Answer 23

Random selection of a unit within a cluster, e.g. one member within the household.

Question 24

Systematic Sampling

Answer 24

Selection of each unit is consistently spaced (interspersion), e.g., select every 4th entry on a list.

Question 25

Benefits and Cons: Systematic Sampling

Answer 25

This sampling methods provides even sampling coverage of the population. And can be an alternative when randomisation is impossible. There is a heavy reliance on assumptions that units are independent and random, relative to spacing. If these assumption fail, then design can be horribly wrong.

Question 26

Simple-Random vs Stratified-Random vs Clustered Sampling

Answer 26

Stratified R- is a more precise estimate of the population than Simple R- for the same sample size. Cluster is typically less precise than either for the same sample size.

Question 27

Convenience / Ad-Hoc Sampling

Answer 27

Selecting units close at hand, e.g. surveying students in front of Business building.

Question 28

Benefits and Cons: Convenience / Ad-Hoc Sampling

Answer 28

This sampling method is cheap, efficient, and easy to implement. But, involves no probabilistic sampling scheme. And can be heavily bias.

Question 29

Why is having a 'probabilistic sampling scheme' important?

Answer 29

Without it, you can't use design-based justification for extrapolation from sample to population. Because extrapolation is based on untestable assumptions.

Question 30

Why is Study Design important?

Answer 30

Using the correct study design allows... - Extrapolation justified by study design - More robust results - Identifying issues that may arise during sampling and analysis, e.g. pseudo-replication. - reduce mismatch between sample and population.

Question 31

What does Mismatch mean in study design?

Answer 31

when the sample is not representative of the population, therefore generalisations may be inaccurate.

Question 32

Why can Mismatch in study design occur?

Answer 32

When the population of interest is not clearly defined, no probabilistic sampling scheme is applied, or voluntary/non-response bias occurs.

Question 33

Selection Bias, and types.

Answer 33

Occurs when selected units systematically differ from the population of interest. This can be exist as sampling bias, non-response bias, or voluntary bias.

Question 34

Sampling Bias

Answer 34

When there is a mismatch that is not justified or accounted for in analysis and interpretation.

Question 35

Non-Response Bias

Answer 35

When a particular subset of the population is less likely to respond.

Question 36

Voluntary Bias

Answer 36

When certain participants self-select their involvement.

Question 37

Response/ Information Bias, and examples.

Answer 37

When the measurement of units is systematically bias or incorrect, e.g. poorly calibrated instruments, leading questions, when true values are not observed (other option in surveys).

Question 38

Two ways bias can be reduced through analysis.

Answer 38

Post-hoc stratification of respondents (categorise) or explaining variation with covariates with linear regression. These changes leads to more model-based inference.

Question 39

What MUST experimental studies have?

Answer 39

Randomisation and Replication

Question 40

How can the impact of 'unwanted variation' be reduce?

Answer 40

- Known variation can be controlled by matching or grouping (stratification, paired studies, etc). - Unknown variation ('random error') can be reduced by increasing replication. - Increasing 'signal to noise ratio' through study design.

Question 41

Why is replication important?

Answer 41

Replication establishes valid experimental results by ensuring, reproducibility, robustness against aberrant results, experimental error (uncertainty). Also makes results more precise because, as replication increases uncertainty decreases.

Question 42

Sampling Distribution

Answer 42

The distribution of a statistic if we could repetitively sample from the population. This is a theoretical property of statistics to make inference.

Question 43

Statistical Inference

Answer 43

The application of the methods of probabilities to the analysis and interpretation of data. In inference we often infer properties of an unknown probability distribution using collected data.

Question 44

What are we saying when we apply a model to our data?

Answer 44

The data will have certain characteristics and properties (dependant on the model used). Therefore, we can prove best fit by running diagnostic tests.

Question 45

What are statistical models? And what are we interested in?

Answer 45

An approximation of reality that describes the data generating process. When we use them we are interested in the model parameters, as they often show underlying variance.

Question 46

Simple Linear Regression, equations and explanation.

Answer 46

y = β0 + β1*x + ε ε ~ Norm(0,σ2) The first line explains the μ as a line. The second line line explains variability around the line.

Question 47

Explain the effect different variances on a distribution when mean is unchanged.

Answer 47

Higher variance will make the curve shorter and more spread. Lower variance will make the curve taller and narrower.

Question 48

What is the purpose of a Hypothesis Test?

Answer 48

Testing whether our observed data occurrence is consistent with our given hypothesis.

Question 49

What parameters are used when standard deviation is unknown?

Answer 49

s (sample standard deviation) and t distribution instead of a normal distribution.

Question 50

Types of Estimators

Answer 50

Internal estimate - 95% confidence interval. Point estimate - 0% confidence interval.

Question 51

Name two methods of getting estimates

Answer 51

Ordinary Least Squares and Maximum Likelihood

Question 52

How does Ordinary Least Squares (OLS) work?

Answer 52

Find estimates that minimise the total square difference between observed and expected values.

Question 53

How does Maximum Likelihood (ML) work?

Answer 53

Finds estimates that maximise the likelihood of observing (in the data) the weight we measured in our model. (most commonly used)

Question 54

Using a Simple T-Test instead of Paired T-Test: what issues occurred from the native analysis? What would we observe if we used the correct analysis? (lecture 6 example)

Answer 54

Because the simple t-test ignores the sampling scheme, treating all the cockles as independent, there are two flaws in our interpretation. Flaw 1: incorrect analysis (analysing cockles instead of quadrates) causing pseudo-replication. Flaw 2: ignoring controllable source of uncertainty (quadrate to quadrate variation). With the correct analysis we would see the CI narrow, because more uncertainty is controlled by accounting for quadrate to quadrate variation.