Evidence from the Scientific Literature

Question 1

Definition of Science

Answer 1

Basic purpose of science is to acquire information that will help us to describe, explain, predict and control phenomena in the world. It has a power to examine and understand phenomena on a level that allows us to predict with varying degrees of accuracy, if not sometimes control, outcomes of events in the natural and human-made world.

Question 2

The Scientific Method

Answer 2

Defining feature of science since 17th century.
Used as a procedure to ensure trustworthiness of findings.
Formulating a hypothesis, testing it, collecting data, analysing the data and drawing a conclusion.

Question 3

Organised Scepticism

Answer 3

Peer review.
Evidence generated by scientists is subject to ‘organised scepticism’. Scrutinises from a position of trust. Better than practitioner, as actively questions evidence- less prone to bias.

Question 4

Replication

Answer 4

Safety check to ensure trustworthiness of scientific claims. Cornerstone of science- exactly repeating studies to see if same result is obtained. In 1998 British Researcher published an article in a medical journal reporting that he had found a link between a common childhood vaccine and autism. Following this, many parents refused to have their children vaccinated. Several epidemics occurred. Soon after, other researchers failed to replicate the findings. Original study was a fraud. Faked his results, due to money. Demonstrates that without checks, scientific evidence is not always reliable.

Question 5

Science is not about proof, it is about the likelihood of an outcome

Answer 5

This uncertainty comes from:
- New information will cast serious doubt
- Always subject to boundary conditions- dependent on context. Goal setting theory- setting moderately difficult goals leads to higher performance.
- As we learn more and more about something, we sometimes discover that our original findings were not quite correct

Question 6

Limitations of human judgement that science aims to overcome

Answer 6

Coincidence
- Could our observation be by chance/a fluke
- Degrees of uncertainty
- Statistical significance: p-values can improve this. Ronald Fisher (1925) suggested that threshold should be set to 5%.

Question 7

Limitation 2 of judgement that science aims to overcome

Answer 7

Methodological Bias- could our obs be due to personal preference or prejudice. I.e. selection bias/sampling bias, social desirability bias, halo effect,

Question 8

Limitation 3

Answer 8

Confounders
Can apply rigorous RM to prevent confounders.

Question 9

Limitation 4

Answer 9

Placebo Effect
Hawthorne Experiments Mayo and Fritz Roethlisberger (1924-1933). Examined relationship between productivity and working conditions of factory workers. Productivity increased when level of light increased. But, figured out it was not the intervention that increased productivity, but instead the knowledge that people were observing them.

Question 10

Problems with scientific literature

Answer 10

File-Drawer problem

Question 10

Limitation 5

Answer 10

Moderators and mediators
Moderator- affects direction and/or strength of the relationship between a predictor and an outcome. Indicate under what conditions a particular effect is likely to be stronger or weaker.
Mediator- variable that specifies how or why a particular effect or relationship occurs.

Question 11

Ways to successfully search for scientific literature

Answer 11

Boolean operators- using AND, OR, NOT.

Truncation- using * at the end of a word allows for results of all forms of that word.

Question 12

Acquiring Scientific Literature

Answer 12

Determine the two most important terms in your PICOC.
Identify alternative and related terms using Google.
Determine whether there is a broader underlying principle.
Pre-test your search terms and determine which terms yield the most relevant articles.
Conduct a search with these pretested search terms and combine the outocme with OR.
Filter for meta-analyses and/or systematic reviews.
Filter for high-quality studies.
Screen the titles and the abstracts of articles for relevance.

Question 13

Importance of Scientific Literature

Answer 13

Good to Great book by Jim Collins (2001)- defines what makes a company go from good to great. Relied on interviews, organisational data, practitioner evidence etc. Very popular book, but some of the businesses he identified as being great began to decline. Shows that experts can come to flawed conclusions when they do not follow the methodological rules of science.

Question 14

Critical Appraisal of Scientific Evidence- Are the findings of the study of practical relevance?

Answer 14

Effect size is necessary to determine practical relevance. Measure of impact of the effect. Standardised, so we can compare across different studies. Application to real world scenarios.

Case study examples:

Question 15

Case Study that demonstrates good appraisal of scientific evidence

Answer 15

Google Project Oxygen (2008)- project that tried to uncover what leadership skills are needed to make the perfect boss at Google. They used findings from scientific literature. Data-driven approach. Methodological. They also used interviews, employee surveys, manager feedback. They made these findings practically relevant (appraisal)

Question 16

Case Study that shows bad appraisal of Scientific Literature

Answer 16

Theranos- Elizabeth Jones. Fraud case. Scientific literature and experts raised doubts about the functionality and feasibility of the technology (blood testing with a single drop of blood). But, this was ignored, so the practical relevance of the validity of the technology was ignored. Demonstrates the importance of appraising scientific evidence, specifically understanding the relevance and applicability of claim.

Question 17

Appraisal- how precise are the findings?

Answer 17

Can use confidence intervals to achieve this.

Question 18

Appraisal- how were the findings measured?

Answer 18

To determine reliability and trustworthiness of evidence. Quantitative over qualitative.