Quiz 3 Flashcards
(118 cards)
1
Q
How do we assess the evidence in news headlines?
A
Go back to the initial study
2
Q
Name 4 strategies (questions) to assess the study evidence and study quality?
A
- What type of evidence is this? (is it the right research design for the research question?)
- Does this research apply to my topic?
- How much were the results affected by bias?
- What are the effects of the results? (Clinical Impact?)
3
Q
What are the 3 best research designs for assessing the effectiveness of a treatment/intervention?
A
- Meta-analysis of RCT studies
- Systematic reviews of RCT studies
- RCT studies
4
Q
What do people usually look at for effectiveness of a treatment/intervention (which is not correct)?
A
Expert opinion/Editorial note published in magazines or scientific journals
5
Q
What is the order for the types of studies in terms of strength for testing the effectiveness of a treatment/intervention?
A
In vitro experiment < animal experiment < Expert opinion/editorial note < case report/case series < Case control studies < Cohort studies < RCT < systematic review < meta-analysis
6
Q
Why is an expert opinion/editorial note biased?
A
Biased with life experiences, your own research. Not objective.
7
Q
Differentiate a systematic review from a meta-analysis
A
Systematic review
- detailed, comprehensive, structured and critical literature review of all research studies that address a particular clinical issue.
- It also produces a systematic evaluation of the quality of included studies according to a pre-determined criterion.
If the studies included in the systematic review have a comparable quantitative data (sample) and a low degree of variation in their findings, a meta-analysis can be performed.
If similar sample type (e.g. African women)…
Meta-analysis
A type of systematic review in which the data of all selected studies are pooled quantitatively and reanalyzed using statistical methods. By combining the samples of all selected studies, the overall sample size is increased, which improves the statistical power of the analysis and the estimation of treatment effects.
8
Q
What are the 3 best research designs for assessing the effectiveness of an etiology/risk factor/prognosis?
A
- Meta-analysis of cohort studies
- Systematic review of cohort studies
- Cohort studies
9
Q
What do people usually look at for prognosis/etiology/risk factors (which is not correct)?
A
People usually look for this type of information in “Expert opinion/Editorial note” published in magazines or even scientific journals.
10
Q
Which type of study is not in the pyramid for Etiology/prognosis that was in the one for treatment intervention? Which is present in etiology that was not present in treatment?
A
- RCT is not in the pyramid
2. Cross-sectional is in the pyramid
11
Q
What is the order for the types of studies in terms of strength for testing the etiology/risk factors/prognosis?
A
In vitro < animals < expert opinion/editorial note < Case report/case series < Cross-sectional < Case-control studies < Cohort < Systematic review < Meta-analysis
12
Q
Name the 10 aspects of effective data
A
- Stored
- Preserved
- Accessible
- Discoverable
- Citable
- Comprehensible
- Reviewed
- Reproducible
- Reusable
- Integrated
13
Q
Explain the “stored” aspect of effective data
A
“data sheet”, excel spreadsheet with all data
14
Q
Explain the “preserved” aspect of effective data
A
Archive data, e.g. hard drive, physical copies or clouds/internet
15
Q
Explain the “accessible” aspect of effective data
A
Both researchers and machines may want to access the data, for example, for meta-analyses or other kinds of re-use. There are a number of different ways researchers can make their data accessible. They can do this either by depositing their data in a public repository, or by using a data sharing system such as Mendeley Data, where researchers create private data sharing spaces that can be opened to larger communities or the wider public.
16
Q
Explain the “discoverable” aspect of effective data
A
An important way to make data more discoverable is to link articles to the data sets these articles are based on. Both Elsevier and other publishers support various mechanisms to set up such links, for instance, through inclusion of data DOIs or data accession numbers, which automatically link to associated data in public databases.
17
Q
Explain the “citable” aspect of effective data
A
GET DATA PUBLISHED.
One of the barriers to data sharing has been that it requires extra work from researchers for little reward. Data citations have the potential to change that because they can be easily incorporated in the current reward system based on article citations. Therefore, researchers should think about providing their data with a unique, persistent and resolvable ID, for which in some cases accession numbers can be used.
18
Q
Explain the “comprehensible” aspect of effective data
A
To enable data to be reused, it needs to be clear which units of measurements were used, how the data was collected and which abbreviations and parameters are used. Publishers can help here, and several publishers now publish dedicated data journals, such as Elsevier’s Data in Brief. In these data journals, scientists can provide a thorough description of their datasets, which makes it easier for other researchers to understand the data, process they used to capture the data, and anomalies in the data (or in the capturing process) that a re-user of the data should be aware of, supporting proper data reuse.
19
Q
Explain the “reviewed” aspect of effective data
A
While it is very common for research articles to be peer reviewed, this is still quite uncommon for research data. However, it is an important step when it comes to quality control and trustworthiness of data.
20
Q
Explain the “Reproducible” aspect of effective data
A
Reproducibility of research results is a big concern for science. To increase the credibility of research results, a Reproducibility Initiative was introduced to validate (for a fee) key experimental results via independent replication. Irreproducibility often originates from missing elements to research data, which are needed in order to achieve the same research results.
21
Q
Explain the “reusable” aspect of effective data
A
The key benefit for the wider research community of having research data being shared is the ability to reuse this data. Only when research data is sufficiently trustworthy and reproducible will other researchers re-use the data. This may be to enlarge a sample or to use information in ways it may not originally have been intended for. It is therefore recommended to allow for attaching a user license to datasets already at the very first step of data sharing: at the time of storage and preservation. This will enable any user to clearly understand what they can and cannot do with the data, and can also help ensure they give researchers and data creators the appropriate credit.
22
Q
Explain the “integrated” aspect of effective data
A
We believe that it is important to integrate these nine aspects of “highly effective research data.” For instance, data should be preserved so that it can be reused. To be citable, it needs to be accessible. But also, in building systems for data reuse or data citation, the practices of current systems for storing and sharing data need to be taken into account. These nine layers and 10th integration step are intended as a guiding principle by which research data management practices can be ordered and checked, rather than as a prescription for perfect performance.
23
Q
What are the aspects of effective data in the “SAVED” category?
A
Stored, preserved
24
Q
What are the aspects of effective data in the “SHARED” category?
A
Accessible, discoverable, citable
25
What are the aspects of effective data in the "TRUSTED" category?
Comprehensible, reviewed, reproducible, reusable
26
What are some attributes of scientific writing?
– Quite different from literary writing
– Formal writing style that is highly technical (i.e., not casual as you would in conversation or email)
– Not overly descriptive or flowery; sentences not complicated; paragraphs are relatively short
– Goal is to clearly convey information
– Objective
– Usually contains statistics and numbers
– Structured
27
Differentiate between a review article and a research article
Journal article
- Describes original research conducted by authors
- Utilizes a structured framework
- Deeper in the topic than background info
Review article
- Integrates, summarizes and provides ideas for extending upon prior research on a particular topic
- Authors study existing literature (versus doing new research)
- Background research --> look for review articles first
28
What are some important aspects of reading articles?
• Takes lots of time to find, select, and collect papers; also to extract relevant information
• Reading and understanding articles is time- consuming and challenging
o Unfamiliar organization
o Unfamiliar terminology and theories
• Read papers at least twice (or 10+ times)
• Take good notes while you do your research (finding articles; reading articles; etc.)
29
Define plagiarism
“Presenting and using another’s published or unpublished work, including theories, concepts, data, source material, methodologies or findings, including graphs and images, as one’s own, without appropriate referencing and, if required, without permission”
30
How to manage your articles in background research?
Make tables
Each row is an article (Author name, issue, journal, DOI number/pubmed number)
Columns:
o Date
o Type of study/research design
o CRAAP (?) - quality - Can use codes for currency (e.g. 1 for last 2 years, 2 for last 5 years etc.)
o Population (Number, Age, sex, ethnicity, location…)
o Dependant variable: Measurement tools (how did they measure Y)
o Independent variable/intervention (how did they measure X)
31
Name the 9 components of a manuscript
* Title,
* Abstract,
* Introduction,
* Materials and Methods,
* Results,
* Discussion,
* Literature Cited
* Acknowledgments
* Tables/Figures
32
Relate the components of a research paper to the scientific process
Observe, hypothesize --> Introduction
Test --> methods and materials
Analyze --> results
Interpret, theorize/apply --> discussion
33
Explain the relevance of a title and what it should contain
- Captures the message, main thrust of the paper
- Factual but can draw people in
- Should be related to conceptual variables; aligns with question
- 10-15 words
34
Explain the goal of an abstract and what it should contain
- < 200 words
- Concisely summarize paper, to the point
- Stand-alone
- Indexed and searched
- Usually structured
```
Background/Introduction (big picture, knowledge gap) 3-4 sentences; 1 sentence for each paragraph of the intro
Objective (hypothesis)
Methods (3 sentences)
Results (3-5 sentences)
Discussion (1-2 sentences)
Implications
```
- Make sure the key topics of the paper are funnelled into the abstract
35
Explain the usual length , the structure and the content of an introduction.
- Usually 1 page single spaced
- 3 to 4 paragraphs; building up a story; 1. Independent variable, 2. Why should we care about independent variable, 3. Challenges with association, 4. Hypothesis
- What did you study and why?
- General --> narrow --> hypothesis (build subject)
- What are knowledge gaps or contentions in field?
- Last paragraph should be the punch line: why is this important? What is the key gap? What questions do you have?
36
Explain the structure and the content of a materials and methods section.
o How you did the study
o Can the reader repeat your study now?
o OK to cite previous approaches, but new ones give full details
o Stats!
```
Standard sections:
Population (describe)
Method underlying independent variable and dependant variable
• How did you measure X? Y?
Data analysis
```
37
Explain the structure and the content of a results section.
What you found but no interpretation
o Tables, figures, text – How decide?
o Captions standalone
Standard sections:
Population results (average age, baseline measures)
Dependant variable (average values, normally distributed? Etc.)
Independent variable
Association between dependant and independent variables
38
Explain the structure and the content of a discussion section.
o What does your work mean?
o Compare/contrast with others; implications? External validity – compare to SIMILAR studies (not animal studies etc.)
o Strengths/weaknesses; Internal validity
o Future work?
o Conclusions?
39
Literature cited:
o Follow journal style
| o Reference managers
40
Acknowledgement section:
o Thanks to helpers, collaborators, funders…
| o May indicate sources of bias
41
What other things must be taken into account when writing a journal article?
o How chose journal? Impact factor and topic (relevance) based on your subject (Impact factor: # of citations/# of published articles)
o Follow guidelines carefully
o Write write write (may re-write paper 50 times before it gets published)
o Expect peer-review comments
42
Explain the science news cycle
Your research shows that A is correlated with B given C and assuming D under E conditions
University PR office releases article with title " Scientists find potential link between A and B (under certain conditions)"
News wire organization --> A causes B, say scientists
Internet --> cable news... --> population
43
Explain Bohannon's spoof study
A study that occured 5-6 years ago because Journalist from science magazine was concerned with the prevalence of “junk” journals that would get anything published
Made up a scientific study
• Claimed that molecule X from lichen species Y inhibits the growth of cancer cell Z.
• Fictitious authors (made up names) are affiliated with fictitious African institutions (made up university)
• Conclusions:
1: cancer cells grow more slowly in a test tube when treated with increasing concentrations of a molecule (results showed opposite trend) --> conclusion was not based on results
2: cells treated with increasing doses of radiation to simulate cancer radiotherapy (not done in the paper)
3: The study author concluded the paper by promising to start treating people with the drug immediately, without further safety testing
• Firm conclusion given
44
What were the outcomes of the spoof study?
• 157 journals accepted; 98 rejected
• Of 255 papers that underwent entire review process, 60% of decisions had no sign of peer review
• Most reviews focused on paper’s layout, formatting, and language
• 36 papers had reviews concerning the science, and of those 16 papers were still accepted
GLOBAL issue --> paper was accepted all over the world
45
What are some signs of a phony/predatory publishing?
Little contact information is given and what is given is suspect
Amateurish page design: clashing colors and graphics, distracting background images, scrolling links, etc.
Text may be full of errors, contain questionable grammatical choices and may lack context
List of seemingly arbitrary keywords are used to boost search engine optimization
False metrics or identifiers such as impact indec, ISJN or CiteFactor
Guarantee of manuscript acceptance and publication or unrealistic turn time
Some elements on the site seem to have a random or indeterminate purpose, like a scrolling text and images that dont link anywhere
List of issues and articles is hard to find, haphazard or non-existant
No statement about ethics or affiliation with industry organizations such as COPE, CSE, ICMJE etc.
The journal website is hosted by an unknown source or free platform that allows users to design their own site
46
What are some signs of a legit publishing?
Contact information is thorough and accurate
Mobile optimization is often a prominent feature
The list of issues and articles is complete and easy to find
Industry standard metrics are clearly displayed
The journal website is hosted by a reputable publisher or technology partner that is well-known
All the links work
Everything on the site has a purpose
Professional, modular page design
Text and navigation are clear, accurate and helpful
Statement about journal's ethics policy or membership in COPE or similar organization
47
Name some signs of bogus e-mails
– Ask for money (for things you didn’t do)
– Written mistakes
– Fonts changing
– Asking for something (an article) in the near future
– May want you as a board member for a bad journal (one you’re not an expert in)
– From a journal in a subject you’re not an expert in
– Google about it – others may also have concerns; can also see lists of conferences in the year – is it part of the list?
48
Name the ICMGE 4 criteria for authorship
1. Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work, writing the grant…
AND
2. Drafting the work or revising it critically for important intellectual content (writing);
AND
3. Final approval of the version to be published; (when submitting a paper, also submit a paper with the signature of all of the authors)
AND
4. Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved (communication aspect; willing to stand up for paper and defend it against criticism)
49
At what point in the process of conducting and reporting on one’s research should decisions concerning authorship and acknowledgments be made?
At the initial stages of the study
50
What are the criteria for aknowledgement?
Contributors who meet fewer than all 4 of the above criteria for authorship should not be listed as authors, but they should be acknowledged. Examples of activities that alone do not qualify a contributor for authorship are
- acquisition of funding;
- general supervision of a research group or general administrative support and writing assistance
- technical editing
- language editing,
- proofreading.
Those whose contributions do not justify authorship may be acknowledged individually or together as a group under a single heading (e.g. "Clinical Investigators" or "Participating Investigators"), and their contributions should be specified (e.g., "served as scientific advisors," "critically reviewed the study proposal," "collected data," "provided and cared for study patients", "participated in writing or technical editing of the manuscript").
Because acknowledgment may imply endorsement by acknowledged individuals of a study’s data and conclusions, editors are advised to require that the corresponding author obtain written permission to be acknowledged from all acknowledged individuals.
51
Are peer-reviewers paid?
Peer-reviewers are volunteers.
| Done to give back to the research community
52
Who are the 4 key people in peer review?
1. Journal staff
2. Editorial board
3. Editors
4. Reviewer
53
What is the role of the journal staff in peer review?
o Oversee receipt of manuscript, communication between author and reviewers, formatting
o Receive papers, quick initial read, ensure proper format
o Payed professionals working for journal
54
What is the role of the editorial board in peer review?
o Read/review papers, select reviewers, monitor quality, recommend actions to Editor
o 3-5 people look at the paper
o Each editor (below) has board members
A board member is assigned a paper by the editor, the board member selects a few reviewers. Reviewers provide a summary and comments, judge the paper. Editorial board member takes a decision and gives it back to the editor
55
What is the role of the editor in peer review?
Make final decisions; decide which editorial board member gets an article.
Each journal has an editor
56
What is the role of the reviewers in peer review?
Provide reviews of manuscripts; make recommendations concerning publication
57
Name attributes of a good reviewer
• Expert
• Objective
• No conflicts of interest (overlaps with your work; same institution; collaborator/friend; financial gain; personal belief) or declare it.
• Good judgment
• Able to think clearly and logically
• Able to write a good critique
(Accurate, Readable, Helpful to editors and authors)
• Reliable in returning reviews
• Able to do the review in the allotted time frame (<4 weeks; few hours per review when experienced)
58
Explain the process of peer review
* Potential reviewer contacted by journal
* Given authors, title, abstract, and time frame for review
* Reviewer agrees to review paper (or declines)
* Reviewer receives paper
* Reviewer performs review
* Reviewer submits review to editors
* Editors examine reviews, obtain additional reviews if needed, and make decision
* Decision goes to author, with comments from reviewers
* Reviewer thanked; may be informed of decision; may receive copy of comments sent to author
59
Why is it hard to find reviewers?
Should get at least 3 people to review a paper
Limited pool of researcher; more pressure for researchers to produce.
Surge in production means that more and more researchers need to review these papers
Pressure on academics to publish + pressure on journals/ companies (Elsevier, Springer…) to sell subscriptions
Those big companies have bought other ones subscription price is now very high
60
What is the content of peer review?
- Review form (Checklist/rubric)
- Comments to editor (authors won't see)
- Comments to authors (editors won't see)
61
What is included in the comments to editor?
Summary of around 100 words/ a quarter of a page
| Frank perspective
62
What is included in the comments to author?
General comments (summary; your interpretation on methods, clinical application, significance)
Specific recommendations (minor and major)
Minor: Spelling mistakes, citations (CRAAP) etc.
Major: Too many spelling mistakes, citing too much outdated works, problem with the science
63
What are the steps in summarizing an article?
In your own words
1. What was the purpose of the study (‘intro’)
-- Context (big picture) --> Knowledge Gap --> Study Objective/Hypothesis [3 sentences]
2. Study details / ‘methods’ (~5 sentences)
-- Study type (design) – experiment or observational study... specific type.... Population/model... [PICO-T model]
-- Independent variable? Dependent variable? Other variables?
-- How did they perform the work?
3. Study findings / “results”
-- Re-state the results in your own words (< 3 sentences)
Then, you CRITIQUE the article
64
How do you review works?
Table with ID, PICOT, Results, bias, etc...
Summary --> complex/critique
Objective --> subjective
To organize background research
Foundation of review paper
65
Peer review: What to look for when you critique a paper? (criteria used to review a scientific article)
* importance and novelty of the work
* appropriateness of the materials, methods and experimental model systems
* rigor of the experimental design (including the inclusion of appropriate controls)
* quality of the data
* appropriateness of the statistical analyses
* rigor of the interpretation of the data
* value of the discussion of the data
* validity of the conclusions drawn in the paper
* The length of the paper
* The writing quality/grammar
* The clarity, accuracy, and completeness of the figures and tables
* The accuracy and adequacy of the introduction which frames the area of the research, of the discussions of prior and related work, and of the citations to the literature
* Do the conclusions match the results? Reflective?
* Methods: Precision, reliability, proper techniques? Validated?
* Missing confounders?
66
What are questions to ask ourselves when evaluating research?
* Where does the research fit with previous findings? Knowledge grows cumulatively...
* Are the arguments coherent? Persuasive? Bias and emotional?
* Are the data analyzed in a systematic manner?
* Are the findings useful? Applicable?
* What are the broader impacts of the work?
67
How is an impact factor calculated?
Total number of times articles are cited during the two previous years ÷ total number of citable articles in the journal during those 2 years
68
What are the 10 top nutrition journals?
1. Annual review of nutrition
2. advances in nutrition
3. American journal of clinical nutrition
4. Nutrition reviews
5. International journal of behavioural nutrition and physical activity
6. International journal of obesity
7. Clinical nutrition
8. Nutrition research reviews
9. Current opinion in clinical nutrition and metabolic care
10. Proceedings of the nutrition society
69
What is the PICO(T) method useful for?
For literature reviews
Useful to define a research question
Useful to appraise the relevance of a research article in your research
70
What does PICOT stand for?
```
P: Population or Patient
– Patient = for case studies
– How do you define your population?
– Who are the relevant people, patients, participants? (Demographic, conditions, etc.)
– Do not write the sample size (N=?)
```
I: Intervention or Exposure
– What is the intervention or exposure of interest? (Drug, food procedure, etc.)
C: Comparison or Control
– What is the alternative or standard practice?
– Who is the control population or intervention?
O: Outcome
– What is the relevant outcome?
T: Type of study / Study design
– Can also stand for time
71
In which steps of research methods is the PICOT method useful?
1. Ask question
2. Find best evidence (guides research)
3. Appraise the evidence (Evaluate whether the article is relevant to your research)
72
What is the population of interest in a case control trial where there are women with fibromyalgia in the case and women without fibromyalgia in the control group?
Women with and without fibromyalgia
73
Who are the 3 associations in the partnership for PEN?
- British Dietetic Association
- Dietitians Association of Australia
- Dietitians of Canada
74
What does PEN include?
o Systematic reviews
o Critically-appraised journal articles
o Practice guidelines
o It uses an Evidence Grading System to provide summaries of evidence and clinical practice recommendations
75
What criteria does PEN use to evaluate the strength and quality of the evidence?
Evidence (research design, sample size, methodology clear?)
Consistency (within the study and with other research)
Clinical impact (Is the intervention feasible in real life? Can it be recommended?)
Generalizability
Applicability
76
Which types of studies can be given a grade A for treatment/intervention studies?
For etiology/prognosis studies?
TREATMENT/INTERV.
- Systematic reviews of RCTs with consistent findings and low risk of bias
- SR including several trials combined in a single well-done meta-analysis with consistent findings
- 2+ high quality RCTs with low risk of bias
ETIOLOGY/PROGNOSIS
- SR of cohort studies with homogeneity
- 2+ well-done prospective cohort studies with consistent results
Results are consistent, clinically important, free of doubts about generalizability and directly applicable to practice settings
77
Which types of studies can be given a grade B for treatment/intervention studies?
For etiology/prognosis studies?
TREATMENT/INTERV.
- SR of RCTs with heterogeneity
- Single RCT with low risk of bias
- 2+ RCTs with a clinically significant conclusion and unclear risk of bias
ETIOLOGY/PROGNOSIS
- SR of cohort studies with homogeneity
- 2+ well-done prospective cohort studies with consistent findings
- SR of case-control studies with homogeneity
- Several independant case-control studies with similar conclusions
minor inconsistencies, minor doubts about clinical significance, minor doubts about generalizability, generally applicable to practice setting
78
Which types of studies can be given a grade C for treatment/intervention studies?
For etiology/prognosis studies?
TREATMENT/INTERV.
- Several RCTs with inconsistent results
- Non-randomized trials
- SR of cohort or case control studies with homogeneity
ETIOLOGY/PROGNOSIS
- SR of cohort or case control studies with heterogeneity
- Single cohort study or 2+ case-control studies unconfirmed by other studies
- 2+ High quality cross sectional studies
Inconsistencies among results, uncertain or moderate clinical impact, uncertainty about the generalizability, likely applicable to practice
79
Discuss why there are concerns over irreproducible research findings
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical models; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.
80
Name 6 reasons why most studies are false
1. Large studies are more likely to yield true results, yet we publish studies with small sample sizes
2. Willingness to publish small effect sizes; with a smaller effect size, findings are less likely to be true. We tend not to look at the effect size; is there a biological, societal relevance?
3. Flexibility in designs definitions, outcomes & analysis reduce chances that results will be true.
(What are people doing when the results are not going their way? They change the design or analysis so that A can cause B (tweaking experiment to prove point))
4. Fishing expeditions to generate new hypothesis or explore unlikely correlations (when you go into a research question without any well reasoned hypothesis (you can not really guess what you will find; you can not predict specific outcomes, etc), rather you leverage the big data world we live in and use computational approaches to help identify associations/correlations which can then be used to discover new hypotheses)
5. Financial & other interests & prejudices reduce the likelihood that results will be true.
6. Competition in research to produce positive findings, especially in hot fields.
81
Describe the replication case by Amgen
They identified 53 of the most important cancer studies in animal and in vitro studies
o Each of those were believed to be some of the most important studies out there (influenced other studies)
Studies that cite a seminal paper are drawing inspiration from that seminal paper. Those few papers inspire 100s of other papers…
• Between 2002 and 2012, Amgen was not able to reproduce 47 out of 53 seminal publications
Spectrum of irreproducibility:
- Data not reproduced by original researchers in own lab
- Specific data reproduced but not overall findings
- Single, non-representative experiment reports
82
Describe the replication case by the field of psychology
Repeated 100 random experiments in papers from 2008 from the 3 high-ranking psych journals
o Experiments looked at cause-effect relationship
o Shared with the community and asked people to try to repeat it
Mean effect size in replication effort was half that on the original effort
97% of original studies had significant results (p<0.05) vs. 36% of the replication studies
83
Why did Amgen do their study? (context)
Past ~15 yrs, efforts to characterize genetic alterations in human cancers = increased understanding of molecular mechanisms
• Hope this would translate to more effective drugs yet research --> clinical success very low
• Why??? Inherently complex nature of disease, sub-optimal models (cells, mice models), etc.
• Also, quality of pre-clinical data which helps ID new biological targets though we assume these to be “truth”
84
What is the impact of unreproducible studies?
o Wasted effort – time, $$$, opportunity cost, etc;
o Findings spur new field with 100s of secondary papers
o Clinical studies launched --> giving treatments to people based on faulty science? Unethical
85
How do unreproducible studies affect society?
o Distorts public policy and public expenditures (e.g. public health, climate science)
o Financial consequences (26 billion per year in US alone on irreproducible preclinical research into new drugs)
o Distorts public trust in science
86
List 13 broad approaches that may be used to improve reproducibility of research findings
1. statistical standards
2. Data handling
3. Research practices
4. Pedagogy
5. Universities
6. Professional associations and journals
7. Scientific industry
8. Private philanthropy
9. Government funding
10. Government regulation
11. Federal legistation
12. State/provincial legistation
13. Government staff and judiciary
87
Explain the "Statistical standards" method to increase research reproducibility
- Avoid making decisions solely on the p-value
- Be more rigorous and use p<0.01 vs p<0.05
i. Willing to make an error 1 out of 20 times?
- Present confidence intervals to better convey the range in which a variable most likely falls
- The p-value was never intended to be a substitute for scientific reasoning
88
Explain the "Data handling" method to increase research reproducibility
– Researchers should make their data available
– Cannot just publish a study and findings and hide the data.
– Researchers should use born-open data (i.e. open access repository that time stamps data when created and updated)
– European Union has an open data plan in which they want all the data generated in the EU
89
Explain the "Research practices" method to increase research reproducibility
– Researchers should pre-register their protocols, filing in advance with an appropriate organization (e.g., journal, scientific society, government agency)
– Adopt standardized schemes to outline methods and materials
– Done to avoid tweaking studies to prove our point
– Means to “control” the researcher who changes midway in a study based on preliminary results
90
What are the goals of open data portals?
For the data to...
1. Be discoverable
2. Be measurable
3. promote use
4. organise for use
5. Be accessible
6. promote standards
7. publish metadata
8. Link data
9. co-locate documentation
10. co-locate tools
91
Explain the "pedagogy" method to increase research reproducibility
– Fields that rely too heavily on statistics (i.e. like ours) to draw conclusions should better educate on the mis-use and mis-understanding
– Teach more holistic approaches as well as reasoning approaches to analyze data
– Integrate more into high school and college math and science classes, especially limits to certainty that statistics can provide. Need to involve more critical thinking
– Will avoid future students to focus on p-values
– The is so much data out there that we need to use our brains to extract information from the data --> use our intelligence to analyze
92
Explain the "Universities" method to increase research reproducibility
– When professors go up for tenure, they should be required to adhere to best practices for research methods
– Statistics 101 (survey level) into core curricula
93
Explain the "Professional associations and journals" method to increase research reproducibility
– They control how the field is run and organized
– Establish regular evaluation of disciplinary norms
– Journals should make peer review process even more transparent and rigorous
– Journals should only publish pre-registered studies (some already do)
– All disciplines should establish a journal devoted to publishing negative findings
94
What is the WEF code of ethics?
```
Commitments of researchers
Researchers should take an oath to promise:
Engage with the public
Pursue the truth
Minimize harm
Engage with decision makers
Support diversity
Be a mentor
Be accountable
```
95
Explain the "scientific industry" method to increase research reproducibility
– Industry should advocate practices that minimize irreproducible research
– Work with academics to formulate standard research practices and protocols to promote reproducible research
– “Journal unite for reproducibility” – initiatives done by strong journals
96
Explain the "private philanthropy" method to increase research reproducibility
– Government does not fund replication of studies
– Fund scientists’ effort to replicate earlier findings
– Megadonors: Bill gates, Mark Zuckerberg
– Fund researchers who strive to develop better methods
– Funding university chairs in "reproducibility studies"
– Establish a prize for most significant negative result in various disciplines - Push to produce significant results that are compelling
– Improve journalism that continues to uncover the reproducibility crises
97
Explain the "government funding" method to increase research reproducibility
– Fund scientist’s effort to replicate earlier findings
– Fund researchers who strive to develop better (+ strict) methods
– Prioritize funding for researchers who pre-register their plans and make data/methods open access
– Adopt new NIH Principles for funding reproducible research
– More funding to broaden statistical literacy
98
What are the 4 areas of focus ('categories') included in a NIH application based on rigor and reproducibility?
1. Scientific premise
2. Scientific rigor
3. Biological variables
4. Authentication
99
Explain what scientific premise is for a NIH application and why it is important to include.
The scientific premise for an application is the research that is used to form the basis for the proposed research question(s)
- The premise of the work needs to be sound
- What is the basis for your proposed research study?
- Strength and weaknesses of past research
- Have you scanned the entire literature or have had a myopic/biased view? Are you systematic? Are you critical? CRAAP? Etc.
- How to diminish this very high number into a manageable size? May focus on human studies, those of the last 10 years...
100
Explain what scientific rigor is for a NIH application and why it is important to include.
Scientific rigor is the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results
• Foundation for achieving robust and unbiased results
• Standards? Sample size estimator? Randomization? Blinding? Replicates? Inclusion and exclusion criteria? Data analyses plan? Etc.
• Are they using the best design to answer the question? Is it based on validated methods? Are they drawing conclusions from biased people? Are they randomized properly? INTERNAL VALIDITY
• Are the researchers themselves aware of the quality of their work?
101
Explain what biological variables are for a NIH application and why it is important to include.
Sex, age, weight, underlying health affect health and disease
• Variables often ignored --> incomplete understanding
• Explain how relevant biological variables are factored into research designs, analysis, and reporting in vertebrate animal and human studies
• May need to pull question from a research and apply it to another population or biological variables
• Key now is sex; frequently ignored in animal studies
Strong justification from the scientific literature must be provided for application proposing to study only one sex
102
Explain what authentication is for a NIH application and why it is important to include.
* How do you ensure the identity and validity of your biological and chemical reagents?
* Do they differ lab to lab? Over time?
* Have varying properties that can influence data?
* Is equipment properly maintained and calibrated?
* Are protocols/SOPs documented and followed? Deviations noted and corrected?
* Are people properly trained, and is training documented?
* Are lab notebooks maintained and reviewed?
The authentication plan should state in one page or less how you will authenticate key resources, including the frequency, as needed for your research
103
Where should scientific premise, scientific rigor, biological variables and authentication be included in the application?
1. Research strategy (significance)
2. Research strategy (approach)
3. Research strategy (approach)
4. Include as an attachment
104
Explain the "government regulation" method to increase research reproducibility
– Ensure new regulations needing scientific justification only use research that meets strict reproducibility standards
– Use strictly the best studies out there for regulations
– Establish committees to determine which regulations are based on reproducible research
– E.g. new food labels
105
Explain the "federal legislation" method to increase research reproducibility
– Pass a “secret science reform act” to prevent agencies from making regulations based on irreproducible research
– Strengthen information quality act
– Fund programs to broaden statistical literacy
– Open government – Canada
i. Trudeau government has decided to make all of their data publicly available
ii. Will probably be true for research in the future
iii. General public will be able to make their own judgements
106
Explain the "state/provincial legislation" method to increase research reproducibility
– Reform “K-12” curricula to include courses on statistical literacy
– Use funding and oversight to encourage universities (CEGEP) to add and strengthen statistical literacy
107
Explain the "government staff and judiciary" method to increase research reproducibility
– Government officials should hire trained staff in statistics and reproducible research to advise them on scientific matters
– Courts should ensure that sound science is used in judicial decision making
– Set approaches to overturn precedents based on irreproducible science
– Relevant courses in law school
108
What is the set of principles that came to consensus at the NIH to reach the goal of reproducibility?
1. Rigorous statistical analysis (A section outlining the journal’s policies for statistical analysis should be included in the Information for Authors, and the journal should have a mechanism to check the statistical accuracy of submissions.)
2. Transparency in reporting
- Journals should have no limit or generous limits on the length of methods sections, while at the same time encouraging efficient and clear presentation to ensure a thorough examination by reviewers.
Journals should use a checklist to ensure the reporting of key methodological and analytical information to reviewers and readers. (e.g. Standards, Replicates, full statistics, Randomization, Blinding, Sample-size estimation, Inclusion and exclusion criteria)
3. Data and material sharing
4. Considerations of refutations
Once a journal publishes a paper, it assumes the obligation to consider publication of a refutation of that paper, subject to its usual standards of quality.
5. Establish best practice guidelines
suggests that journals establish best practices for dealing with image-based data (for example, screening for manipulation, storing full-resolution archival versions) and for describing experiments in full.
An example for animal experiments is to report the source, species, strain, sex, age, husbandry and inbred and strain characteristics for transgenic animals. For cell lines, one might report the source, authentication and mycoplasma contamination status.
The existence of these guidelines does not obviate the need for replication or independent verification of research results, but should make it easier to perform such replication.
109
Explain what Brian Wansink is accused of
“committed academic misconduct in his research and scholarship, including misreporting of research data, problematic statistical techniques, failure to properly document and preserve research results, and inappropriate authorship.”
110
Define what a clinical study is.
Clinical trials are research studies performed in people that are aimed at evaluating a medical, surgical, or behavioral intervention. Individual-focused; involves people.
111
Why are lower-income countries losing out because of predatory publishing?
Pressure for researchers to publish is as high as here (if not higher)
In those countries, they have less funds to do research so they have less chances of being publish in important journals.
They turn towards phony journal --> quick way to get a paper published; they may not know the journal phony.
Also, the journal subscription fees are very high (they often do not have access to legitimate journals.
They turn to open access journals which may be phony journals --> if you only have this information; you tend to think those are legitimate
112
What are the 6 points in the pyramid of Bloom's taxonomy?
1. remember
2. understand
3. apply
4. analyze
5. evaluate
6. create
113
Explain the "remember" aspect of Bloom's taxonomy
Recall facts and basic concepts (define, duplicate, list, memorize, repeat, state)
114
Explain the "understand" aspect of Bloom's taxonomy
Explain ideas or concepts (classify, describe, discuss, explain, identify, locate, recognize, report, select, translate)
115
Explain the "apply" aspect of Bloom's taxonomy
Use information in new situations (execute, implement, solve, use, demonstrate, interpret, operate, schedule, sketch)
116
Explain the "analyse" aspect of Bloom's taxonomy
Draw connections among ideas (differentiate, organize, relate, compare, contrast, distinguish, examine, experiment, question, test)
117
Explain the "evaluate" aspect of Bloom's taxonomy
Justify or stand a decision (appraise, argue, defend, judge, select, support, value, critique, weigh)
118
Explain the "create" aspect of Bloom's taxonomy
Produce new or original work (design, assemble, construct, conjecture, develop, formulate, author, investigate)
