Quiz 2 Flashcards
Defining Surveys & Experiments
- Surveys are used to provide a quantitative or numeric description of trends, attitudes, or opinions of a population by studying a sample of that population
- Experiments are used to test the impact of a treatment or intervention on an outcome, controlling for all other factors that may influence that outcome; a sample is identified and generalizations are made to the population
Components of Survey Method Plan
-Survey design, population and sample, instrumentation, variables in the study, and data analysis and interpretation
-Question checklist examples:
- Is the purpose stated, are the reasons for choosing the design mentioned?
- Is the nature (cross-sectional vs longitudinal) identified?
- Will the population be stratified?
- How many people will be in the sample? What is the timeline for administering?
- What instruments, procedures, etc.?
- What steps will be taken in data analysis to show returns, check for bias, etc.?
Survey Design
-Provide a purpose and rationale for using a survey for the proposed study
-Indicate why a survey is the preferred type of data collection
-Indicate the type of design (cross-sectional or longitudinal)
-Specify the medium of data collection and strengths/weakness of this method
Population & Sample
-Identify the population in the study; state the size of the population, means of identifying individuals, and the availability of sample frames
-Specify if sampling will be single-stage (direct contact) or multi-stage (using groups)
-Identify selection process as probability or non-probability
-Indicate is the population and subsequent sample will be stratified based on specific population characteristics
-Indicate the number of people in the sample and the procedures used to compute this number (present as percentage or fraction of population
Instrumentation
-Name the survey instrument used to collect the data (designed or modified for this research, specify)
-When using an existing instrument, describe the established validity (content, predictive or concurrent, and construct validity)
-Mention whether scores resulting from past use of the instrument demonstrate reliability (test-retest correlations)
-When one modified or combine an instrument the original validity and reliability may not hold for the new instrument (must be established)
-Include sample items from the instrument so readers can see actual items used
-Label major content sections in the instrument: cover letter, items (demographics, attitude items, behavior items, factual items), closing instructions, and type of scales used
-Discuss plan and rationalize the pilot testing and field test the survey
Variables in the Study
-Useful in the methods section to relate the variables to back to the research question and items on the instrument
-Allows the reader to easier determine how the data collection connects to the variables and question or hypothesis
-Allows for cross-referencing the variable, the questions or hypothesis, and specific survey items
Data Analysis & Interpretation
-Present steps: report response rate, determine response bias (effect of nonresponses on survey estimates), discuss plan to provide descriptive analyses, check instrument’s scales, statistics and statistical computer program for inferential statistical analyses, and present and interpret results
-Report how the results answered the research question or hypotheses
-Discuss the implication of the results for practice or future research on the topic; draw inference and conclusions from results
An Experimental Method Plan
-Components of experimental method plan: participants, materials, procedures, measures
-Question checklist examples:
Who are the participants?
What is the population which the results of the participants will be generalized?
Was random selection used?
What is the treatment condition and how was it operationalized?
What experimental design is used?
What are the steps in the procedure?
What are potential threats to internal and external validity and how will they be addressed?
What stats will be used in analysis?
Participants
-Describe the selection as random or nonrandom
-Indicate if it is a true experiment or not
-Identify other features in the experimental design that will influence the outcome
-Describe the assignment of participants to groups and procedure for determining group size (level of statistical significance, amount of power desired, effect size)
-The experiment is planned so that the size of each treatment group provides the greatest sensitivity that the effect on the outcome actually is due to the experimental manipulation
Variables
-Specifying the variables in an experiment identifies the group receiving the experimental treatment and the outcomes being measured
-Make groups, identify the IVs including treatment variables, and DVs (outcomes)
Instrumentation & Materials
-Describe the instrument(s) participants complete in the experiment (development, items, scales, reliability and validity reports)
-Thoroughly discuss material used for the treatment
-Experimental procedures: identify type (pre-experimental, true, quasi, and single subject), identify type of comparisons (within group or between subject), and provide a visual model to illustrate the research design used
Consider Threats to Validity
-Internal: procedures, treatments, experiences of participants
History, maturation, regression, selection, mortality, diffusion of treatment, compensatory demoralization or rivalry, testing, and instrumentation
-External: characteristics of sample/setting/timing
Interaction of selection and treatment, interaction of setting and treatment, and interaction of history and treatment
-Statistical conclusion: inadequate statistical power to generalize
-Construct: inadequate definitions and measures of variables
Procedure
-Administer measures of DV to participants
-Assign participants to match pairs on the basis of their scores
-Randomly assign one member of each pair to the control and experimental group
-Expose the experimental group to the treatment
-Measure DVs to experimental and control groups
-Compare performance on posttest using statistical significance
Data Analysis & Interpreting Results
-Report descriptive statistics (measures of central tendency and variability), conduct inferential statistical tests (t-test, ANOVA, ANCOVA), use line graphs for single subject designs, and report confidence intervals and effect sizes in addition to statistical test
-Interpreting results: discuss results, limitations, and implications
Components of Qualitative Methods
-Tell readers about the design being used in the study
- discuss the sample for the study
-discuss the data collection
- outline data analysis steps
- discuss how to present the data, interpret it, validate it, and indicate potential outcomes of the study
- include a methods section that mentions the nature of the final written product
Characteristics of Qualitative research
- Review the needs of potential audiences for the proposal
- discuss characteristics of qualitative research if audience is not knowledgeable
- Characteristics include:
- natural setting
Attempt to find audience that best represent concerns you hope to address. - researcher as key instrument
Interviewing, observing etc. follow protocol for every interview the same way. (built in deviation) - multiple sources of data
Interviews, observations, documents, videos etc. organize into themes across all sources - inductive and deductive data analysis
Build patterns from bottom up into more abstract information. Back and forth inductive and deductive thinking from themes - participants meanings
Keep a focus on learning the meaning that the participants hold and not the meaning you bring to the research - emergent design
Initial plan for research cannot be prescribed tightly and phases may change or shift after you begin to collect data. - reflexivity
Reflect how the participants culture, background ect. shapes interpretations of meaning. You want to take note about your own response on what your findings are as well. - holistic account
Report multiple perspectives, identify many factors involved, and generally sketching larger picture
Strategies of Inquiry
-Focus on data collection, analysis, and writing
-5 popular examples:
Narrative - focus on individuals
Phenomenology- focus on individuals
Ethnography - focusing on broad cultural dynamics etc.
Case study- focusing on processes, activities, or events
Grounded theory - focusing on processes, activities, or events
Qualitative Design
-In writing a procedure for a qualitative proposal:
- Identify the specific design that you will be using and provide references to the literature
-Provide background info about the design - how has it developed and adapted
-Discuss why it is an appropriate strategy to use in the proposed study - simple language and explanations
-Identify how the use of the design will shape the many aspects of the design process(title, problem, question, data collection, analysis, and write up)
The Researchers Role
-Researcher has sustained, intensive experience with participants
-Explain strategic, ethical, and personal issues that can arise
-Researchers should:
- Discuss prior experience with participants, setting, or research problem
- Indicate how the experiences may potentially shape the interpretations the researchers make during the study
- Comment on the connection between the researchers and participants and the research site that may unduly influence the researchers interpretations
Indicate steps to get IRB permissions
-Discuss steps to gain entry into the setting
- Why was the site chosen, what activities will occur during the study?
- How will the study be disruptive?
- What will the gatekeeper gain from the study?
-Comment about ethical issues that may arise and indicate how the research will address each
Data Collection Procedures
-Identify the individuals and sites for the study
-Indicate the number of sites and participants involved in the study
-Select the type of data to be collected:
- Qualitative observations: researcher may be completely concealed or may be known
- Qualitative interviews: focus group, individual, email, phone, blog
- Qualitative documents: minutes of meetings or newspapers for retrospective
- Qualitative audio-visual materials: photos, videos, art pieces, sound bytes, film
-Include data collection types that go beyond typical observations and interviews
-These unusual forms create reader interest in a proposal and can capture useful info that observations and interviews may miss
Data Recording Procedures
-Observational protocol: record info while observing
- Record descriptive notes, reflective notes, and demo info during observations
-Interview protocol: for asking questions and recording answers
- A heading
- Instructions for interviewer to follow
- The questions
- Ice breaker, 4-5 questions, concluding question
- Probes for the 4-5 questions (ask to elaborate for further details etc.)
- Space between questions to record answers
- A final thank you statement
- A log to keep a record of documents collected for analysis
Data Analysis and interpretation
-Specify the steps in analyzing the various forms of qualitative data by segmenting and taking them apart - make sense of the data
-Data analysis will proceed hand in hand with data collection and write up of findings
-Dense and rich data means presentation will be an aggregate with small numbers of themes
-Specify the use of computer data analysis program if used and the name/use / benefits of using that program
-Analysis steps embedded within specific qualitative designs
-Blend the general steps with the specific research strategy steps
-Steps of data analysis and interpretation:
1. Organize and prepare data for analysis (transcribe interviews etc.)
2. Read or look at all the data
3. Start coding all the data
Qualitative validity
-Researcher uses procedures to check accuracy of findings
Triangulate
Member checking
Rich, thick description
Clarify bias
Negative info
Prolonged time in the field
Peer debriefing
External auditor
-Qualitative reliability
Researcher uses an approach that is consistent across different analysts and projects
-Qualitative generalization
Focus on qualitative research on the particularity, not generalizability
Qualitative Write Up
-Discuss strategies for writing up the qualitative findings
-Develop description and themes
-Match write up to strategy of inquiry
-Use quotes
-Include some conversation
-Use first person form
-Use metaphors and analogies
-Discuss how findings will be related to theories and literature
Cronbach’s Alpha
-Coefficient of reliability requiring the following assumptions be met: normality, linear nature, tau equivalence, and independence (errors are independent)
-Used when variables are at interval or ratio level of measurement
-Explained as a function of the number of questions or items in a measure, the between pairs of items average covariance, and the overall variance of the total measured score - Internal consistency, homogeneity between items
-Alpha values range from 0 to 1, with 1 representing the presence of no measurement error - Over 0.7 is general benchmark for appropriate reliability
-Calculated by taking the score from each scale item and correlating them with the total score for each observation and then comparing that with the variance for all individual item scores (covariance vs variance)
Chi-Square
-Simplest method to analyze variables that are measured on the categorical level (nonparametric test)
-Examples of questions that can be addressed: Is there an association between infant mortality and income? Is whether or not a person smokes related to whether a person drinks coffee? Is satisfaction level related to types of health insurance coverage?
-Assumptions: all observations are independent, expected count or cases in each cell should be greater than 1, and expected count or cases in no more than 20% of cells should be less than 5
Doing & Interpreting Chi-Square Tests
-Null hypothesis: there is no association between the two categorical variables
-Alternative hypothesis: there is an association between the two categorical variables
-Calculated with sum of all the [(observed frequency - expected frequency) squared / expected frequency)
-Evaluating the p-value: small means reject the null, large means do not reject
Reporting Chi-Square Tests
-What to report:
size of corresponding chi-square statistics
associated degrees of freedom
p-value
-Sample report: there was an association between the age group and whether or not they use antihistamine, chi-square (1) = 5.00, p = 0.025
Fisher’s Exact Test
-One of the assumptions of chi-square test is that the expected count in all cells should be greater than 1 and no more than 20% of the cells should be less than 5
-If this is violated in 2 x 2 contingency table, Fisher’s exact test should be used instead of chi-square test (note that Fisher’s exact test is only for 2 x 2 contingency table)
