Research and Program evaluation Flashcards
What are some trends in counseling research?
- More studies seem to be sporting multiple authors and female authors
- Increased attention to multicultural issues
- Field-based professionals and practitioners submitting fewer contributions
- Meta-studies used to summarize findings related to a given topic or theme
- A majority of studies use grad students and adults as subjects
- qualitative research (as done by Freud and Piaget) seems to be making a comeback.
- N1= single subject designs seem to be making a comeback
- counselors and grad students feel they need more training in APA publication guidelines to submit to journals
- Kurt Lewin’s concept of action research is popular.
- Using the internet to conduct an experiment (rapid data collection, lower research costs, large sample sizes)
- Neuroscience is being used to guide diagnostic and treatment procedures.
What is Cohen’s d effect size (ES) statistic?
This statistic is used to gauge how strong a relationship exists (small .2, medium .5, large .8)
Why are N=1 single subject designs making a comeback?
- Only one person is required and counselors are interested in individual change
- The setting is usual real world vs. lab
- Generally easier for consumers of mental health services to understand this type of studies since typically don’t need complex statistical analysis
A common single-subject (N=1) design uses the ABABA design. This model tracks the client with an extended baseline throughout treatment to outcome. Single subject research is idiographic while studies of groups to discover general principles are called nomothetic
What is Kurt Lewin’s concept of action research?
This is a type of research intended to improve the situation (vs. just advance knowledge) with local people/clients who will be better off at the end of the research. Self-surveys are often used to conduct Action Research. Action research bridges the gap between research and application/practice.
What is the most valuable type of research?
the experiment, used to discover cause and effect relationships. Experimental research is the process of gathering data to make evaluative comparisons regarding different situations.
What must an experiment have?
An experiment must have the conditions of treatment controlled via the experimenter and random assignments. An experiment attempts to eliminate all extraneous variables.
What is a quasi-experiment?
A quasi experiment is like a true experiment BUT the groups are not ramdomly assigned. In a quasi experiment, you cannot state with any degree of statistical conference that the IV caused the DV (dependent variable). One popular type of quasi experiment is known as the ex post facto study or causal comparative study. Ex post facto means after the fact, connoting a correlations study or research in which intact, preexisting groups are utilized. This, however, can threaten internal and external validity.
What is internal validity?
This refers to whether the dependent variables were truly influenced by experimental independent variable or whether other factors had an impact.
What is external validity?
This refers to whether the experimental research results can be generalized to larger populations (I.e. other people, settings, or conditions).
What is a factor analysis?
This is a statical procedure that uses the important or underlying “factors” in an attempt to summarize a lot of variables. So a test which measures a counselors ability may try to describe the 3 most important variables (“factors”) that make an effective helper, although literally hundersds of factors may exist. USung factor analysis procedures, a brief test that measures the 3 major factors may be able to predict who will be an effective counselor as accurately as 10 other tests that examine hundreds of variables/factors.
What is a chi-square?
This is a nonparametric statistical measure that tests whether an obtained distribution differs significantly from the distribution that the researcher expected. You must have mutually exclusive categories to use a chi square.
What is parsimony?
Parsimony is when you interpret results in the simplest way. In research, we strive for parsimony in that the easiest and least-complex explanation is said to be the best. The simplest explanation of finding is always preferred. For example, factor analysis is parsimonious in the sense that it is concerned with data reduction.
What is Occam’s razor?
It suggests that experimenters interpret results in the simplest manner.
What are “bubbles” in research?
This refers to flaws in research – like when you try to put a sticker on a car and there are always air bubbles.
What is a confound?
This is when an undesirable or excess variable “confounds” or flaws the experiment (I.e. if there is an experiment testing a new form of therapy but the person is seeing another therapist on the side). The only experimental variable should be the independent variable (IV). A confounded experiment is, by nature, invalid.
All correlational research is said to be confounded.
What periodical publishes more counseling research articles than any other periodical in our field?
The APA’s Journal of Counseling Psychology.
What is the difference between basic research and applied research?
Basic research is conducted to advance our understanding of theory. Applied research (aka action research or experience-near research) is conducted to advance our knowledge of how theories, skills, and techniques can be used in practical application. Often counselors asserts that much of the research isn’t relevant to the actual counseling process.
What is an independent variable?
An independent variable (IV) is a variable that the researcher manipulates, controls, alters or wises to experiment with.
What is a dependent variable (DV)?
A dependent variable expresses the outcome of the data.
What is a causal comparative design?
This is a type of experiment that is a true experiment EXCEPT that the groups are not randomly designed. Data gland from a causal comparative ex post facto (after the fact) design can be analyzed with a test of significant like a t test or anova, just like any true experiment.
What does it mean for an experimenter to be guided by ethics?
In all Experiments, a counselor/researcher should be guided by ethics. This means:
- subjects are informed of any risks
- negative after-effects are removed
- subjects can withdraw at anytime
- confidentiality will be protected
- results of research reports will be presented in an accurate format that isn’t misleading
- the counselor/researcher will only use techniques they are trained in
What are control groups and experimental groups
Both the control and experimental groups should have the same characteristics except the members of the control group will not have the experimental treatment applied to them. In an experiment, the control group does not receive the independent variable (IV). The experimental group receives the independent variable. The basic supposition is that the averages (means) of the groups do not differ significantly at the beginning of the experiment.
Note that if you cannot randomly assign the subjects to two groups, the research will be considered a quasi-experiment.
What is an organismic variable?
This is a variable that a researcher cannot control, yet exists - such as heigh, weight, or gender. To determine whether there is an organismic IV, you must ask yourself if there is an experimental variable being examined that you cannot manipulate.
What is hypothesis testing and who pioneered it?
Hypothesis testing was pioneered by RA Fisher. A hypothesis is a hunch or an educated guess which can be tested utilizing the experimental model – so it is a statement which can be tested regarding the relationship of the independent variable and dependent variable.
What is a null hypothesis?
This hypothesis suggests that there will not be a significant difference between the experimental group which received the IV and the control group which did not. Essentially, the null hypothesis is simply that the IV did not impact the DV.
What is a meta analysis?
A meta analysis is a study that analyzes the results of numerous studies.
What is an alternative hypothesis (or affirmative hypothesis)?
This asserts that the independent variable has indeed caused a change in the dependent variable.
What is a test of significance?
This is a type of statistical test that is used to determine whether a difference in the groups’ scores is “significant” or just due to chance factors. In this case, a t test would be determined if a significant different between two means exists. This has been called the two groups or two randomized groups research design. In this study, the two groups were independent of each other in the sense that the change (or lack thereof) in one group did not influence the other group, those is is known as an independent group comparison. If the researcher had measured the same group of subjects without the IV and with the IV, it would be known as a repeated measures comparison design.
What is a correlation coefficient?
This is a way of measuring correlational research
What is a “between subjects design”?
This is when a research study uses different subjects for each condition. If the same subjects are employed, it could be referred to as a within subjects design.
What does P mean in relation to a test of significance?
P in this context means probability or the level of significance. Traditionally, the probability in social science research has been set at .05 or lower (I.e. .01 or .001). The .05 level indicates that differences would occur via chance only 5 times in 100. The significant level must be set before the experiment begins!
The smaller the value of P the more stringent the level of significance.
What is a parameter?
A parameter is technically a value obtained from a population which a statistic is a value drawn from a sample. A parameter summarizes a characteristic of a population (I.e. the average male’s height is 5’9).
What is the accepted probability level in social sciences?
.05 or less. The two most popular levels of significant are .05 and .01
What does P = .05 really mean?
That there is only a 5% chance that the difference between the control group and the experimental group is due to chance factors. This could also be referred to as a 95% confidence interval which means that the results would be due to change only 5 times out of 100. When P = .05, differences in the experimental group and the control group are evident at the end of the experiment and the odds are only 1 in 20 that this can be explained by change. Reminder: P= level of significance (aka level of confidence or confidence level)
What are Type I and Type II errors?
- A type I error, also known as an alpha error, occurs when a researcher rejects the null hypothesis when it’s true
- A type II error, also known as a beta error, occurs when you accept a null hypothesis when it’s false.
The probability of committing a type I error (saying there is an experimental impact when there’s not) equals the level of significance (P). Therefore the level of significance is often referred to as the alpha level.
What is the power of a statistical test
the power of a statistical test is what it’s called when you calculate 1-beta (Beta errors). In this respect, “power” connotes a statistical test’s ability to reject correctly a false null hypothesis.
Parametric tests have more power than nonparametric statistical tests. Paremetric test are used only with interval and ratio error.
How does increasing sample size impact Type I and Type II errors?
Increasing sample sizes helps to lower the risk of chance/error factors. Differences revealed via large samples are more likely to be genuine than differences revealed using a smaller sample size.
How do alpha and beta errors increase or decrease if a researcher changes the significance level from .05 to .001
Alpha errors would decrease (I.e. there is a higher standard to prove that the IV had an impact, so it is less likely that you would think something happened when it really hadn’t) BUT because there’s a higher threshold to prove that, so you risk thinking nothing happened when it really had, so beta errors would increase.
What is a t test?
A t test is a simplistic form of the analysis of variance. The t test is used to ascertain whether two sample means are significantly different. The researcher sets the level of significance and then runs the experiment. The t test is computed using 3 key data values: the difference between the mean values from each data set, the standard deviation of each group, and the number of data values in each group. This yields a t value.
The researcher then goes to a t table found in the index of most statistics’ texts. If the t value obtained statistically is lower than the t value (sometimes called the critical t), in the table, then you’ll accept the null hypothesis. The computation must exceed the number cited in the table in order to reject the null.
The t test only applies when there are two groups. If there are more groups you need to use the analysis of variance (ANOVA).
What is ANOVA?
ANOVA, the analysis of variance, is used to figure out whether there are specific differences between more than 2 groups (if there are only 2 groups, can use a t test). The results of an ANOVA field an F-statistic. The researcher then consults an F table for a critical value of F. If the F obtained exceeds the critical F value in the table, then the null hypothesis is rejected.
This is considered a one-way test which is used for testing one independent variable. If there are two independent variables, you could use a two-way ANOVA (analysis of variance), if there are 3 IVs, you would use a three-way ANOVA, etc. These multi-way ANOVAs are also called MANOVA.
What are the major statistical tests (to test for significance difference)?
- t test - tests for significant difference between two groups. Compares t obtained to a t table for a critical t value
- ANOVA (analysis of variance) - tests for significant difference between more than two groups. Yields an F statistic - researcher compares to an F table for a critical value of F.
- ANCOVA (analysis of covariance) - tests two or more groups while controlling for extraneous variables that are often called covariates
- Kruskal-Wallis - used instead of the one-way ANOVA when data are nonparametric and you wish to test whether two correlated means differ significantly
- Mann-Whitney U test - used to determine whether two uncorrelated means differ significantly when data are nonparametric
- Spearman correlation or Kendall’s tau - Used in place of the Pearson r when parametric assumptions cannot be utilized
- chi-square nonparametric test - examines whether obtained frequencies differ significantly from expected frequencies
What is MANOVA?
This is a test for statistical significnance. MANOVA stands for multivariate analysis of variance to test more than one dependent variable. Two-way and three-way ANOVAs are considered MANOVAs.
What is a correlation coefficient?
This is how the results of a correlation experiment is reported and is a statistic that indicates the degree of magnitude of linear relationship between two variables. The correlation coefficient is often abbreviated using an r . A coefficient of correlation makes a statement regarding the association of two variables and how a change in one is related to a change in the other. Correlations range from 0.00 (no relationship) to 1.0 or -1.0 which signify perfect relationships.
Note that a positive correlation is not a stronger relationship than a negative one of the same numerical value. A correlation of -.70 is still indicative of a stronger relationship than a positive relationship of .60. The minus sign merely describes the fact that as one variable goes up, the other goes down.
The Pearson Product-moment correlation r is used for interval or ratio data while the spearman rho correlation is used for ordinal data.
What are positive and negative correlations?
Positive correlation is evident when both variables change in the same direction (test scores go up when studying goes up). A negative correlation is evident when the variables are inversely associated - one goes up and the other goes down.
What are the terms bivariate and multivariate?
A bivariate correlational paradigm means it is a paradigm that is trying to describe the nature and relationship of two variables. If the researcher is looking at more than two variables, they would use the term multivariate to describe the correlational paradigm.
What is an N = 1 study and when might it be used?
This is a study with only one person in it. This is basically a case study approach and is popular with behaviorists who seek out overt behavioral change. The client’s dysfunctional behavior is measured (a baseline measure), the treatment is implemented, and then the behavior is measured again (a new baseline). This is sometimes delineated using A, B, C in which A = baseline, B = intervention implementation, and C = second or alternative form of intervention.
These studies are often called idiographic studies or single subject designs.
The original case study methodology was popularized by Freud, though he did not rely on baseline measures. Case studies can be misleading because the results are not necessarily generalizable.
What are single-blind and double-blind experiments?
In a single blind experiment, the subject wouldn’t know if she is a member o the control of experimental group - but the researcher does know.
A double blind study is one in which both the participant and the researcher don’t know which group the participant is in.