MET 13 - Testing reliability

Question 1

What is reliability?

Answer 1

• If a psychological measure can be used multiple times with the same results it is said to be reliable
• For example, a reliable test of intelligence would yield the same results on the same person each time the measurement is taken

Question 2

What is external reliability?

Answer 2

To get the same results again and it is test using test retest

Question 3

What is internal reliability?

Answer 3

Consistency within a measure itself and it is tested using inter-rater reliability

Question 4

What is inter-rater reliability?

Answer 4

• Correlating the judgements of two or more ratings of behaviour, when using rating scale
• Can happen in a questionnaire or interview when behaviour has been rated or in an observation when behaviour has been rated

Question 5

How does inter-rater reliability work?

Answer 5

• Mainly for observational research
• Two or more observers are used to increase reliability of the observation
• Data is collected independently
• Data is correlated statistically to determine reliability
• A correlation co-efficient of over 0.8 would be needed to say your observations were reliable

Question 6

What is inter-observer reliability?

Answer 6

• The extent to which there is agreement between two or more observers involved in observations of a behaviour
• This is measured by correlating the observations of two or more observers
• A general rule is that if (total number of agreements)/ (total number of observations) > +0.80, the data has high inter-observer reliability

Question 7

How does test-retest work?

Answer 7

• When tests are carried out again, the same result should be achieved
• The test-retest method is commonly applied when assessing the reliability of questionnaires. e.g. of personality or IQ and interviews
• The test is administered twice, and the results are compared
• If scores are obtained, then a correlation coefficient can be calculated
• A reasonably reliable result would be one where the correlation coefficient was +.80 or more
• Note that one of the challenges with this method is deciding on the time lapse between tests
• There must be sufficient time to be sure that the participant is not simply recalling their previous responses but not too much time in case the attitude or ability being tested actually changes

Question 8

How can reliability be improved in experiments?

Answer 8

Use standardised procedures and instructions in experiments to ensure consistency over time

Question 9

How can reliability be improved in questionnaire?

Answer 9

• Use test-retest tests the reliability over time
• Comparing 2 sets of data should produce a correlation that exceeds =.80
• Low test-rest means questions should be removed or rewritten
• In questionnaires that are found to be unreliable, ambiguous or complex questions may be removed
• Sometimes this involves replacing open questions (open to individual interpretation) with closed ones with a more restricted range of responses

Question 10

How can reliability be improved in observations?

Answer 10

• Careful operationalisation and explanation of the behavioural categories
• Ensure categories are agreed before the study and are measurable
• They should be as independent as possible, i.e. avoiding overlap
• Use multiple researchers to have interobserver reliability

Question 11

How can reliability be improved in interviews?

Answer 11

• Using the same interviewer with all participants or at least training them all in the same way, e.g. to avoid leading questions
• Not asking leading or ambiguous questions
• Leading/ambiguous questions can be avoided by using structured interviews as questions are fixed and less free flowing, so it is more likely to be reliable